[GLUTEN-8340][VL] Enable from_json function

[zhli1142015]

What changes were proposed in this pull request?

Fixes: #8340

How was this patch tested?

UT.

[github-actions]

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/apache/incubator-gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

• Other pull requests

[github-actions]

Run Gluten ClickHouse CI on ARM

[github-actions]

Run Gluten ClickHouse CI on ARM

[github-actions]

Run Gluten ClickHouse CI on ARM

[zhli1142015]

cc @PHILO-HE , thanks.

[PHILO-HE]

Looks good! Some minor comments.

[zhouyuan]

@zhli1142015 thanks a lot for the patch, would you please help to create one issue to track this? I assume there will be some follow up patches required

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

#8340

[zhli1142015]

@zhli1142015 thanks a lot for the patch, would you please help to create one issue to track this? I assume there will be some follow up patches required

Yes, #8340. This patch targets phase 1 only.

[zhouyuan]

pending: facebookincubator/velox#11709

[ayushi-agarwal]

val jsonData = Seq(
"""{"platformId": "IPHONE", "userId": "123", "sessionId": "abc"}""",
"""{"platformId": "ANDROID", "userId": "456", "sessionId": "def"}""",
"""{"platformId": "IPHONE", "userId": "789", "sessionId": "ghi"}"""
)
val df = spark.createDataFrame(jsonData.map(Tuple1(_))).toDF("json_column")
df.printSchema()

df.write.mode("overwrite").parquet("output/json_parquet_data")
val parquetDF = spark.read.parquet("output/json_parquet_data")

val r2 = parquetDF.collect()
r2.foreach(println)

val schema = new StructType().add("platformId", StringType).add("userId", StringType).add("sessionId", StringType)
val filteredDF = parquetDF.withColumn("parsed_json", from_json(col("json_column"), schema)).select("parsed_json")
val result = filteredDF.collect()

This prints null result when offloaded.
@zhli1142015 Are structs not supported? Shall we add a check to not offload for struct types?

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[ayushi-agarwal]

val jsonData = Seq( """{"platformId": "IPHONE", "userId": "123", "sessionId": "abc"}""", """{"platformId": "ANDROID", "userId": "456", "sessionId": "def"}""", """{"platformId": "IPHONE", "userId": "789", "sessionId": "ghi"}""" ) val df = spark.createDataFrame(jsonData.map(Tuple1(_))).toDF("json_column") df.printSchema()

df.write.mode("overwrite").parquet("output/json_parquet_data")
val parquetDF = spark.read.parquet("output/json_parquet_data")

val r2 = parquetDF.collect()
r2.foreach(println)

val schema = new StructType().add("platformId", StringType).add("userId", StringType).add("sessionId", StringType)
val filteredDF = parquetDF.withColumn("parsed_json", from_json(col("json_column"), schema)).select("parsed_json")
val result = filteredDF.collect()

This prints null result when offloaded. @zhli1142015 Are structs not supported? Shall we add a check to not offload for struct types?

@zhli1142015 Will this be fixed by the last change you made, shall we add this as a test which check for results matching with and without offload?

[zhli1142015]

Yes, this issue has been resolved. The null value occurred because the schema's case did not match that of the input field name. I did the test mostly on 1.2 which doesn't have such problem. I'm not sure which commit caused this difference.

[github-actions]

Run Gluten ClickHouse CI on ARM

[github-actions]

Run Gluten ClickHouse CI on ARM

[github-actions]

Run Gluten ClickHouse CI on ARM

[github-actions]

Run Gluten ClickHouse CI on ARM