[CELEBORN-1123] Support fallback to non-columnar shuffle for schema that cannot be obtained from shuffle dependency

[gaochao0509]

What changes were proposed in this pull request?

Support fallback to non-columnar shuffle for schema that cannot be obtained from shuffle dependency.

Why are the changes needed?

When columnar shuffle is enabled, it was found that the shuffle class operator of Spark RDD is not supported. It's recommended to support fallback to non-columnar shuffle for schema that cannot be obtained from shuffle dependency.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

• CelebornColumnarShuffleReaderSuite#columnarShuffleReaderNewSerializerInstance
• ColumnarHashBasedShuffleWriterSuiteJ#createColumnarShuffleWriter

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (7263f64) 46.61% compared to head (0d93af9) 46.64%.
Report is 5 commits behind head on main.

❗ Current head 0d93af9 differs from pull request most recent head 5a8926c. Consider uploading reports for the commit 5a8926c to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2101      +/-   ##
==========================================
+ Coverage   46.61%   46.64%   +0.03%     
==========================================
  Files         166      166              
  Lines       10695    10699       +4     
  Branches      977      977              
==========================================
+ Hits         4984     4989       +5     
  Misses       5386     5386              
+ Partials      325      324       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[waitinfuture]

Hi @kerwin-zk , could you take a look at this PR?

[SteNicholas]

LGTM. cc @kerwin-zk

[pan3793]

@waitinfuture this feature is not covered by CI, any changes require manual testing, maybe we can put a modified spark tgz in OSS, and write an IT to download it then run tests in CI?

[cfmcgrady]

@waitinfuture this feature is not covered by CI, any changes require manual testing, maybe we can put a modified spark tgz in OSS, and write an IT to download it then run tests in CI?

maybe the Github Action cache is enough?

[gaochao0509]

@kerwin-zk, could you help to review this pull request?

[kerwin-zk]

Add UT to shuffle writer and reader respectively when schema is null and not null.

[pan3793]

maybe the Github Action cache is enough?

I don't get your point, we need a modified spark binary tgz to test it, also, we can apply patch and rebuild from spark source each time, but the cost is too high

[gaochao0509]

@kerwin-zk, I have added CelebornColumnarShuffleReaderSuite#columnarShuffleReaderNewSerializerInstance and ColumnarHashBasedShuffleWriterSuiteJ#createColumnarShuffleWriter. PTAL.

[SteNicholas]

LGTM. Thanks for @gaochao0509 updates.