Compute and use the initial string offset when building nested large string cols with chunked parquet reader
#17702
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
mhaseeb123
added
5 - DO NOT MERGE
mhaseeb123
changed the title
str_offset when reading nested large string cols with chunked Parquet reader.str_offset when reading nested large string cols with chunked Parquet reader.
|
CC: @etseidl would love your review here as well if possible! |
mhaseeb123
commented
Jan 14, 2025
mhaseeb123
commented
Jan 14, 2025
| @@ -51,6 +51,9 @@ TEST_F(MergeTest, MergeLargeStrings) | |||
| CUDF_TEST_EXPECT_COLUMNS_EQUIVALENT(c, input); | |||
| } | |||
|
|
|||
| // Unset the LIBCUDF_LARGE_STRINGS_THRESHOLD if already set. | |||
There was a problem hiding this comment.
Setting env var to a smaller number may result in this test to fail (and rightfully so) so locally unsetting it here
mhaseeb123
added
3 - Ready for Review
mhaseeb123
changed the title
str_offset when reading nested large string cols with chunked Parquet reader.nested large string cols with chunked parquet reader
PointKernel
reviewed
Jan 14, 2025
mhaseeb123
commented
Jan 15, 2025
| * atomically update the initial string offset to be used during large string column construction | ||
| */ | ||
| template <int block_size> | ||
| __device__ void compute_string_offsets(page_state_s* const state, |
There was a problem hiding this comment.
Comments about the name of this utility function are welcome! 🙂
PointKernel
approved these changes
Jan 15, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Closes #17692.
This PR enables computing the
str_offsetrequired to correctly compute the offsets columns for nested large strings columns with chunked Parquet reader whenchunk_read_limitis small resulting in multiple output table chunks per subpass.Checklist