[Early feedback] [HUDI-8163] Refactor UnMergedLogHandler with iterators #12608
Conversation
| .build(); | ||
| scanner.scan(false); | ||
| Iterator<HoodieRecord<?>> recordIterator = scanner.iterator(); |
There was a problem hiding this comment.
Is it possible we return an iterator too for method getUnmergedLogFileRecords to the list add/get can be simplified.
| @@ -617,6 +617,8 @@ && compareTimestamps(logBlock.getLogBlockHeader().get(INSTANT_TIME), GREATER_THA | |||
| /** | |||
| * Iterate over the GenericRecord in the block, read the hoodie key and partition path and call subclass processors to | |||
| * handle it. | |||
| * TODO: | |||
| * 1. what is the purpose of this method? should the HoodieRecord be added to a queue and consumed by an iterator? | |||
| */ | |||
| private void processDataBlock(HoodieDataBlock dataBlock, Option<KeySpec> keySpecOpt) throws Exception { | |||
There was a problem hiding this comment.
This should be the main entry for handling the log blocks, each kind of log block would trigger invocation of specific row-level handling methods.
| } | ||
|
|
||
| public Builder withRecordDeletionCallback(RecordDeletionCallback recordDeletionCallback) { | ||
| this.recordDeletionCallback = recordDeletionCallback; |
There was a problem hiding this comment.
The recordDeletionCallback is only consumer for deleted reords from the delete avro block, we may need another iterator impl specifically for it.
| while (recordIterator.hasNext()) { | ||
| try { | ||
| records.add(recordIterator.next()); | ||
| } catch (Exception e) { |
There was a problem hiding this comment.
The code is exactly the same if we collect all the records in the collection again , maybe we do not remove those two callbacks and just add a new iterator impl for the case that really need it: the Flink streaming reader.
Change Logs
Refactor code to deprecate usage of callbacks and replace with iterators
Impact
Describe any public API or user-facing feature change or any performance impact.
Risk level (write none, low medium or high below)
If medium or high, explain what verification was done to mitigate the risks.
Documentation Update
Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".
ticket number here and follow the instruction to make
changes to the website.
Contributor's checklist