Unbatch Fundus #357
Merged
Unbatch Fundus #357
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# Conflicts: # src/fundus/scraping/pipeline.py # src/fundus/scraping/scraper.py
addie9800
requested changes
Mar 31, 2024
Co-authored-by: Adrian Breiding <[email protected]>
# Conflicts: # pyproject.toml # src/fundus/scraping/html.py
|
@addie9800 Thanks for the review. This one is ready for another round. Maybe @dobbersc also want to give it a look. |
I'm happy for now, except for the two still open comments I just replied to. |
This was referenced Apr 6, 2024
dobbersc
requested changes
Apr 13, 2024
dobbersc
reviewed
Apr 13, 2024
dobbersc
reviewed
Apr 13, 2024
Co-authored-by: Conrad Dobberstein <[email protected]>
dobbersc
approved these changes
Apr 17, 2024
addie9800
approved these changes
Apr 17, 2024
addie9800
added a commit
that referenced
this pull request
Apr 19, 2024
Add query parameter [Based on #357]
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR removes the remaining async logic from the main crawler and in the process merges CCNewsCrawler and MainCrawler logic. While this is a major redesign from the inside there is not much changing on the outside, but for the few things it does change, I adjusted the documentation accordingly.
The main advantage of this redesign is that Fundus main crawler no longer operates in batches but runs every publisher independently using threads and queues. Now, bad connections, like timeouts and denied access no longer halt the pipeline because the crawler has to wait for a new batch.
Closes #344