Add query parameter [Based on #357] #403
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
addie9800
added
the
DON'T MERGE
addie9800
added a commit
that referenced
this pull request
Apr 14, 2024
addie9800
removed
the
DON'T MERGE
MaxDall
requested changes
Apr 18, 2024
There was a problem hiding this comment.
@addie9800 Thanks for adding. Looks like a great feature :)
| @@ -18,7 +18,7 @@ | |||
|
|
|||
| def get_test_article(enum: PublisherEnum, url: Optional[str] = None) -> Optional[Article]: | |||
| if url is not None: | |||
| source = WebSource([url], publisher=enum.publisher_name) | |||
| source = WebSource([url], publisher=enum.publisher_name, query_parameters=enum.query_parameter) | |||
There was a problem hiding this comment.
Since this is about crawling a specified URL we should not pass the query parameter here.
There was a problem hiding this comment.
Makes sense, I'll remove that
src/fundus/scraping/html.py
Outdated
| delay: Optional[Delay] = None, | ||
| ): | ||
| self.url_source = url_source | ||
| self.publisher = publisher | ||
| self.url_filter = url_filter | ||
| self.request_header = request_header or _default_header | ||
| self.query_parameters = query_parameters |
There was a problem hiding this comment.
i would suggest
self.query_parameters = query_parameters or {}and remove
if self.query_parameters is not None:
MaxDall
requested changes
Apr 19, 2024
MaxDall
approved these changes
Apr 19, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is a rebrand of and closes #392 where the code is changed to work with #357
The major change compared to the former version ist that the query_parameter attribute accepts a dictionary of key value pairs. Adding Heise will be done in a separate PR