Switch _ResultIterator to using _Parser #52
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
spbnick
force-pushed
the
switch_resultiterator_to_using_parser
branch
2 times, most recently
from
95e6a64 to
4ccadd1
Compare
|
Ah, one undesirable side-effect of this is that we now throw |
|
Added a commit fixing parsing of RS-separated streams (RFC 7464) |
spbnick
force-pushed
the
switch_resultiterator_to_using_parser
branch
2 times, most recently
from
8407b2f to
5f00fd0
Compare
Add an implementation of parse_json() function accepting either text or a text iterator and producing an iterable returning parsed values. Add a naive implementation of parse_json_file() function accepting a text file object and producing an iterable returning parsed values. This allows parsing JSON and JSON streams without passing them through a program.
Let Python give us the length of the "bytes" it already knows, instead of doing an strlen(). This improves performance a bit.
In addition to (Unicode) strings, also accept "bytes" (and corresponding iterators) as input to the parser. This allows skipping the decode/encode step when reading raw data from a file or socket, e.g. with os.read(). This introduces small, but measurable performance increase for such cases.
Add support for returning native jv values (wrapped in _JV class) from the _JSONParser iterator, if the "packed" argument is specified as true. This enables future use of _JSONParser in _ResultIterator, replacing its own copy of parsing code.
Remove the "dumpopts" variable from _ResultIterator.__next__(). It was forgotten when the function was switched to using _jv_to_python().
Switch the _ResultIterator to using an instance of _Parser instead of bytes, as the input. This avoids the duplication of the parsing code, and enables eventual processing of JSON streams with programs.
Fix the parsing loop to use the proper interface, thus fixing parsing
RS-separated streams (RFC 7464). Without this, the program would abort
on assertion failure, like this:
python3: src/jv_parse.c:684: jv_parser_set_buf: Assertion
`(p->curr_buf == 0 || p->curr_buf_pos == p->curr_buf_length) &&
"previous buffer not exhausted"' failed.
spbnick
force-pushed
the
switch_resultiterator_to_using_parser
branch
from
5f00fd0 to
ce6bddf
Compare
|
Closing this outdated PR for another attempt. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is an attempt to reuse the _Parser for feeding programs with input data, which enables eventual processing of JSON streams with JQ programs.
Might not be the ideal approach, tell me what you think!
Please see more explanations in individual commit messages.
Based on and requires #51.