Add support for streaming input #48
Conversation
spbnick
force-pushed
the
support_streaming_input
branch
2 times, most recently
from
0eada66 to
5761a35
Compare
Add support for accepting iterators as values for `value` and `text`, output of which (values and strings respectively) is fed gradually into the parser, yielding values as they are read. This is useful for parsing and processing large streams of objects.
spbnick
force-pushed
the
support_streaming_input
branch
from
5761a35 to
d91e4af
Compare
|
I think I'd prefer to add a separate method rather than overloading the existing method for clarity -- for instance, adding an |
|
@mwilliamson, separate method works for me and makes sense! Especially since having an iterator producing values for input doesn't make much sense. However, I've already worked past this locally, trying to speed up things and lower memory requirements. I have code for going directly from jv's to Python data, bypassing JSON representation. It's trivial, but it's twice as fast on large data sets. I really only need stream parsing at this moment (which doesn't exist for Python otherwise), without passing the data through the program (which has speed and memory (?) overhead). So, I'm working on having a separate parser interface exposed (and perhaps used by "program"). I might be able to post it next week (with a separate method, as you request). |
|
Thank you for taking a look at this, and I'm glad you're not against having support for streaming :) |
|
Closing this in favor of #49. Would like to get around to implementing this on top of that. |
Add support for accepting iterators as values for
valueandtext,output of which (values and strings respectively) is fed gradually into
the parser, yielding values as they are read.
This is useful for parsing and processing large streams of objects.
This is a rather quick solution, and might need improvement, but I'd like to know if such a contribution could be accepted in principle.