Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

0.3.0b8 json gen adjs #11

Merged
merged 2 commits into from
Apr 1, 2024
Merged

0.3.0b8 json gen adjs #11

merged 2 commits into from
Apr 1, 2024

Conversation

carueda
Copy link
Member

@carueda carueda commented Apr 1, 2024

No description provided.

@carueda
Copy link
Member Author

carueda commented Apr 1, 2024

change resulted in a reduction of the time to write json files from about 100 minutes to... about 30 seconds

The files look good by spot check, and they have all the same file sizes as the earlier run that had redundant creation of the dataframe. I will proceed with checking full temporal coverage, however I do see one issue just from examining the json files -- There are extra input flac files in some of the daily json files. We need at most one file from the previous day to cover the beginning of a day, however series of dates include two files from the previous day, one that definitely does not have data for the day being processed. I would guess it is caused by a fixed buffer definition for tapping into files that began the previous day, rather than finding the first file that has data for a day. It probably does not affect processing, so we can choose to ignore it, but I do think we ought to be able to code for correct constraint of only files that contain data for a day. For the record, the dates in 2020 with the extra files are 2/10 through 3/13, and 12/10 through 12/31.

@carueda carueda merged commit 274a788 into main Apr 1, 2024
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant