Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Export process with non-nwb external files #1085

Open
samuelbray32 opened this issue Sep 4, 2024 · 1 comment
Open

Export process with non-nwb external files #1085

samuelbray32 opened this issue Sep 4, 2024 · 1 comment
Labels
enhancement New feature or request infrastructure Unix, MySQL, etc. settings/issues impacting users

Comments

@samuelbray32
Copy link
Collaborator

Describe the bug

  • The docker export process uses Dandi and the AnalysisNwbfile table to log accessed files and share intermediate and final results

  • For historical and/or data type reasons, some tables store external files in a different format

    • ClusterlessDecodingV1
    • SortedSpikesDecodingV1
    • v0.SpikeSortingRecording
    • v0.SpikeSorting
    • v0.SpikeSorting()
    • v0.Waveforms()
    • v0.QualityMetrics()
  • These external files are not logged in the export process, nor are they shareable through Dandi

Solutions

  • Short term
    • If these are "intermediate" analysis files (not directly accessed in producing final results/figures) this shoudn't raise any errors when running in the Docker container.
    • File would not be available to someone trying to do new analysis from intermediate results in Docker container
  • Medium term
  • Long term
    • Increase priority of storing files as nwb in new pipelines
    • Already done in spikesorting.v1 (Thanks @khl02007 )
    • Non-trivial for some filetypes. Can rely on re-compute in these cases (?)
@samuelbray32 samuelbray32 added enhancement New feature or request infrastructure Unix, MySQL, etc. settings/issues impacting users labels Sep 4, 2024
@edeno
Copy link
Collaborator

edeno commented Sep 4, 2024

Just a note that @rly did make a nwb extension for xarray so we could potentially use that for the decoding:
https://github.com/rly/ndx-xarray

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request infrastructure Unix, MySQL, etc. settings/issues impacting users
Projects
None yet
Development

No branches or pull requests

2 participants