Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

participant information #253

Open
karl-koschutnig opened this issue Dec 13, 2024 · 16 comments
Open

participant information #253

karl-koschutnig opened this issue Dec 13, 2024 · 16 comments

Comments

@karl-koschutnig
Copy link

Hi Marcel,

I was wondering if it would be possible to add an option in the bidseditor for specifying participant-related values. It’s great that the tool automatically generates a participants.tsv and corresponding JSON file, but it would be really helpful to be able to define variables like sex (and potentially others) directly through the interface.

For instance, the current participants.json includes a "Level" entry for age, which doesn’t quite fit unless we’re categorizing participants into discrete groups like “young” or “old.” A format similar to what’s used in the neurobagel project might be a good reference.

What do you think?

Best,
Karl

@marcelzwiers
Copy link
Collaborator

marcelzwiers commented Dec 13, 2024

So if I understood you properly, you'd like to be able to specify dynamic values for the participants.tsv file? Something like (each key is a column in the participants-table):

participants:
  participant_id: <<filepath:/raw/sub-(.*?)/>>
  session_id: <<filepath:/raw/sub-.*?/ses-(.*?)/>>
  group:
  age: <<PatientAge>>
  sex: <<PatientSex>>
  size: <<PatientSize>>
  weight: <<PatientWeight>>

Currently, this info is collected in the plugins and then written to the participants-file. But that's hard coded, I guess it may indeed be useful to be able to add to that using the GUI

@karl-koschutnig
Copy link
Author

Thanks for the quick reply (as always!!)

Yeah, a GUI option would be nice. Indeed, it would be great to have a way to change the corresponding json file. So for example we get the sex from the DCM-Tag and also want to set the levels in the json file.
That ultimate renders to:

"sex": {
    "Description": "sex of the participant as reported by the participant",
    "Levels": {
        "M": "male",
        "F": "female"
    }
},

@marcelzwiers
Copy link
Collaborator

I will give it a try to see how it works out. Btw, do you have Presentation logfiles in your dataflow? Because in the upcoming version I added a new feature to convert that to BIDS too (i.e. to a sub*_events.tsv file) and I can always use feedback :-)

@karl-koschutnig
Copy link
Author

Great - thanks a lot!

Yes, we do have some "old" studies that have used neurons. I am happy to help and try it out!
We invested much time in dealing with events files and made a little psychopy module (https://psychopy-bids.readthedocs.io/en/stable/).

@marcelzwiers
Copy link
Collaborator

I played around a bit with the interface and I can make something to customize the participants.tsv file. The json-sidecar is much harder though, because of its nested structure. Yet, I understood it's the sidecar that you are after, right? Or would you also like to tweak the table?

@karl-koschutnig
Copy link
Author

I know this is not easy to implement. I really appreciate your effort.
Have you had the chance to look at the neurobagel approach (link in the first post). There, you "upload" a tsv and can annotate basic stuff. Something like that would be great.
In my world, it would be perfect to have some basic participants info in the exact same format after coining them. Later, I would tweak the table (but not in the conversion process).

@marcelzwiers
Copy link
Collaborator

I am still thinking about the best way to implement this, as part of a refactoring. It may take a while...

@marcelzwiers
Copy link
Collaborator

marcelzwiers commented Jan 15, 2025

Ok, I paved the way a bit and adjusted bidscoin such that the contents of the participant.tsv file can now be specified in the template bidsmap. What I didn't do (yet) is to make this editable in the bidseditor, and to offer a possibility to annotate the data in a json sidecar like neurobagel does.

@karl-koschutnig
Copy link
Author

Thanks a lot!

I just downloaded the latest version and had a first try (under macos - just saying ,-). Bidsmapper worked as expected but the bidscoiner prompted an error regarding the participants file.

Traceback (most recent call last):
File "/Users/User/opt/miniconda3/bin/bidscoiner", line 8, in
sys.exit(main())
File "/Users/User/opt/miniconda3/lib/python3.9/site-packages/bidscoin/bidscoiner.py", line 315, in main
raise error
File "/Users/User/opt/miniconda3/lib/python3.9/site-packages/bidscoin/bidscoiner.py", line 311, in main
bidscoiner(**vars(args))
File "/Users/User/opt/miniconda3/lib/python3.9/site-packages/bidscoin/bidscoiner.py", line 279, in bidscoiner
personals = plugin.Interface().personals(bidsmap, datasource)
File "/Users/User/opt/miniconda3/lib/python3.9/site-packages/bidscoin/plugins/init.py", line 65, in personals
for key, value in bidsmap.dataformat(datasource.dataformat).participant:
ValueError: too many values to unpack (expected 2)

@marcelzwiers
Copy link
Collaborator

marcelzwiers commented Jan 15, 2025

Yes, that could be, because I just committed the patch without that much testing. I'll have a look

marcelzwiers added a commit that referenced this issue Jan 15, 2025
@marcelzwiers
Copy link
Collaborator

Ok, I think I fixed it, can you try again?

@karl-koschutnig
Copy link
Author

You fixing > me testing ;-)

The error stays the same. Here is what I did. I guess I did something wrong:

  • createa new pyhton env (python -m venv bidscoin)
  • upgrade pip (/Volumes/data/work/bidscoin/bidscoin/bin/python -m pip install --upgrade pip )
  • install bidscoin (pip install "bidscoin[dcm2niix2bids]@git+https://github.com/Donders-Institute/bidscoin"); btw. the command on readthedocs doesn't work on mac os. I had to use ""
  • bidsmapper sourcedata .
    returned this:

bidsmapper sourcedata .

A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.0.2 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.

If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2.

Traceback (most recent call last): File "/Users/karlkoschutnig/opt/miniconda3/bin/bidsmapper", line 5, in
from bidscoin.bidsmapper import main
File "/Users/karlkoschutnig/opt/miniconda3/lib/python3.9/site-packages/bidscoin/bidsmapper.py", line 22, in
from bidscoin.bids import BidsMap
File "/Users/karlkoschutnig/opt/miniconda3/lib/python3.9/site-packages/bidscoin/bids.py", line 13, in
import pandas as pd
File "/Users/karlkoschutnig/opt/miniconda3/lib/python3.9/site-packages/pandas/init.py", line 49, in
from pandas.core.api import (
File "/Users/karlkoschutnig/opt/miniconda3/lib/python3.9/site-packages/pandas/core/api.py", line 28, in
from pandas.core.arrays import Categorical
File "/Users/karlkoschutnig/opt/miniconda3/lib/python3.9/site-packages/pandas/core/arrays/init.py", line 1, in
from pandas.core.arrays.arrow import ArrowExtensionArray
File "/Users/karlkoschutnig/opt/miniconda3/lib/python3.9/site-packages/pandas/core/arrays/arrow/init.py", line 5, in
from pandas.core.arrays.arrow.array import ArrowExtensionArray
File "/Users/karlkoschutnig/opt/miniconda3/lib/python3.9/site-packages/pandas/core/arrays/arrow/array.py", line 50, in
from pandas.core import (
File "/Users/karlkoschutnig/opt/miniconda3/lib/python3.9/site-packages/pandas/core/ops/init.py", line 8, in
from pandas.core.ops.array_ops import (
File "/Users/karlkoschutnig/opt/miniconda3/lib/python3.9/site-packages/pandas/core/ops/array_ops.py", line 56, in
from pandas.core.computation import expressions
File "/Users/karlkoschutnig/opt/miniconda3/lib/python3.9/site-packages/pandas/core/computation/expressions.py", line 21, in
from pandas.core.computation.check import NUMEXPR_INSTALLED
File "/Users/karlkoschutnig/opt/miniconda3/lib/python3.9/site-packages/pandas/core/computation/check.py", line 5, in
ne = import_optional_dependency("numexpr", errors="warn")
File "/Users/karlkoschutnig/opt/miniconda3/lib/python3.9/site-packages/pandas/compat/_optional.py", line 135, in import_optional_dependency
module = importlib.import_module(name)
File "/Users/karlkoschutnig/opt/miniconda3/lib/python3.9/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/Users/karlkoschutnig/opt/miniconda3/lib/python3.9/site-packages/numexpr/init.py", line 24, in
from numexpr.interpreter import MAX_THREADS, use_vml, BLOCK_SIZE1
AttributeError: _ARRAY_API not found
INFO |
INFO | -------------- START BIDSmapper ------------
INFO | >>> bidsmapper sourcefolder=/Volumes/data/work/bidscoin/sourcedata bidsfolder=/Volumes/data/work/bidscoin bidsmap=bidsmap.yaml template=/Users/karlkoschutnig/.bidscoin/4.5.0/templates/bidsmap_dccn.yaml plugins=[] subprefix=None sesprefix=None store=False force=False
INFO | No bidsmap file found: /Volumes/data/work/bidscoin/code/bidscoin/bidsmap.yaml
INFO | Reading: /Users/karlkoschutnig/.bidscoin/4.5.0/templates/bidsmap_dccn.yaml
INFO | Checking the bidsmap run-items:
INFO | Missing 'beh' run-item in: bidsmap[Presentation][beh] (NB: this may perhaps be fine / a deprecated item)
INFO | Missing 'eeg' run-item in: bidsmap[Presentation][eeg] (NB: this may perhaps be fine / a deprecated item)
INFO | Missing 'ieeg' run-item in: bidsmap[Presentation][ieeg] (NB: this may perhaps be fine / a deprecated item)
INFO | Missing 'meg' run-item in: bidsmap[Presentation][meg] (NB: this may perhaps be fine / a deprecated item)
INFO | Missing 'meg' run-item in: bidsmap[Presentation][meg] (NB: this may perhaps be fine / a deprecated item)
INFO | Missing 'meg' run-item in: bidsmap[Presentation][meg] (NB: this may perhaps be fine / a deprecated item)
INFO | Missing 'headshape' run-item in: bidsmap[Presentation][meg] (NB: this may perhaps be fine / a deprecated item)
INFO | Missing 'markers' run-item in: bidsmap[Presentation][meg] (NB: this may perhaps be fine / a deprecated item)
INFO | Missing 'nirs' run-item in: bidsmap[Presentation][nirs] (NB: this may perhaps be fine / a deprecated item)
INFO | Missing 'bold' run-item in: bidsmap[Presentation][func] (NB: this may perhaps be fine / a deprecated item)
INFO | Missing 'cbv' run-item in: bidsmap[Presentation][func] (NB: this may perhaps be fine / a deprecated item)
INFO | Missing 'sbref' run-item in: bidsmap[Presentation][func] (NB: this may perhaps be fine / a deprecated item)
INFO | Missing 'noRF' run-item in: bidsmap[Presentation][func] (NB: this may perhaps be fine / a deprecated item)
SUCCESS | All datatypes and options in the template bidsmap are valid
INFO | Mapping: /Volumes/data/work/bidscoin/sourcedata/sub-126BPCP021120/ses-1 (subject 1/1)
VERBOSE | Executing plugin: dcm2niix2bids -> /Volumes/data/work/bidscoin/sourcedata/sub-126BPCP021120/ses-1
VERBOSE | Ignoring hidden file/folder: .DS_Store
INFO | Discovered sample: [DICOM] /Volumes/data/work/bidscoin/sourcedata/sub-126BPCP021120/ses-1/005-func_task-Rest_acq-Fs2-30.0/00001.dcm
INFO | Discovered sample: [DICOM] /Volumes/data/work/bidscoin/sourcedata/sub-126BPCP021120/ses-1/007-anat-T1w-2.06/00001.dcm
VERBOSE | Executing plugin: events2bids -> /Volumes/data/work/bidscoin/sourcedata/sub-126BPCP021120/ses-1
VERBOSE | Ignoring hidden file/folder: .DS_Store
INFO | Opening the bidseditor
2025-01-15 14:49:49.078 python3[57285:5095316] +[IMKClient subclass]: chose IMKClient_Modern
2025-01-15 14:49:49.078 python3[57285:5095316] +[IMKInputSession subclass]: chose IMKInputSession_Modern
2025-01-15 14:49:53.929 python3[57285:5095316] The class 'NSSavePanel' overrides the method identifier. This method is implemented by class 'NSWindow'
INFO | Checking the bidsmap run-items:
SUCCESS | All run-items in the bidsmap are valid
INFO | Saving bidsmap in: /Volumes/data/work/bidscoin/code/bidscoin/bidsmap.yaml
INFO | -------------- FINISHED! -------------------
INFO |
SUCCESS | No BIDScoin errors or warnings were reported
INFO |
INFO | For the complete log see: /Volumes/data/work/bidscoin/code/bidscoin/bidsmapper.log
NB: That folder may contain privacy sensitive information, e.g. pathnames in logfiles and provenance data samples

Finally the coiner:

INFO | -------------- START BIDScoiner 4.5.0: BIDS 1.10.0 ------------
INFO | >>> bidscoiner sourcefolder=/Volumes/data/work/bidscoin/sourcedata bidsfolder=/Volumes/data/work/bidscoin participant=None force=False bidsmap=bidsmap.yaml
INFO | Creating dataset description file: /Volumes/data/work/bidscoin/dataset_description.json
INFO | Creating a template README file (adjust it to your needs): /Volumes/data/work/bidscoin/README
INFO | Reading: /Volumes/data/work/bidscoin/code/bidscoin/bidsmap.yaml
INFO | Checking the bidsmap run-items:
SUCCESS | All run-items in the bidsmap are valid
INFO | ------------------- Subject 1/1 -------------------
VERBOSE | Ignoring hidden file/folder: .DS_Store
INFO | >>> Coining dcm2niix2bids datasources in: /Volumes/data/work/bidscoin/sourcedata/sub-126BPCP021120/ses-1
VERBOSE | Executing plugin: dcm2niix2bids
VERBOSE | Ignoring hidden file/folder: .DS_Store
INFO | --> Coining: [DICOM] /Volumes/data/work/bidscoin/sourcedata/sub-126BPCP021120/ses-1/005-func_task-Rest_acq-Fs2-30.0/00001.dcm
VERBOSE | Command:
dcm2niix -b y -z y -i n -l n -f "sub-126BPCP021120_ses-1_task-functaskRestacqFs2_bold" -o "/Volumes/data/work/bidscoin/sub-126BPCP021120/ses-1/func" "/Volumes/data/work/bidscoin/sourcedata/sub-126BPCP021120/ses-1/005-func_task-Rest_acq-Fs2-30.0"
VERBOSE | Output:
Chris Rorden's dcm2niiX version v1.0.20241211 Clang16.0.0 x86-64 (64-bit MacOS)
Found 420 DICOM file(s)
Convert 420 DICOM as /Volumes/data/work/bidscoin/sub-126BPCP021120/ses-1/func/sub-126BPCP021120_ses-1_task-functaskRestacqFs2_bold (96x96x60x420)
Conversion required 38.612474 seconds (36.139294 for core code).

INFO | --> Coining: [DICOM] /Volumes/data/work/bidscoin/sourcedata/sub-126BPCP021120/ses-1/007-anat-T1w-2.06/00001.dcm
VERBOSE | Command:
dcm2niix -b y -z y -i n -l n -f "sub-126BPCP021120_ses-1_acq-anat_T1w" -o "/Volumes/data/work/bidscoin/sub-126BPCP021120/ses-1/anat" "/Volumes/data/work/bidscoin/sourcedata/sub-126BPCP021120/ses-1/007-anat-T1w-2.06"
VERBOSE | Output:
Chris Rorden's dcm2niiX version v1.0.20241211 Clang16.0.0 x86-64 (64-bit MacOS)
Found 1 DICOM file(s)
Convert 1 DICOM as /Volumes/data/work/bidscoin/sub-126BPCP021120/ses-1/anat/sub-126BPCP021120_ses-1_acq-anat_T1w (256x256x224x1)
Conversion required 3.036508 seconds (2.993987 for core code).

VERBOSE | Writing acquisition time data to: /Volumes/data/work/bidscoin/sub-126BPCP021120/ses-1/sub-126BPCP021120_ses-1_scans.tsv
Traceback (most recent call last):
File "/Users/user/opt/miniconda3/bin/bidscoiner", line 8, in
sys.exit(main())
File "/Users/user/opt/miniconda3/lib/python3.9/site-packages/bidscoin/bidscoiner.py", line 315, in main
raise error
File "/Users/user/opt/miniconda3/lib/python3.9/site-packages/bidscoin/bidscoiner.py", line 311, in main
bidscoiner(**vars(args))
File "/Users/user/opt/miniconda3/lib/python3.9/site-packages/bidscoin/bidscoiner.py", line 279, in bidscoiner
personals = plugin.Interface().personals(bidsmap, datasource)
File "/Users/user/opt/miniconda3/lib/python3.9/site-packages/bidscoin/plugins/init.py", line 65, in personals
for key, value in bidsmap.dataformat(datasource.dataformat).participant:
ValueError: too many values to unpack (expected 2)

@marcelzwiers
Copy link
Collaborator

I don't know about your numpy warnings/errors, but I guess they are unrelated to the current issue. I think your installation did not install the latest code (with the path), because your error still shows the old code. That is your line 65 was:

for key, value in bidsmap.dataformat(datasource.dataformat).participant:

But I patched it to:

for key, value in bidsmap.dataformat(datasource.dataformat).participant.items():

@karl-koschutnig
Copy link
Author

Thanks!
I tried multiple installations - nothing really worked. I finally cloned your repo and changed the line manually. Now it works!

VERBOSE | Writing acquisition time data to: /Volumes/data/work/bidscoin/sub-126BPCP021120/ses-1/sub-126BPCP021120_ses-1_scans.tsv
VERBOSE | Writing sub-126BPCP021120 subject data to: /Volumes/data/work/bidscoin/participants.tsv
VERBOSE | Writing subject meta data to: /Volumes/data/work/bidscoin/participants.json
VERBOSE | Ignoring hidden file/folder: .DS_Store
INFO | >>> No events2bids datasources found in '/Volumes/data/work/bidscoin/sourcedata/sub-126BPCP021120/ses-1'
INFO | -------------- FINISHED! ------------
INFO |
SUCCESS | No BIDScoin errors or warnings were reported
INFO |
INFO | For the complete log see: /Volumes/data/work/bidscoin/code/bidscoin/bidscoiner.log
NB: That folder may contain privacy sensitive information, e.g. pathnames in logfiles and provenance data samples

@marcelzwiers
Copy link
Collaborator

Sorry to hear that, I guess you also tried the pip --force-reinstall option? That helped me out in the past...

@marcelzwiers
Copy link
Collaborator

I guess it's a bug (or missing functionality) in pip, it only looks at the package version number when installing. So if you install from github, it doesn't look at the latest commits, and nothing will be updated if the version number (here 4.5.0) didn't change (which it didn't). Hence the force-reinstall

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants