Week 05/01

[02agarwalt]

Remainder of year:

@ebridge2

• run lots of data. EC2 works, so the sky is the limit. Visualize the normal way for each dataset. Start with BNU1, BNU2, DC1, HNU1, SWU4. as mcuh as possible by 05/01.
  
  This was run in the last 10 hours after the memory/processor optimization below, so speed and efficiency are amazing now.
• play with memory more. 05/01. specs
• figure out how to run one subject through CPAC for QA comparison. 05/01 goal. Notes: Don't have proof of execution. Apparently CPAC doesn't produce crash logs when it fails anymore or something, because I ran it and it exited fine (ie, no reported crashes), but the "quality control" page says this:
  
  After execution which means that the software never actually recorded itself finishing, and CPAC makes QA on termination so they didnt actually make any QA.
  Also, for some reason it appears impossible to generate .npy timeseries files (you can only generate 1d averages). Here is the documentation for how to construct your config file:
  
  Here is my config file:
  
  Here are my outputs:
  
  Where the csv is the 1d average file, and the npy timeseries are nowhere in sight >:(

Their bids docker container hasn't been updated in months, so I am thinking maybe it is out of date. Might try to talk w a developer, and if that doesn't work, I'll spin up an EC2 instance w CPAC and see if that might work. Alternative is to take their data derivates (ie, brains) and run them manually through our QA.

• finish up preliminary documentation for Ewok to merge into main documentation, and produce demonstration that what we are doing is working. Discriminability analysis for majority of CoRR; come up with better ways to visualize outputs. 05/08 goal.
• presentation for class started and pipeline portion completed or with TODOs as necessary. 04/27 goal. presentation
• compare CPAC outputs to FNGS qa. 05/04
• notebook running FNGS demo on cortex. reach for 05/01. due by 05/04.

@02agarwalt

• finish up cloud deployment scripts.
• get website back up to speed so it works with the existing batch functionality
  Deliverable: Demo

reach for next week:

• add functionality to website so it can accept a folder from a user, upload it to S3, and alert the user via email
• re-design website
  Specific tasks:
  change job definition name to ndmg-fmri
  change compute environment name to ndmg-fmri-env
  change job queue name to ndmg-fmri-queue
  incorporate custom ami
  restructure website: two separate tabs for submission and query
  change job directory label to token
  make none option for stc
  incorporate data upload feature
  incorporate email feature
  improve design

@ewalke31

• task fMRI justification of our nuisance correction alternative method. Refine/debug/check previous code and make 2 simulations (one where we expect a significant difference in our simulated "white matter", one where we do not) to illustrate that our analysis was actually correct. Then, discriminability results on task fMRI correlation matrices to show that using frequency filtering is more consistent by task. Ask eric for advice on a dataset to do this with.
  https://neurodatadesign.github.io/fngs/ewalke31/nuisance_correction/week_0403/nuiscorr_ftest_v3.html
• finish up documentation for pipeline.
  https://neurodatadesign.github.io/fngs/ewalke31/tutorials/week_0410/Schematic.html
• replace pipeline schematic with eric's new schematic.

[jovo]

ok, figure out which are you committed to this week, and make the rest reach.

do 1 of these for each of you this week