Skip to content

Commit

Permalink
docs: updated with latest justfile
Browse files Browse the repository at this point in the history
  • Loading branch information
@danellecline
danellecline committed Dec 30, 2024
1 parent 074b6b5 commit bf649c4
Show file tree
Hide file tree
Showing 2 changed files with 75 additions and 37 deletions.
74 changes: 42 additions & 32 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -69,38 +69,48 @@ Recipes are available to run the pipelines. To see the available recipes, run t
```shell
just list
```

| Recipe Name | Description |
|----------------------------------|-------------------------------------------------------------------------------------------------------------|
| `list` | List recipes |
| `install` | Setup the environment |
| `update_trackers` | Update the environment. Run this command after checking out any code changes |
| `plot-tsne-vss project='uav'` | Generate a tsne plot of the VSS database |
| `optimize-vss project='uav'` | Optimize the VSS database |
| `calc-acc-vss project='uav'` | Calculate the accuracy of the VSS database; run after download, then optimize |
| `reset-vss-all` | Reset the VSS database, removing all data. Proceed with caution!! |
| `reset-vss project='uav'` | Reset the VSS database, removing all data. Run before init-vss or when creating the database. Run with e.g. `uav` |
| `remove-vss project='uav'` | Remove an entry from the VSS database, e.g., `remove-vss i2map --doc 'doc:marine organism:*'` |
| `init-vss project='uav'` | Initialize the VSS database for a project |
| `load-vss project='uav'` | Load already computed exemplars into the VSS database |
| `cluster-uav` | Cluster mission in `aipipeline/projects/uav/data/missions2process.txt` |
| `detect-uav` | Detect mission in `aipipeline/projects/uav/data/missions2process.txt` |
| `detect-uav-test` | Detect mission data in `aipipeline/projects/uav/data/missions2process.txt` |
| `load-uav-images` | Load UAV mission images in `aipipeline/projects/uav/data/missions2process.txt` |
| `load-uav type="cluster"` | Load UAV detections/clusters in `aipipeline/projects/uav/data/missions2process.txt` |
| `fix-uav-metadata` | Fix UAV metadata lat/lon/alt |
| `compute-saliency project='uav'` | Compute saliency for downloaded VOC data and update the Tator database |
| `crop project='uav'` | Crop detections from VOC formatted downloads |
| `download-crop project='uav'` | Download and crop |
| `download project='uav'` | Download only |
| `predict-vss project='uav'` | Predict images using the VSS database, e.g., `predict-vss image_dir='/tmp/download'` |
| `run-ctenoA-test` | Run the strided inference on a single video |
| `run-ctenoA-prod` | Run the strided inference on a collection of videos in a TSV file |
| `run-mega-inference` | Run the mega strided inference only on a single video |
| `run-mega-track-bio-video` | Run the mega strided tracking pipeline on a single video for the bio project for 30 seconds |
| `run-mega-track-bio-dive` | Run the mega strided tracking pipeline on an entire dive for the bio project |
| `run-mega-track-i2map` | Run the mega strided tracking pipeline on a single video for the i2map project |

| Recipe Name | Description |
|----------------------------------|--------------------------------------------------------------------------------------------------------------------|
| list | List recipes |
| install | Setup the environment |
| cp-env | Copy the default .env file to the project |
| update_trackers | Update the environment. Run this command after checking out any code changes |
| cp-core | Copy core dev code to the project on doris |
| cp-dev-uav | Copy uav dev code to the project on doris |
| cp-dev-bio | Copy bio dev code to the project on doris |
| cp-dev-i2map | Copy i2map dev code to the project on doris |
| plot-tsne-vss project='uav' | Generate a tsne plot of the VSS database |
| optimize-vss project='uav' | Optimize the VSS database |
| calc-acc-vss project='uav' | Calculate the accuracy of the VSS database; run after download, then optimize |
| reset-vss-all | Reset the VSS database, removing all data. Proceed with caution!! |
| reset-vss project='uav' | Reset the VSS database, removing all data. Run before init-vss or when creating the database. |
| remove-vss project='uav' | Remove an entry from the VSS database |
| init-vss project='uav' | Initialize the VSS database for a project |
| load-vss project='uav' | Load already computed exemplars into the VSS database |
| load-cfe-isiis-videos | Load CFE ISII mission videos |
| cluster-uav | Cluster mission in UAV data |
| detect-uav | Detect mission in UAV data |
| detect-uav-test | Detect mission data in UAV data |
| load-uav-images | Load UAV mission images |
| load-uav type="cluster" | Load UAV detections/clusters |
| fix-uav-metadata | Fix UAV metadata lat/lon/alt |
| compute-saliency project='uav' | Compute saliency for VOC data |
| crop project='uav' | Crop detections from VOC formatted downloads |
| download-crop project='uav' | Download and crop |
| download project='uav' | Download only |
| predict-vss project='uav' | Predict images using the VSS database |
| run-ctenoA-test | Run strided inference on a single video |
| run-ctenoA-prod | Run strided inference on a collection of videos |
| run-mega-inference | Run mega strided inference on a single video |
| run-mega-track-bio-video | Run mega tracking pipeline on a single video for bio project |
| run-mega-track-bio-dive | Run mega tracking pipeline on an entire dive for bio project |
| run-mega-track-i2map-video | Run mega tracking pipeline on a single video for i2map project |
| run-mega-track-test-1min | Run mega tracking pipeline on a single video for testing |
| run-mega-track-test-fastapiyv5 | Test pipeline using FastAPI |
| gen-cfe-data | Generate training data for the CFE project |
| gen-i2map-data | Generate training data for the i2map project |
| gen-i2mapbulk-data | Generate training data for the i2map project from the bulk server |
| gen-uav-data | Generate training data for the UAV project |

--

Expand Down
38 changes: 33 additions & 5 deletions justfile
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
#!/usr/bin/env just --justfile

# Source the .env file
# copy the default .env file
## Source the .env file and support an alternative name
set dotenv-load := true
set dotenv-filename := x'${ENV_FILE:-.env}'

# List recipes
list:
Expand All @@ -16,6 +18,9 @@ install: update_trackers
cd deps/co-tracker && conda run -n aipipeline pip install -e .
cd .. && mkdir checkpoints && cd checkpoints && wget https://huggingface.co/facebook/cotracker3/resolve/main/scaled_offline.pth

# Copy the default .env file to the project
cp-env:
@cp .env.mantis .env
# Update the environment. Run this command after checking out any code changes
update_trackers:
conda env update_trackers --file environment.yml --prune
Expand All @@ -38,6 +43,11 @@ cp-dev-bio:
cp ./aipipeline/projects/bio/config/ /Volumes/dcline/code/aipipeline/aipipeline/projects/bio/config/
cp ./aipipeline/projects/bio/model/*.py /Volumes/dcline/code/aipipeline/aipipeline/projects/bio/model/

# Copy i2map dev code to the project on doris
cp-dev-i2map:
cp ./aipipeline/projects/i2map/config/* /Volumes/dcline/code/aipipeline/aipipeline/projects/i2map/config/
cp ./aipipeline/projects/i2mapbulk/config/* /Volumes/dcline/code/aipipeline/aipipeline/projects/i2mapbulk/config/

# Generate a tsne plot of the VSS database
plot-tsne-vss project='uav':
#!/usr/bin/env bash
Expand Down Expand Up @@ -281,10 +291,9 @@ run-mega-track-bio-dive dive='/mnt/M3/mezzanine/Ventana/2022/09/4432' gpu_id='0'
find "{{dive}}" -name '*.m*' ! -name "._*.m*" -type f | xargs -P 1 -n 1 -I {} bash -c 'process_file "{}"'
# Run the mega strided tracking pipeline on a single video for the i2map project
run-mega-track-i2map video='/mnt/M3/master/i2MAP/2019/02/20190204/i2MAP_20190205T102700Z_200m_F031_17.mov' gpu_id='0':
run-mega-track-i2map-video video='/mnt/M3/master/i2MAP/2019/02/20190204/i2MAP_20190205T102700Z_200m_F031_17.mov' gpu_id='0':
#!/usr/bin/env bash
export PYTHONPATH=.:deps/biotrack:.
#videos=("i2MAP_20190205T102700Z_200m_F031_17.mov" "i2MAP_20190205T104923Z_400m_F031_19.mov" "i2MAP_20190205T104923Z_400m_F031_19.mov")
time python3 aipipeline/projects/bio/process.py \
--config ./aipipeline/projects/i2map/config/config.yml \
--det-model /mnt/DeepSea-AI/models/megadet \
Expand All @@ -293,7 +302,8 @@ run-mega-track-i2map video='/mnt/M3/master/i2MAP/2019/02/20190204/i2MAP_20190205
--stride-fps 15 --max-seconds 60 --imshow --skip-load \
--video {{video}} --gpu-id {{gpu_id}}
run-mega-track-test-yv5:
# Run the mega strided tracking pipeline on a single video to test the pipeline
run-mega-track-test-1min:
#!/usr/bin/env bash
export PYTHONPATH=.:/Users/dcline/Dropbox/code/biotrack:.
time python3 aipipeline/projects/bio/predict.py \
Expand All @@ -304,7 +314,6 @@ run-mega-track-test-yv5:
--stride-fps 15 --max-seconds 60 --imshow --skip-load \
--video aipipeline/projects/bio/data/V4361_20211006T163256Z_h265_1min.mp4

run-mega-track-test-fastapiyv5:
#!/usr/bin/env bash
export PYTHONPATH=.:/Users/dcline/Dropbox/code/biotrack:.
Expand All @@ -315,3 +324,22 @@ run-mega-track-test-fastapiyv5:
--stride-fps 15 --max-seconds 60 --imshow --skip-load \
--endpoint-url http://FastAP-FastA-0RIu3xAfMhUa-337062127.us-west-2.elb.amazonaws.com/predict \
--video aipipeline/projects/bio/data/V4361_20211006T163256Z_h265_1min.mp4
# Generate training data for the CFE project
gen-cfe-data:
just --justfile {{justfile()}} download-crop cfe --skip-clean True --version Baseline

# Generate training data for the i2map project
gen-i2map-data:
just --justfile {{justfile()}} download-crop i2map --skip-clean True --version Baseline

# Generate training data for the i2map project from the bulk server, run with ENV_FILE=.env.i2map just gen-i2mapbulk-data
gen-i2mapbulk-data:
just --justfile {{justfile()}} download-crop i2mapbulk --skip-clean True --version Baseline
just --justfile {{justfile()}} download-crop i2mapbulk --skip-clean True --version dino_vits8_20240207_022529

# Generate training data for the uav project
gen-uav-data:
just --justfile {{justfile()}} download-crop i2map --skip-clean True --version Baseline
just --justfile {{justfile()}} download-crop i2map --skip-clean True --version yolov5x6-uavs-oneclass-uav-vit-b-16
just --justfile {{justfile()}} download-crop i2map --skip-clean True --version uav-yolov5-30k-vs

0 comments on commit bf649c4

Please sign in to comment.