Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add GlacierMappingAlps dataset #2508

Open
wants to merge 17 commits into
base: main
Choose a base branch
from

Conversation

dcodrut
Copy link

@dcodrut dcodrut commented Jan 9, 2025

This PR adds a a Multi-modal Dataset for Glacier Mapping (Segmentation) in the European Alps.

The dataset consists of Sentinel-2 images from 2015 (mainly), 2016 and 2017, and binary segmentation masks for glaciers, based on an inventory built by glaciology experts (Paul et al. 2020).
Given that glacier ice is not always visible in the images, due to seasonal snow, shadow/cloud cover and, most importantly, debris cover, the dataset also includes additional features that can help in the segmentation task.

Here's a sample extracted with the plot function implemented in the dataset:
image

A preprint is available here that describes in detail how the dataset was constructed.
For a shorter description check also: https://huggingface.co/datasets/dcodrut/dl4gam_alps.

@github-actions github-actions bot added documentation Improvements or additions to documentation datasets Geospatial or benchmark datasets testing Continuous integration testing labels Jan 9, 2025
Copy link
Collaborator

@nilsleh nilsleh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for the contribution :) I hope this first round of review is able to resolve the import xarray error that is preventing the unit tests from running. Once they are working, will take another look for anything else!

torchgeo/datasets/glacier_mapping_alps.py Outdated Show resolved Hide resolved
torchgeo/datasets/glacier_mapping_alps.py Outdated Show resolved Hide resolved
torchgeo/datasets/glacier_mapping_alps.py Outdated Show resolved Hide resolved
torchgeo/datasets/glacier_mapping_alps.py Outdated Show resolved Hide resolved
torchgeo/datasets/glacier_mapping_alps.py Outdated Show resolved Hide resolved
@nilsleh nilsleh added this to the 0.7.0 milestone Jan 9, 2025
@github-actions github-actions bot added the dependencies Packaging and dependencies label Jan 9, 2025
@dcodrut
Copy link
Author

dcodrut commented Jan 9, 2025

@microsoft-github-policy-service agree

@dcodrut
Copy link
Author

dcodrut commented Jan 9, 2025

Thanks a lot for the contribution :) I hope this first round of review is able to resolve the import xarray error that is preventing the unit tests from running. Once they are working, will take another look for anything else!

I hope it will be somehow useful. Happy to contribute and learn new things on the way :)
Thanks for the comments and please let me know if there are additional things I should change.

@adamjstewart
Copy link
Collaborator

I'm completely fine with adding xarray and netcdf4 as optional dependencies. We've been talking about adding xarray support for a long time. We're still not sure if it will be directly through xarray or something like rioxarray, but that doesn't matter for this PR.

@adamjstewart
Copy link
Collaborator

Can you resolve the merge conflicts so the tests run?

@adamjstewart
Copy link
Collaborator

adamjstewart commented Jan 15, 2025

Can you resolve the merge conflicts? These requirements files unfortunately get updated a lot, causing merge issues.

adamjstewart
adamjstewart previously approved these changes Jan 15, 2025
Copy link
Collaborator

@adamjstewart adamjstewart left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code looks fantastic, just a few minor formatting comments. Thanks for taking the time to contribute your dataset to TorchGeo, let's hope it makes it easier for people to actually use and do cool science with!

requirements/datasets.txt Outdated Show resolved Hide resolved
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These also need to be added to pyproject.toml. We'll need to find minimum versions that allow the tests to pass. I can help with this if you want.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found the minimum versions, manually. Do you use any tool for this?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nah, I do it manually too, you can use binary search to speed it up though.

)


class GlacierMappingAlps(NonGeoDataset):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we rename the dataset to DL4GAM or DL4GAMAlps? Or is DL4GAM the project/model and GlacierMappingAlps is the name of the dataset? It's up to you, it's your dataset.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was not sure when I named it. Thanks for the suggestion, I think the second (DL4GAMAlps) makes more sense, and it's also consistent with the HF dataset.

torchgeo/datasets/glacier_mapping_alps.py Outdated Show resolved Hide resolved
torchgeo/datasets/glacier_mapping_alps.py Outdated Show resolved Hide resolved
torchgeo/datasets/glacier_mapping_alps.py Outdated Show resolved Hide resolved
torchgeo/datasets/glacier_mapping_alps.py Outdated Show resolved Hide resolved
torchgeo/datasets/glacier_mapping_alps.py Outdated Show resolved Hide resolved
torchgeo/datasets/glacier_mapping_alps.py Outdated Show resolved Hide resolved
docs/api/datasets/non_geo_datasets.csv Outdated Show resolved Hide resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
datasets Geospatial or benchmark datasets dependencies Packaging and dependencies documentation Improvements or additions to documentation testing Continuous integration testing
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants