Challenge 22 - XAI for Weather Forecasting Models (Transformer Embeddings)

[RubenRT7]

Challenge 22 - XAI for Weather Forecasting Models (Transformer Embeddings)

Stream 2 - Machine Learning for Earth Sciences applications

Goal

Welcome to the XAI Transformer Embedding Challenge for Weather Forecasting Models! In this challenge, participants will explore the explainability of Transformer embeddings in the context of weather forecasting models, using models like PanguWeather. The goal is to develop insights and techniques that enhance the interpretability of these AI models. Additionally, participants are encouraged to consider the possibility of utilizing visualization techniques similar to the ones developed by BertViz, Quantus or SHAP tailored to weather forecasting data.

Mentors and skills

• Mentors: Mariana Clare, Jesper Dramsch, Ana Prieto Nemesio, Mario Santa Cruz (all ECMWF)
• Skills required:
  • Python programming (numpy, pandas, xarray, matplotlib, cartopy …)
  • Machine learning model development (scikit-learn, Pytorch, Tensorflow and/or Jax)
  • Processing and visualization of scientific data (netCDF, HDF, GRIB, …)
  • Theoretical and practical knowledge of Machine Learning tools
  • Some experience/knowledge of meteorology

---

Challenge description

Traditional weather forecasting models often rely on solving physical equations to make a forecast. However, the emergence of deep learning and transformer-based models has shown potential in improving forecast accuracy. Transformers, renowned for their effectiveness in natural language processing tasks, have been adapted to time series forecasting, including weather prediction due to their ability to capture complex spatiotemporal patterns in weather data. However, as these models become more complex, understanding how they arrive at their predictions becomes crucial for trust, accountability, and further model improvements.

Ideas for the challenge implementation:

• Data Exploration and Analysis:
  The challenge involves exploring and analyzing the weather data used to train the AI weather forecasting models to gain insights into the patterns, trends, and relationships between different weather variables.
• Transformer Embeddings Understanding:
  Participants will leverage pre-trained Transformer models (such as PanguWeather, AIFS* etc.) fine-tuned on weather forecasting tasks to analyze the embeddings generated by these models and understand how they encode information about weather patterns and features.
• Explainability Techniques Development:
  Participants will develop explainability techniques tailored to Transformer embeddings in the context of weather forecasting, to enhance model interpretability, providing insights into how specific weather events or patterns are represented in the embeddings and how they contribute to the final predictions.
• Evaluation and Interpretation:
  Participants will evaluate the effectiveness of their explainability techniques using relevant metrics. They will interpret the insights gained from their techniques and assess their utility in improving the interpretability and trustworthiness of weather forecasting models.
  Participants will also be encouraged to implement visualization techniques tailored to weather forecasting data, to visualize attention mechanisms and model predictions (inspired by BertViz).

[mariebrl]

Hello,
Thank you for creating this challenge. I am very interested in writing a proposal for it, if anyone wants to team up let me know.

Best,

Marie

[mrvozdas]

Hello, Thank you for creating this challenge. I am very interested in writing a proposal for it, if anyone wants to team up let me know.

Best,

Marie

Hello Marie,

I want to team up with you if that's ok with you.

Best,

Merve

[SamMajumder]

Hi Marie, Merve and mentors

I am interested in this challenge. I'd like to participate.

Best,
Sambadi

[hoonerg]

Hello, Thank you for creating this challenge. I am very interested in writing a proposal for it, if anyone wants to team up let me know.

Best,

Marie

Hi Marie,
I'd like to collaborate on the challenge.

[Pei4579]

Hello, Thank you for creating this challenge. I am very interested in writing a proposal for it, if anyone wants to team up let me know.

Best,

Marie

Hi Marie,

I'm also interested in this challenge. And I have been working on XAI for my project recently. May I team up with you?

Best,
Peichun

[mariebrl]

Hello,
That's great, thank you for joining the team! I created a repository and invited you so we can start brainstorming ideas for the proposal. Let me know if you did not receive the invite
Best,
Marie

[Pei4579]

Hello, That's great, thank you for joining the team! I created a repository and invited you so we can start brainstorming ideas for the proposal. Let me know if you did not receive the invite Best, Marie

I received your invitation to a repository, thank you!
After everyone joins shall we have an online meeting or something? For setting some milestones, and deadlines also knowing what's our scope.

[Pei4579]

Hello, That's great, thank you for joining the team! I created a repository and invited you so we can start brainstorming ideas for the proposal. Let me know if you did not receive the invite Best, Marie

Oh, yep. Should we have a messenger group or something like that?

[mariebrl]

Hello, I created a discussion channel in the repository, we can start with this to share ideas. But I think you're right it would be good if we could schedule an online meeting in the coming days.
Let's move the discussion to the repository.
Best,
Marie

[jayteeG]

Hello, Thank you for creating this challenge. I am very interested in writing a proposal for it, if anyone wants to team up let me know.

Best,

Marie

Hi,

I am also interested in joining the team if still taking applications :). My background is in data science/engineering and geophysical fluid dynamics.

Best,

James

[mariebrl]

Hi James, yes, of course, let me add you to the repository

Best,
Marie

[scarletto999]

Hello @mariebrl,
Seems that I am late to the party. After today's webinar recommendation to join a team I thought to contact you and I see that you already found a lot of team members. Can I still join you?
Cheers, Scarlet

[scarletto999]

Hello @mc4117,
Could you clarify the project conditions: do you want us to train a transformer weather model and explain it or do you want to provide a pre-trained transformer which we explain post-hoc?
Cheers,
Scarlet

[mc4117]

Hi Scarlet @scarletto999, GIven the time limitations, we think it would probably be better to use a pre-trained transformer and explain post-hoc

[mariebrl]

Hello @mc4117,
Regarding the AIFS model, is it possible to share the documentation with us so we can get a better understanding of its architecture?
Kind regards,
Marie

[mc4117]

Hi Marie,

You should find more about the model architecture here https://www.ecmwf.int/en/about/media-centre/aifs-blog/2024/first-update-aifs

Mariana

[scarletto999]

Hello Mentors,

we wonder if you expect us to propose a general approach suitable for "weather transformers" or if we should be more specific about an approach to explain PanguWeather? There is no documentation yet on AIFS and thus it is impossible to come up with a method tailored to its architecture.

Scarlet

[sandupal]

Hello, Thank you for creating this challenge. I am very interested in writing a proposal for it, if anyone wants to team up let me know.
Best,
Marie

Hello Marie,

I want to team up with you if that's ok with you.

Best,

Merve

Hi,
I am interested in teaming up. Let me know how I can connect with you?
Thanks

[kam3545]

Hi,
I am also interested in joining the team.
My background is in regional weather forecasting and data assimilation.
Regards

Prashant

[chicha1986]

Hello,
I interesting about this program, I work on the ensemble forecast using opensource AI weather model, and accelerate it.
I major in verification NWP models, and have been to work with XAI(LRP).
Wish have chance to join your teams.
Best regards,

chicha

[thinhngo-x]

Hello,

I would like to participate in this challenge. May I ask, what are the expected deliverables for this challenge? Like code, reports, etc..?

Best regards,

[oriollacave]

Hi, I'm also interested in this challenge. I have experience on NWP, data processing, validating but would like to explore XAI weather models. There might some people who could team up in a different team if there is already too many people on @scarletto999 @sandupal @kam3545 @chicha1986 @thinhngo-x . Just show you interest and we can create a repo to work on.

[anaprietonem]

Hello,

I would like to participate in this challenge. May I ask, what are the expected deliverables for this challenge? Like code, reports, etc..?

Best regards,

Hello in terms of deliverables, for each of the challenges, teams are expected to prepare a presentation to present their work and findings as well as to share the code in github

[Teklehaim]

Dear weather team,
Recently, my former study coordinator shared about this exciting project. I would like to join and contribute. Please let me know if could join you.I have good knowledge in meteorology related to energy, programming in python.
Thank you.

Regards,

Teklehaimanot

[lheyer]

Hi everyone here,
is it still possible to join a team here?
I have experience in Python (many different frameworks, incl. pandas, numpy, xarray, pytorch, cartopy & matplotlib), with netCDF & HDF, and my background lies in physics and i have a interest, though not so much knowledge so far, in meteorology since I worked at the german weather service for a year.

kind regards,
Lisa

[trakasa]

Dear @Teklehaim,
many thanks for your interest in Code for Earth and this challenge!

If you want to participate in Code for Earth, you need to submit a proposal, explaining how you would solve the stated problem of the challenge.
You can do this alone or with a team of people.

The deadline for submission ends 9. April and then we will evaluate the proposals. Those selected will work over the summer on their projects and the mentors will provide some guidance.

For more information, please read the FAQ and Terms & Conditions on our website.

Additionally you might want to listen to the recording of the Q&A session on Youtube.

I hope this gets you going!
And questions to the mentors can be asked here...

Bye, Athina

Dear weather team, Recently, my former study coordinator shared about this exciting project. I would like to join and contribute. Please let me know if could join you.I have good knowledge in meteorology related to energy, programming in python. Thank you.

Regards,

Teklehaimanot

[Teklehaim]

Thank you very much @trakasa for the information. I sent my application today.
Look forward to collaborating in this amazing projects.

Regards,
Teklehaimanot

[oriollacave]

Dear @Teklehaim, many thanks for your interest in Code for Earth and this challenge!

If you want to participate in Code for Earth, you need to submit a proposal, explaining how you would solve the stated problem of the challenge. You can do this alone or with a team of people.

The deadline for submission ends 9. April and then we will evaluate the proposals. Those selected will work over the summer on their projects and the mentors will provide some guidance.

For more information, please read the FAQ and Terms & Conditions on our website.

Additionally you might want to listen to the recording of the Q&A session on Youtube.

I hope this gets you going! And questions to the mentors can be asked here...

Bye, Athina

Dear weather team, Recently, my former study coordinator shared about this exciting project. I would like to join and contribute. Please let me know if could join you.I have good knowledge in meteorology related to energy, programming in python. Thank you.
Regards,
Teklehaimanot

Hi,

I was checking the apply form and it does not work. It should until today end of the day, right?

Thank you

[trakasa]

Dear @oriollacave
thanks for the information and letting us know about your issues with the submission form. This should have been sorted out and you should be able to submit your proposal.

Please let us know, if you still have any issues.

Apologies for the inconvenience and thanks for your patience!

Bye, Athina

Hi,

I was checking the apply form and it does not work. It should until today end of the day, right?

Thank you

[trakasa]

@oriollacave ... forgot to mention: yes, deadline for submission is tonight at 23:59 CET ;-)