Skip to content
This repository has been archived by the owner on Dec 22, 2022. It is now read-only.

Unreliable remote I/O w/ open_restart #59

Open
nbren12 opened this issue Feb 3, 2021 · 0 comments
Open

Unreliable remote I/O w/ open_restart #59

nbren12 opened this issue Feb 3, 2021 · 0 comments

Comments

@nbren12
Copy link
Contributor

nbren12 commented Feb 3, 2021

Lately, we had hanging tests on fv3net due to I/O problems with fv3gfs.util.open_restart. See ai2cm/fv3net#951. This only happened in the context of an FV3 run. open_restart worked fine with a simple testing script.

This PR solved these problems by using fsspec.get to pre-download the restart files: ai2cm/fv3net#978.

My guess is that it's related to multiple MPI jobs launching downloads and getting stuck. I've also found that fsppec.open does many more HTTP requests than fsspec.get. Maybe we should switch to use that here.

Since remote I/O paradigms are in constant flux, it might be simpler to deprecate this remote I/O capability, and focus fv3gfs-util on local usage.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant