Cache csv download data and shas #1746
Conversation
We don't need to rebuild the caches after import, but I was concerned that if we don't, we'll still sometimes get the behaviour we saw yesterday on OpenCodelists, where the check endpoint made everything go slow. Given that it takes a minimal time to re-cache 600+ dm+d codelist versions though, it's possible that dm+d isn't actually the problem, it's more studies that have huge numbers of codelists in general. However, there's also a chance of a race if two users try to download a codelist that needs cached at the same time (also CI runs/job-server checks, since the check endpoint will also re-cache if necessary). The import runs automatically at about 2am, so there's probably less risk of a race with users if it re-caches then. I could make the re-cache after import an optional flag, so it does it in the automated 2am import, but not if we re-run an import manually (more likely to be during working hours) |
I don't think it's worth adding the complexity for this. |
Fixes #1744
And hopefully also fixes #1740
The opensafely CLI, study github actions and jobserver now all call the new codelist check endpoing /api/v1/check. Mostly this is a pretty fast call, but for very large dm+d codelists, and/or studies that have a lot of codelists, especially dm+d ones, it can be slow, and it seems to be the cause of tech-support issue #1740
We can cache these shas so we don't have to go through the VMP mapping checks every time. dm+d is release once a week, so at most we need to re-cache weekly. There are under 700 dm+d codelist versions that would need to be re-cached, and locally this is quick to do, so it can be done as part of the weekly import.
Since caching the shas involves fetching the CSV data for downloading, we can also cache that. It's caching with a key that includes the function parameters so that we can cache different calls (e.g. OSI will always pass
fixed_headers=True)