[WIP] Add update.py semi-automatic release crawler #1

octomike · 2020-05-10T19:10:47Z

This PR is an attempt to reactivate this repo and help keep MCR images up to date in an automated or at least semi-automated way.
The included update.py helper script tries to parse the Matlab MCR release page and creates branches/tags containing a Dockerfile based on one ore more templates.

Each Matlab release name (R2020a etc) gets a branch that contains a single
Dockerfile and each release version (9.8.0 or 9.8.5 for 9.8 Update 5) becomes
a tag in that branch.
So a sample history for R2020a could look something like this:

 + [R2020a] <9.8.1-core> Auto-Update
 + <9.8.1> Auto-Update
 + Merged master
 | \
 |  + Update in templates
 | /
 + <9.8.0-core> Auto-Update
 + <9.8.0> Auto-Update
 + <master> Import

(Circle)CI should then fire a docker build and push for every tag only. Shared
tags for the major version (e.g. 9.8 always pointing to the latest 9.8 tag are
done in Circle CI to avoid duplicate builds).

I successfully tested the script by running update.py and pushing the created branches first and then all the tags to populate a copy here: https://hub.docker.com/r/octomike/matlab-compiler-runtime/tags

The script also supports Dockerfile variants and I used that to create a "core" variant that only installs parts of the huge MCR package and also afterwards drastically deletes hopefully unnecessary folders. That last bit is completely done on a hunch and probably subject to further testing and bugs.

But, I was able to successfully run the shipped pipelines for https://github.com/BIDS-Apps/SPM/, which I assume would be the main beneficiary here. The singularity image size went down from 2.8G to 885M in that case.

There is also an open WIP PR for SPM basing it on matlab-compiler-runtime.

sappelhoff

good idea to try to automate the updating of the MCR docker file. 👍

seeing the tags of the images on https://hub.docker.com/r/octomike/matlab-compiler-runtime/tags I just think that including the actual Matlab version in the tag name might be nice (e.g., R2012a).

Another question: given that we merge this PR, will any existing pipelines depending on the current bids/matlab-compiler-runtime break?

.circleci/config.yml

sappelhoff · 2020-05-11T14:20:59Z

update.py

+tags for the major version (e.g. 9.8 always pointing to the latest 9.8 tag are
+done in Circle CI to avoid duplicate builds).
+
+Update.py should be run often enough to catch individual matlab releae updates.


Suggested change

Update.py should be run often enough to catch individual matlab releae updates.

Update.py should be run often enough to catch individual matlab release updates.

I don't get the "semi updating" process yet. Does it mean a potential maintainer regularly runs update.py from their machine and then commits and pushes any changes?

Are tags generated automatically?

Right now update.py is simply generating git tags in separate branches and afterwards outputs necessary git commands to trigger circleCI actually building new images based on git tags.

Example output would be:

Adding R2020a/9.8.0 Adding R2020a/9.8.0-core Adding R2019b/9.7.5 Adding R2019b/9.7.5-core Adding R2019a/9.6.8 Adding R2019a/9.6.8-core Adding R2018b/9.5.0 Adding R2018b/9.5.0-core Adding R2018a/9.4.0 Adding R2018a/9.4.0-core New tags have been added, verify and update to git with: git push --all git push origin 9.4.0-core git push origin 9.4.0 git push origin 9.5.0-core git push origin 9.5.0 git push origin 9.6.8-core git push origin 9.6.8 git push origin 9.7.5-core git push origin 9.7.5 git push origin 9.8.0-core git push origin 9.8.0

Those last git push lines can easily be automated away of course but my experience with Matlab tells me that it will still require minor changes once in a while, so this semi-automated process boils down to:

run update.py

inspect and potentially fix new releases (minor changes in the URL probably)

git push all the things

That shouldn't take more than 5 minutes every other month or so.

gllmflndn · 2020-05-11T16:05:30Z

Mentioning your work to @kaczmarj as it might be relevant to Neurodocker.

octomike · 2020-05-11T16:23:47Z

Another question: given that we merge this PR, will any existing pipelines depending on the current bids/matlab-compiler-runtime break?

Right now there is only one tag on dockerhub: R2016b. As I do not change that image and because there is no floating latest tag all existing images that reference bids/matlab-compiler-runtime:R2016b won't be affected. This PR simply enables other projects to build on top of different MCR versions.

sappelhoff · 2020-05-11T16:28:45Z

okay, thanks

This PR simply enables other projects to build on top of different MCR versions.

I think the proposed changes are very reasonable.

What do you think @gllmflndn?

@chrisgorgo any concerns from your side to update this repo?

octomike · 2020-05-11T16:31:11Z

think that including the actual Matlab version in the tag name might be nice (e.g., R2012a).

The naming convention for Matlab/MCR seems a bit inconsistent to me. I suspect that with every biannual release (R20XXc) they also increase the second digit in their version (9.N -> 9.M), but I'm not sure. They do have a 5-part version on other places (I have seen something like 9.7.0-123456 Update X).

So if a user wants to have say R2020a - what image should this refer to? The original release? Or the most recent major patch release (9.X.Y if R2020a initially was 9.X).

kaczmarj · 2020-05-11T16:45:42Z

The naming convention for Matlab/MCR seems a bit inconsistent to me. I suspect that with every biannual release (R20XXc) they also increase the second digit in their version (9.N -> 9.M), but I'm not sure. They do have a 5-part version on other places (I have seen something like 9.7.0-123456 Update X).

@octomike - this is usually true, but not always. i made a mapping of the most recent matlab versions -> mcr versions in https://github.com/ReproNim/neurodocker/blob/master/neurodocker/interfaces/interfaces.py#L139-L176 (as @gllmflndn linked above)

octomike · 2020-05-11T17:27:21Z

😩 Thanks for mentioning this!

Anyway, since the Matlab runtime version is picked up by update.py it should handle cases where a new Matlab release is not consistent with a minor upgrade (like what happened early 2016).

To avoid confusion I think I'll just add a rolling tag R20XXc to the top of its branch as well. So people can either pick a base image based on the MCR version number or the corresponding Matlab release name.

Does that make sense?

octomike · 2020-05-11T17:39:49Z

Btw, how do you feel about that awful -core variant hack? :)

sappelhoff · 2020-05-11T17:42:16Z

So people can either pick a base image based on the MCR version number or the corresponding Matlab release name. Does that make sense?

yes 👍

Btw, how do you feel about that awful -core variant hack? :)

not sure -> if I get you correctly then this has not been tested so far. So maybe we should do the testing beforehand? E.g., via your octomike images?

Is a smaller image size the only advantage of this hack?

octomike · 2020-05-11T18:36:34Z

if I get you correctly then this has not been tested so far.

My only test was with SPM's default pipeline configs and that worked out okay. But it can easily fail as soon as some other function calls a missing MCR function or toolbox.

Is a smaller image size the only advantage of this hack?

Yes it is, but it makes handling MCR-based much easier in my opinion.

Ultimately the maintainer can just deactivate variants in update.py

gllmflndn · 2020-05-11T19:23:55Z

Dockerfile-core.template

@@ -0,0 +1,42 @@
+FROM bids/base_validator


This is the only BIDS-related item: why not base directly on,e.g., ubuntu?

I have no idea why this is the base image. Is it containing something that BIDS-Apps expect?

Can't remember now but given the --skip-bids-validator flag, I assume that a BIDS-App would by default validate the BIDS input and therefore expect bids-validator to be available.

bids/base_validator is ubuntu with bids-validator installed. But it seems to be very outdated as well: https://github.com/BIDS-Apps/base_validator/blob/master/Dockerfile

That one could probably be based on node:current-slim

If bids/base_validator is meant to be a base image on which to base BIDS app then node:current-slim would probably not be first choice. Even fmriprep is not based on bids/base_validator but a very specific image (ubuntu:xenial-20200114). Not sure one can impose an base, therefore bids/base_validator and bids/matlab-compiler-runtime are more examples than base images?

Not sure one can impose an base, therefore bids/base_validator and bids/matlab-compiler-runtime are more examples than base images?

perhaps. 🤷‍♂️ I'll ask about this in an upcoming meeting with other BIDS folks who may know.

Dockerfile-core.template

gllmflndn · 2020-05-11T21:19:32Z

I wonder if this shouldn't be hosted elsewhere as it can be used by a MATLAB-based BIDS-App but it's potentially more generic than that.
Also, if it's a recipe to install a given MATLAB Runtime in a Docker container, could it benefit being used in a multi-stage build?
As for the "core" variant, I think it would be a valuable option but it might be sometimes difficult to draw the line as different tools might have different requirements. For example, by removing libQt* and qtwebengine, does it break the SPM GUI (you can secretly access it from the --gui flag)?

octomike · 2020-05-12T08:38:58Z

I wonder if this shouldn't be hosted elsewhere as it can be used by a MATLAB-based BIDS-App but it's potentially more generic than that.

Well true, this should be managed by the Matlab folks. On the other hand then we wouldn't be able to easily include the bids validator.

Also, if it's a recipe to install a given MATLAB Runtime in a Docker container, could it benefit being used in a multi-stage build?

If I understand multi stage builds correctly then the advantage is that the resulting image does not contain any bits of the (bloated) build environment. This is potentially super useful for small, statically built go apps, but not necessarily in this case. But I can check how big the difference is if the final stage does not include xorg.

As for the "core" variant, I think it would be a valuable option but it might be sometimes difficult to draw the line as different tools might have different requirements. For example, by removing libQt* and qtwebengine, does it break the SPM GUI (you can secretly access it from the --gui flag)?

I don't know, possibly. My approach would be to test different SPM use cases with -core once in a while and see if it breaks. But that doesn't have to happen in this repo.

gllmflndn · 2020-05-12T09:09:26Z

Well true, this should be managed by the Matlab folks.

Not with the MATLAB Runtime (yet?) but there is some work in this direction.

On the other hand then we wouldn't be able to easily include the bids validator.

I guess this is the point I want to make: should your work be standalone, i.e. provide a (community-driven) way to install any MATLAB Runtime in a Docker container (e.g. potentially usable in Neurodocker) or something specific to MATLAB-based BIDS apps? For example, as a user I would still want to be able to choose the base image and if you were to make that an option here then, in my understanding, you would start reinventing Neurodocker. That's why I was looking if multi-stage builds could be a way to re-use installation recipes without having to base from or copy-paste.

gllmflndn · 2020-05-12T09:23:27Z

On a side note, about the name of this repo, MATLAB Compiler Runtime (MCR) seems to be referred to as MATLAB Runtime now.

.circleci/config.yml

Remi-Gau · 2023-01-04T23:41:36Z

Will go ahead and merge this.

The idea would be to provide base images that people can use to construct bids apps.

But there is no reason why this script could not be used to recreate something more generic outside of the bids context.

Remi-Gau · 2023-01-12T22:02:09Z

quick FYI that at the last BIDS maintainers we decided that our maintenance effort would be probably best invested into helping making that neurodocker can actually help build recipes for dockerfile that people may want to use to build their bids apps.

So it is likely that this repo and its corresponding dockerhub will be archived.

Add update.py semi-auto release crawler

c82d748

sappelhoff reviewed May 11, 2020

View reviewed changes

gllmflndn mentioned this pull request May 11, 2020

[WIP] Use different base image containing MCR bids-apps/SPM#23

Open

Typo

9e95510

octomike changed the title ~~Add update.py semi-automatic release crawler~~ [WIP] Add update.py semi-automatic release crawler May 11, 2020

Tag matlab release name (R20XXc) as well

ceabc10

gllmflndn reviewed May 11, 2020

View reviewed changes

Dockerfile-core.template Outdated Show resolved Hide resolved

octomike added 3 commits May 11, 2020 21:39

circle: guess branch on CIRCLE_TAG build

85b28fa

circle: persist guessed branch

17426d6

Rename MCRPath to MCR_HOME

f8d1237

change octomike to bids and format

dd743ee

sappelhoff reviewed Jan 4, 2023

View reviewed changes

.circleci/config.yml Outdated Show resolved Hide resolved

Remi-Gau added 4 commits January 4, 2023 22:25

refactor into bash scripts

f78da21

add and apply pre-commit

baa6dd9

Merge branch 'master' into feature_release_crawler

9028e80

refactor

94d05bf

Remi-Gau added 3 commits January 4, 2023 22:50

format

ccd763e

tmp

b391051

ready to go

ba95f26

Remi-Gau merged commit 54ad6e1 into bids-apps:master Jan 4, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Add update.py semi-automatic release crawler #1

[WIP] Add update.py semi-automatic release crawler #1

octomike commented May 10, 2020

sappelhoff left a comment

sappelhoff May 11, 2020 •

edited

Loading

octomike May 11, 2020

gllmflndn commented May 11, 2020

octomike commented May 11, 2020

sappelhoff commented May 11, 2020

octomike commented May 11, 2020

kaczmarj commented May 11, 2020

octomike commented May 11, 2020

octomike commented May 11, 2020

sappelhoff commented May 11, 2020

octomike commented May 11, 2020

gllmflndn May 11, 2020

octomike May 11, 2020

gllmflndn May 11, 2020

sappelhoff May 12, 2020 •

edited

Loading

octomike May 12, 2020

gllmflndn May 12, 2020

sappelhoff May 12, 2020 •

edited

Loading

gllmflndn commented May 11, 2020

octomike commented May 12, 2020

gllmflndn commented May 12, 2020

gllmflndn commented May 12, 2020

Remi-Gau commented Jan 4, 2023

Remi-Gau commented Jan 12, 2023

	Update.py should be run often enough to catch individual matlab releae updates.
	Update.py should be run often enough to catch individual matlab release updates.

[WIP] Add update.py semi-automatic release crawler #1

[WIP] Add update.py semi-automatic release crawler #1

Conversation

octomike commented May 10, 2020

sappelhoff left a comment

Choose a reason for hiding this comment

sappelhoff May 11, 2020 • edited Loading

Choose a reason for hiding this comment

octomike May 11, 2020

Choose a reason for hiding this comment

gllmflndn commented May 11, 2020

octomike commented May 11, 2020

sappelhoff commented May 11, 2020

octomike commented May 11, 2020

kaczmarj commented May 11, 2020

octomike commented May 11, 2020

octomike commented May 11, 2020

sappelhoff commented May 11, 2020

octomike commented May 11, 2020

gllmflndn May 11, 2020

Choose a reason for hiding this comment

octomike May 11, 2020

Choose a reason for hiding this comment

gllmflndn May 11, 2020

Choose a reason for hiding this comment

sappelhoff May 12, 2020 • edited Loading

Choose a reason for hiding this comment

octomike May 12, 2020

Choose a reason for hiding this comment

gllmflndn May 12, 2020

Choose a reason for hiding this comment

sappelhoff May 12, 2020 • edited Loading

Choose a reason for hiding this comment

gllmflndn commented May 11, 2020

octomike commented May 12, 2020

gllmflndn commented May 12, 2020

gllmflndn commented May 12, 2020

Remi-Gau commented Jan 4, 2023

Remi-Gau commented Jan 12, 2023

sappelhoff May 11, 2020 •

edited

Loading

sappelhoff May 12, 2020 •

edited

Loading

sappelhoff May 12, 2020 •

edited

Loading