Plotting Learning Curve for best model found after optimization in talos

[mmrahman-utexas]

I am trying to plot the learning curve for the best model found after the optimization in talos. Is there a way to do it. If the answer is yes, how can I plot the learning curve?

[mikkokotila]

You can't with Talos. But you can see get a live training plot during the experiment by following the example here.

[JohanMollevik]

I am also interested in this, and not only for the best model. If this data could be saved for all models it would be a nice enhancement.

mikkokotila: would it fit talos design to add this if it could be nicely implemented?

[mikkokotila]

In that case we have to store history from Keras in the Scan() object. Then later we can figure out some meaningful things to do with in terms of Reporting() and Evaluate(). I have to take a look at the compilation of various signals in the output .csv and Scan() object, so I'll add this at the same time.

[JohanMollevik]

Great! As log as the data is there it will be very usefull, as you say displaying it can be figured out later.

What I would do with the data is to generate an image file on disk with the learning graph for each permutation that interested me (or a folder with all of them).

I have not dvelved into talos reporting facilities enought to see if such an operation would fit into them.

[mikkokotila]

I'd be open to create a simple dashboard that runs in a web browser. Something that runs on Flask, so it's also a REST API which allows rapidly deploying custom dashboards, etc.

First for experiment analysis, and eventually also for real-time monitoring. That's basically what will need anyways for meaningful "during the experiment" changes. So we'd end up with a REST API where you can command LIVE experiments. Basically the dashboard could end up more like a real-time strategy game where you watch the experiment and work together with the machine to "play the game".

[JohanMollevik]

Sounds ambitious, I would be more than satisfied with a simple json file with the data :)

[mikkokotila]

Sure, let's start with a list of dictionaries stored in to Scan() object where each dictionary is a history from Keras, one dictionary per permutation.

[JohanMollevik]

Great

[mikkokotila]

Somewhat related with #207. Related, see ExperimentLogCallback in talos.utils.ExperimentLogCallback for a way to get access to the data. Building the plotting around it would be coming sometime soon. I'm still going for the above game-like approach. That's very much inline with the Talos way. Also check out local_function reducer. Together these two now give us two way communication with the experiment, during the experiment.

Javascript frontend work is somewhat painful and tedious in comparison to backend python works, but I'll get to it probably sometime during the autumn.

[mikkokotila]

I'm closing this as this now lives in Gamify.