Long term investigation: stream data to an existing database

[traversaro]

At the moment (2021/02) the yarp-telemetry library provides support for saving data in memory and then periodically (or when the user desires) the data is saved in .mat file. While this is the nice property of permit to easily save file in a completely standalone way, without depending on any infrastructure to be set up, in practice for long running tests that will save a lot of data, this will probably be not feasible both for the memory usage, and for the fact that it is not feasible to deal with a log of sparse .mat files.

For this reason, in the future it could make sense to have, instead of a periodic save to file thread, a periodic thread that send these files to a database, from which users then can extract the data for visualization or processing, eventually even saving the query in .mat format.

Interesting pointers:

• https://github.com/questdb/questdb
• app-logger: Proposal for a C++ library component to log channels per process #17 (comment)

[drdanz]

We should also consider the hypothesis to stream the data immediately, without the periodic thread, so that we can avoid all copies of the data, and periodic delays when the data is sent (i.e. test whether it is more convenient having a constant small delay, vs having big periodical delay).