Synchronisation use case

[gedw99]

Siot looks to be getting stable .

Is this use case full of hairy sync gremlins?

I install siot on my laptop and configure a complex node graph
I also deploy the workers that use the siot api and act as workers.

i deploy it to fly . I have to deploy the db too since it models the node graph. I also deploy the binaries of the workers and daemon use them..

users hit fly and change sone settings as well as workers do work

3 months later I want to work on the project , so I synchronise my laptop version with the fly version . I guess using the nats leaf concepts .

would this be viable ?

[cbrake]

We currently don't use NATS leaf nodes as I was not able to figure out how to do the type of synchronization I need:

• bi-directional
• granular
• bandwidth efficient
• CRDT properties

SIOT functions similarly, but has its own syncing logic.

Your use case sounds like a good fit. In SIOT, you can mirror nodes, so you could mirror the same subset of nodes to all workers if they all share the same config. Workers could simply be more SIOT instances, then sync is taken care of to your workers. Then develop a custom SIOT client for your workload.

[gedw99]

Thanks @cbrake

will try this on weekend.

Sync is based on your CRDT points architecture right ? LWW…

So if on my laptop i change something in config or worke it should just work. The db is 1 gb by now and so once I start up my siot on my laptop is should sync with fly. Bu my laptop is NATed, so we could use NATS pull consumers instead to get my laptop synchronised.

Are there any dummy example golang clients anywhere I can use to build up a chain of worker nodes quickly ?

[cbrake]

You can see existing built-in clients here:

https://github.com/simpleiot/simpleiot/tree/master/client

I don't think we have any standalone examples. But you can use the client API:

https://pkg.go.dev/github.com/simpleiot/[email protected]/client

This has Go function that wrap NATS to do most things we support.

This API is a little out of control -- perhaps I should move the built-in clients to a subdir so this dir is a little cleaner.

[gedw99]

thanks.

Are there any dummy example workers in github that use this client anywhere ? So its easy to test out many nodes

[cbrake]

none that I know of offhand. The main applications thus far have been remote monitoring/control of industrial/building applications. Let me know how it goes, what does not work, etc -- feedback is appreciated.

[gedw99]

@cbrake

Ok show and tell :)

My Use case is Monitoring buildings and Actuation of those buildings based on the monitoring.
A typical actuation use case is HVAC ( heating ventilation and Air conditioning).
Modbus and other protocols is typical for HVAC. OpenTherm is another one getting uptake.

I did my masters thesis on this area years ago with the CSIRO.

Now am working on an EU project that does this and exploring how far we can take SIOT. So far it looks like it can work.
There is a big push for this because of the war and gas crisis.
Its not open yet, but hopefully soon.

Machine learning:
Needed in order to make good recommendations to the actuation system and Facility Managers ( HIL - Humans in the loop ).

GUI:
I am working on a Node based UX to make it easier to design and visualise large node graphs.
A really poor example of Node based GUI but anyway: https://bimcorner.com/wp-content/uploads/2022/07/Elefront-Grasshopper.png
Using GIOUI golang to build this at the moment. NATS works really well with GIOUI. It works on web these days because nats.go compiles to WASM.
The existing SIOT Admin is great too but needs additional tooling / UI for non techy Users and Machine Learning people.
Happy to include this in SIOT if there is interest.

Telemetry and monitoring:
I used influx but found scaling issues.
Victoria Metrics is leaner and scales out much better.
Also happy to put this into SIOT if wanted.

Running lots of NODES ( workers ) and making it easy for Techy and Non Techy actors is also needed:
I started a discussion about this here:
https://github.com/orgs/simpleiot/discussions/514#discussioncomment-5114134

[cbrake]

Thanks for sharing your use case -- that does sound interesting!

I looked into VictoriaMetrics -- it looks very good and will definitely be trying that out. Thanks so much for the recommendation!

https://community.tmpdir.org/t/victoriametrics-simple-reliable-monitoring-for-everyone/931