database migration

[gedw99]

This might have been touched on again, but i keep hitting this, with a voice in the back of my brain saying there is no schema evolution of the data.

Here is the thing. Siot has Protobufs to so that we have Schema evolution of the Data.
But the data lives inside Genji, and so if the Schema changes we need to have migration scripts for the data in genji.

I did read the ADR ( or doc) about the design of Points and how its designed to do CRDT later.
I guess the points is designed to be Schema indepentent and the CRDT to allow sync of the data.

But i just worry that there is no escape hatch as they say, where you will eventually need to change the architecture or something like that, and you just need to migrate data.

The solution would be to add a Migration concept to the Protobufs and stream it through the nodes. SO its event based migrations.
The migration script is just some genji change script.
As long as all nodes advertise their Version of the software and DB things will work with systems are different versions of software and DB. The protobufs protect from a crash.

For NATS in general i have been thingking about this, independent of the SIOT projects. NATS has KV and Objects now which can retain data as long as you want and so are like mini database. Nats does not care about doing a data migration on its data. Instead it wants you to use just live with it. Code that consumes NATS KV or any messages just has to handle data schema diffs using Protobuf or whatever, but in terms of data residing inside data stores, there is no migration. thats just how it is.

Anyways, I hope that someone has some sort of answer about this. Maybe i am being paranoid.

I just worry that all these Nodes might need a stop the world locally, and migrate the underlying genji store because a big change happened that the current Points approach cant handle

[cbrake]

I did read the ADR ( or doc) about the design of Points and how its designed to do CRDT later. I guess the points is designed to be Schema indepentent and the CRDT to allow sync of the data.

Yes, the idea is points can represent map or array type data, so theoretically capable of representing anything. And node hierarchies give you deeper structure if needed.

One thing this scheme lacks is transactions that include multiple points in multiple nodes, but that really seems beyond the scope of an event bus.

But i just worry that there is no escape hatch as they say, where you will eventually need to change the architecture or something like that, and you just need to migrate data.

Yes, that is a good question.

The solution would be to add a Migration concept to the Protobufs and stream it through the nodes. SO its event based migrations. The migration script is just some genji change script. As long as all nodes advertise their Version of the software and DB things will work with systems are different versions of software and DB. The protobufs protect from a crash.

Are you thinking each node as a version number field as well?

For NATS in general i have been thingking about this, independent of the SIOT projects. NATS has KV and Objects now which can retain data as long as you want and so are like mini database. Nats does not care about doing a data migration on its data. Instead it wants you to use just live with it. Code that consumes NATS KV or any messages just has to handle data schema diffs using Protobuf or whatever, but in terms of data residing inside data stores, there is no migration. thats just how it is.

I've been thinking about NATS KV and wondering if I even need a separate database. However, I've not come up with a way to do granular point type data in a node where data is flowing in both directions for a node, and needs to be merge intelligently. I'd really like to figure out how to use NATS jetstream for more stuff, but it seems jetstream is more applicable for problems like orders -> shipping flows where you are passing along a complex object that flows in just one direction.

Note sure if you saw the latest genji release notes:

https://github.com/genjidb/genji/releases/tag/v0.15.0

Looks like good stuff. However, I still leaning toward sqlite and putting nodes and points in separate tables. There are advantages of sticking with a more conventional db structure, and that would make migrations easy. I'm also suspecting that serializing and de-serializing objects in a KV store is actually rather expensive -- even with fast encoding like protobuf. Having points in separate tables would also allow indexing of various point types, which may be useful.

I just worry that all these Nodes might need a stop the world locally, and migrate the underlying genji store because a big change happened that the current Points approach cant handle

That is a good point.

[gedw99]

I did read the ADR ( or doc) about the design of Points and how its designed to do CRDT later. I guess the points is designed to be Schema independent and the CRDT to allow sync of the data.

Yes, the idea is points can represent map or array type data, so theoretically capable of representing anything. And node hierarchies give you deeper structure if needed.

One thing this scheme lacks is transactions that include multiple points in multiple nodes, but that really seems beyond the scope of an event bus.

agree...

But i just worry that there is no escape hatch as they say, where you will eventually need to change the architecture or something like that, and you just need to migrate data.

Yes, that is a good question.

The solution would be to add a Migration concept to the Protobufs and stream it through the nodes. SO its event based migrations. The migration script is just some genji change script. As long as all nodes advertise their Version of the software and DB things will work with systems are different versions of software and DB. The protobufs protect from a crash.

Are you thinking each node as a version number field as well?

Yes it might make sense actually, but am not confident of my opinion. A parent Node and its children have a version i guess ?

For NATS in general i have been thingking about this, independent of the SIOT projects. NATS has KV and Objects now which can retain data as long as you want and so are like mini database. Nats does not care about doing a data migration on its data. Instead it wants you to use just live with it. Code that consumes NATS KV or any messages just has to handle data schema diffs using Protobuf or whatever, but in terms of data residing inside data stores, there is no migration. thats just how it is.

I've been thinking about NATS KV and wondering if I even need a separate database. However, I've not come up with a way to do granular point type data in a node where data is flowing in both directions for a node, and needs to be merge intelligently. I'd really like to figure out how to use NATS jetstream for more stuff, but it seems jetstream is more applicable for problems like orders -> shipping flows where you are passing along a complex object that flows in just one direction.

Its hard to really answer what you said because i don't know enough about SIOT but here are my 2 cents....

1. You need genji because you need to be able to query the DB after the data coming off NATS has been swallowed.
2. NATS KV and Object can have a forever retention but its useless to you. What it is good for of course is listening for changes based on the subject namespace, and updating genji based on that. For example, I consume data based on KV subject namespace to update a XML structure. I do the same for NATS Objects to update the File system. Your just updating NODES in genji because you can then query Genji later and do computing on it.

The only thing that matters is if and where you need to transform the data.
I think you made things overly complex because NATS has moved forward. If you stood back from SIOT you could just use NATS KV to map directly to Nodes ( genji ). The NATS Objects can map directly to Files. There is NO transformation if you just wanting to get HA with everything the same in many physical places.

It is just CQRS but make easy. Write side is through nats, which leads to the read side in genji.
Regarding "both directions", i think about it this way....
On the write side, Domain events are handled by a Command, which produces events ( with payloads) into NATS. The important part is that a command handler can fire many events into nats !!
On the read side, workers handle events coming off NATS and transform the data into genji. It is very simple when you have genji. I see SIOT is just an abstraction layer on top of this pattern.

Note sure if you saw the latest genji release notes:

https://github.com/genjidb/genji/releases/tag/v0.15.0

Looks like good stuff. However, I still leaning toward sqlite and putting nodes and points in separate tables. There are advantages of sticking with a more conventional db structure, and that would make migrations easy. I'm also suspecting that serializing and de-serializing objects in a KV store is actually rather expensive -- even with fast encoding like protobuf. Having points in separate tables would also allow indexing of various point types, which may be useful.

Genji now has binary stability. +1
I have not benchmarked SQlite against genji. I know the CGO free SQLite is 2 times slower than the standard SQLite.
Without data it's impossible to have an opinion. I really think you should bench the new Genji using Pebble before you draw any conclusions. I suspect Pebble is going to be fast as hell.
Also SIOT is chatty, not chunky. You might want to do some batching if that's possible.

Protobufs
You dont need to support GRPC, GRPC-Web or OpenAPI.

Protobufs are NOT fast.. They actually SUCK if you want perf. They are great if you need the whole kitchen sink of GRPC, GRPC-web, Envoy, etc.

https://github.com/bufbuild/connect-go is protobufs without the legacy shit from google. worth thinking about. The NATS tea are already playing with it because they can use the NATS ROund tripper with it to make life easy with NATS and GRPC.

https://github.com/inkeliz/karmem is another way that has the properties of Protobufs without the crap you dont need.

I just worry that all these Nodes might need a stop the world locally, and migrate the underlying genji store because a big change happened that the current Points approach cant handle

That is a good point.

Anyways about schema migration.

1. If map SIOT nodes one to one to NATS KV and Objects, then Genji migration becomes easy.
2. there is no 2 :)

[cbrake]

As always, lots of good information/links to think about!

1. You need genji because you need to be able to query the DB after the data coming off NATS has been swallowed.

Currently, I don't every access the db directly -- at least from most clients. Nodes/node children are fetched using a NATS request, and then updates come in as NATS point messages. I'm currently writing some utility code to make this easy.

1. If map SIOT nodes one to one to NATS KV and Objects, then Genji migration becomes easy.

Are you proposing that the data be duplicated in Genji and NATS KV? NATS KV handles replication/HV and Genji is used for reading?