ElasticSearch Migration from 5 to 6 possible?

[james-deee]

Hello, we were looking at migrating an existing instance of our Conductor system and wanted to migrate from ElasticSearch 5 to 6. We were successful in terms of doing a snapshot from 5 and a restore into 6. However, it looks like the expected indices are pretty different in 6.

Does anyone know if it's possible to do some re-indexing of the ElasticSearch conductor index into the various other expected conductor_workflow and conductor_task indices? I know we can re-index in ES, but it's not clear to me all of the changes that would be involved.

Is this type of thing even possible? Would someone be able to point me to some explicit places where I could compare the 2 expected indices created?

Or is is just not possible and not worth trying to do, and we should just stick with 5?

Thanks!

[james-deee]

We did a re-index of the ES 5 conductor index to split out the data to conductor_workflow and conductor_task. Search "appears" to work correctly. Does anyone know if there would have to be other changes made? Do events need to go into their own index? I didnt see any event types in the original ES 5 conductor index though.

@kishorebanala maybe? Sorry to tag directly, but just wanted to see if we could be setting ourselves up for disaster if we just only re-index those 2 things.

Thanks.

[apanicker-nflx]

We did a re-index of the ES 5 conductor index to split out the data to conductor_workflow and conductor_task. Search "appears" to work correctly.

This is correct, the workflow and task types that were part of the conductor index have been separated out into explicit indices.

Does anyone know if there would have to be other changes made? Do events need to go into their own index? I didnt see any event types in the original ES 5 conductor index though.

Following the same pattern, you would need to additionally split out data in conductor_tasklog index to event, message and task_log indices. These indices store the indexed data for the events that the event handlers listens to, payloads from the events and the task logs from the task results respectively.

if we could be setting ourselves up for disaster if we just only re-index those 2 things.

No, the event, message and task_log indices are auxiliary and only useful for debugging purposes, hence should not impact executions in anyway.

[james-deee]

@apanicker-nflx Thanks! This was perfect.

We were able to make the migration successfully by re-indexing things as you mentioned above.