SWITCH/CONDITION tasks make big workflows

[sarmuru2]

As we need to give entire list of
tasks in decisionCases node for SWITCH/CONDITION tasks, its increasing the workflow size to Mbs. For example if we have 6/7 decision tasks and 12 worker tasks, it becomes nested which brings up 1.7 Mb workflow size. In other words it we need to form close to ~5000 task entries in the workflow. It takes close to 2 minutes to add that workflow into the conductor system. Everytime to load the workflow also it takes lot of time which increases the workflow execution time.

To explain it litter further, I have defined a workflow with two decision tasks and 10 simple tasks. Instead of creating 12 unique tasks we ended up in creating 17 tasks. The difference will be huge if we increase the number of decision tasks. Attaching the sample workflow with two decision tasks.

This issue can be fixed by adding a goto task inside each decision task as specified in this thread. #1031
But this PR is not merged into conductor master.

Is there any solution for this issue in the netflix conductor? Is there a way to reduce the number of tasks when we define more number of decision tasks? Any one faced the similar problem and how did you overcome that?

[v1r3n]

Hi @sarmuru2 the number of tasks will not impact the overall runtime or the execution of the workflows - if that the concern.
What type of issues do you see with additional tasks - other than having extra tasks? would love to get some insights.

I think goto can be added, but generally it is not recommended as with all the programming languages, it makes it harder to debug.

[sarmuru2]

hi v1r3n, thanks for the reply. for 6 decision tasks, creating 1.7 mb file is too much which is affecting the time required to put the workflow definition into the database. It is taking 2 minutes to put 1.7 mb workflow definition into database. We are putting the workflow definition into the database from the ui thread which makes Save button hanging for 2 minutes which is not acceptable. On executing workflow conductor is loading workflow object multiple times from database. As workflow definition is added as a field in Workflow object, WorkflowDefinition is also queried whenever Workflow object is queried. Also to load 1.7 mb from database that too multiple times will definitely affect execution time as well.

[v1r3n]

Hi @sarmuru2 thanks for the context. What version of conductor are you using? And what the backend database you use for Conductor? There are improvements in 3.x that addresses some of these concerns - if you would like to try it out, give it a spin for the same workflow on https://play.orkes.io/ and see if that works better, I think even with 1.7MB payload you might be able to get much better response times (minus any network latencies) on play.orkes.io.

[sarmuru2]

Hi v1r3n, We are using latest from main branch. Our main problem is we are submitting a 1.7 MB file for simple workflow which has 6 decision tasks. Is there any fix which reduces this big workflow config in 3.x? can you elaborate more on that? How to upload my workflow in https://play.orkes.io? will PUT to https://play.orkes.io/api/workflow/ works?

[sarmuru2]

We are using redis as backend database. I tried uploading workflow through PUT request to https://play.orkes.io/api/metadata/workflow/ after adding X-Authorization header but getting Error validating token response.

[v1r3n]

Hi @sarmuru2 seems like the token is either not valid or expired. Can you try directly here?
https://play.orkes.io/swagger-ui/index.html?configUrl=/api-docs/swagger-config#/metadata-resource/update

1. Obtain the token from the play.orkes.io as described here?
2. Try the update API to create the workflow

Let me know if this works, or we can chat here for more realtime collaboration.

[sarmuru2]

thats ok. we are trying dynamic task instead of switch/decision task.