How to restart the video/chat convert process only?

[EunHaSeoHwa]

Hi!

When an error occurred during the chat convert process or something, I could restart that step in the queue page in the former version of Ganymede.
In the Ganymede ver.2, it operates the whole procedure (Create Folder->Download Thumbnail->Save Info->...) when I restart the workflow. So, if the stream is ended, it doesn't move on to the video/chat convert process because it is stuck in the download thumbnail step even if the video and chat files have been downloaded already.
Is there any way to restart the video/chat convert step by ignoring the download thumbnail process?

Thanks.

[Zibbp]

Hi,

You should be able to restart a specific step, such as the chat conversion. Navigate to the workflows page and find the desired workflow, for this ConvertTwitchChatWorkflow

Click on that workflow entry and ensure it's the correct workflow (looking at the video ID and external video ID) then click the orange restart icon.

Do you see any failed workflows on the workflows page?

[EunHaSeoHwa]

I can't find it, maybe because I restarted it.
I found something in the ganymede-temporal logs.

2024/01/09 23:53:14 stdout {"level":"error","ts":"2024-01-09T14:53:14.423Z","msg":"Update workflow execution operation failed.","shard-id":1,"address":"127.0.0.1:44229","wf-namespace-id":"cc013d00-60d0-4cef-8b2e-80642154ffe2","wf-id":"16e0c3c0-edd7-46c1-8bf6-3903d75a128d_5","wf-run-id":"d0e88a03-3e35-4701-a1a3-f9715676441e","store-operation":"update-wf-execution","error":"UpdateWorkflowExecution failed. Failed to start transaction. Error: context deadline exceeded","logging-call-at":"transaction_impl.go:512"

2024/01/09 23:53:41 stdout {"level":"error","ts":"2024-01-09T14:53:41.661Z","msg":"Persistent fetch operation Failure","shard-id":1,"address":"127.0.0.1:44229","wf-namespace-id":"cc013d00-60d0-4cef-8b2e-80642154ffe2","wf-id":"16e0c3c0-edd7-46c1-8bf6-3903d75a128d_5","wf-run-id":"d0e88a03-3e35-4701-a1a3-f9715676441e","store-operation":"get-wf-execution","error":"GetWorkflowExecution: failed to get request cancel info. Error: Failed to get request cancel info. Error: context deadline exceeded","logging-call-at":"transaction_impl.go:478"

2024/01/09 23:53:41 stdout {"level":"error","ts":"2024-01-09T14:53:41.771Z","msg":"Fail to process task","shard-id":1,"address":"127.0.0.1:44229","component":"timer-queue-processor","wf-namespace-id":"cc013d00-60d0-4cef-8b2e-80642154ffe2","wf-id":"16e0c3c0-edd7-46c1-8bf6-3903d75a128d_5","wf-run-id":"d0e88a03-3e35-4701-a1a3-f9715676441e","queue-task-id":1053004,"queue-task-visibility-timestamp":"2024-01-09T14:53:32.338Z","queue-task-type":"ActivityTimeout","queue-task":{"NamespaceID":"cc013d00-60d0-4cef-8b2e-80642154ffe2","WorkflowID":"16e0c3c0-edd7-46c1-8bf6-3903d75a128d_5","RunID":"d0e88a03-3e35-4701-a1a3-f9715676441e","VisibilityTimestamp":"2024-01-09T14:53:32.338497Z","TaskID":1053004,"TimeoutType":4,"EventID":5,"Attempt":1,"Version":0},"wf-history-event-id":5,"error":"GetWorkflowExecution: failed to get request cancel info. Error: Failed to get request cancel info. Error: context deadline exceeded","lifecycle":"ProcessingFailed","logging-call-at":"lazy_logger.go:68",

I cut some of the following lines.
Is it helpful to find something?

[Zibbp]

Those logs aren't too helpful, I log the important stuff in the API container. If/when you're able to recreate the issue archiving a livestream, please paste the errors logs in the API container here.

[EunHaSeoHwa]

Hi again.

Some archiving tasks are stuck again, like the captured image. And in this time, I didn't restart any workflows.
I am just confirming that the ConvertTwitchLiveChatWorkflow or similar workflows are not listed on the closed workflows list.

I attached some logs of the ganymede-api container. Because of too many lines, I picked up some logs that appeared repeatedly or looked related.

2024/01/11 01:29:43,stdout,"{\"level\":\"warn\",\"Error\":\"context deadline exceeded\",\"Namespace\":\"default\",\"TaskQueue\":\"chat-download\",\"WorkerID\":\"78d8dff24d5d\",\"WorkerType\":\"WorkflowWorker\",\"time\":\"2024-01-11T01:28:08+09:00\",\"message\":\"Failed to poll for task.\"}

2024/01/11 01:29:41,stderr,"{\"level\":\"debug\",\"time\":\"2024-01-11T01:29:41+09:00\",\"message\":\"finished vod video convert for 40961823239\"}

2024/01/11 01:29:43,stdout,"{\"level\":\"warn\",\"ActivityID\":\"9\",\"ActivityType\":\"DownloadTwitchLiveChat\",\"Attempt\":1,\"Error\":\"context canceled\",\"Namespace\":\"default\",\"RunID\":\"3689bcde-a927-4ec5-ac76-986e3753bfa1\",\"TaskQueue\":\"archive\",\"WorkerID\":\"78d8dff24d5d\",\"WorkflowID\":\"98e105a3-d617-49c4-9003-fc4b6d34b0c0_37\",\"WorkflowType\":\"DownloadTwitchLiveChatWorkflow\",\"time\":\"2024-01-11T01:28:20+09:00\",\"message\":\"RecordActivityHeartbeat with error\"}

2024/01/11 01:29:43,stdout,"{\"level\":\"warn\",\"ActivityID\":\"5\",\"ActivityType\":\"PostprocessVideo\",\"Attempt\":3,\"Error\":\"context canceled\",\"Namespace\":\"default\",\"RunID\":\"9cfe1094-b1af-4339-80a1-976f16e7fdcd\",\"TaskQueue\":\"video-convert\",\"WorkerID\":\"78d8dff24d5d\",\"WorkflowID\":\"af990199-e647-4c95-8137-84ab2320d2fb_17\",\"WorkflowType\":\"PostprocessVideoWorkflow\",\"time\":\"2024-01-11T01:28:31+09:00\",\"message\":\"RecordActivityHeartbeat with error\"}

2024/01/11 01:29:43,stdout,"{\"level\":\"warn\",\"ActivityID\":\"9\",\"ActivityType\":\"DownloadTwitchLiveChat\",\"Attempt\":1,\"Error\":\"context canceled\",\"Namespace\":\"default\",\"RunID\":\"a1365c11-2238-4c9e-9a5a-40c08893620b\",\"TaskQueue\":\"archive\",\"WorkerID\":\"78d8dff24d5d\",\"WorkflowID\":\"8a5a1f22-98bf-4b58-b5f3-79cb2f0cf925_37\",\"WorkflowType\":\"DownloadTwitchLiveChatWorkflow\",\"time\":\"2024-01-11T01:28:39+09:00\",\"message\":\"RecordActivityHeartbeat with error\"}

2024/01/11 01:29:50,stdout,"{\"level\":\"info\",\"Error\":\"unable to mark activity as canceled without activity being request canceled first\",\"Namespace\":\"default\",\"TaskQueue\":\"video-convert\",\"WorkerID\":\"78d8dff24d5d\",\"WorkerType\":\"ActivityWorker\",\"time\":\"2024-01-11T01:29:50+09:00\",\"message\":\"Task processing failed with error\"}

2024/01/11 01:29:52,stdout,"{\"level\":\"debug\",\"Attempt\":1,\"CachedPreviousStartedEventID\":12,\"Namespace\":\"default\",\"PreviousStartedEventID\":12,\"RunID\":\"3b5d64e3-2fed-45d8-b49e-23c38a63c0e3\",\"TaskFirstEventID\":1,\"TaskQueue\":\"archive\",\"TaskStartedEventID\":18,\"WorkerID\":\"78d8dff24d5d\",\"WorkflowID\":\"98e105a3-d617-49c4-9003-fc4b6d34b0c0_32\",\"time\":\"2024-01-11T01:29:52+09:00\",\"message\":\"Cached state staled, new task has unexpected events\"}

[Zibbp]

I've made some changes that are available using the :main image tag for the API container. Can you pull that and give it another try?

[EunHaSeoHwa]

I saw the comment (#354 (comment)), so I pulled the :main image and restarted the container yesterday. But it still has the same issue on a few streams.

Queue page

Video download logs at the Queue page

Chat download logs at the Queue page

API container logs

2024/01/17 23:02:24,stdout,"{\"level\":\"warn\",\"ActivityID\":\"5\",\"ActivityType\":\"PostprocessVideo\",\"Attempt\":1,\"Error\":\"context canceled\",\"Namespace\":\"default\",\"RunID\":\"7acb2a30-91de-4f82-a16b-c3be6f9f449f\",\"TaskQueue\":\"video-convert\",\"WorkerID\":\"d6b38bb22e06\",\"WorkflowID\":\"0f507a96-5651-486e-9793-8ae8e539ee66_14\",\"WorkflowType\":\"PostprocessVideoWorkflow\",\"time\":\"2024-01-17T23:02:24+09:00\",\"message\":\"RecordActivityHeartbeat with error\"}

2024/01/17 23:02:25,stdout,"{\"level\":\"warn\",\"ActivityID\":\"5\",\"ActivityType\":\"DownloadTwitchLiveVideo\",\"Attempt\":1,\"Error\":\"context canceled\",\"Namespace\":\"default\",\"RunID\":\"d3445dbe-46b6-46b0-940a-5375fd1ac27f\",\"TaskQueue\":\"archive\",\"WorkerID\":\"d6b38bb22e06\",\"WorkflowID\":\"33a95a79-2ae4-4046-ac32-00a0793aec72_5\",\"WorkflowType\":\"DownloadTwitchLiveVideoWorkflow\",\"time\":\"2024-01-17T23:02:25+09:00\",\"message\":\"RecordActivityHeartbeat with error\"}

2024/01/17 23:02:26,stdout,"{\"level\":\"warn\",\"ActivityID\":\"9\",\"ActivityType\":\"DownloadTwitchLiveChat\",\"Attempt\":1,\"Error\":\"context canceled\",\"Namespace\":\"default\",\"RunID\":\"f88f9282-2bfa-401d-8d25-2ee77427c8f3\",\"TaskQueue\":\"archive\",\"WorkerID\":\"d6b38bb22e06\",\"WorkflowID\":\"5eb48310-4fd5-4a34-83f3-4ed24477af38_37\",\"WorkflowType\":\"DownloadTwitchLiveChatWorkflow\",\"time\":\"2024-01-17T23:02:26+09:00\",\"message\":\"RecordActivityHeartbeat with error\"}

2024/01/17 23:02:33,stderr,"{\"level\":\"debug\",\"time\":\"2024-01-17T23:02:33+09:00\",\"message\":\"finished vod video convert for 40982358487\"}

2024/01/17 23:02:33,stderr,"{\"level\":\"debug\",\"time\":\"2024-01-17T23:02:33+09:00\",\"message\":\"stopping heartbeat postprocess-video-40982358487\"}

2024/01/17 23:02:33,stdout,"{\"level\":\"error\",\"ActivityType\":\"PostprocessVideo\",\"Attempt\":1,\"Error\":\"context canceled\",\"Namespace\":\"default\",\"RunID\":\"7acb2a30-91de-4f82-a16b-c3be6f9f449f\",\"TaskQueue\":\"video-convert\",\"WorkerID\":\"d6b38bb22e06\",\"WorkflowID\":\"0f507a96-5651-486e-9793-8ae8e539ee66_14\",\"time\":\"2024-01-17T23:02:33+09:00\",\"message\":\"Activity error.\"}

2024/01/17 23:02:37,stdout,"{\"level\":\"info\",\"Error\":\"unable to mark activity as canceled without activity being request canceled first\",\"Namespace\":\"default\",\"TaskQueue\":\"video-convert\",\"WorkerID\":\"d6b38bb22e06\",\"WorkerType\":\"ActivityWorker\",\"time\":\"2024-01-17T23:02:37+09:00\",\"message\":\"Task processing failed with error\"}

2024/01/17 23:02:38,stderr,"{\"level\":\"debug\",\"time\":\"2024-01-17T23:02:38+09:00\",\"message\":\"Error notification is disabled\"}

2024/01/17 23:02:38,stderr,"{\"level\":\"debug\",\"time\":\"2024-01-17T23:02:38+09:00\",\"message\":\"Error notification is disabled\"}

2024/01/17 23:02:41,stdout,"{\"level\":\"debug\",\"Attempt\":1,\"CachedPreviousStartedEventID\":3,\"Namespace\":\"default\",\"PreviousStartedEventID\":3,\"RunID\":\"771ef86f-c707-47bd-89bb-50fb3f639e64\",\"TaskFirstEventID\":1,\"TaskQueue\":\"archive\",\"TaskStartedEventID\":11,\"WorkerID\":\"d6b38bb22e06\",\"WorkflowID\":\"a0d6e279-de39-4206-8ece-a92f3585bd1f_9\",\"time\":\"2024-01-17T23:02:41+09:00\",\"message\":\"Cached state staled, new task has unexpected events\"}

2024/01/17 23:02:41,stderr,"{\"level\":\"debug\",\"time\":\"2024-01-17T23:02:41+09:00\",\"message\":\"checking if tasks are done for video 40982358487\"}