AzureStorageOrchestrationService - LockNextTaskActivityWorkItem assert failure

[KonradCzarny]

In one of our orchestrators that is running multiple activities in parallel, we've encountered following warning:

[Warning] Internal assert failure: Work item queue message with ID = {Id} is being processed multiple times concurrently

Activity is being launched via standard TaskOrchestrationContext.CallActivityAsync method. It resulted in orchestrator being stuck in limbo, and after activity ended - not being possible to start next activity from the orchestration queue.

What I have found in the OrchestrationService class, is following piece of code:

                if (!this.activeActivitySessions.TryAdd(message.Id, session))
                {
                    // This means we're already processing this message. This is never expected since the message
                    // should be kept invisible via background calls to RenewTaskActivityWorkItemLockAsync.
                    this.settings.Logger.AssertFailure(
                        this.azureStorageClient.QueueAccountName,
                        this.settings.TaskHubName,
                        $"Work item queue message with ID = {message.Id} is being processed multiple times concurrently.");
                    return null;
                }

According to the comment, this assert should never fail - however it did in our scenario. My question is: any ideas what could be the reason for that, and how to prevent it happening in future?

[cgillum]

@KonradCzarny thanks for reporting this. One possible explanation is that the message renewal operation might be failing in the background. What version of the Microsoft.Azure.DurableTask.AzureStorage nuget package are you using?