Azure Durable function Orchestrator function replay throwing Non-Deterministic workflow detected after nuget package upgrades

[Kruti-Joshi]

Error -
Non-Deterministic workflow detected: A previous execution of this orchestration scheduled an activity task with sequence ID 0 and name 'AzureMonitorConnectorActivityFunction' (version ''), but the current replay execution hasn't (yet?) scheduled this task. Was a change made to the orchestrator code after this instance had already started running?

From the error message, it looks like the first call itself hasn't taken place.

Package Update Info

before package update

    <PackageReference Include="Microsoft.Identity.ServiceEssentials.AspNetCore" Version="1.25.0" />
    <PackageReference Include="Microsoft.Identity.ServiceEssentials.TokenAcquisitionIdWeb" Version="1.25.0" />
    <PackageReference Include="Microsoft.Identity.Web.DownstreamApi" Version="2.18.2" />
    <PackageReference Include="Microsoft.Identity.Web" Version="2.18.1" />
    <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.DurableTask" Version="1.0.0" />

We had the below code for auth using federated MI in the code -

 public TokenCredential GetClientCredential(string tenantId)
    {
        try
        {
            logger.LogInformation($"Getting client credentials for tenant {tenantId} using clientId: {authOptions.ClientId}");

            ManagedIdentityClientAssertion managedIdentityClientAssertion = new ManagedIdentityClientAssertion(authOptions.FederatedManagedIdentityClientId);

            ClientAssertionCredential credential = new ClientAssertionCredential(tenantId, authOptions.ClientId, managedIdentityClientAssertion.GetSignedAssertion);

            return credential;
        }
        catch (Exception ex)
        {
            logger.LogError(ex, $"Failed to get client credential for tenant {tenantId}.");
            throw;
        }
    }

after package update

    <PackageReference Include="Microsoft.Identity.ServiceEssentials.AspNetCore" Version="1.29.0" />
    <PackageReference Include="Microsoft.Identity.ServiceEssentials.TokenAcquisitionIdWeb" Version="1.29.0" />
    <PackageReference Include="Microsoft.Identity.Web.DownstreamApi" Version="3.2.0" />
    <PackageReference Include="Microsoft.Identity.Web" Version="3.2.0" />
    <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.DurableTask" Version="1.1.6" />

With the upgrade of Microsoft.Azure.Functions.Worker.Extensions.DurableTask to 1.1.6, there was a breaking change that the sync method for 'GetSignedAssertion' no longer exists and is replaced by 'GetSignedAssertionAsync'. To support this, we made the below change -

public TokenCredential GetClientCredential(string tenantId)
    {
        try
        {
            logger.LogInformation($"Getting client credentials for tenant {tenantId} using clientId: {authOptions.ClientId}");

            string authority = $"https://login.microsoftonline.com/{tenantId}";

            logger.LogInformation($"Getting client credentials for tenant {tenantId} using authority: {authority}");

            AssertionRequestOptions assertionOptions = new ()
            {
                ClientID = authOptions.ClientId,
                TokenEndpoint = authority,
            };

            Func<CancellationToken,Task<string>> returnMSIToken = async (CancellationToken cancellationToken) =>
            {
                string msiToken = await new ManagedIdentityClientAssertion(authOptions.FederatedManagedIdentityClientId)
                    .GetSignedAssertionAsync(assertionOptions)
                    .ConfigureAwait(false);

                return msiToken;
            };

            logger.LogInformation($"Getting client credentials for tenant {tenantId} using MSIToken: {authOptions.FederatedManagedIdentityClientId}");

            ClientAssertionCredential credential = new ClientAssertionCredential(tenantId, authOptions.ClientId, returnMSIToken);

            logger.LogInformation($"Successfully acquired client credentials for tenant {tenantId}.");
            return credential;
        }
        catch (Exception ex)
        {
            logger.LogError(ex, $"Failed to get client credential for tenant {tenantId}.");
            throw;
        }
    }

This method is being wrapped in a sync call from the orchestrator, and this has been the only change in our code with the package upgrades.

Orchestrator and Service calls -

public async Task RunOrchestrator([OrchestrationTrigger] TaskOrchestrationContext context)
{
    .
    .
    .
    try
    {
        // Call the method synchronously, as orchestrator code cannot contain 'non-durable async calls' [https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-code-constraints?tabs=csharp#async-apis]
        connectors = rpaasService.GetAllAsync().GetAwaiter().GetResult();
    }
    catch (Exception ex)
   {
        logger.LogError(ex, $"Error gettingresponse. Check the inner exception for details.");
        throw;
    }
    .
    .
}

public async Task<IEnumerable<Response>> GetAllAsync()
{
    .
    .
    .
   ClientAssertionCredential credential = (ClientAssertionCredential)GetClientCredential(tenantId);
   TokenRequestContext tokenRequestContext = new TokenRequestContext(new string[] { scope });
   accessToken = await credential.GetTokenAsync(tokenRequestContext);
   return accessToken;
   .
   .

Since the orchestrator is calling GetAllAsync() synchronously, we shouldn't be running into this issue, right?
We do some processing after we get this response and only then trigger the first activity function.

Kindly help us figure out what the issue is.