Is this thread safe?

[bsambrone]

I was reading about multiple instances here, and I'm not sure if this is the answer to my question or not. I'd like to essentially for a Parallel.ForEach and have 4 threads running using a Scoped class I have (1 copy for each thread), and in said Scoped class I'm injecting IOpenAiService using the regular ol services.AddOpenAIService(); . Is this OK to do? Does the IOpenAiService work like a Singleton registration, or is it Scoped as well?

[belaszalontai]

AFAIK this opeanAI SDK is a wrapper of HttpClient using IHttpClientBuilder interface. In other words it is a typed HttpClient.
According to this https://learn.microsoft.com/en-us/dotnet/architecture/microservices/implement-resilient-applications/use-httpclientfactory-to-implement-resilient-http-requests and this https://learn.microsoft.com/en-us/aspnet/core/fundamentals/http-requests?view=aspnetcore-7.0 documents, the services.AddOpenAIService() is transient so a new instance and also a new HttpClient instance will be instantiated.

The HttpClient is thread safe itself, so I can say that this SDK is thread safe.

@kayhantolga, @BroMarduk Anyway I do not really understand why injected HttpClient is instantiated and disposed in this code? https://github.com/betalgo/openai/blob/master/OpenAI.SDK/Managers/OpenAIService.cs
And if it is necessary why a _disposeHttpClient variable is needed?

[kayhantolga]

https://github.com/betalgo/openai/blob/master/OpenAI.SDK/Extensions/OpenAIServiceCollectionExtensions.cs is a file you can check to see how we use injection. We use a pretty standard implementation of HttpClient. For more details, I would suggest checking the documentation of HttpClient. If you encounter any issues, I would be happy to debug them together.

[BroMarduk]

@belaszalontai The best way to use the OpenAIService class is to create and pass in a HttpClient using the Factory from the calling app in OpenAIService either directly or using a Singleton HttpClient and DI. There is plenty of documentation on this (see IHttpClientFactory) In this case the calling app owns the HttpClient lifetime, and it can be reused by this class and in the application. In addition, the _disposeHttpClient is false so no dispose will take place in the class as we don't want to dispose of the HttpClient from the app. But in this case where the HttpClient is null an HttpClient needs to be created and then the class is managing its lifetime. In this case the code creates an HttpClient and keeps it around until the class itself is disposed. _disposeHttpClient is set to true and when the OpenAIService gets garbage collected, so does the HttpClient.

tl;dr; If you inject an HttpClient into the class, nothing will get disposed. If the class creates it because it didn't have one (httpClient was null) then it gets disposed with the class.

[Artein]

I am using this package with Hangfire + custom-made "jobs". In a nutshell, it seems like it is thread-safe (no problems for the last year while we are prototyping). But yes, for other third-party calls, we are using IHttpClientFactory (read somewhere that it is better).

But I don't use existing AddOpenAIService(). Just copied its implementation and made my changes on top:

// Taken from OpenAI.GPT3.Extensions.OpenAIServiceCollectionExtensions.cs (to pass Timeout to HttpClient)
        builder.Services.AddOptions<OpenAiOptions>()
            .Configure(options =>
            {
                options.ApiKey = openaiKeySection.Value;
                options.Organization = openaiOrganizationSection.Value;
                options.DefaultModelId = defaultModelProvider.ModelId;
                options.Validate();
            });

        var httpClientBuilder = builder.Services.AddHttpClient<IOpenAIService, OpenAIService>(options =>
        {
            options.Timeout = TimeSpan.FromMinutes(graceAISection.GetValue("RequestTimeoutMinutes", 5));
            options.DefaultRequestHeaders.ConnectionClose = false;
        });

        httpClientBuilder.SetHandlerLifetime(TimeSpan.FromMinutes(5)); // default: 2 minutes

        var rateLimitSection = graceAISection.GetSection("RateLimit");
        var rpm = rateLimitSection.GetValue<double>("Chat:RPM");
        var medianFirstRetryDelay = TimeSpan.FromSeconds(60 / rpm);
        var retryCount = rateLimitSection.GetValue<int>("RetryCount");

        if (retryCount > 0)
        {
            var retryPolicy = Policy.HandleResult<HttpResponseMessage>(response =>
                {
                    if (response.StatusCode == HttpStatusCode.TooManyRequests)
                    {
                        // Log.Logger.InformationHere("{Message} {@Response}", "Too many requests", response);
                        return true;
                    }

                    return false;
                })
                .WaitAndRetryAsync(Backoff.DecorrelatedJitterBackoffV2(medianFirstRetryDelay, retryCount));

            httpClientBuilder.AddPolicyHandler(retryPolicy);
        }