Race condition problem during database initialization of some modules

[Shtong]

Is there an existing issue for this?

• I have searched the existing issues

Description

There is a concurrency problem in the code used to initialize the database for some of the official ABP modules. This problem essentially creates concurrent database operations on the same DbContext instance, which is not a scenario officially supported by EF Core, as the DbContext class is not thread safe.

In practice, I haven't seen any side effect of this problem in production scenarios. However, we have been encountering issues caused by this problem on several projects when running unit tests containing large amounts of data seeding, or when the offending modules have a lot of data to initialize. The unit tests fails because the SQLite drivers used by the tests are very sensitive to concurrency problems, more so than other drivers used in production environments.

The offending modules are :

• AbpSettingManagementDomainModule
• AbpFeatureManagementDomainModule
• AbpPermissionManagementDomainModule

The root of the issue is the use of Thread.Run(...) to start the database initialization sequences. Because Thread.Run will queue the provided method on the thread pool, the thread pool may decide to execute them all at once on different threads, or during some other data operation taking place on the main thread.

Reproduction Steps

Since this is a race condition problem, it cannot be reprodured reliably. To improve your chances of reproducing it, create a project that inserts a lot of data during the data seeding (executed on the main thread), and/or during the database initialization of the Setting, Permission or Features modules. The objective is to spend more time doing database work in the initialization phase of the tests.

After that, just running any unit test is enough to try and replicate, as the crash will happen during the initialization phase of ABP.

Expected behavior

Unit tests starts every time without errors

Actual behavior

Unit tests will randomly throw exceptions during initialization (I am currently working on a project where an exception happens once every 10-15 runs). Here is an example of such an exception:

[xUnit.net 00:00:03.68]     MyApp.Prices.PricingServiceTests.ComputePriceTest [FAIL]
  Échoué MyApp.Prices.PricingServiceTests.ComputePriceTest [1 ms]
  Message d'erreur :
   Volo.Abp.AbpInitializationException : An error occurred during the initialize Volo.Abp.Modularity.OnApplicationInitializationModuleLifecycleContributor phase of the module MyApp.AppPricingTestBaseModule, MyApp.TestBase, Version=3.0.0.0, Culture=neutral, PublicKeyToken=null: Object reference not set to an instance of an object.. See the inner exception for details.
---- System.NullReferenceException : Object reference not set to an instance of an object.
  Arborescence des appels de procédure :
     at Volo.Abp.Modularity.ModuleManager.InitializeModules(ApplicationInitializationContext context)
   at Volo.Abp.AbpApplicationBase.InitializeModules()
   at Volo.Abp.AbpApplicationWithExternalServiceProvider.Initialize(IServiceProvider serviceProvider)
   at Volo.Abp.Testing.AbpIntegratedTest`1..ctor()
   at MyApp.AppPricingTestBase`1..ctor()
   at MyApp.AppPricingDomainTestBase..ctor()
   at MyApp.Prices.PricingServiceTests..ctor()
   at System.RuntimeType.CreateInstanceDefaultCtor(Boolean publicOnly, Boolean wrapExceptions)
----- Inner Stack Trace -----
   at Microsoft.Data.Sqlite.SqliteConnection.RemoveCommand(SqliteCommand command)
   at Microsoft.Data.Sqlite.SqliteCommand.Dispose(Boolean disposing)
   at System.ComponentModel.Component.Dispose()
   at System.Data.Common.DbCommand.DisposeAsync()
   at Microsoft.EntityFrameworkCore.Storage.RelationalDataReader.DisposeAsync()
   at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.ToListAsync[TSource](IQueryable`1 source, CancellationToken cancellationToken)
   at Volo.Abp.SettingManagement.EntityFrameworkCore.EfCoreSettingRepository.GetListAsync(String providerName, String providerKey, CancellationToken cancellationToken)
   at Castle.DynamicProxy.AsyncInterceptorBase.ProceedAsynchronous[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo)
   at Volo.Abp.Castle.DynamicProxy.CastleAbpMethodInvocationAdapterWithReturnValue`1.ProceedAsync()
   at Volo.Abp.Uow.UnitOfWorkInterceptor.InterceptAsync(IAbpMethodInvocation invocation)
   at Volo.Abp.Castle.DynamicProxy.CastleAsyncAbpInterceptorAdapter`1.InterceptAsync[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo, Func`3 proceed)
   at Volo.Abp.SettingManagement.SettingManagementStore.SetCacheItemsAsync(String providerName, String providerKey, String currentName, SettingCacheItem currentCacheItem)
   at Volo.Abp.SettingManagement.SettingManagementStore.GetCacheItemAsync(String name, String providerName, String providerKey)
   at Volo.Abp.SettingManagement.SettingManagementStore.GetOrNullAsync(String name, String providerName, String providerKey)
   at Castle.DynamicProxy.AsyncInterceptorBase.ProceedAsynchronous[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo)
   at Volo.Abp.Castle.DynamicProxy.CastleAbpMethodInvocationAdapterWithReturnValue`1.ProceedAsync()
   at Volo.Abp.Uow.UnitOfWorkInterceptor.InterceptAsync(IAbpMethodInvocation invocation)
   at Volo.Abp.Castle.DynamicProxy.CastleAsyncAbpInterceptorAdapter`1.InterceptAsync[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo, Func`3 proceed)
   at Volo.Abp.Settings.TenantSettingValueProvider.GetOrNullAsync(SettingDefinition setting)
   at Volo.Abp.Settings.SettingProvider.GetOrNullValueFromProvidersAsync(IEnumerable`1 providers, SettingDefinition setting)
   at Volo.Abp.Settings.SettingProvider.GetOrNullAsync(String name)
   at Volo.Abp.Settings.SettingProviderExtensions.GetAsync[T](ISettingProvider settingProvider, String name, T defaultValue)
   at Volo.Abp.Identity.AbpIdentityOptionsManager.OverrideOptionsAsync(String name, IdentityOptions options)
   at Volo.Abp.Identity.IdentityDataSeeder.SeedAsync(String adminEmail, String adminPassword, Nullable`1 tenantId)
   at Castle.DynamicProxy.AsyncInterceptorBase.ProceedAsynchronous[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo)
   at Volo.Abp.Castle.DynamicProxy.CastleAbpMethodInvocationAdapterWithReturnValue`1.ProceedAsync()
   at Volo.Abp.Uow.UnitOfWorkInterceptor.InterceptAsync(IAbpMethodInvocation invocation)
   at Volo.Abp.Castle.DynamicProxy.CastleAsyncAbpInterceptorAdapter`1.InterceptAsync[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo, Func`3 proceed)
   at Volo.Abp.Data.DataSeeder.SeedAsync(DataSeedContext context)
   at Castle.DynamicProxy.AsyncInterceptorBase.ProceedAsynchronous(IInvocation invocation, IInvocationProceedInfo proceedInfo)
   at Volo.Abp.Castle.DynamicProxy.CastleAbpMethodInvocationAdapter.ProceedAsync()
   at Volo.Abp.Uow.UnitOfWorkInterceptor.InterceptAsync(IAbpMethodInvocation invocation)
   at Volo.Abp.Castle.DynamicProxy.CastleAsyncAbpInterceptorAdapter`1.InterceptAsync(IInvocation invocation, IInvocationProceedInfo proceedInfo, Func`3 proceed)
   at MyApp.AppPricingTestBaseModule.<>c__DisplayClass3_0.<<SeedTestData>b__0>d.MoveNext() in C:\Users\myusername\source\repos\AppPricing\test\MyApp.TestBase\AppPricingTestBaseModule.cs:line 78
--- End of stack trace from previous location ---
   at Nito.AsyncEx.Synchronous.TaskExtensions.WaitAndUnwrapException(Task task)
   at Nito.AsyncEx.AsyncContext.<>c__DisplayClass15_0.<Run>b__0(Task t)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
   at Nito.AsyncEx.Synchronous.TaskExtensions.WaitAndUnwrapException(Task task)
   at Nito.AsyncEx.AsyncContext.Run(Func`1 action)
   at Volo.Abp.Threading.AsyncHelper.RunSync(Func`1 action)
   at MyApp.AppPricingTestBaseModule.SeedTestData(ApplicationInitializationContext context) in C:\Users\myusername\source\repos\AppPricing\test\MyApp.TestBase\AppPricingTestBaseModule.cs:line 74
   at MyApp.AppPricingTestBaseModule.OnApplicationInitialization(ApplicationInitializationContext context) in C:\Users\myusername\source\repos\AppPricing\test\MyApp.TestBase\AppPricingTestBaseModule.cs:line 69
   at Volo.Abp.Modularity.OnApplicationInitializationModuleLifecycleContributor.Initialize(ApplicationInitializationContext context, IAbpModule module)
   at Volo.Abp.Modularity.ModuleManager.InitializeModules(ApplicationInitializationContext context)

Échoué!  - échec :     1, réussite :    10, ignorée(s) :     0, total :    11, durée : 8 s - MyApp.Domain.Tests.dll (net8.0)

Regression?

No response

Known Workarounds

If you do not need the data initialization done by the Settings/Features/Permissions modules during testing, you can disable their data initialization, which will prevent the separate threads from being created in the impacted test module (usually any module referencing your Domain module):

public class MyAppTestBaseModule : AbpModule
{
  public override void ConfigureServices(ServiceConfigurationContext context)
  {
    // ... other initialization tasks here ...
   
        // Disable the settings module data initialization
        Configure<SettingManagementOptions>(options =>
        {
            options.IsDynamicSettingStoreEnabled = false;
            options.SaveStaticSettingsToDatabase = false;
        });

        // Disable the permissions module data initialization
        Configure<PermissionManagementOptions>(options =>
        {
            options.IsDynamicPermissionStoreEnabled = false;
            options.SaveStaticPermissionsToDatabase = false;
        });

        // Disable the features module data initialization
        Configure<FeatureManagementOptions>(options =>
        {
            options.IsDynamicFeatureStoreEnabled = false;
            options.SaveStaticFeaturesToDatabase = false;
        });
  }

Version

8.0.4

User Interface

MVC

Database Provider

EF Core (Default)

Tiered or separate authentication server

Tiered

Operation System

Windows (Default)

Other information

No response

[realLiangshiwei]

Entity Framework Core does not support multiple parallel operations being run on the same DbContext instance

ABP will create a new database context instead of using the same one

abp/modules/setting-management/src/Volo.Abp.SettingManagement.Domain/Volo/Abp/SettingManagement/StaticSettingSaver.cs

Line 95 in e31121d

using (var unitOfWork = UnitOfWorkManager.Begin(requiresNew: true, isTransactional: true))

The unit tests fails because the SQLite drivers used by the tests are very sensitive to concurrency problems

Maybe related to : #19065

[Shtong]

Yes, this seems to be the same issue. Closing as duplicate