Open Closed

Creating a new tenant deadlocks in SQL Server #4563


User avatar
0
scott7106 created

reference #4278 -- this ticket is SQLLite, but I am running into this with SQL Server (Developer Edition)

  • ABP Framework version: v7.0.1
  • UI type: Angular
  • DB provider: EF Core (SQL Server)
  • Tiered (MVC) or Identity Server Separated (Angular): no
  • Exception message and stack trace:
  • Steps to reproduce the issue:"

Trying to create a new tenant.

The process is running MigrateAndSeedForTenantAsync from the ZiplineTenantDatabaseMigrationHandler (default code, we have not changed this). I am getting a deadlock here with the query to get the newly created tenant record deadlocking the insert and the query times out. Once the query trying to get the tenant record times out, the insert completes. The tenant record gets created, but none of the other data seeding for the tenant occurs. I changed the database timeout for the DbContext to 120 seconds which gives me plenty of time to diagnose and see the deadlock, but I am not seeing how to resolve the issue.

[13:15:09 ERR] An exception occurred while iterating over the results of a query for context type 'OtisEd.Zipline.EntityFrameworkCore.ZiplineDbContext'. Microsoft.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception (258): The wait operation timed out. at Microsoft.Data.SqlClient.SqlCommand.<>c.<ExecuteDbDataReaderAsync>b__209_0(Task1 result) at System.Threading.Tasks.ContinuationResultTaskFromResultTask2.InnerInvoke() at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) --- End of stack trace from previous location --- at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread) --- End of stack trace from previous location --- at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable1.AsyncEnumerator.InitializeReaderAsync(AsyncEnumerator enumerator, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func4 operation, Func4 verifySucceeded, CancellationToken cancellat ionToken) at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable1.AsyncEnumerator.MoveNextAsync() ClientConnectionId:180b7a1a-bb77-45ad-a20c-1683ba85021a Error Number:-2,State:0,Class:11 Microsoft.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception (258): The wait operation timed out. at Microsoft.Data.SqlClient.SqlCommand.<>c.<ExecuteDbDataReaderAsync>b__209_0(Task1 result) at System.Threading.Tasks.ContinuationResultTaskFromResultTask2.InnerInvoke() at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) --- End of stack trace from previous location --- at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread) --- End of stack trace from previous location --- at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable1.AsyncEnumerator.InitializeReaderAsync(AsyncEnumerator enumerator, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func4 operation, Func4 verifySucceeded, CancellationToken cancellat ionToken) at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable1.AsyncEnumerator.MoveNextAsync() ClientConnectionId:180b7a1a-bb77-45ad-a20c-1683ba85021a Error Number:-2,State:0,Class:11 [13:15:09 DBG] 'ZiplineDbContext' disposed. [13:15:09 DBG] Disposing connection to database 'Zipline7' on server 'localhost'. [13:15:09 DBG] Disposed connection to database '' on server '' (0ms). [13:15:09 ERR] Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding. Microsoft.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception (258): The wait operation timed out. at Microsoft.Data.SqlClient.SqlCommand.<>c.<ExecuteDbDataReaderAsync>b__209_0(Task1 result) at System.Threading.Tasks.ContinuationResultTaskFromResultTask2.InnerInvoke() at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) --- End of stack trace from previous location --- at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread) --- End of stack trace from previous location --- at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable1.AsyncEnumerator.InitializeReaderAsync(AsyncEnumerator enumerator, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func4 operation, Func4 verifySucceeded, CancellationToken cancellat ionToken) at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable1.AsyncEnumerator.MoveNextAsync() at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleOrDefaultAsync[TSource](IAsyncEnumerable1 asyncEnumerable, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleOrDefaultAsync[TSource](IAsyncEnumerable1 asyncEnumerable, CancellationToken cancellationToken) at Volo.Abp.Domain.Repositories.EntityFrameworkCore.EfCoreRepository3.FindAsync(TKey id, Boolean includeDetails, CancellationToken cancellationToken) at Castle.DynamicProxy.AsyncInterceptorBase.ProceedAsynchronous[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo) at Volo.Abp.Castle.DynamicProxy.CastleAbpMethodInvocationAdapterWithReturnValue1.ProceedAsync() at Volo.Abp.Uow.UnitOfWorkInterceptor.InterceptAsync(IAbpMethodInvocation invocation) at Volo.Abp.Castle.DynamicProxy.CastleAsyncAbpInterceptorAdapter1.InterceptAsync[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo, Func3 proceed) at Volo.Saas.Tenants.TenantStore.GetCacheItemAsync(Nullable`1 id, String name) at Volo.Saas.Tenants.TenantStore.FindAsync(Guid id) at OtisEd.Zipline.Data.ZiplineTenantDatabaseMigrationHandler.MigrateAndSeedForTenantAsync(Guid tenantId, String adminEmail, String adminPassword) in C:\AppDev\otised\Miscellaneous\OtisEd.Zipline\aspn et-core\src\OtisEd.Zipline.Domain\Data\ZiplineTenantDatabaseMigrationHandler.cs:line 99 ClientConnectionId:180b7a1a-bb77-45ad-a20c-1683ba85021a Error Number:-2,State:0,Class:11 [13:15:09 ERR] ---------- Exception Data ---------- HelpLink.ProdName = Microsoft SQL Server HelpLink.ProdVer = 15.00.2101 HelpLink.EvtSrc = MSSQLServer HelpLink.EvtID = -2 HelpLink.BaseHelpUrl = https://go.microsoft.com/fwlink HelpLink.LinkId = 20476


6 Answer(s)
  • User Avatar
    0
    liangshiwei created
    Support Team Fullstack Developer

    Hi,

    Can you use the CLI to create a new project to reproduce the problem?

  • User Avatar
    0
    scott7106 created

    No. I have not been able to reproduce this in a new project. The project with the issue was created using Suite v7.0.1. We have upgraded several of our projects to the latest version of ABP. This issue is only occurring in one of them. It is something specific to this project. However, I have not been able to isolate the problem.

    This line of code: var tenantConfiguration = await _tenantStore.FindAsync(tenantId);

    From the MigrateAndSeedForTenantAsync method in the *TenantDatabaseMigrationHandler is blocking the insert statement for the new tenant record. Once this line of code fails with a timeout, the tenant record gets created. But there is no data seeded for the new tenant.

    The tenant record creation should have been completed before MigrateAndSeedForTenantAsync is called. However, that is not the case in this project.

    I can email you a link to a zip file of the project.

  • User Avatar
    0
    liangshiwei created
    Support Team Fullstack Developer

    Hi,

    Ok, please. shiwei.liang@volosoft.com

  • User Avatar
    0
    liangshiwei created
    Support Team Fullstack Developer

    Hi,

    I can create a tenant without any error

    BTW, you can try :

    using (var uow = _unitOfWorkManager.Begin(requiresNew: false, isTransactional: false))
    {
        var tenantConfiguration = await _tenantStore.FindAsync(tenantId);
        if (tenantConfiguration?.ConnectionStrings != null &&
            !tenantConfiguration.ConnectionStrings.Default.IsNullOrWhiteSpace())
        {
            foreach (var migrator in _dbSchemaMigrators)
            {
                await migrator.MigrateAsync();
            }
        }
    
        await uow.CompleteAsync();
    }
    
  • User Avatar
    0
    scott7106 created

    I will try that. Did you try running it without it being on the debugger? Since it is a timing issue, the debugger could potentially affect the issue.

  • User Avatar
    0
    liangshiwei created
    Support Team Fullstack Developer

    Did you try running it without it being on the debugger

    Yes, I tried

Made with ❤️ on ABP v8.2.0-preview Updated on March 25, 2024, 15:11