Open Closed

No Connection available on Redis server #5568


User avatar
0
chris.b@mindmill.co.uk created

We have a problem affecting our production environment where we are unable to connect to Redis and it brings the sites down. Below is the error from the Auth Server logs.

  • ABP Framework version: v6.2
  • UI Type: MVC
  • Database System: EF Core (SQL Server)
  • Tiered (for MVC) or Auth Server Separated (for Angular): yes
  • Exception message and full stack trace:
* 2023-08-11 05:19:20.878 +00:00 [ERR] No connection is active/available to service this operation: SET GTCManagementProduction:_Phoenix.GTCManagement.AuthServer_AbpExternalLocalizationSaving, mc: 1/1/0, mgr: 10 of 10 available, clientName: app1-vm-prod, IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=1,Free=32766,Min=4,Max=32767), v: 2.2.4.27433
StackExchange.Redis.RedisConnectionException: No connection is active/available to service this operation: SET GTCManagementProduction:_Phoenix.GTCManagement.AuthServer_AbpExternalLocalizationSaving, mc: 1/1/0, mgr: 10 of 10 available, clientName: app1-vm-prod, IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=1,Free=32766,Min=4,Max=32767), v: 2.2.4.27433
   at StackExchange.Redis.ConnectionMultiplexer.ThrowFailed[T](TaskCompletionSource`1 source, Exception unthrownException) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 2760
--- End of stack trace from previous location ---
   at Medallion.Threading.Redis.RedLock.RedLockAcquire.WaitForAcquireAsync(IReadOnlyDictionary`2 tryAcquireTasks) in /_/DistributedLock.Redis/RedLock/RedLockAcquire.cs:line 144
   at Medallion.Threading.Redis.RedLock.RedLockAcquire.TryAcquireAsync() in /_/DistributedLock.Redis/RedLock/RedLockAcquire.cs:line 60
   at Medallion.Threading.Redis.RedLock.RedLockAcquire.TryAcquireAsync() in /_/DistributedLock.Redis/RedLock/RedLockAcquire.cs:line 94
   at Medallion.Threading.Redis.RedisDistributedLock.TryAcquireAsync(CancellationToken cancellationToken) in /_/DistributedLock.Redis/RedisDistributedLock.cs:line 73
   at Medallion.Threading.Internal.BusyWaitHelper.WaitAsync[TState,TResult](TState state, Func`3 tryGetValue, TimeoutValue timeout, TimeoutValue minSleepTime, TimeoutValue maxSleepTime, CancellationToken cancellationToken) in /_/DistributedLock.Core/Internal/BusyWaitHelper.cs:line 29
   at Medallion.Threading.Internal.Helpers.Convert[TDerived,TBase](ValueTask`1 task, ValueTaskConversion _) in /_/DistributedLock.Core/Internal/Helpers.cs:line 24
   at Volo.Abp.DistributedLocking.MedallionAbpDistributedLock.TryAcquireAsync(String name, TimeSpan timeout, CancellationToken cancellationToken)
   at Volo.Abp.LanguageManagement.External.ExternalLocalizationSaver.SaveAsync()
   at Castle.DynamicProxy.AsyncInterceptorBase.ProceedAsynchronous(IInvocation invocation, IInvocationProceedInfo proceedInfo)
   at Volo.Abp.Castle.DynamicProxy.CastleAbpMethodInvocationAdapter.ProceedAsync()
   at Volo.Abp.Uow.UnitOfWorkInterceptor.InterceptAsync(IAbpMethodInvocation invocation)
   at Volo.Abp.Castle.DynamicProxy.CastleAsyncAbpInterceptorAdapter`1.InterceptAsync(IInvocation invocation, IInvocationProceedInfo proceedInfo, Func`3 proceed)
   at Volo.Abp.LanguageManagement.LanguageManagementDomainModule.<>c__DisplayClass6_1.dqYSTJLxrtjUOO4OmDO.MoveNext()
2023-08-11 05:19:20.878 +00:00 [ERR] ---------- Exception Data ----------
Redis-Multiplexer-Connects = 1/1/0
Redis-Manager = 10 of 10 available
Redis-Client-Name = app1-vm-prod
Redis-ThreadPool-IO-Completion = (Busy=0,Free=1000,Min=1,Max=1000)
Redis-ThreadPool-Workers = (Busy=1,Free=32766,Min=4,Max=32767)
Redis-Busy-Workers = 1
Redis-Version = 2.2.4.27433
redis-command = SET GTCManagementProduction:_Phoenix.GTCManagement.AuthServer_AbpExternalLocalizationSaving
request-sent-status = WaitingToBeSent
  • Steps to reproduce the issue:
  • Once the Redis service has been restarted, the issue cannot be reproduced.

1 Answer(s)
  • User Avatar
    0
    jfistelmann created

    What do your redis logs say?

    On what version is your redis and what specifically are you using?

Made with ❤️ on ABP v8.2.0-preview Updated on March 25, 2024, 15:11