Open Closed

Auth server Production issue with Redis CLUSTERDOWN MIcroservices K8s #3141


0
ronaksbhavsar created
  • ABP Framework version: v5.2.0
  • UI type: Angular
  • DB provider: EF Core
  • Tiered (MVC) or Identity Server Separated (Angular): yes
  • Exception message and stack trace: at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 2817 2022-05-25 21:46:14 StackExchange.Redis.RedisServerException: CLUSTERDOWN The cluster is down 2022-05-25 21:46:14 [16:16:14 WRN] CLUSTERDOWN The cluster is down 2022-05-25 21:46:14 at Volo.Abp.Caching.DistributedCache2.Get(TCacheKey key, Nullable1 hideErrors, Boolean considerUow) 2022-05-25 21:46:14 at Microsoft.Extensions.Caching.StackExchangeRedis.RedisCache.Get(String key) 2022-05-25 21:46:14 at Microsoft.Extensions.Caching.StackExchangeRedis.RedisCache.GetAndRefresh(String key, Boolean getData) 2022-05-25 21:46:14 at Microsoft.Extensions.Caching.StackExchangeRedis.RedisExtensions.HashMemberGet(IDatabase cache, String key, String[] members) 2022-05-25 21:46:14 at StackExchange.Redis.RedisDatabase.HashGet(RedisKey key, RedisValue[] hashFields, CommandFlags flags) in /_/src/StackExchange.Redis/RedisDatabase.cs:line 324 2022-05-25 21:46:14 at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in //src/StackExchange.Redis/RedisBase.cs:line 54 2022-05-25 21:46:14 at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 2817 2022-05-25 21:46:14 StackExchange.Redis.RedisServerException: CLUSTERDOWN The cluster is down 2022-05-25 21:46:14 [16:16:14 WRN] CLUSTERDOWN The cluster is down 2022-05-25 21:46:14 at Volo.Abp.Caching.DistributedCache2.<>c__DisplayClass50_0.<Set>g__SetRealCache|0() 2022-05-25 21:46:14 at Microsoft.Extensions.Caching.StackExchangeRedis.RedisCache.Set(String key, Byte[] value, DistributedCacheEntryOptions options) 2022-05-25 21:46:14 at StackExchange.Redis.RedisDatabase.ScriptEvaluate(String script, RedisKey[] keys, RedisValue[] values, CommandFlags flags) in //src/StackExchange.Redis/RedisDatabase.cs:line 1189 2022-05-25 21:46:14 at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in /_/src/StackExchange.Redis/RedisBase.cs:line 54 2022-05-25 21:46:14 at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in //src/StackExchange.Redis/ConnectionMultiplexer.cs:line 2817 2022-05-25 21:46:14 StackExchange.Redis.RedisServerException: CLUSTERDOWN The cluster is down 2022-05-25 21:46:14 [16:16:14 WRN] CLUSTERDOWN The cluster is down 2022-05-25 21:46:14 at Volo.Abp.Caching.DistributedCache2.Get(TCacheKey key, Nullable1 hideErrors, Boolean considerUow) 2022-05-25 21:46:14 at Microsoft.Extensions.Caching.StackExchangeRedis.RedisCache.Get(String key) 2022-05-25 21:46:14 at Microsoft.Extensions.Caching.StackExchangeRedis.RedisCache.GetAndRefresh(String key, Boolean getData) 2022-05-25 21:46:14 at Microsoft.Extensions.Caching.StackExchangeRedis.RedisExtensions.HashMemberGet(IDatabase cache, String key, String[] members) 2022-05-25 21:46:14 at StackExchange.Redis.RedisDatabase.HashGet(RedisKey key, RedisValue[] hashFields, CommandFlags flags) in //src/StackExchange.Redis/RedisDatabase.cs:line 324 2022-05-25 21:46:14 at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in /_/src/StackExchange.Redis/RedisBase.cs:line 54 2022-05-25 21:46:14 at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in //src/StackExchange.Redis/ConnectionMultiplexer.cs:line 2817 2022-05-25 21:46:14 StackExchange.Redis.RedisServerException: CLUSTERDOWN The cluster is down 2022-05-25 21:46:14 [16:16:14 WRN] CLUSTERDOWN The cluster is down 2022-05-25 21:46:14 at Volo.Abp.Caching.DistributedCache2.Get(TCacheKey key, Nullable1 hideErrors, Boolean considerUow) 2022-05-25 21:46:14 at Microsoft.Extensions.Caching.StackExchangeRedis.RedisCache.Get(String key) 2022-05-25 21:46:14 at Microsoft.Extensions.Caching.StackExchangeRedis.RedisCache.GetAndRefresh(String key, Boolean getData) 2022-05-25 21:46:14 at Microsoft.Extensions.Caching.StackExchangeRedis.RedisExtensions.HashMemberGet(IDatabase cache, String key, String[] members) 2022-05-25 21:46:14 at StackExchange.Redis.RedisDatabase.HashGet(RedisKey key, RedisValue[] hashFields, CommandFlags flags) in //src/StackExchange.Redis/RedisDatabase.cs:line 324 2022-05-25 21:46:14 at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in /_/src/StackExchange.Redis/RedisBase.cs:line 54 2022-05-25 21:46:14 at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in //src/StackExchange.Redis/ConnectionMultiplexer.cs:line 2817 2022-05-25 21:46:14 StackExchange.Redis.RedisServerException: CLUSTERDOWN The cluster is down 2022-05-25 21:46:14 [16:16:14 WRN] CLUSTERDOWN The cluster is down 2022-05-25 21:46:14 at Volo.Abp.Caching.DistributedCache2.<>c__DisplayClass50_0.<Set>g__SetRealCache|0() 2022-05-25 21:46:14 at Microsoft.Extensions.Caching.StackExchangeRedis.RedisCache.Set(String key, Byte[] value, DistributedCacheEntryOptions options) 2022-05-25 21:46:14 at StackExchange.Redis.RedisDatabase.ScriptEvaluate(String script, RedisKey[] keys, RedisValue[] values, CommandFlags flags) in /_/src/StackExchange.Redis/RedisDatabase.cs:line 1189 2022-05-25 21:46:14 at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in //src/StackExchange.Redis/RedisBase.cs:line 54 2022-05-25 21:46:14 at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 2817 2022-05-25 21:46:14 StackExchange.Redis.RedisServerException: CLUSTERDOWN The cluster is down 2022-05-25 21:46:14 [16:16:14 WRN] CLUSTERDOWN The cluster is down 2022-05-25 21:46:14 at Volo.Abp.Caching.DistributedCache2.Get(TCacheKey key, Nullable1 hideErrors, Boolean considerUow) 2022-05-25 21:46:14 at Microsoft.Extensions.Caching.StackExchangeRedis.RedisCache.Get(String key) 2022-05-25 21:46:14 at Microsoft.Extensions.Caching.StackExchangeRedis.RedisCache.GetAndRefresh(String key, Boolean getData) 2022-05-25 21:46:14 at Microsoft.Extensions.Caching.StackExchangeRedis.RedisExtensions.HashMemberGet(IDatabase cache, String key, String[] members) 2022-05-25 21:46:14 at StackExchange.Redis.RedisDatabase.HashGet(RedisKey key, RedisValue[] hashFields, CommandFlags flags) in /_/src/StackExchange.Redis/RedisDatabase.cs:line 324 2022-05-25 21:46:14 at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in //src/StackExchange.Redis/RedisBase.cs:line 54 2022-05-25 21:46:14 at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor`1 processor, ServerEndPoint server) in //src/StackExchange.Redis/ConnectionMultiplexer.cs:line 2817 2022-05-25 21:46:14 StackExchange.Redis.RedisServerException: CLUSTERDOWN The cluster is down 2022-05-25 20:54:40 [15:24:40 ERR] An error occurred while reading the key ring. Log labels

app authserver

container authserver

filename /var/log/pods/{appName}-stag_authserver-7494b9846f-brl9d_ff534c8b-8c20-4340-87b8-f1652332979a/authserver/0.log

job {appName}-stag/authserver

namespace {appName}-stag

pod authserver-7494b9846f-brl9d

pod_template_hash 7494b9846f

stream stdout Detected fields

ts 2022-05-25T15:24:40.647Z

tsNs 1653492280647510856 2022-05-25 20:54:35 [15:24:35 INF] Starting {appName}.AuthServer. 2022-05-25 20:52:36 [15:22:36 INF] Application is shutting down... 2022-05-25 20:52:35 [15:22:35 INF] Disposed RabbitMQ Channel Pool (1 channels in 928.38 ms). 2022-05-25 20:52:34 [15:22:34 INF] Disposing channel pool (1 channels).

  • Steps to reproduce the issue:" facing an issue on Auth server

2 Answer(s)
  • 0
    liangshiwei created
    Support Team

    Hi,

    CLUSTERDOWN The cluster is down

    It seems not related to ABP but Redis, can you check the Redis server?

  • 0
    ronaksbhavsar created

    Hi,

    CLUSTERDOWN The cluster is down

    It seems not related to ABP but Redis, can you check the Redis server?

    yes, redis related issue it was.

    Thanks for your time.