佳佳的博客

Appearance

RedisResponseException: No more data & Zero length respose ​

🏷️ Redis

运维切换了新的 Redis 架构之后，偶尔会报 No more data 或 Zero length respose 的错误。

架构 ​

客户端 C# 使用的 ServiceStack.Redis 3.9.60.0 包，通过连接池 PooledRedisClientManager 管理连接；
Redis 是一主多从；
中间通过 HAproxy 代理，实现 LB；

异常信息 ​

客户端 ​

cs
ServiceStack.Redis.RedisResponseException
No more data, sPort: 53059, LastCommand:
ServiceStack.Redis
   在 ServiceStack.Redis.RedisNativeClient.CreateResponseError(String error)
   在 ServiceStack.Redis.RedisNativeClient.ExpectSuccess()
   在 ServiceStack.Redis.RedisNativeClient.Set(String key, Byte[] value)
   在 ServiceStack.Redis.RedisClient.Set[T](String key, T value)
   在 ServiceStack.Redis.RedisClient.Set[T](String key, T value, DateTime expiresAt)
   在 Framework.Utils.RedisProvider.Set[T](String key, T value, Int32 minutes)
   在 MX.OBP.WebApp.Login.InitPage()
   在 System.Web.UI.Control.LoadRecursive()
   在 System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
cs
ServiceStack.Redis.RedisResponseException
Zero length respose, sPort: 52673, LastCommand:
ServiceStack.Redis
   在 ServiceStack.Redis.RedisNativeClient.CreateResponseError(String error)
   在 ServiceStack.Redis.RedisNativeClient.ParseSingleLine(String r)
   在 ServiceStack.Redis.RedisNativeClient.GetBytes(String key)
   在 ServiceStack.Redis.RedisClient.Get[T](String key)
   在 Framework.Utils.RedisProvider.Get[T](String key)

HAProxy ​

查看 HAProxy 的日志，对应上面的客户端的出错信息，日志能看到 cD 和 CD 错误状态。

CD
The client unexpectedly aborted during data transfer.
This can be caused by a browser crash, by an intermediate equipment between the client and haproxy which decided to actively break the connection,
by network routing issues between the client and haproxy, or by a keep-alive session between the server and the client terminated first by the client.
当传输数据时客户端异常退出

cD
The client did not send nor acknowledge any data for as long as the timeout client delay.
This is often caused by network failures on client side, or the client simply leaving the net uncleanly.
在 data 阶段，客户端超时

ServiceStack.Redis 源码 ​

下面贴出了上面错误消息中调用的部分源码，可以看出不管是保存还是获取，在返回的 Bstream 中没有值时，就会报上面的错误。

RedisNativeClient.cs

cs
namespace ServiceStack.Redis
{
  public class RedisNativeClient : IRedisNativeClient, IDisposable
  {
    public void Set(string key, byte[] value)
    {
      if (key == null)
        throw new ArgumentNullException(nameof (key));
      value = value ?? new byte[0];
      if (value.Length > 1073741824)
        throw new ArgumentException("value exceeds 1G", nameof (value));
      this.SendExpectSuccess(Commands.Set, key.ToUtf8Bytes(), value);
    }

    protected void SendExpectSuccess(params byte[][] cmdWithBinaryArgs)
    {
      if (!this.SendCommand(cmdWithBinaryArgs))
        throw this.CreateConnectionError();
      if (this.Pipeline != null)
        this.Pipeline.CompleteVoidQueuedCommand(new Action(this.ExpectSuccess));
      else
        this.ExpectSuccess();
    }

    protected void ExpectSuccess()
    {
      int num = this.SafeReadByte();
      if (num == -1)
        throw this.CreateResponseError("No more data");
      string str = this.ReadLine();
      if (num == 45)
        throw this.CreateResponseError(!str.StartsWith("ERR") || str.Length < 4 ? str : str.Substring(4));
    }

    public byte[] GetBytes(string key)
    {
      if (key == null)
        throw new ArgumentNullException(nameof (key));
      return this.SendExpectData(Commands.Get, key.ToUtf8Bytes());
    }

    protected byte[] SendExpectData(params byte[][] cmdWithBinaryArgs)
    {
      if (!this.SendCommand(cmdWithBinaryArgs))
        throw this.CreateConnectionError();
      if (this.Pipeline == null)
        return this.ReadData();
      this.Pipeline.CompleteBytesQueuedCommand(new Func<byte[]>(this.ReadData));
      return (byte[]) null;
    }

    private byte[] ReadData()
    {
      return this.ParseSingleLine(this.ReadLine());
    }

    private byte[] ParseSingleLine(string r)
    {
      if (r.Length == 0)
        throw this.CreateResponseError("Zero length respose");
      switch (r[0])
      {
        case '$':
          if (r == "$-1")
            return (byte[]) null;
          int result;
          if (!int.TryParse(r.Substring(1), out result))
            throw this.CreateResponseError("Invalid length");
          byte[] buffer = new byte[result];
          int offset = 0;
          while (result > 0)
          {
            int num = this.Bstream.Read(buffer, offset, result);
            if (num <= 0)
              throw this.CreateResponseError("Unexpected end of Stream");
            offset += num;
            result -= num;
          }
          if (this.Bstream.ReadByte() != 13 || this.Bstream.ReadByte() != 10)
            throw this.CreateResponseError("Invalid termination");
          return buffer;
        case '-':
          throw this.CreateResponseError(r.StartsWith("-ERR") ? r.Substring(5) : r.Substring(1));
        case ':':
          return r.Substring(1).ToUtf8Bytes();
        default:
          throw this.CreateResponseError("Unexpected reply: " + r);
      }
    }

    private int SafeReadByte()
    {
      return this.Bstream.ReadByte();
    }

    private RedisResponseException CreateResponseError(string error)
    {
      this.HadExceptions = true;
      RedisResponseException responseException = new RedisResponseException(string.Format("{0}, sPort: {1}, LastCommand: {2}", (object) error, (object) this.clientPort, (object) this.lastCommand));
      RedisNativeClient.log.Error((object) responseException.Message);
      throw responseException;
    }
  }
}

对策 ​

o(╯□╰)o

因为更改架构之前直连 Redis 是正常的，所以猜测是 HAProxy 连接 Redis 超时导致的。

Redis 的超时时间设置的是 5min，HAProxy 的超时时间是 10s，所以将 HAProxy 的超时时间设为和 Redis 一样看看是否有效果。

参考 ​