redis_config

1 Redis 连接

1.1 配置文件

butterfly/conf/servicer/redis_xx.py

url_xx = "redis://@localhost:6379/0?socket_timeout=0.2&socket_connect_timeout=0.1&retry_on_timeout=false"

如果有密码，则在 @ 前添加账号和密码，如 redis://default:xxxx@localhost:6379/0

1.2 读取配置

from xlib import db

_db = db.redisdb_get(None, "redis_xx", "url_xx")
cache = _db.cache()

key = "ceshi"
value = "ceshi"
timeout = 2

# True
print cache.set(key, value, timeout)
# ceshi
print cache.get(key)
# 1
print cache.delete(key)

2 Redis连接配置中的超时配置

• socket_connect_timeout

• socket_timeout

• retry_on_timeout

2.1 socket_timeout

• 此配置参数是指Redis发出命令接收响应的时间不能超过此参数设置时间. 如果超过了此时间, 将会抛出异常:redis.exceptions.TimeoutError: Timeout reading from socket, 即读取响应超时.

• 如何来演示socket_timeout触发的超时问题呢? 可以从我们经常在list类型数据上进行BLPOP操作着手.

  from redis import StrictRedis
  
  redis = StrictRedis(host="192.168.0.111",
                      port=6380,
                      db=13,
                      socket_timeout=5,
                      socket_connect_timeout=2,
                      decode_responses=True)
  
  
  a = redis.blpop("test_key", timeout=10)

  如上所示: 当test_key列表为空时, BLPOP命令将会进行阻塞, 阻塞的timeout设置为10秒, 但Redis连接的socket_timeout设置为5秒. 此时会触发异常.

  原因如下: 当我们在进行BLPOP操作时, 阻塞的过程其实也是等待响应的过程(waiting for reading from socket). 因为socket_timeout设置为5秒, 所以阻塞5秒后会触发超时异常, 而非等待10秒后得到的None.

• 解决方式: 将类BLPOP型命令的超时时间设置少于socket_timeout设置的时间即可.

  from redis import StrictRedis
  
  redis = StrictRedis(host="192.168.0.111",
                      port=6380,
                      db=13,
                      socket_timeout=10,
                      socket_connect_timeout=2,
                      decode_responses=True)
  
  
  a = redis.blpop("test_key", timeout=5)

• 实际业务中遇到此问题是在使用redis_lock过程中, 其实原理和上述一致(实际redis_lock底层也是使用了BLPOP命令), 只是锁的过期时间需要小于socket_timeout设置的时间.示例代码如下:

  # -*- coding: utf-8 -*-
  import time
  import threading
  
  import redis_lock
  from redis import StrictRedis
  
  redis = StrictRedis(host="192.168.0.111",
                      port=6380,
                      db=13,
                      socket_timeout=5,
                      socket_connect_timeout=2,
                      decode_responses=True)
  
  
  def func1():
      with redis_lock.Lock(redis_client=redis, name="test_key", expire=10):
          print("线程01获取到了锁")
          time.sleep(120)
  
  
  def func2():
      with redis_lock.Lock(redis_client=redis, name="test_key", expire=10):
          print("线程02获取到了锁")
          time.sleep(120)
  
  
  if __name__ == '__main__':
      t1 = threading.Thread(target=func1)
      t2 = threading.Thread(target=func2)
      t1.start()
      t2.start()
      t1.join()
      t2.join()

2.2 socket_connect_timeout

• 此配置参数指Redis建立连接超时时间. 当设置此参数时, 如果在此时间内没有建立连接, 将会抛出异常redis.exceptions.TimeoutError: Timeout connecting to server.

• 可以用以下代码做测试, 比如, 将socket_connect_timeout设置尽可能小, 则很容易模拟实际的连接超时问题.

  from redis import StrictRedis
  
  redis = StrictRedis(host="192.168.0.111",
                      port=6380,
                      db=13,
                      socket_connect_timeout=0.001)

• 对redis-py源码进行分析, 实际redis建立TCP连接使用socket包进行的连接. 源码如下:

      def connect(self):
          if self._sock:  # 如果连接已存在,则直接返回
              return
          try:
              sock = self._connect()
          except socket.timeout:  # 此处即捕获socket的timeout异常
              raise TimeoutError("Timeout connecting to server")
  
      def _connect(self):
          "Create hardware TCP socket connection"
          """此处即实际Redis使用socket进行tcp连接的方法,在此处会在连接超时时抛出异常"""
          err = None
          for res in socket.getaddrinfo(self.host, self.port, self.socket_type,
                                        socket.SOCK_STREAM):
              family, socktype, proto, canonname, socket_address = res
              sock = None
              try:
                  sock = socket.socket(family, socktype, proto)
                  # TCP_NODELAY
                  sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
  
                  # TCP_KEEPALIVE
                  if self.socket_keepalive:
                      sock.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
                      for k, v in iteritems(self.socket_keepalive_options):
                          sock.setsockopt(socket.IPPROTO_TCP, k, v)
  
                  # set the socket_connect_timeout before we connect
                  sock.settimeout(self.socket_connect_timeout)
  
                  # connect
                  sock.connect(socket_address)
  
                  # set the socket_timeout now that we're connected
                  sock.settimeout(self.socket_timeout)
                  return sock
  
              except socket.error as _:
                  err = _
                  if sock is not None:
                      sock.close()
  
          if err is not None:
              raise err  # 此处抛出实际连接异常的错误信息
          raise socket.error("socket.getaddrinfo returned an empty list")

2.3 retry_on_timeout

• 此参数为布尔型. 默认为False.

• 当设置False时, 一个命令超时后, 将会直接抛出timeout异常.

• 当设置为True时, 命令超时后,将会重试一次, 重试成功则正常返回; 失败则抛出timeout异常.

引用

Need clarification and better documentation on socket_timeout, socket_connect_timeout, and blocking commands