Redis ORM

1 历史

之前一直使用的 meetbill/redis-orm 库

> xxxxxxxxxx 表示 instance id: 如 39NzUg3sOmD7ZDe1
+------------------------------------------------------------+
| string: redisorm:{model_name}:object:{xxxxxxxxxx}          | 通过 pickle 序列化为二进制数据进行存储
| string: redisorm:{model_name}:object:{xxxxxxxxxx}:expire   | 记录对象的过期时间
| set, redisorm:{model_name}:tags:{field_name1:field_value1} | 记录了此 tag 的 instance id 集合
| set, redisorm:{model_name}:tags:{field_name2:field_value2} | # 使用 set 交集可以获取多个 tag 的 instance 集合
| set, redisorm:{model_name}:__all__                         | 记录了此 model 的所有 instance id
| zset, redisorm:{model_name}:__expire__                     | 使用 zset 可方便批量删除过期 instance
+------------------------------------------------------------+

需求：
(1) 二级索引：支持，通过 tag 方式写入特定的 set，然后使用 Smembers 进行返回对应的 id，然后再根据 id 查询对应的对象
(2) model: 支持，通过设置固定的前缀实现不同 model 的区分
(3) 搜索：仅支持普通搜索，即搜索固定 tag

但有如下不足：
(1) 存储的 key 为 redisorm:{model_name}:object:{xxxxxxxxxx} 方式，instance id 是个无意义的字符串，查找不方便
    假如设置了唯一键的话，应该设置为唯一键标识，查找时可以直接拼接对应的 key，无需查找两次
(2) 无法根据范围查找 instance
    比如 field_value 为数字时，无法根据查询特定条件的 instance

看到 coleifer/walrus 满足条件，进而逐渐替换为 coleifer/walrus

2 model

例子

import datetime
from xlib import db
from xlib.db import redisorm
from xlib.middleware import funcattr
from xlib.httpgateway import Request

class User(db.RedisModel):
    name = redisorm.TextField(primary_key=True)
    dob = redisorm.DateField(index=True)

@funcattr.api
def redisorm_demo(req):
    """
    redisorm 例子
    """
    assert isinstance(req, Request)
    # 增
    User.create(name='Charlie', dob=datetime.date(1983, 1, 1))

2.1 增

# 操作
User.create(name='Charlie', dob=datetime.date(1983, 1, 1))

# 内部变更
1607656303.134010 [0 127.0.0.1:43505] "DEL" "user:id.Charlie"
1607656303.134322 [0 127.0.0.1:43505] "HMSET" "user:id.Charlie" "dob" "410198400.0" "name" "Charlie"
1607656303.134667 [0 127.0.0.1:43505] "SADD" "user:all" "user:id.Charlie"
1607656303.135038 [0 127.0.0.1:43505] "SADD" "user:dob.absolute.410198400.0" "user:id.Charlie"
1607656303.135410 [0 127.0.0.1:43505] "ZADD" "user:dob.continuous" "410198400.0" "user:id.Charlie"
1607656303.135756 [0 127.0.0.1:43505] "SADD" "user:name.absolute.Charlie" "user:id.Charlie"

2.2 查

2.2.1 查询单个数据

# 操作
user = User.load('Charlie')
print user         # <User: Charlie>
print user.name    # Charlie
print user.bob     # 1983-01-01

# 内部变更
1607657219.497825 [0 127.0.0.1:21021] "EXISTS" "user:id.Charlie"
1607657219.498066 [0 127.0.0.1:21021] "HGETALL" "user:id.Charlie"

假设 key 不存在时，会返回如下异常

Traceback (most recent call last):
  File "/home/meetbill/test/butterfly/handlers/x/__init__.py", line 84, in redisorm_demo
    user = User.load('Charliex')
  File "/home/meetbill/test/butterfly/xlib/db/redisorm/models.py", line 799, in load
    raise KeyError('Object not found.')
KeyError: 'Object not found.'

2.2.2 检索所有记录

names_dobs = [
    ('Huey', datetime.date(2011, 6, 1)),
    ('Zaizee', datetime.date(2012, 5, 1)),
    ('Mickey', datetime.date(2007, 8, 1))]
for name, dob in names_dobs:
    User.create(name=name, dob=dob)

for user in User.all():
    print user.name

# 内部变更
----------------------------------------------------------- all
1607660345.415714 [0 127.0.0.1:27484] "SSCAN" "user:all" "0"
1607660345.416097 [0 127.0.0.1:27484] "EXISTS" "user:id.Zaizee"
1607660345.416310 [0 127.0.0.1:27484] "HGETALL" "user:id.Zaizee"
1607660345.416956 [0 127.0.0.1:27484] "EXISTS" "user:id.Mickey"
1607660345.417174 [0 127.0.0.1:27484] "HGETALL" "user:id.Mickey"
1607660345.417792 [0 127.0.0.1:27484] "EXISTS" "user:id.Huey"
1607660345.418009 [0 127.0.0.1:27484] "HGETALL" "user:id.Huey"
1607660345.418602 [0 127.0.0.1:27484] "EXISTS" "user:id.Charlie"
1607660345.418822 [0 127.0.0.1:27484] "HGETALL" "user:id.Charlie"

2.2.3 排序

排序

# 操作
for user in User.query(order_by=User.name):
    print user.name

# 内部变更
1607661155.310246 [0 127.0.0.1:32299] "SORT" "user:all" "BY" "*->name" "ALPHA"
1607661155.310630 [0 127.0.0.1:32299] "EXISTS" "user:id.Charlie"
1607661155.310868 [0 127.0.0.1:32299] "HGETALL" "user:id.Charlie"
1607661155.311516 [0 127.0.0.1:32299] "EXISTS" "user:id.Huey"
1607661155.311745 [0 127.0.0.1:32299] "HGETALL" "user:id.Huey"
1607661155.312347 [0 127.0.0.1:32299] "EXISTS" "user:id.Mickey"
1607661155.312569 [0 127.0.0.1:32299] "HGETALL" "user:id.Mickey"
1607661155.313186 [0 127.0.0.1:32299] "EXISTS" "user:id.Zaizee"
1607661155.313407 [0 127.0.0.1:32299] "HGETALL" "user:id.Zaizee"

2.2.4 过滤（根据范围查询时需要使用 lua)

使用 lua，后端为 Redis 集群时可能会失败

范围查询

# 操作
for user in User.query(User.dob <= datetime.date(2009, 1, 1)):
    print user.dob

# 内部变更
1607661321.459083 [0 127.0.0.1:57637] "EVALSHA" "a40edace4ab12ab8926998b7ae581d1c132ad9c7" "2" "user:dob.continuous" "temp.ada9e509-6097-4a61-abad-9775e964ba84" "-inf" "1230739200.0"
1607661321.459375 [0 127.0.0.1:57637] "SCRIPT" "LOAD" "-- Filter.[nlocal zset_key = KEYS[1][nlocal dest_key = KEYS[2][nlocal low = ARGV[1][nlocal high = ARGV[2]\nlocal values = redis.call('ZRANGEBYSCORE', zset_key, low, high)\nif # values > 0 then\n  for i, value in ipairs(values) do\n    redis.call('SADD', dest_key, value)\n  end\nend\nreturn # values\n"
1607661321.459786 [0 127.0.0.1:57637] "EVALSHA" "a40edace4ab12ab8926998b7ae581d1c132ad9c7" "2" "user:dob.continuous" "temp.ada9e509-6097-4a61-abad-9775e964ba84" "-inf" "1230739200.0"
1607661321.459813 [0 lua] "ZRANGEBYSCORE" "user:dob.continuous" "-inf" "1230739200.0"
1607661321.459834 [0 lua] "SADD" "temp.ada9e509-6097-4a61-abad-9775e964ba84" "user:id.Mickey"
1607661321.460063 [0 127.0.0.1:57637] "EXPIRE" "temp.ada9e509-6097-4a61-abad-9775e964ba84" "15"
1607661321.460311 [0 127.0.0.1:57637] "SSCAN" "temp.ada9e509-6097-4a61-abad-9775e964ba84" "0"
1607661321.460605 [0 127.0.0.1:57637] "EXISTS" "user:id.Mickey"
1607661321.460822 [0 127.0.0.1:57637] "HGETALL" "user:id.Mickey"

等式

# 操作
for user in User.query(User.num == 20):
    print user.name

# 内部变更
1607667218.945403 [0 127.0.0.1:10498] "SSCAN" "user:num.absolute.20" "0"
1607667218.945817 [0 127.0.0.1:10498] "EXISTS" "user:id.Mickey"
1607667218.946053 [0 127.0.0.1:10498] "HGETALL" "user:id.Mickey"

不等式 (set 交集运算）

备注

涉及 set 交集运算，如果后端是集群，则涉及的 key 应该加 hashtag, 需要修改 redis-orm 内部实现

# 操作
for user in User.query(User.num != 20):
    print user.name

# 内部变更
1607667233.159826 [0 127.0.0.1:12635] "SDIFFSTORE" "temp.a64b09c2-a7d3-4a9a-b551-34eb559bad87" "user:all" "user:num.absolute.20"
1607667233.160112 [0 127.0.0.1:12635] "EXPIRE" "temp.a64b09c2-a7d3-4a9a-b551-34eb559bad87" "15"
1607667233.160366 [0 127.0.0.1:12635] "SSCAN" "temp.a64b09c2-a7d3-4a9a-b551-34eb559bad87" "0"
1607667233.160711 [0 127.0.0.1:12635] "EXISTS" "user:id.Zaizee"
1607667233.160933 [0 127.0.0.1:12635] "HGETALL" "user:id.Zaizee"
1607667233.161656 [0 127.0.0.1:12635] "EXISTS" "user:id.Huey"
1607667233.161887 [0 127.0.0.1:12635] "HGETALL" "user:id.Huey"
1607667233.162555 [0 127.0.0.1:12635] "EXISTS" "user:id.Charlie"
1607667233.162794 [0 127.0.0.1:12635] "HGETALL" "user:id.Charlie"

2.3 改

# 操作
user = User.load('Charlie')
user.dob = datetime.date(2012, 4, 1)
user.save()

# 内部变更
--------------------------------------------------------------------load
1607657501.874712 [0 127.0.0.1:12035] "EXISTS" "user:id.Charlie"
1607657501.875011 [0 127.0.0.1:12035] "HGETALL" "user:id.Charlie"
--------------------------------------------------------------------save(load+delete+create)
1607657501.875731 [0 127.0.0.1:12035] "EXISTS" "user:id.Charlie"
1607657501.875994 [0 127.0.0.1:12035] "HGETALL" "user:id.Charlie"
1607657501.876420 [0 127.0.0.1:12035] "SREM" "user:all" "user:id.Charlie"
1607657501.876837 [0 127.0.0.1:12035] "SREM" "user:dob.absolute.410198400.0" "user:id.Charlie"
1607657501.877078 [0 127.0.0.1:12035] "SCARD" "user:dob.absolute.410198400.0"
1607657501.877334 [0 127.0.0.1:12035] "DEL" "user:dob.absolute.410198400.0"
1607657501.877695 [0 127.0.0.1:12035] "ZREM" "user:dob.continuous" "user:id.Charlie"
1607657501.877956 [0 127.0.0.1:12035] "ZCARD" "user:dob.continuous"
1607657501.878191 [0 127.0.0.1:12035] "DEL" "user:dob.continuous"
1607657501.878550 [0 127.0.0.1:12035] "SREM" "user:name.absolute.Charlie" "user:id.Charlie"
1607657501.878822 [0 127.0.0.1:12035] "SCARD" "user:name.absolute.Charlie"
1607657501.879095 [0 127.0.0.1:12035] "DEL" "user:name.absolute.Charlie"
1607657501.879352 [0 127.0.0.1:12035] "DEL" "user:id.Charlie"

1607657501.879703 [0 127.0.0.1:12035] "DEL" "user:id.Charlie"
1607657501.880046 [0 127.0.0.1:12035] "HMSET" "user:id.Charlie" "dob" "1333209600.0" "name" "Charlie"
1607657501.880407 [0 127.0.0.1:12035] "SADD" "user:all" "user:id.Charlie"
1607657501.880767 [0 127.0.0.1:12035] "SADD" "user:dob.absolute.1333209600.0" "user:id.Charlie"
1607657501.881161 [0 127.0.0.1:12035] "ZADD" "user:dob.continuous" "1333209600.0" "user:id.Charlie"
1607657501.881522 [0 127.0.0.1:12035] "SADD" "user:name.absolute.Charlie" "user:id.Charlie"

2.4 删

# 操作
>>> nobody = User.create(name='nobody', dob=datetime.date(1990, 1, 1))
>>> nobody.delete()

# 内部变更
--------------------------------------------------------------------create
1607657921.866741 [0 127.0.0.1:15678] "DEL" "user:id.nobody"
1607657921.867068 [0 127.0.0.1:15678] "HMSET" "user:id.nobody" "dob" "631123200.0" "name" "nobody"
1607657921.867377 [0 127.0.0.1:15678] "SADD" "user:all" "user:id.nobody"
1607657921.867736 [0 127.0.0.1:15678] "SADD" "user:dob.absolute.631123200.0" "user:id.nobody"
1607657921.868114 [0 127.0.0.1:15678] "ZADD" "user:dob.continuous" "631123200.0" "user:id.nobody"
1607657921.868463 [0 127.0.0.1:15678] "SADD" "user:name.absolute.nobody" "user:id.nobody"

--------------------------------------------------------------------delete
1607657921.868903 [0 127.0.0.1:15678] "EXISTS" "user:id.nobody"
1607657921.869124 [0 127.0.0.1:15678] "HGETALL" "user:id.nobody"
1607657921.869530 [0 127.0.0.1:15678] "SREM" "user:all" "user:id.nobody"
1607657921.869883 [0 127.0.0.1:15678] "SREM" "user:dob.absolute.631123200.0" "user:id.nobody"
1607657921.870107 [0 127.0.0.1:15678] "SCARD" "user:dob.absolute.631123200.0"
1607657921.870321 [0 127.0.0.1:15678] "DEL" "user:dob.absolute.631123200.0"
1607657921.870652 [0 127.0.0.1:15678] "ZREM" "user:dob.continuous" "user:id.nobody"
1607657921.870873 [0 127.0.0.1:15678] "ZCARD" "user:dob.continuous"
1607657921.871202 [0 127.0.0.1:15678] "SREM" "user:name.absolute.nobody" "user:id.nobody"
1607657921.871414 [0 127.0.0.1:15678] "SCARD" "user:name.absolute.nobody"
1607657921.871624 [0 127.0.0.1:15678] "DEL" "user:name.absolute.nobody"
1607657921.871838 [0 127.0.0.1:15678] "DEL" "user:id.nobody"

3 自定义连接以及 namespace

import datetime
from xlib import db
from xlib.db import redisorm

my_redis = db.my_caches["model"]

class User(db.RedisModel):
    _database_ = my_redis
    _namespace_ = "ceshi"
    name = redisorm.TextField(primary_key=True)
    dob = redisorm.DateField(index=True)

print User.create(name='Charlie', dob=datetime.date(1983, 1, 1))

对应 redis 记录

1617456233.589480 [0 127.0.0.1:34622] "DEL" "ceshi|user:id.Charlie"
1617456233.589727 [0 127.0.0.1:34622] "HMSET" "ceshi|user:id.Charlie" "dob" "410198400.0" "name" "Charlie"
1617456233.589893 [0 127.0.0.1:34622] "SADD" "ceshi|user:all" "ceshi|user:id.Charlie"
1617456233.590065 [0 127.0.0.1:34622] "SADD" "ceshi|user:dob.absolute.410198400.0" "ceshi|user:id.Charlie"
1617456233.590237 [0 127.0.0.1:34622] "ZADD" "ceshi|user:dob.continuous" "410198400.0" "ceshi|user:id.Charlie"
1617456233.590382 [0 127.0.0.1:34622] "SADD" "ceshi|user:name.absolute.Charlie" "ceshi|user:id.Charlie"

3.1 例子

import datetime
from xlib import db
from xlib.db import redisorm
from xlib.middleware import funcattr

class Bianque(db.RedisModel):
    id = redisorm.TextField(primary_key=True)
    instance = redisorm.TextField(index=True)
    name = redisorm.TextField(index=True)
    value = redisorm.TextField(index=True)
    c_time = redisorm.DateTimeField()


def create(instance, name, value):
    """
    Args:
        instance: instance_name
        item: item_name
        value: item_value
    """
    Bianque.create(
            id = "{instance}|{name}".format(instance=instance, name=name),
            instance = instance,
            name=name,
            value=value,
            c_time=datetime.datetime.now())

def get(instance, item):
    try:
        data = Bianque.load('{instance}|{item}'.format(instance=instance, item=item))
        return data
    except KeyError:
        return None

if __name__ == "__main__":
    # -------------------------------demo
    create("ceshi", "ceshi1", "1")
    item=get("ceshi", "ceshi1")
    # 1
    print item.value
    item=get("ceshi", "ceshi1x")
    # None
    print item
    # -------------------------------
    _cache = db.my_caches["default"]
    item = _cache.hgetall("bianque:id.{instance}|{item_name}".format(instance="ceshi", item_name="ceshi1"))
    # {'instance': 'ceshi', 'name': 'ceshi1', 'id': 'ceshi|ceshi1', 'value': '1', 'c_time': '1628165163.900614'}
    print item
    item = _cache.hgetall("bianque:id.{instance}|{item_name}".format(instance="ceshi", item_name="ceshi1x"))
    # {}
    print item

3.2 注意

3.2.1 在 ListField 中添加值

使用 TextField 时直接可以用 cls.create() 给 redis 设置值，但是 ListField 时会报错

ListField 中插入值必须先实例化一个对象，如 SetField 例子

class Note(Model):
    _database_ = db
    text = TextField()
    timestamp = DateTimeField(
        default=datetime.datetime.now,
        index=True)
    tags = SetField()

 

>>> note = Note.create(content='my first note')
>>> note.tags
<Set "note:container.tags.note:id.3": 0 items>
>>> note.tags.add('testing', 'walrus')

>>> Note.load(note._id).tags
<Set "note:container.tags.note:id.3": 0 items>

4 实践

4.1 Redis key

4.1.1 EventHistory 数据库

主键是自增 ID

model

from xlib import db
from xlib.db import redisorm


class EventHistory(db.RedisModel):
    """
    Bianque class
    """
    _database_ = db.my_caches["baichuan"]
    # reqid
    event_id = redisorm.TextField()
    # source
    event_source = redisorm.TextField(index=True)
    # target
    event_target = redisorm.TextField(index=True)
    # type
    event_type = redisorm.TextField()
    # event status msg
    event_status = redisorm.TextField(index=True)
    # data
    event_data = redisorm.PickledField()
    # action_status
    action_status = redisorm.TextField()
    # action_jobid
    action_jobid = redisorm.IntegerField()
    # event_expire(90d)
    event_expire = redisorm.ExpireField(default=7776000)
    # create_time
    create_time = redisorm.DateTimeField()

如 ORM 类为 EventHistory
则会有如下的 key(<> 表示变量)
{eventhistory}:all                                     # set
{eventhistory}:<field_name>.absolute.<field_value>     # set
{eventhistory}:<field_name>.continuous                 # zset
{eventhistory}:_id.absolute.<id>                       # set
{eventhistory}:_id.continuous                          # zset
eventhistory:id.<id>


如果主键 key 是自增 id, 则存储当前 id 的 key 为：
string {<model_name>}:_id._sequence

所以清空数据的话，将 {eventhistory} 和 eventhistory 开头的 key 删除即可

4.1.2 Event 数据库

model

from xlib import db
from xlib.db import redisorm


class Event(db.RedisModel):
    """
    Bianque class
    """
    _database_ = db.my_caches["baichuan"]
    # source + target (key: event:id.<source>:<target_id>)
    event_name = redisorm.TextField(primary_key=True)
    # reqid
    event_id = redisorm.TextField()
    # source
    event_source = redisorm.TextField(index=True)
    # target
    event_target = redisorm.TextField(index=True)
    # type
    event_type = redisorm.TextField(index=True)
    # status
    event_flag = redisorm.TextField(index=True)
    # event status msg
    event_status = redisorm.TextField(index=True)
    # data
    event_data = redisorm.PickledField()
    # data hash
    event_data_hash = redisorm.TextField()
    # count(event_data 重复次数，若与上次一样，进行递增，否则清零)
    event_count = redisorm.IntegerField(default=0)
    # event update time(若 event_data 重复, 仅更新 update_time ，但不更新 create_time)
    update_time = redisorm.DateTimeField()
    # event create time
    create_time = redisorm.DateTimeField()
    # event_expire(2d)
    event_expire = redisorm.ExpireField(default=172800)

redis> SRANDMEMBER {event}:all 1
1) "event:id.check_redis_config:3821"

---------------------value 是字符串的索引
redis> keys {event}:event_source.*
 1) "{event}:event_source.absolute.check_proxy_appid"
 2) "{event}:event_source.absolute.metaserver_connect"
 3) "{event}:event_source.absolute.metaserver_num"
 4) "{event}:event_source.absolute.check_replication_num"
 5) "{event}:event_source.absolute.check_replication"
 6) "{event}:event_source.absolute.check_redis_safenum"
 7) "{event}:event_source.absolute.check_redis_memory_usage"
 8) "{event}:event_source.absolute.check_redis_config"
 9) "{event}:event_source.absolute.check_redis_outkbps"
10) "{event}:event_source.absolute.check_proxy_cpu"
11) "{event}:event_source.absolute.check_pega_memory_usage"
12) "{event}:event_source.absolute.ceshi"
13) "{event}:event_source.absolute.check_whitelist"
14) "{event}:event_source.absolute.check_proxy_num"

-------------------- 查询 expire 过期数据
redis> "ZRANGEBYSCORE" "{event}:event_expire.continuous" "-inf" "(1685083956" "limit" "0" "1000"
 1) "event:id.check_pega_memory_usage:4487"
 2) "event:id.check_pega_memory_usage:4517"
备注： 1685083956 就是当前时间

4.2 Redis 内部操作

4.2.1 记录总数

> SCARD {eventhistory}:all
(integer) 7447504

4.2.2 排序

EventHistory.query(EventHistory.event_flag == "OK", order_by=EventHistory._id)
> "SORT" "{eventhistory}:event_flag.absolute.OK" "BY" "*->_id"  LIMIT 0 10
 1) "eventhistory:id.261"
 2) "eventhistory:id.263"
 3) "eventhistory:id.265"
 4) "eventhistory:id.266"
 5) "eventhistory:id.267"
 6) "eventhistory:id.268"
 7) "eventhistory:id.269"
 8) "eventhistory:id.270"
 9) "eventhistory:id.271"
10) "eventhistory:id.272"
(17.04s)

-------------------------------------------------------
EventHistory.query(order_by=EventHistory._id)
> "SORT" "{eventhistory}:all" "BY" "*->_id" LIMIT 0 10
 1) "eventhistory:id.1"
 2) "eventhistory:id.2"
 3) "eventhistory:id.3"
 4) "eventhistory:id.4"
 5) "eventhistory:id.5"
 6) "eventhistory:id.6"
 7) "eventhistory:id.7"
 8) "eventhistory:id.8"
 9) "eventhistory:id.9"
10) "eventhistory:id.10"
(16.05s)

-------------------------------------------------------
EventHistory.query(EventHistory._id < 20, order_by=EventHistory._id)
1669348526.990706 [0 127.0.0.1:18600] "EVALSHA" "a40edace4ab12ab8926998b7ae581d1c132ad9c7" "2" "{eventhistory}:_id.continuous" "{eventhistory}.temp.cbf44a74-6e4f-4e3e-b013-04e39c8e1629" "-inf" "(20"
1669348526.990767 [0 lua] "SADD" "{eventhistory}.temp.cbf44a74-6e4f-4e3e-b013-04e39c8e1629" "eventhistory:id.1"
1669348526.990778 [0 lua] "SADD" "{eventhistory}.temp.cbf44a74-6e4f-4e3e-b013-04e39c8e1629" "eventhistory:id.2"
1669348526.990792 [0 lua] "SADD" "{eventhistory}.temp.cbf44a74-6e4f-4e3e-b013-04e39c8e1629" "eventhistory:id.3"
1669348526.990800 [0 lua] "SADD" "{eventhistory}.temp.cbf44a74-6e4f-4e3e-b013-04e39c8e1629" "eventhistory:id.4"
1669348526.990809 [0 lua] "SADD" "{eventhistory}.temp.cbf44a74-6e4f-4e3e-b013-04e39c8e1629" "eventhistory:id.5"
1669348526.990818 [0 lua] "SADD" "{eventhistory}.temp.cbf44a74-6e4f-4e3e-b013-04e39c8e1629" "eventhistory:id.6"
1669348526.990827 [0 lua] "SADD" "{eventhistory}.temp.cbf44a74-6e4f-4e3e-b013-04e39c8e1629" "eventhistory:id.7"
1669348526.990836 [0 lua] "SADD" "{eventhistory}.temp.cbf44a74-6e4f-4e3e-b013-04e39c8e1629" "eventhistory:id.8"
1669348526.990844 [0 lua] "SADD" "{eventhistory}.temp.cbf44a74-6e4f-4e3e-b013-04e39c8e1629" "eventhistory:id.9"
1669348526.990853 [0 lua] "SADD" "{eventhistory}.temp.cbf44a74-6e4f-4e3e-b013-04e39c8e1629" "eventhistory:id.10"
1669348526.990863 [0 lua] "SADD" "{eventhistory}.temp.cbf44a74-6e4f-4e3e-b013-04e39c8e1629" "eventhistory:id.11"
1669348526.990872 [0 lua] "SADD" "{eventhistory}.temp.cbf44a74-6e4f-4e3e-b013-04e39c8e1629" "eventhistory:id.12"
1669348526.990884 [0 lua] "SADD" "{eventhistory}.temp.cbf44a74-6e4f-4e3e-b013-04e39c8e1629" "eventhistory:id.13"
1669348526.990893 [0 lua] "SADD" "{eventhistory}.temp.cbf44a74-6e4f-4e3e-b013-04e39c8e1629" "eventhistory:id.14"
1669348526.990902 [0 lua] "SADD" "{eventhistory}.temp.cbf44a74-6e4f-4e3e-b013-04e39c8e1629" "eventhistory:id.15"
1669348526.990911 [0 lua] "SADD" "{eventhistory}.temp.cbf44a74-6e4f-4e3e-b013-04e39c8e1629" "eventhistory:id.16"
1669348526.990920 [0 lua] "SADD" "{eventhistory}.temp.cbf44a74-6e4f-4e3e-b013-04e39c8e1629" "eventhistory:id.17"
1669348526.990929 [0 lua] "SADD" "{eventhistory}.temp.cbf44a74-6e4f-4e3e-b013-04e39c8e1629" "eventhistory:id.18"
1669348526.990938 [0 lua] "SADD" "{eventhistory}.temp.cbf44a74-6e4f-4e3e-b013-04e39c8e1629" "eventhistory:id.19"
1669348526.991180 [0 127.0.0.1:18600] "EXPIRE" "{eventhistory}.temp.cbf44a74-6e4f-4e3e-b013-04e39c8e1629" "15"
1669348526.991320 [0 127.0.0.1:18600] "SORT" "{eventhistory}.temp.cbf44a74-6e4f-4e3e-b013-04e39c8e1629" "BY" "*->_id"

-------------------------------------------------------
EventHistory.query((EventHistory._id < 20) & (EventHistory._id > 10), order_by=EventHistory._id)
(EventHistory._id > 10) 这个非常耗时
(EventHistory._id < 20) 如果数量量大时，也会非常耗时

4.2.3 删除

1683118133.410823 [0 127.0.0.1:51140] "EXISTS" "eventhistory:id.3467694"
1683118133.410905 [0 127.0.0.1:51140] "HGETALL" "eventhistory:id.3467694"
1683118133.411170 [0 127.0.0.1:51140] "EXISTS" "eventhistory:id.3467694"
1683118133.411268 [0 127.0.0.1:51140] "HGETALL" "eventhistory:id.3467694"

1683118133.411522 [0 127.0.0.1:51140] "SREM" "{eventhistory}:all" "eventhistory:id.3467694"

// 字段是字符串(event_source)
1683118133.411654 [0 127.0.0.1:51140] "SREM" "{eventhistory}:event_source.absolute.check_proxy_num" "eventhistory:id.3467694"
1683118133.411756 [0 127.0.0.1:51140] "SCARD" "{eventhistory}:event_source.absolute.check_proxy_num"

// 字段是字符串(event_target)
1683118133.411883 [0 127.0.0.1:51140] "SREM" "{eventhistory}:event_target.absolute.6074" "eventhistory:id.3467694"
1683118133.411980 [0 127.0.0.1:51140] "SCARD" "{eventhistory}:event_target.absolute.6074"

// 字段是字符串(event_status)
1683118133.412105 [0 127.0.0.1:51140] "SREM" "{eventhistory}:event_status.absolute.OK" "eventhistory:id.3467694"
1683118133.412195 [0 127.0.0.1:51140] "SCARD" "{eventhistory}:event_status.absolute.OK"

// 字段是数字，设置的仅保存 continuous 数据 (event_expire)
1683118133.412329 [0 127.0.0.1:51140] "ZREM" "{eventhistory}:event_expire.continuous" "eventhistory:id.3467694"
1683118133.412421 [0 127.0.0.1:51140] "ZCARD" "{eventhistory}:event_expire.continuous"

// 主键
1683118133.412546 [0 127.0.0.1:51140] "SREM" "{eventhistory}:_id.absolute.3467694" "eventhistory:id.3467694"
1683118133.412640 [0 127.0.0.1:51140] "SCARD" "{eventhistory}:_id.absolute.3467694"
1683118133.412726 [0 127.0.0.1:51140] "DEL" "{eventhistory}:_id.absolute.3467694"

1683118133.412842 [0 127.0.0.1:51140] "ZREM" "{eventhistory}:_id.continuous" "eventhistory:id.3467694"
1683118133.412939 [0 127.0.0.1:51140] "ZCARD" "{eventhistory}:_id.continuous"

1683118133.413029 [0 127.0.0.1:51140] "DEL" "eventhistory:id.3467694"

4.2.4 删除所有索引

索引 key

<{model_name}>:<field_name>.absolute.<field_value>

例子

127.0.0.1:6387> exists {eventhistory}:event_type.absolute.trigger
(integer) 1
127.0.0.1:6387> debug object {eventhistory}:event_type.absolute.trigger
Value at:0x7f50f2c2baf0 refcount:1 encoding:hashtable serializedlength:253424056 lru:5377023 lru_seconds_idle:21743
(6.94s)

4.2.5 删除单个索引

索引 key

<{model_name}>:<field_name>.absolute.<field_value>

主键 key

<{model_name}>:id.<field_value>

例子

"SREM" "{event}:event_flag.absolute.ERR" "event:id.check_whitelist:5033"

4.2.6 泄露 key

<{model_name}>.temp.<uuid>

用于查询时生成的临时 key，生成完临时 key 后，会给 key 加过期时间
如果往 key 添加元素的时候超时退出，此 key 就不会被删除
这种 key 可以直接删除掉

如："{eventhistory}.temp.1bcaa699-c65d-4cae-a47c-a90bcccf8b34"

4.2.7 主键 key 自增 id

如果主键 key 是自增 id, 则存储当前 id 的 key 为：

string {<model_name>}:_id._sequence

4.3 规范

4.3.1 禁止修改主键 key

主键 key 修改后会导致存量 key 无法找到主键，故而无法删除

删除记录时会使用到主键 key

4.4 异常情况处理

4.4.1 索引中可以找到，但是却没有找到 key

4.4.2 索引数据和记录数据不一致

4.5 批量删除 key

redis-cli --scan --pattern "event:*" | xargs -L 2000 redis-cli del

4.6 多个可选的搜索条件

model = model_audit.Audit

expression = (model.event_target == target)
if source is not None:
    expression = expression & (model.event_source == source)

query = model.query(expression, order_by=model._id.desc())

传送门

• meetbill/redis-orm: 使用 str 存储对象

  • meetbill/redis-orm 使用手册
• coleifer/walrus

  • coleifer/walrus 官方文档