🦋
Butterfly 开发手册
  • Butterfly
  • project-framework
    • 一 why
    • 二 what
    • 三 how
      • HTTP 协议基础
      • HTTP 服务器模型
      • WSGIGateway
      • 框架之路由系统
      • 框架之 MIDDLEWARE
      • 框架之日志管理
      • 框架之 HTTP 接口
        • 请求流程
        • handler 类型
          • (1) json
          • (2) file
          • (3) HTTP
        • API 设计风格
        • 响应格式自定义
        • cookie
      • 基础公共库之模板引擎
        • 配置文件渲染
      • 基础公共库之本地缓存
      • 数据访问层之资源访问 RAL（百度特有协议支持）
      • 调试和测试之调试
      • 调试和测试之单元测试
      • 状态机
  • project-handlers
    • 云原生
      • 可观测性
        • Tracing
    • 通用服务
      • 流程编排(星桥)
      • 任务队列(百川)
        • 异步任务
        • baichuan_go_woker
      • 认证鉴权(火眼)
      • 任务调度(如期)
      • 配置管理(五行)
      • 大盘统计(沧海)
      • 包管理(百仓)
      • tag 服务(画像)
    • 春风(DevOps)
      • 定期巡检(扁鹊)
        • 全量检查-Redis
        • 同步探测-Redis
      • 根因诊断(寻踪)
      • 故障自愈(青囊)
      • 数据分析(庖丁)
      • 应急预案(游刃)
    • 奕秋(服务管理)
    • 纵横(资源管理)
      • 资源管理
    • 其他
      • 数据同步
      • 智能客服
      • 资源定位
      • 工单系统
      • 评论服务
      • feed 流
      • 混沌工程
      • 软件测试
  • project-third
    • grafana
      • ES
    • NoSQL
      • ES
        • Elasticsearch as Database
        • Elasticsearch py
        • FAQ
      • Redis
        • AOF parse
        • RedisShake
        • RDB parse
        • modules
      • Like Redis
        • rosedb
    • kibana
    • zk
  • Project-Other
    • net/web
      • urllib2
    • 他山之石
      • django/tonado/flask/bottle
      • wsgi & asgi
      • pecan
    • 其他杂记
      • serverless
      • 云原生
Powered by GitBook
On this page

Last updated 4 months ago

调试和测试之 debug（调试）

1.debug

1.1 对 handler 进行调试

日常我们主要是编写 handler，如果想要看 handler 运行是否符合预期，可以通过 test_handler.py 进行调试

1.1.1 输出 usage

python test_handler.py

执行 test_handler.py 后会输出当前所有的 handler 及请求参数

1.1.2 debug 某个 handler

实际执行的时候，是彩色输出的

python test_handler.py /apidemo/hello meetbill

1.2 框架相关

1.2.1 查看执行目录

1.2.2 系统调用

通过 strace 可以看访问 ip 等信息

另外当 Linux 命令卡住时，可以通过 strace command 获取系统调用信息

1.2.3 Linux 下多线程查看工具 (pstree、ps、pstack)

  1. pstree 以树结构显示进程

7916 进程共启动了 16 个子线程，加上主线程是 17 个线程

  1. ps -Lf

$ ps -Lf 7916

进程共启动了 17 个线程

  1. pstack 显示每个进程的栈跟踪

1.2.4 查看 Butterfly stack trace

此功能在 handler 中进行实现，代码见 /handler/demo_stackdump

使用

例子 (butterfly/logs/stack.log 内容）

2 gdb

2.1 使用方法

2.1.1 调试进程

查看 C 调用栈

查看 Python 调用栈

2.2 官方文档

https://wiki.python.org/moin/DebuggingWithGdb

2.3 查看 core 保存路径

  • ulimit -c 是查看创建的核心转储的最大大小

    • 0: 不保存

    • unlimited: 不限制大小

  • cat /proc/sys/kernel/core_pattern 生成的文件路径

2.4 查看 core 文件例子

例子

可以看到程序是在停止的时候，在如下部分发生的问题

3 lsof(lists openfiles)

在 Unix 中一切（包括网络套接口）都是文件。lsof 的功能是列出打开文件（lists openfiles）

3.1 使用 -p 查看指定进程 ID 已打开的文件

lsof -p {PID}

3.2 使用 @host:port 显示基于主机与端口的连接

4 mem_top 排查内存泄露问题

将会将将日志打印在 logs/common/common.log

  • refs 回显信息：占用内存最大的 topN 变量中的元素个数

  • bytes 回显信息：占用内存最大的 topN 变量每个变量所占用的 bytes

  • types 回显信息：占用内存最大的 topN 类型变量个数

4.1 例子

5 传送门

5.1 根据 python 代码自动生成流程图

5.2 GDB 教程

5.3 mem_top

– 把你的 Python 和 JavaScript 代码转换为流程图。

- 这个库可以把你的 Python 应用的流程（调用图）进行可视化。

Usage:
test_handler.py /x/hello 'str_info'
test_handler.py /auth/ssologin  'ticket='
test_handler.py /report/log
test_handler.py /auth/verification
test_handler.py /main
test_handler.py /tpl
test_handler.py /apidemo/ping
test_handler.py /apidemo/hello 'str_info'
test_handler.py /x/ping
Source path:... test_handler.py
>>>>>|12:59:38.334264 4524121536-MainThread call        66             def main():
------12:59:38.335328 4524121536-MainThread line        67                 return func(*args)
    Source path:... /Users/meetbill/test/crontab/handlers/apidemo/__init__.py
    Starting var:.. req = <xlib.httpgateway.Request object at 0x10227c7d0>
    Starting var:.. str_info = 'meetbill'
    >>>>>|12:59:38.335414 4524121536-MainThread call        40 def hello(req, str_info):
    ------12:59:38.341516 4524121536-MainThread line        41     isinstance(req, Request)
    ------12:59:38.341742 4524121536-MainThread line        42     return retstat.HTTP_OK, {"stat":"OK","str_info": str_info}, [(__info, __version)]
    |<<<<<12:59:38.341985 4524121536-MainThread return      42     return retstat.HTTP_OK, {"stat":"OK","str_info": str_info}, [(__info, __version)]
    Return value:.. (200, {'stat': 'OK', 'str_info': 'meetbill'}, [('http_demo', '1.0.1')])
Source path:... test_handler.py
|<<<<<12:59:38.342153 4524121536-MainThread return      67                 return func(*args)
Return value:.. (200, {'stat': 'OK', 'str_info': 'meetbill'}, [('http_demo', '1.0.1')])
Elapsed time: 00:00:00.008072
=============================================================
(200, {'stat': 'OK', 'str_info': 'meetbill'}, [('http_demo', '1.0.1')])
=============================================================
ll /proc/${pid}/cwd
$ strace -p 17553
...
poll([{fd=11, events=POLLIN}], 1, 1000) = 0
futex(0x6a1a10, FUTEX_WAKE, 1)          = 0
poll([{fd=11, events=POLLIN, revents=POLLIN}], 1, 1000) = 1
accept(11, {sa_family=AF_INET, sin_port=htons(33054), sin_addr=inet_addr("127.0.0.1")}, [2198038101723447312]) = 13
futex(0x6a1a10, FUTEX_WAKE, 1)          = 0
futex(0x6a1a10, FUTEX_WAKE, 1)          = 0
...
$ pstree -p 7916
python2.7(7916)─┬─{python2.7}(8262)
                ├─{python2.7}(8263)
                ├─{python2.7}(8264)
                ├─{python2.7}(8265)
                ├─{python2.7}(8266)
                ├─{python2.7}(8267)
                ├─{python2.7}(8268)
                ├─{python2.7}(8270)
                ├─{python2.7}(8271)
                ├─{python2.7}(8273)
                ├─{python2.7}(8276)
                ├─{python2.7}(8277)
                ├─{python2.7}(8278)
                ├─{python2.7}(8279)
                ├─{python2.7}(8280)
                └─{python2.7}(8281)
UID        PID  PPID   LWP  C NLWP STIME TTY      STAT   TIME CMD
304804    7916     1  7916  0   17 13:00 pts/18   Sl     0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque
304804    7916     1  8262  0   17 13:00 pts/18   Sl     0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque
304804    7916     1  8263  0   17 13:00 pts/18   Sl     0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque
304804    7916     1  8264  0   17 13:00 pts/18   Sl     0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque
304804    7916     1  8265  0   17 13:00 pts/18   Sl     0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque
304804    7916     1  8266  0   17 13:00 pts/18   Sl     0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque
304804    7916     1  8267  0   17 13:00 pts/18   Sl     0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque
304804    7916     1  8268  0   17 13:00 pts/18   Sl     0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque
304804    7916     1  8270  0   17 13:00 pts/18   Sl     0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque
304804    7916     1  8271  0   17 13:00 pts/18   Sl     0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque
304804    7916     1  8273  0   17 13:00 pts/18   Sl     0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque
304804    7916     1  8276  0   17 13:00 pts/18   Sl     0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque
304804    7916     1  8277  0   17 13:00 pts/18   Sl     0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque
304804    7916     1  8278  0   17 13:00 pts/18   Sl     0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque
304804    7916     1  8279  0   17 13:00 pts/18   Sl     0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque
304804    7916     1  8280  0   17 13:00 pts/18   Sl     0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque
304804    7916     1  8281  0   17 13:00 pts/18   Sl     0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque
$ pstack 7916
Thread 17 (Thread 1107855712 (LWP 8262)):
#0  0x0000003f0b90a65f in sem_wait () from /lib64/tls/libpthread.so.0
#1  0x00007f0d88cdbeb8 in PyThread_acquire_lock ()
#2  0x00007f0d88cdf602 in lock_PyThread_acquire_lock ()
#3  0x00007f0d88cb19cb in PyEval_EvalFrameEx ()
#4  0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#5  0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#6  0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#7  0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#8  0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#9  0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#10 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#11 0x00007f0d88c457e8 in function_call ()
#12 0x00007f0d88c2189d in PyObject_Call ()
#13 0x00007f0d88c30c14 in instancemethod_call ()
#14 0x00007f0d88c2189d in PyObject_Call ()
#15 0x00007f0d88cac3e1 in PyEval_CallObjectWithKeywords ()
#16 0x00007f0d88cdff81 in t_bootstrap ()
#17 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0
#18 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6
#19 0x0000000000000000 in ?? ()
Thread 16 (Thread 1118345568 (LWP 8263)):
#0  0x0000003f0b90a65f in sem_wait () from /lib64/tls/libpthread.so.0
#1  0x00007f0d88cdbeb8 in PyThread_acquire_lock ()
#2  0x00007f0d88cdf602 in lock_PyThread_acquire_lock ()
#3  0x00007f0d88cb19cb in PyEval_EvalFrameEx ()
#4  0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#5  0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#6  0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#7  0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#8  0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#9  0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#10 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#11 0x00007f0d88c457e8 in function_call ()
#12 0x00007f0d88c2189d in PyObject_Call ()
#13 0x00007f0d88c30c14 in instancemethod_call ()
#14 0x00007f0d88c2189d in PyObject_Call ()
#15 0x00007f0d88cac3e1 in PyEval_CallObjectWithKeywords ()
#16 0x00007f0d88cdff81 in t_bootstrap ()
#17 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0
#18 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6
#19 0x0000000000000000 in ?? ()
Thread 15 (Thread 1128835424 (LWP 8264)):
#0  0x0000003f0b90a65f in sem_wait () from /lib64/tls/libpthread.so.0
#1  0x00007f0d88cdbeb8 in PyThread_acquire_lock ()
#2  0x00007f0d88cdf602 in lock_PyThread_acquire_lock ()
#3  0x00007f0d88cb19cb in PyEval_EvalFrameEx ()
#4  0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#5  0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#6  0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#7  0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#8  0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#9  0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#10 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#11 0x00007f0d88c457e8 in function_call ()
#12 0x00007f0d88c2189d in PyObject_Call ()
#13 0x00007f0d88c30c14 in instancemethod_call ()
#14 0x00007f0d88c2189d in PyObject_Call ()
#15 0x00007f0d88cac3e1 in PyEval_CallObjectWithKeywords ()
#16 0x00007f0d88cdff81 in t_bootstrap ()
#17 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0
#18 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6
#19 0x0000000000000000 in ?? ()
Thread 14 (Thread 1139325280 (LWP 8265)):
#0  0x0000003f0b90a65f in sem_wait () from /lib64/tls/libpthread.so.0
#1  0x00007f0d88cdbeb8 in PyThread_acquire_lock ()
#2  0x00007f0d88cdf602 in lock_PyThread_acquire_lock ()
#3  0x00007f0d88cb19cb in PyEval_EvalFrameEx ()
#4  0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#5  0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#6  0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#7  0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#8  0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#9  0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#10 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#11 0x00007f0d88c457e8 in function_call ()
#12 0x00007f0d88c2189d in PyObject_Call ()
#13 0x00007f0d88c30c14 in instancemethod_call ()
#14 0x00007f0d88c2189d in PyObject_Call ()
#15 0x00007f0d88cac3e1 in PyEval_CallObjectWithKeywords ()
#16 0x00007f0d88cdff81 in t_bootstrap ()
#17 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0
#18 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6
#19 0x0000000000000000 in ?? ()
Thread 13 (Thread 1149815136 (LWP 8266)):
#0  0x0000003f0b90a65f in sem_wait () from /lib64/tls/libpthread.so.0
#1  0x00007f0d88cdbeb8 in PyThread_acquire_lock ()
#2  0x00007f0d88cdf602 in lock_PyThread_acquire_lock ()
#3  0x00007f0d88cb19cb in PyEval_EvalFrameEx ()
#4  0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#5  0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#6  0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#7  0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#8  0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#9  0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#10 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#11 0x00007f0d88c457e8 in function_call ()
#12 0x00007f0d88c2189d in PyObject_Call ()
#13 0x00007f0d88c30c14 in instancemethod_call ()
#14 0x00007f0d88c2189d in PyObject_Call ()
#15 0x00007f0d88cac3e1 in PyEval_CallObjectWithKeywords ()
#16 0x00007f0d88cdff81 in t_bootstrap ()
#17 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0
#18 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6
#19 0x0000000000000000 in ?? ()
Thread 12 (Thread 1160304992 (LWP 8267)):
#0  0x0000003f0b90a65f in sem_wait () from /lib64/tls/libpthread.so.0
#1  0x00007f0d88cdbeb8 in PyThread_acquire_lock ()
#2  0x00007f0d88cdf602 in lock_PyThread_acquire_lock ()
#3  0x00007f0d88cb19cb in PyEval_EvalFrameEx ()
#4  0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#5  0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#6  0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#7  0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#8  0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#9  0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#10 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#11 0x00007f0d88c457e8 in function_call ()
#12 0x00007f0d88c2189d in PyObject_Call ()
#13 0x00007f0d88c30c14 in instancemethod_call ()
#14 0x00007f0d88c2189d in PyObject_Call ()
#15 0x00007f0d88cac3e1 in PyEval_CallObjectWithKeywords ()
#16 0x00007f0d88cdff81 in t_bootstrap ()
#17 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0
#18 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6
#19 0x0000000000000000 in ?? ()
Thread 11 (Thread 1170794848 (LWP 8268)):
#0  0x0000003f0b90a65f in sem_wait () from /lib64/tls/libpthread.so.0
#1  0x00007f0d88cdbeb8 in PyThread_acquire_lock ()
#2  0x00007f0d88cdf602 in lock_PyThread_acquire_lock ()
#3  0x00007f0d88cb19cb in PyEval_EvalFrameEx ()
#4  0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#5  0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#6  0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#7  0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#8  0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#9  0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#10 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#11 0x00007f0d88c457e8 in function_call ()
#12 0x00007f0d88c2189d in PyObject_Call ()
#13 0x00007f0d88c30c14 in instancemethod_call ()
#14 0x00007f0d88c2189d in PyObject_Call ()
#15 0x00007f0d88cac3e1 in PyEval_CallObjectWithKeywords ()
#16 0x00007f0d88cdff81 in t_bootstrap ()
#17 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0
#18 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6
#19 0x0000000000000000 in ?? ()
Thread 10 (Thread 1085798752 (LWP 8270)):
#0  0x0000003f0b90a65f in sem_wait () from /lib64/tls/libpthread.so.0
#1  0x00007f0d88cdbeb8 in PyThread_acquire_lock ()
#2  0x00007f0d88cdf602 in lock_PyThread_acquire_lock ()
#3  0x00007f0d88cb19cb in PyEval_EvalFrameEx ()
#4  0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#5  0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#6  0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#7  0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#8  0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#9  0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#10 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#11 0x00007f0d88c457e8 in function_call ()
#12 0x00007f0d88c2189d in PyObject_Call ()
#13 0x00007f0d88c30c14 in instancemethod_call ()
#14 0x00007f0d88c2189d in PyObject_Call ()
#15 0x00007f0d88cac3e1 in PyEval_CallObjectWithKeywords ()
#16 0x00007f0d88cdff81 in t_bootstrap ()
#17 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0
#18 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6
#19 0x0000000000000000 in ?? ()
Thread 9 (Thread 1096288608 (LWP 8271)):
#0  0x0000003f0b90a65f in sem_wait () from /lib64/tls/libpthread.so.0
#1  0x00007f0d88cdbeb8 in PyThread_acquire_lock ()
#2  0x00007f0d88cdf602 in lock_PyThread_acquire_lock ()
#3  0x00007f0d88cb19cb in PyEval_EvalFrameEx ()
#4  0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#5  0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#6  0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#7  0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#8  0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#9  0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#10 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#11 0x00007f0d88c457e8 in function_call ()
#12 0x00007f0d88c2189d in PyObject_Call ()
#13 0x00007f0d88c30c14 in instancemethod_call ()
#14 0x00007f0d88c2189d in PyObject_Call ()
#15 0x00007f0d88cac3e1 in PyEval_CallObjectWithKeywords ()
#16 0x00007f0d88cdff81 in t_bootstrap ()
#17 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0
#18 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6
#19 0x0000000000000000 in ?? ()
Thread 8 (Thread 1181284704 (LWP 8273)):
#0  0x0000003f0b90a65f in sem_wait () from /lib64/tls/libpthread.so.0
#1  0x00007f0d88cdbeb8 in PyThread_acquire_lock ()
#2  0x00007f0d88cdf602 in lock_PyThread_acquire_lock ()
#3  0x00007f0d88cb19cb in PyEval_EvalFrameEx ()
#4  0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#5  0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#6  0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#7  0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#8  0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#9  0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#10 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#11 0x00007f0d88c457e8 in function_call ()
#12 0x00007f0d88c2189d in PyObject_Call ()
#13 0x00007f0d88c30c14 in instancemethod_call ()
#14 0x00007f0d88c2189d in PyObject_Call ()
#15 0x00007f0d88cac3e1 in PyEval_CallObjectWithKeywords ()
#16 0x00007f0d88cdff81 in t_bootstrap ()
#17 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0
#18 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6
#19 0x0000000000000000 in ?? ()
Thread 7 (Thread 1191774560 (LWP 8276)):
#0  0x0000003f0b90a65f in sem_wait () from /lib64/tls/libpthread.so.0
#1  0x00007f0d88cdbeb8 in PyThread_acquire_lock ()
#2  0x00007f0d88cdf602 in lock_PyThread_acquire_lock ()
#3  0x00007f0d88cb19cb in PyEval_EvalFrameEx ()
#4  0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#5  0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#6  0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#7  0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#8  0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#9  0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#10 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#11 0x00007f0d88c457e8 in function_call ()
#12 0x00007f0d88c2189d in PyObject_Call ()
#13 0x00007f0d88c30c14 in instancemethod_call ()
#14 0x00007f0d88c2189d in PyObject_Call ()
#15 0x00007f0d88cac3e1 in PyEval_CallObjectWithKeywords ()
#16 0x00007f0d88cdff81 in t_bootstrap ()
#17 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0
#18 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6
#19 0x0000000000000000 in ?? ()
Thread 6 (Thread 1202264416 (LWP 8277)):
#0  0x0000003f0b90a65f in sem_wait () from /lib64/tls/libpthread.so.0
#1  0x00007f0d88cdbeb8 in PyThread_acquire_lock ()
#2  0x00007f0d88cdf602 in lock_PyThread_acquire_lock ()
#3  0x00007f0d88cb19cb in PyEval_EvalFrameEx ()
#4  0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#5  0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#6  0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#7  0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#8  0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#9  0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#10 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#11 0x00007f0d88c457e8 in function_call ()
#12 0x00007f0d88c2189d in PyObject_Call ()
#13 0x00007f0d88c30c14 in instancemethod_call ()
#14 0x00007f0d88c2189d in PyObject_Call ()
#15 0x00007f0d88cac3e1 in PyEval_CallObjectWithKeywords ()
#16 0x00007f0d88cdff81 in t_bootstrap ()
#17 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0
#18 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6
#19 0x0000000000000000 in ?? ()
Thread 5 (Thread 1212754272 (LWP 8278)):
#0  0x0000003f0b90a65f in sem_wait () from /lib64/tls/libpthread.so.0
#1  0x00007f0d88cdbeb8 in PyThread_acquire_lock ()
#2  0x00007f0d88cdf602 in lock_PyThread_acquire_lock ()
#3  0x00007f0d88cb19cb in PyEval_EvalFrameEx ()
#4  0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#5  0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#6  0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#7  0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#8  0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#9  0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#10 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#11 0x00007f0d88c457e8 in function_call ()
#12 0x00007f0d88c2189d in PyObject_Call ()
#13 0x00007f0d88c30c14 in instancemethod_call ()
#14 0x00007f0d88c2189d in PyObject_Call ()
#15 0x00007f0d88cac3e1 in PyEval_CallObjectWithKeywords ()
#16 0x00007f0d88cdff81 in t_bootstrap ()
#17 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0
#18 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6
#19 0x0000000000000000 in ?? ()
Thread 4 (Thread 1223244128 (LWP 8279)):
#0  0x0000003f0b90a65f in sem_wait () from /lib64/tls/libpthread.so.0
#1  0x00007f0d88cdbeb8 in PyThread_acquire_lock ()
#2  0x00007f0d88cdf602 in lock_PyThread_acquire_lock ()
#3  0x00007f0d88cb19cb in PyEval_EvalFrameEx ()
#4  0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#5  0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#6  0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#7  0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#8  0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#9  0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#10 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#11 0x00007f0d88c457e8 in function_call ()
#12 0x00007f0d88c2189d in PyObject_Call ()
#13 0x00007f0d88c30c14 in instancemethod_call ()
#14 0x00007f0d88c2189d in PyObject_Call ()
#15 0x00007f0d88cac3e1 in PyEval_CallObjectWithKeywords ()
#16 0x00007f0d88cdff81 in t_bootstrap ()
#17 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0
#18 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6
#19 0x0000000000000000 in ?? ()
Thread 3 (Thread 1233733984 (LWP 8280)):
#0  0x0000003f0b90a65f in sem_wait () from /lib64/tls/libpthread.so.0
#1  0x00007f0d88cdbeb8 in PyThread_acquire_lock ()
#2  0x00007f0d88cdf602 in lock_PyThread_acquire_lock ()
#3  0x00007f0d88cb19cb in PyEval_EvalFrameEx ()
#4  0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#5  0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#6  0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#7  0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#8  0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#9  0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#10 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#11 0x00007f0d88c457e8 in function_call ()
#12 0x00007f0d88c2189d in PyObject_Call ()
#13 0x00007f0d88c30c14 in instancemethod_call ()
#14 0x00007f0d88c2189d in PyObject_Call ()
#15 0x00007f0d88cac3e1 in PyEval_CallObjectWithKeywords ()
#16 0x00007f0d88cdff81 in t_bootstrap ()
#17 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0
#18 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6
#19 0x0000000000000000 in ?? ()
Thread 2 (Thread 1244223840 (LWP 8281)):
#0  0x0000003f0b90a65f in sem_wait () from /lib64/tls/libpthread.so.0
#1  0x00007f0d88cdbeb8 in PyThread_acquire_lock ()
#2  0x00007f0d88cdf602 in lock_PyThread_acquire_lock ()
#3  0x00007f0d88cb19cb in PyEval_EvalFrameEx ()
#4  0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#5  0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#6  0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#7  0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#8  0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#9  0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#10 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#11 0x00007f0d88c457e8 in function_call ()
#12 0x00007f0d88c2189d in PyObject_Call ()
#13 0x00007f0d88c30c14 in instancemethod_call ()
#14 0x00007f0d88c2189d in PyObject_Call ()
#15 0x00007f0d88cac3e1 in PyEval_CallObjectWithKeywords ()
#16 0x00007f0d88cdff81 in t_bootstrap ()
#17 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0
#18 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6
#19 0x0000000000000000 in ?? ()
Thread 1 (Thread 139696105322208 (LWP 7916)):================================》主线程
#0  0x0000003f0b0bd0c2 in poll () from /lib64/tls/libc.so.6
#1  0x00007f0d853c5730 in internal_select ()
#2  0x00007f0d853c61ca in sock_accept ()
#3  0x00007f0d88cafa21 in PyEval_EvalFrameEx ()
#4  0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#5  0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#6  0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#7  0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#8  0x00007f0d88cb3822 in PyEval_EvalCode ()
#9  0x00007f0d88ccc49c in run_mod ()
#10 0x00007f0d88cccafc in PyRun_FileExFlags ()
#11 0x00007f0d88ccda11 in PyRun_SimpleFileExFlags ()
#12 0x00007f0d88cddd26 in Py_Main ()
#13 0x0000003f0b01c4bb in __libc_start_main () from /lib64/tls/libc.so.6
#14 0x00000000004006da in _start ()
#15 0x00007fff62a773d8 in ?? ()
#16 0x000000000000001c in ?? ()
#17 0x0000000000000003 in ?? ()
#18 0x00007fff62a776db in ?? ()
#19 0x00007fff62a776e5 in ?? ()
#20 0x00007fff62a7771e in ?? ()
#21 0x0000000000000000 in ?? ()
$ curl "http://127.0.0.1:8585/demo_stackdump/stackdump"

stack trace 日志记录在 logs/stack.log
----------------------------------------------------------------
dump_time=2021-04-09 12:19:55 reqid=2A5BE75641F1BC5A
1 ActiveThreadCount: 18
2 KnownActiveThreadNames:
  MainThread-------------------------------- 主线程
  APScheduler------------------------------- 任务调度线程
  CP Server Thread-1------------------------Butterfly handler 处理线程，以下类似
  CP Server Thread-11
  CP Server Thread-5
  CP Server Thread-9
  CP Server Thread-14
  CP Server Thread-7
  CP Server Thread-13
  CP Server Thread-15
  CP Server Thread-3
  CP Server Thread-2
  CP Server Thread-4
  CP Server Thread-6
  CP Server Thread-12
  CP Server Thread-8
  CP Server Thread-10
  CP Server Thread-16

3 ThreadInfo:
Thread: 1082784096 CP Server Thread-16
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap
  self.__bootstrap_inner()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner
  self.run()
File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1643, in run
  conn = self.server.requests.get()
File: "/home/users/meetbill/.jumbo/lib/python2.7/Queue.py", line 168, in get
  self.not_empty.wait()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 243, in wait
  waiter.acquire()

Thread: 1093196128 CP Server Thread-15
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap
  self.__bootstrap_inner()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner
  self.run()
File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1643, in run
  conn = self.server.requests.get()
File: "/home/users/meetbill/.jumbo/lib/python2.7/Queue.py", line 168, in get
  self.not_empty.wait()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 243, in wait
  waiter.acquire()

Thread: 1089005920 CP Server Thread-3
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap
  self.__bootstrap_inner()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner
  self.run()
File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1643, in run
  conn = self.server.requests.get()
File: "/home/users/meetbill/.jumbo/lib/python2.7/Queue.py", line 168, in get
  self.not_empty.wait()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 243, in wait
  waiter.acquire()

Thread: 1076640096 CP Server Thread-13
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap
  self.__bootstrap_inner()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner
  self.run()
File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1643, in run
  conn = self.server.requests.get()
File: "/home/users/meetbill/.jumbo/lib/python2.7/Queue.py", line 168, in get
  self.not_empty.wait()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 243, in wait
  waiter.acquire()

Thread: 1103497568 CP Server Thread-6
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap
  self.__bootstrap_inner()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner
  self.run()
File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1643, in run
  conn = self.server.requests.get()
File: "/home/users/meetbill/.jumbo/lib/python2.7/Queue.py", line 168, in get
  self.not_empty.wait()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 243, in wait
  waiter.acquire()

Thread: 1094404448 CP Server Thread-5
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap
  self.__bootstrap_inner()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner
  self.run()
File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1643, in run
  conn = self.server.requests.get()
File: "/home/users/meetbill/.jumbo/lib/python2.7/Queue.py", line 168, in get
  self.not_empty.wait()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 243, in wait
  waiter.acquire()

Thread: 1100007776 CP Server Thread-2
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap
  self.__bootstrap_inner()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner
  self.run()
File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1643, in run
  conn = self.server.requests.get()
File: "/home/users/meetbill/.jumbo/lib/python2.7/Queue.py", line 168, in get
  self.not_empty.wait()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 243, in wait
  waiter.acquire()

Thread: 140004998178528 MainThread
File: "/home/users/meetbill/butterfly/main.py", line 30, in <module>
  server.start()
File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 2104, in start
  self.tick()
File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 2158, in tick
  s, addr = self.socket.accept()
File: "/home/users/meetbill/.jumbo/lib/python2.7/socket.py", line 202, in accept
  sock, addr = self._sock.accept()

Thread: 1102469472 CP Server Thread-7
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap
  self.__bootstrap_inner()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner
  self.run()
File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1643, in run
  conn = self.server.requests.get()
File: "/home/users/meetbill/.jumbo/lib/python2.7/Queue.py", line 168, in get
  self.not_empty.wait()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 243, in wait
  waiter.acquire()

Thread: 1095801184 CP Server Thread-8
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap
  self.__bootstrap_inner()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner
  self.run()
File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1643, in run
  conn = self.server.requests.get()
File: "/home/users/meetbill/.jumbo/lib/python2.7/Queue.py", line 168, in get
  self.not_empty.wait()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 243, in wait
  waiter.acquire()

Thread: 1100343648 CP Server Thread-11
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap
  self.__bootstrap_inner()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner
  self.run()
File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1643, in run
  conn = self.server.requests.get()
File: "/home/users/meetbill/.jumbo/lib/python2.7/Queue.py", line 168, in get
  self.not_empty.wait()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 243, in wait
  waiter.acquire()

Thread: 1075272032 CP Server Thread-14
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap
  self.__bootstrap_inner()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner
  self.run()
File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1643, in run
  conn = self.server.requests.get()
File: "/home/users/meetbill/.jumbo/lib/python2.7/Queue.py", line 168, in get
  self.not_empty.wait()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 243, in wait
  waiter.acquire()

Thread: 1100863840 APScheduler
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap
  self.__bootstrap_inner()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner
  self.run()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 504, in run
  self.__target(*self.__args, **self.__kwargs)
File: "/home/users/meetbill/butterfly/xlib/apscheduler/schedulers/blocking.py", line 42, in _main_loop
  self._event.wait(wait_seconds)
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 403, in wait
  self.__cond.wait(timeout)
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 262, in wait
  _sleep(delay)

Thread: 1084520800 CP Server Thread-4
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap
  self.__bootstrap_inner()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner
  self.run()
File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1643, in run
  conn = self.server.requests.get()
File: "/home/users/meetbill/.jumbo/lib/python2.7/Queue.py", line 168, in get
  self.not_empty.wait()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 243, in wait
  waiter.acquire()

Thread: 1097337184 CP Server Thread-9
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap
  self.__bootstrap_inner()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner
  self.run()
File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1643, in run
  conn = self.server.requests.get()
File: "/home/users/meetbill/.jumbo/lib/python2.7/Queue.py", line 168, in get
  self.not_empty.wait()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 243, in wait
  waiter.acquire()

Thread: 1091410272 CP Server Thread-12
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap
  self.__bootstrap_inner()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner
  self.run()
File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1643, in run
  conn = self.server.requests.get()
File: "/home/users/meetbill/.jumbo/lib/python2.7/Queue.py", line 168, in get
  self.not_empty.wait()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 243, in wait
  waiter.acquire()

Thread: 1087252832 CP Server Thread-10
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap
  self.__bootstrap_inner()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner
  self.run()
File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1643, in run
  conn = self.server.requests.get()
File: "/home/users/meetbill/.jumbo/lib/python2.7/Queue.py", line 168, in get
  self.not_empty.wait()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 243, in wait
  waiter.acquire()

Thread: 1099741536 CP Server Thread-1
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap
  self.__bootstrap_inner()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner
  self.run()
File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1651, in run
  conn.communicate()
File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1460, in communicate
  req.respond()
File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 901, in respond
  self.server.gateway(self).respond()
File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 2426, in respond
  response = self.req.server.wsgi_app(self.env, self.start_response)
File: "/home/users/meetbill/butterfly/wsgiapp.py", line 161, in application
  status, headders, content = wsgigw.process(environ)
File: "/home/users/meetbill/butterfly/xlib/httpgateway.py", line 195, in process
  httpstatus, headers, content = protocol(req)
File: "/home/users/meetbill/butterfly/xlib/protocol_json.py", line 171, in __call__
  ret = self._func(**params)
File: "/home/users/meetbill/butterfly/handlers/demo_stackdump/__init__.py", line 79, in stackdump
  for filename, lineno, name, line in traceback.extract_stack(stack):
gdb python <pid of running process>
(gdb) bt
(gdb) py-bt
# ulimit -c
unlimited

# cat /proc/sys/kernel/core_pattern
/home/work/core.%e.%p.%t
#gdb `which python ` core_path
# gdb `which python ` core.python2.7.25133.1617075053
...
gdb info
...
[New LWP 25192]
[New LWP 25193]
[New LWP 25195]
[New LWP 25196]
[New LWP 25197]
[New LWP 25194]
[New LWP 25198]
[New LWP 25199]
[New LWP 25201]
[New LWP 25200]
[New LWP 25202]
[New LWP 25204]
[New LWP 25206]
[New LWP 25203]
[New LWP 25205]
[New LWP 25207]
[New LWP 25191]
[New LWP 25208]
[New LWP 25133]

warning: File "/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
To enable execution of this file add
	add-auto-load-safe-path /lib64/libthread_db-1.0.so
line to your configuration file "/home/work/.gdbinit".
To completely disable this security protection add
	set auto-load safe-path /
line to your configuration file "/home/work/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
	info "(gdb)Auto-loading safe path"

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.

warning: File "/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Core was generated by `python2.7 /home/work/chunfeng/chunfeng_app2_4600/main.py 954216e9'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f85e85265c5 in PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2995
2995	Python/ceval.c: No such file or directory.
[Current thread is 1 (LWP 25192)]
(gdb)
#0  0x00007f85e85265c5 in PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2995
可以输入 bt 查看更多，其从上到下是错误从底层到最外层的顺序。
COMMAND ：进程的名称
PID     ：进程标识符
USER    ：进程所有者
FD      ：文件描述符，应用程序通过文件描述符识别该文件。如 cwd、txt 等
        (1) cwd 值表示应用程序的当前工作目录，这是该应用程序启动的目录，除非它本身对这个目录进行更改
        (2) txt 类型的文件是程序代码，如应用程序二进制文件本身或共享库，其次数值表示应用程序的文件描述符，这是打开该文件时返回的一个整数。
        (3) u 表示该文件被打开并处于读取 / 写入模式，而不是只读 r 或只写 (w) 模式。同时还有大写 的 W 表示该应用程序具有对整个文件的写锁。
            该文件描述符用于确保每次只能打开一个应用程序实例。
            初始打开每个应用程序时，都具有三个文件描述符，从 0 到 2，分别表示标准输入、输出和错误流。
            所以大多数应用程序所打开的文件的 FD 都是从 3 开始。
TYPE    ：文件类型，如 DIR、REG 等
        (1) 文件和目录分别称为 REG 和 DIR
        (2) CHR 表示字符
        (3) BLK 表示块设备
        (4) UNIX -- UNIX 域套接字
            FIFO -- 先进先出 (FIFO) 队列
            IPv4 -- 网际协议 (IP) 套接字
DEVICE  ：指定磁盘的名称
SIZE    ：文件的大小
NODE    ：索引节点（文件在磁盘上的标识）
NAME    ：打开文件的确切名称
# lsof -i@xx.xx.xx.xx:22
kill -12 <pid>
refs:
49452	<type 'dict'> {'xlib.util.pluginbase._internalspace._spef7874fe95975333c5ff649d4c981dca.vm_manager': <module 'xlib.util.pluginbase._internalspace._spef7874fe95975333c5ff649d4c981dca.vm_manager' from '/home/work/chu
5928	<type 'dict'> {'_sp349460523b2000c7ebcfd35e0025e291': <module 'xlib.util.pluginbase._internalspace._sp349460523b2000c7ebcfd35e0025e291' (built-in)>, '_spdf494a60169fd9552a956256d05ccc00': <module 'xlib.util.pluginb
954	<type 'dict'> {'TOO_MANY_KEY_PARTS': 1070, 'WRONG_PARAMCOUNT_TO_PROCEDURE': 1107, 'SP_COND_MISMATCH': 1319, 'XAER_OUTSIDE': 1400, 'KILL_DENIED_ERROR': 1095, 'TABLE_CANT_HANDLE_SPKEYS': 1464, 'DUMP_NOT_IMPLEMENTED':
570	<type 'dict'> {'SocketType': <class 'socket._socketobject'>, 'getaddrinfo': <built-in function getaddrinfo>, 'AI_NUMERICSERV': 1024, 'PACKET_OTHERHOST': 3, 'AI_NUMERICHOST': 4, 'StringIO': <built-in function String
520	<type 'dict'> {'DateField': <class 'xlib.db.peewee.DateField'>, 'ModelNamedTupleCursorWrapper': <class 'xlib.db.peewee.ModelNamedTupleCursorWrapper'>, 'ColumnMetadata': <class 'xlib.db.peewee.ColumnMetadata'>, 'Bit
496	<type 'dict'> {'restore': <function restore at 0x299cde8>, 'zrangebylex': <function zrangebylex at 0x299f7d0>, 'renamenx': <function renamenx at 0x299cd70>, 'client_id': <function client_id at 0x29999b0>, 'acl_save
490	<type 'dict'> {'SocketType': <type '_socket.socket'>, 'getaddrinfo': <built-in function getaddrinfo>, 'IPPROTO_RAW': 255, 'AI_NUMERICSERV': 1024, 'PACKET_OTHERHOST': 3, 'AI_NUMERICHOST': 4, 'IPPORT_RESERVED': 1024,
490	<type 'dict'> {'SocketType': <type '_socket.socket'>, 'getaddrinfo': <built-in function getaddrinfo>, 'AI_NUMERICSERV': 1024, 'PACKET_OTHERHOST': 3, 'AI_NUMERICHOST': 4, 'IPPORT_RESERVED': 1024, 'AF_IRDA': 23, 'MSG
462	<type 'dict'> {'tcsendbreak': <built-in function tcsendbreak>, 'B110': 3, 'tcdrain': <built-in function tcdrain>, 'CIBAUD': 269418496, 'FF1': 32768, 'FF0': 0, 'FIONCLEX': 21584, 'ONLCR': 4, 'B600': 8, 'TCSANOW': 0,
462	<type 'dict'> {'tcsendbreak': <built-in function tcsendbreak>, 'B110': 3, 'tcdrain': <built-in function tcdrain>, 'TCIOFLUSH': 2, 'FF1': 32768, 'FF0': 0, 'FIONCLEX': 21584, 'ONLCR': 4, 'B600': 8, 'TCSANOW': 0, 'TIO
448	<type 'dict'> {'WTERMSIG': <built-in function WTERMSIG>, 'lseek': <built-in function lseek>, 'EX_IOERR': 74, 'EX_NOHOST': 68, 'seteuid': <built-in function seteuid>, 'pathsep': ':', 'execle': <function execle at 0x
412	<type 'list'> [<weakref at 0x7fa50428c050; to 'type' at 0x7fa5046695e0 (type)>, <weakref at 0x7fa50428c0a8; to 'type' at 0x7fa50466b700 (weakref)>, <weakref at 0x7fa50428c100; to 'type' at 0x7fa50466ba40 (weakcalla
364	<type 'dict'> {'empty_provider': <pkg_resources.EmptyProvider instance at 0x34a23b0>, 'ExtractionError': <class 'pkg_resources.ExtractionError'>, 'functools': <module 'functools' from '/home/work/.jumbo/lib/python2
359	<class 'xlib.db.redis.client.CaseInsensitiveDict'> {u'UNWATCH': <function bool_ok at 0x2995848>, u'SLAVEOF': <function bool_ok at 0x2995848>, u'OBJECT': <function parse_object at 0x2994c80>, u'GEODIST': <function float_or_none at 0x29957d0>, u'EXPIRE'
359	<class 'xlib.db.redis.client.CaseInsensitiveDict'> {u'UNWATCH': <function bool_ok at 0x2995848>, u'SLAVEOF': <function bool_ok at 0x2995848>, u'OBJECT': <function parse_object at 0x2994c80>, u'GEODIST': <function float_or_none at 0x29957d0>, u'EXPIRE'
358	<type 'dict'> {u'UNWATCH': <function bool_ok at 0x2995848>, u'SLAVEOF': <function bool_ok at 0x2995848>, u'MOVE': <type 'bool'>, u'GEODIST': <function float_or_none at 0x29957d0>, u'TIME': <function <lambda> at 0x2
338	<type 'dict'> {'WTERMSIG': <built-in function WTERMSIG>, 'lseek': <built-in function lseek>, 'EX_IOERR': 74, 'EX_NOHOST': 68, 'seteuid': <built-in function seteuid>, 'O_RSYNC': 1052672, 'fstatvfs': <built-in functi
338	<type 'dict'> {'WTERMSIG': <built-in function WTERMSIG>, 'lseek': <built-in function lseek>, 'EX_IOERR': 74, 'EX_NOHOST': 68, 'getpgrp': <built-in function getpgrp>, 'major': <built-in function major>, 'fstatvfs':
286	<type 'dict'> {'bytearray': <type 'bytearray'>, 'IndexError': <type 'exceptions.IndexError'>, 'all': <built-in function all>, 'help': Type help() for interactive help, or help(object) for help about object., 'vars'
286	<type 'dict'> {1: Charset(id=1, name='big5', collation='big5_chinese_ci'), 2: Charset(id=2, name='latin2', collation='latin2_czech_cs'), 3: Charset(id=3, name='dec8', collation='dec8_swedish_ci'), 4: Charset(id=4,

bytes:
3146008	 {'xlib.util.pluginbase._internalspace._spef7874fe95975333c5ff649d4c981dca.vm_manager': <module 'xlib.util.pluginbase._internalspace._spef7874fe95975333c5ff649d4c981dca.vm_manager' from '/home/work/chu
196888	 {'_sp349460523b2000c7ebcfd35e0025e291': <module 'xlib.util.pluginbase._internalspace._sp349460523b2000c7ebcfd35e0025e291' (built-in)>, '_spdf494a60169fd9552a956256d05ccc00': <module 'xlib.util.pluginb
49432	 {'TOO_MANY_KEY_PARTS': 1070, 'WRONG_PARAMCOUNT_TO_PROCEDURE': 1107, 'SP_COND_MISMATCH': 1319, 'XAER_OUTSIDE': 1400, 'KILL_DENIED_ERROR': 1095, 'TABLE_CANT_HANDLE_SPKEYS': 1464, 'DUMP_NOT_IMPLEMENTED':
12568	 {'bytearray': <type 'bytearray'>, 'IndexError': <type 'exceptions.IndexError'>, 'all': <built-in function all>, 'help': Type help() for interactive help, or help(object) for help about object., 'vars'
12568	 {'bytearray': <type 'bytearray'>, 'all': <built-in function all>, 'vars': <built-in function vars>, 'SyntaxError': <type 'exceptions.SyntaxError'>, 'unicode': <type 'unicode'>, 'sorted': <built-in fun
12568	 {'WTERMSIG': <built-in function WTERMSIG>, 'lseek': <built-in function lseek>, 'EX_IOERR': 74, 'EX_NOHOST': 68, 'seteuid': <built-in function seteuid>, 'pathsep': ':', 'execle': <function execle at 0x
12568	 {'EISDIR': 21, 'EPROTONOSUPPORT': 93, 'ETXTBSY': 26, 'ETOOMANYREFS': 109, 'EDEADLOCK': 35, 'ETIME': 62, 'ESOCKTNOSUPPORT': 94, 'ENOTTY': 25, 'ENAMETOOLONG': 36, 'ENETRESET': 102, 'EMULTIHOP': 72, 'EMS
12568	 {'WTERMSIG': <built-in function WTERMSIG>, 'lseek': <built-in function lseek>, 'EX_IOERR': 74, 'EX_NOHOST': 68, 'seteuid': <built-in function seteuid>, 'O_RSYNC': 1052672, 'fstatvfs': <built-in functi
12568	 {'WTERMSIG': <built-in function WTERMSIG>, 'lseek': <built-in function lseek>, 'EX_IOERR': 74, 'EX_NOHOST': 68, 'getpgrp': <built-in function getpgrp>, 'major': <built-in function major>, 'fstatvfs':
12568	 {'isstring': <function isstring at 0x7fa504193aa0>, '_optimize_unicode': <function _optimize_unicode at 0x7fa504193938>, 'SRE_FLAG_VERBOSE': 64, '_identityfunction': <function _identityfunction at 0x7
12568	 {'_PATTERNENDERS': set([')', '|']), 'REPEAT_ONE': 'repeat_one', 'DIGITS': set(['1', '0', '3', '2', '5', '4', '7', '6', '9', '8']), 'makedict': <function makedict at 0x7fa504200758>, 'CATEGORY_LOC_NOT_
12568	 {'REPEAT_ONE': 'repeat_one', 'makedict': <function makedict at 0x7fa504200758>, 'CATEGORY_LOC_NOT_WORD': 'category_loc_not_word', 'SRE_INFO_CHARSET': 4, 'SRE_INFO_PREFIX': 1, 'SRE_FLAG_DEBUG': 128, 'M
12568	 {'latin_1_encode': <built-in function latin_1_encode>, 'getreader': <function getreader at 0x7fa5041bb0c8>, 'readbuffer_encode': <built-in function readbuffer_encode>, 'BOM': '\xff\xfe', 'StreamWriter
12568	 {'functools': <module 'functools' from '/home/work/.jumbo/lib/python2.7/functools.pyc'>, 'PY3': False, 'ModuleSixMovesUrllibRobotparser': <class 'xlib.util.six.ModuleSixMovesUrllibRobotparser'>, 'get_
12568	 {'gt': <built-in function gt>, '__concat__': <built-in function __concat__>, 'imul': <built-in function imul>, 'ge': <built-in function ge>, '__irshift__': <built-in function __irshift__>, 'setslice':
12568	 {'Union': <type '_ctypes.Union'>, 'c_wchar': <class 'ctypes.c_wchar'>, 'c_bool': <class 'ctypes.c_bool'>, 'c_double': <class 'ctypes.c_double'>, 'CFUNCTYPE': <function CFUNCTYPE at 0x7fa4fe2ef0c8>, '_
12568	 {'logMultiprocessing': 1, '__path__': ['/home/work/.jumbo/lib/python2.7/logging'], 'LogRecord': <class 'logging.LogRecord'>, 'logProcesses': 1, 'addLevelName': <function addLevelName at 0x227fde8>, '_
12568	 {'BadStatusLine': <class 'httplib.BadStatusLine'>, 'UnknownProtocol': <class 'httplib.UnknownProtocol'>, '_UNKNOWN': 'UNKNOWN', 'SWITCHING_PROTOCOLS': 101, 'urlsplit': <function urlsplit at 0x2328758>
12568	 {'SocketType': <class 'socket._socketobject'>, 'getaddrinfo': <built-in function getaddrinfo>, 'AI_NUMERICSERV': 1024, 'PACKET_OTHERHOST': 3, 'AI_NUMERICHOST': 4, 'StringIO': <built-in function String
12568	 {'SocketType': <type '_socket.socket'>, 'getaddrinfo': <built-in function getaddrinfo>, 'IPPROTO_RAW': 255, 'AI_NUMERICSERV': 1024, 'PACKET_OTHERHOST': 3, 'AI_NUMERICHOST': 4, 'IPPORT_RESERVED': 1024,

types:
44136	 <type 'dict'>
30965	 <type 'list'>
21183	 <type 'module'>
11067	 <type 'function'>
5348	 <type 'tuple'>
3270	 <type 'cell'>
2961	 <class 'xlib.util.pluginbase.PluginSource'>
2961	 <class 'xlib.util.pluginbase._PluginSourceModule'>
2961	 <class 'xlib.util.pluginbase.PluginBaseState'>
1694	 <type 'weakref'>
1234	 <type 'wrapper_descriptor'>
1108	 <type 'builtin_function_or_method'>
905	 <type 'type'>
842	 <type 'getset_descriptor'>
711	 <type 'method_descriptor'>
449	 <type 'set'>
427	 <type 'property'>
381	 <type 'member_descriptor'>
198	 <class '_weakrefset.WeakSet'>
168	 <type 'frame'>
code2flow
pycallgraph
Low-level Python debugging with GDB
https://github.com/denis-ryzhkov/mem_top
  • 调试和测试之 debug（调试）
  • 1.debug
  • 1.1 对 handler 进行调试
  • 1.2 框架相关
  • 2 gdb
  • 2.1 使用方法
  • 2.2 官方文档
  • 2.3 查看 core 保存路径
  • 2.4 查看 core 文件例子
  • 3 lsof(lists openfiles)
  • 3.1 使用 -p 查看指定进程 ID 已打开的文件
  • 3.2 使用 @host:port 显示基于主机与端口的连接
  • 4 mem_top 排查内存泄露问题
  • 4.1 例子
  • 5 传送门
  • 5.1 根据 python 代码自动生成流程图
  • 5.2 GDB 教程
  • 5.3 mem_top
Previous数据访问层之资源访问 RAL（百度特有协议支持）Next调试和测试之单元测试
  1. project-framework
  2. 三 how

调试和测试之调试