调试和测试之调试
调试和测试之 debug（调试）
1.debug
1.1 对 handler 进行调试
日常我们主要是编写 handler，如果想要看 handler 运行是否符合预期，可以通过 test_handler.py 进行调试
1.1.1 输出 usage
python test_handler.py
Usage:
test_handler.py /x/hello 'str_info'
test_handler.py /auth/ssologin 'ticket='
test_handler.py /report/log
test_handler.py /auth/verification
test_handler.py /main
test_handler.py /tpl
test_handler.py /apidemo/ping
test_handler.py /apidemo/hello 'str_info'
test_handler.py /x/ping执行 test_handler.py 后会输出当前所有的 handler 及请求参数
1.1.2 debug 某个 handler
实际执行的时候，是彩色输出的
python test_handler.py /apidemo/hello meetbill
Source path:... test_handler.py
>>>>>|12:59:38.334264 4524121536-MainThread call 66 def main():
------12:59:38.335328 4524121536-MainThread line 67 return func(*args)
Source path:... /Users/meetbill/test/crontab/handlers/apidemo/__init__.py
Starting var:.. req = <xlib.httpgateway.Request object at 0x10227c7d0>
Starting var:.. str_info = 'meetbill'
>>>>>|12:59:38.335414 4524121536-MainThread call 40 def hello(req, str_info):
------12:59:38.341516 4524121536-MainThread line 41 isinstance(req, Request)
------12:59:38.341742 4524121536-MainThread line 42 return retstat.HTTP_OK, {"stat":"OK","str_info": str_info}, [(__info, __version)]
|<<<<<12:59:38.341985 4524121536-MainThread return 42 return retstat.HTTP_OK, {"stat":"OK","str_info": str_info}, [(__info, __version)]
Return value:.. (200, {'stat': 'OK', 'str_info': 'meetbill'}, [('http_demo', '1.0.1')])
Source path:... test_handler.py
|<<<<<12:59:38.342153 4524121536-MainThread return 67 return func(*args)
Return value:.. (200, {'stat': 'OK', 'str_info': 'meetbill'}, [('http_demo', '1.0.1')])
Elapsed time: 00:00:00.008072
=============================================================
(200, {'stat': 'OK', 'str_info': 'meetbill'}, [('http_demo', '1.0.1')])
=============================================================1.2 框架相关
1.2.1 查看执行目录
ll /proc/${pid}/cwd1.2.2 系统调用
$ strace -p 17553
...
poll([{fd=11, events=POLLIN}], 1, 1000) = 0
futex(0x6a1a10, FUTEX_WAKE, 1) = 0
poll([{fd=11, events=POLLIN, revents=POLLIN}], 1, 1000) = 1
accept(11, {sa_family=AF_INET, sin_port=htons(33054), sin_addr=inet_addr("127.0.0.1")}, [2198038101723447312]) = 13
futex(0x6a1a10, FUTEX_WAKE, 1) = 0
futex(0x6a1a10, FUTEX_WAKE, 1) = 0
...通过 strace 可以看访问 ip 等信息
另外当 Linux 命令卡住时，可以通过 strace command 获取系统调用信息
1.2.3 Linux 下多线程查看工具 (pstree、ps、pstack)
pstree 以树结构显示进程
$ pstree -p 7916
python2.7(7916)─┬─{python2.7}(8262)
├─{python2.7}(8263)
├─{python2.7}(8264)
├─{python2.7}(8265)
├─{python2.7}(8266)
├─{python2.7}(8267)
├─{python2.7}(8268)
├─{python2.7}(8270)
├─{python2.7}(8271)
├─{python2.7}(8273)
├─{python2.7}(8276)
├─{python2.7}(8277)
├─{python2.7}(8278)
├─{python2.7}(8279)
├─{python2.7}(8280)
└─{python2.7}(8281)7916 进程共启动了 16 个子线程，加上主线程是 17 个线程
ps -Lf
$ ps -Lf 7916
UID PID PPID LWP C NLWP STIME TTY STAT TIME CMD
304804 7916 1 7916 0 17 13:00 pts/18 Sl 0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque
304804 7916 1 8262 0 17 13:00 pts/18 Sl 0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque
304804 7916 1 8263 0 17 13:00 pts/18 Sl 0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque
304804 7916 1 8264 0 17 13:00 pts/18 Sl 0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque
304804 7916 1 8265 0 17 13:00 pts/18 Sl 0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque
304804 7916 1 8266 0 17 13:00 pts/18 Sl 0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque
304804 7916 1 8267 0 17 13:00 pts/18 Sl 0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque
304804 7916 1 8268 0 17 13:00 pts/18 Sl 0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque
304804 7916 1 8270 0 17 13:00 pts/18 Sl 0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque
304804 7916 1 8271 0 17 13:00 pts/18 Sl 0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque
304804 7916 1 8273 0 17 13:00 pts/18 Sl 0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque
304804 7916 1 8276 0 17 13:00 pts/18 Sl 0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque
304804 7916 1 8277 0 17 13:00 pts/18 Sl 0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque
304804 7916 1 8278 0 17 13:00 pts/18 Sl 0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque
304804 7916 1 8279 0 17 13:00 pts/18 Sl 0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque
304804 7916 1 8280 0 17 13:00 pts/18 Sl 0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque
304804 7916 1 8281 0 17 13:00 pts/18 Sl 0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque进程共启动了 17 个线程
pstack 显示每个进程的栈跟踪
$ pstack 7916
Thread 17 (Thread 1107855712 (LWP 8262)):
#0 0x0000003f0b90a65f in sem_wait () from /lib64/tls/libpthread.so.0
#1 0x00007f0d88cdbeb8 in PyThread_acquire_lock ()
#2 0x00007f0d88cdf602 in lock_PyThread_acquire_lock ()
#3 0x00007f0d88cb19cb in PyEval_EvalFrameEx ()
#4 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#5 0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#6 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#7 0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#8 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#9 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#10 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#11 0x00007f0d88c457e8 in function_call ()
#12 0x00007f0d88c2189d in PyObject_Call ()
#13 0x00007f0d88c30c14 in instancemethod_call ()
#14 0x00007f0d88c2189d in PyObject_Call ()
#15 0x00007f0d88cac3e1 in PyEval_CallObjectWithKeywords ()
#16 0x00007f0d88cdff81 in t_bootstrap ()
#17 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0
#18 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6
#19 0x0000000000000000 in ?? ()
Thread 16 (Thread 1118345568 (LWP 8263)):
#0 0x0000003f0b90a65f in sem_wait () from /lib64/tls/libpthread.so.0
#1 0x00007f0d88cdbeb8 in PyThread_acquire_lock ()
#2 0x00007f0d88cdf602 in lock_PyThread_acquire_lock ()
#3 0x00007f0d88cb19cb in PyEval_EvalFrameEx ()
#4 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#5 0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#6 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#7 0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#8 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#9 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#10 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#11 0x00007f0d88c457e8 in function_call ()
#12 0x00007f0d88c2189d in PyObject_Call ()
#13 0x00007f0d88c30c14 in instancemethod_call ()
#14 0x00007f0d88c2189d in PyObject_Call ()
#15 0x00007f0d88cac3e1 in PyEval_CallObjectWithKeywords ()
#16 0x00007f0d88cdff81 in t_bootstrap ()
#17 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0
#18 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6
#19 0x0000000000000000 in ?? ()
Thread 15 (Thread 1128835424 (LWP 8264)):
#0 0x0000003f0b90a65f in sem_wait () from /lib64/tls/libpthread.so.0
#1 0x00007f0d88cdbeb8 in PyThread_acquire_lock ()
#2 0x00007f0d88cdf602 in lock_PyThread_acquire_lock ()
#3 0x00007f0d88cb19cb in PyEval_EvalFrameEx ()
#4 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#5 0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#6 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#7 0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#8 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#9 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#10 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#11 0x00007f0d88c457e8 in function_call ()
#12 0x00007f0d88c2189d in PyObject_Call ()
#13 0x00007f0d88c30c14 in instancemethod_call ()
#14 0x00007f0d88c2189d in PyObject_Call ()
#15 0x00007f0d88cac3e1 in PyEval_CallObjectWithKeywords ()
#16 0x00007f0d88cdff81 in t_bootstrap ()
#17 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0
#18 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6
#19 0x0000000000000000 in ?? ()
Thread 14 (Thread 1139325280 (LWP 8265)):
#0 0x0000003f0b90a65f in sem_wait () from /lib64/tls/libpthread.so.0
#1 0x00007f0d88cdbeb8 in PyThread_acquire_lock ()
#2 0x00007f0d88cdf602 in lock_PyThread_acquire_lock ()
#3 0x00007f0d88cb19cb in PyEval_EvalFrameEx ()
#4 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#5 0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#6 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#7 0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#8 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#9 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#10 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#11 0x00007f0d88c457e8 in function_call ()
#12 0x00007f0d88c2189d in PyObject_Call ()
#13 0x00007f0d88c30c14 in instancemethod_call ()
#14 0x00007f0d88c2189d in PyObject_Call ()
#15 0x00007f0d88cac3e1 in PyEval_CallObjectWithKeywords ()
#16 0x00007f0d88cdff81 in t_bootstrap ()
#17 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0
#18 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6
#19 0x0000000000000000 in ?? ()
Thread 13 (Thread 1149815136 (LWP 8266)):
#0 0x0000003f0b90a65f in sem_wait () from /lib64/tls/libpthread.so.0
#1 0x00007f0d88cdbeb8 in PyThread_acquire_lock ()
#2 0x00007f0d88cdf602 in lock_PyThread_acquire_lock ()
#3 0x00007f0d88cb19cb in PyEval_EvalFrameEx ()
#4 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#5 0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#6 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#7 0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#8 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#9 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#10 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#11 0x00007f0d88c457e8 in function_call ()
#12 0x00007f0d88c2189d in PyObject_Call ()
#13 0x00007f0d88c30c14 in instancemethod_call ()
#14 0x00007f0d88c2189d in PyObject_Call ()
#15 0x00007f0d88cac3e1 in PyEval_CallObjectWithKeywords ()
#16 0x00007f0d88cdff81 in t_bootstrap ()
#17 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0
#18 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6
#19 0x0000000000000000 in ?? ()
Thread 12 (Thread 1160304992 (LWP 8267)):
#0 0x0000003f0b90a65f in sem_wait () from /lib64/tls/libpthread.so.0
#1 0x00007f0d88cdbeb8 in PyThread_acquire_lock ()
#2 0x00007f0d88cdf602 in lock_PyThread_acquire_lock ()
#3 0x00007f0d88cb19cb in PyEval_EvalFrameEx ()
#4 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#5 0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#6 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#7 0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#8 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#9 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#10 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#11 0x00007f0d88c457e8 in function_call ()
#12 0x00007f0d88c2189d in PyObject_Call ()
#13 0x00007f0d88c30c14 in instancemethod_call ()
#14 0x00007f0d88c2189d in PyObject_Call ()
#15 0x00007f0d88cac3e1 in PyEval_CallObjectWithKeywords ()
#16 0x00007f0d88cdff81 in t_bootstrap ()
#17 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0
#18 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6
#19 0x0000000000000000 in ?? ()
Thread 11 (Thread 1170794848 (LWP 8268)):
#0 0x0000003f0b90a65f in sem_wait () from /lib64/tls/libpthread.so.0
#1 0x00007f0d88cdbeb8 in PyThread_acquire_lock ()
#2 0x00007f0d88cdf602 in lock_PyThread_acquire_lock ()
#3 0x00007f0d88cb19cb in PyEval_EvalFrameEx ()
#4 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#5 0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#6 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#7 0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#8 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#9 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#10 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#11 0x00007f0d88c457e8 in function_call ()
#12 0x00007f0d88c2189d in PyObject_Call ()
#13 0x00007f0d88c30c14 in instancemethod_call ()
#14 0x00007f0d88c2189d in PyObject_Call ()
#15 0x00007f0d88cac3e1 in PyEval_CallObjectWithKeywords ()
#16 0x00007f0d88cdff81 in t_bootstrap ()
#17 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0
#18 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6
#19 0x0000000000000000 in ?? ()
Thread 10 (Thread 1085798752 (LWP 8270)):
#0 0x0000003f0b90a65f in sem_wait () from /lib64/tls/libpthread.so.0
#1 0x00007f0d88cdbeb8 in PyThread_acquire_lock ()
#2 0x00007f0d88cdf602 in lock_PyThread_acquire_lock ()
#3 0x00007f0d88cb19cb in PyEval_EvalFrameEx ()
#4 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#5 0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#6 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#7 0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#8 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#9 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#10 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#11 0x00007f0d88c457e8 in function_call ()
#12 0x00007f0d88c2189d in PyObject_Call ()
#13 0x00007f0d88c30c14 in instancemethod_call ()
#14 0x00007f0d88c2189d in PyObject_Call ()
#15 0x00007f0d88cac3e1 in PyEval_CallObjectWithKeywords ()
#16 0x00007f0d88cdff81 in t_bootstrap ()
#17 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0
#18 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6
#19 0x0000000000000000 in ?? ()
Thread 9 (Thread 1096288608 (LWP 8271)):
#0 0x0000003f0b90a65f in sem_wait () from /lib64/tls/libpthread.so.0
#1 0x00007f0d88cdbeb8 in PyThread_acquire_lock ()
#2 0x00007f0d88cdf602 in lock_PyThread_acquire_lock ()
#3 0x00007f0d88cb19cb in PyEval_EvalFrameEx ()
#4 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#5 0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#6 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#7 0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#8 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#9 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#10 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#11 0x00007f0d88c457e8 in function_call ()
#12 0x00007f0d88c2189d in PyObject_Call ()
#13 0x00007f0d88c30c14 in instancemethod_call ()
#14 0x00007f0d88c2189d in PyObject_Call ()
#15 0x00007f0d88cac3e1 in PyEval_CallObjectWithKeywords ()
#16 0x00007f0d88cdff81 in t_bootstrap ()
#17 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0
#18 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6
#19 0x0000000000000000 in ?? ()
Thread 8 (Thread 1181284704 (LWP 8273)):
#0 0x0000003f0b90a65f in sem_wait () from /lib64/tls/libpthread.so.0
#1 0x00007f0d88cdbeb8 in PyThread_acquire_lock ()
#2 0x00007f0d88cdf602 in lock_PyThread_acquire_lock ()
#3 0x00007f0d88cb19cb in PyEval_EvalFrameEx ()
#4 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#5 0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#6 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#7 0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#8 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#9 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#10 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#11 0x00007f0d88c457e8 in function_call ()
#12 0x00007f0d88c2189d in PyObject_Call ()
#13 0x00007f0d88c30c14 in instancemethod_call ()
#14 0x00007f0d88c2189d in PyObject_Call ()
#15 0x00007f0d88cac3e1 in PyEval_CallObjectWithKeywords ()
#16 0x00007f0d88cdff81 in t_bootstrap ()
#17 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0
#18 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6
#19 0x0000000000000000 in ?? ()
Thread 7 (Thread 1191774560 (LWP 8276)):
#0 0x0000003f0b90a65f in sem_wait () from /lib64/tls/libpthread.so.0
#1 0x00007f0d88cdbeb8 in PyThread_acquire_lock ()
#2 0x00007f0d88cdf602 in lock_PyThread_acquire_lock ()
#3 0x00007f0d88cb19cb in PyEval_EvalFrameEx ()
#4 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#5 0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#6 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#7 0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#8 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#9 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#10 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#11 0x00007f0d88c457e8 in function_call ()
#12 0x00007f0d88c2189d in PyObject_Call ()
#13 0x00007f0d88c30c14 in instancemethod_call ()
#14 0x00007f0d88c2189d in PyObject_Call ()
#15 0x00007f0d88cac3e1 in PyEval_CallObjectWithKeywords ()
#16 0x00007f0d88cdff81 in t_bootstrap ()
#17 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0
#18 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6
#19 0x0000000000000000 in ?? ()
Thread 6 (Thread 1202264416 (LWP 8277)):
#0 0x0000003f0b90a65f in sem_wait () from /lib64/tls/libpthread.so.0
#1 0x00007f0d88cdbeb8 in PyThread_acquire_lock ()
#2 0x00007f0d88cdf602 in lock_PyThread_acquire_lock ()
#3 0x00007f0d88cb19cb in PyEval_EvalFrameEx ()
#4 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#5 0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#6 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#7 0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#8 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#9 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#10 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#11 0x00007f0d88c457e8 in function_call ()
#12 0x00007f0d88c2189d in PyObject_Call ()
#13 0x00007f0d88c30c14 in instancemethod_call ()
#14 0x00007f0d88c2189d in PyObject_Call ()
#15 0x00007f0d88cac3e1 in PyEval_CallObjectWithKeywords ()
#16 0x00007f0d88cdff81 in t_bootstrap ()
#17 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0
#18 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6
#19 0x0000000000000000 in ?? ()
Thread 5 (Thread 1212754272 (LWP 8278)):
#0 0x0000003f0b90a65f in sem_wait () from /lib64/tls/libpthread.so.0
#1 0x00007f0d88cdbeb8 in PyThread_acquire_lock ()
#2 0x00007f0d88cdf602 in lock_PyThread_acquire_lock ()
#3 0x00007f0d88cb19cb in PyEval_EvalFrameEx ()
#4 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#5 0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#6 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#7 0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#8 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#9 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#10 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#11 0x00007f0d88c457e8 in function_call ()
#12 0x00007f0d88c2189d in PyObject_Call ()
#13 0x00007f0d88c30c14 in instancemethod_call ()
#14 0x00007f0d88c2189d in PyObject_Call ()
#15 0x00007f0d88cac3e1 in PyEval_CallObjectWithKeywords ()
#16 0x00007f0d88cdff81 in t_bootstrap ()
#17 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0
#18 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6
#19 0x0000000000000000 in ?? ()
Thread 4 (Thread 1223244128 (LWP 8279)):
#0 0x0000003f0b90a65f in sem_wait () from /lib64/tls/libpthread.so.0
#1 0x00007f0d88cdbeb8 in PyThread_acquire_lock ()
#2 0x00007f0d88cdf602 in lock_PyThread_acquire_lock ()
#3 0x00007f0d88cb19cb in PyEval_EvalFrameEx ()
#4 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#5 0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#6 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#7 0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#8 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#9 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#10 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#11 0x00007f0d88c457e8 in function_call ()
#12 0x00007f0d88c2189d in PyObject_Call ()
#13 0x00007f0d88c30c14 in instancemethod_call ()
#14 0x00007f0d88c2189d in PyObject_Call ()
#15 0x00007f0d88cac3e1 in PyEval_CallObjectWithKeywords ()
#16 0x00007f0d88cdff81 in t_bootstrap ()
#17 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0
#18 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6
#19 0x0000000000000000 in ?? ()
Thread 3 (Thread 1233733984 (LWP 8280)):
#0 0x0000003f0b90a65f in sem_wait () from /lib64/tls/libpthread.so.0
#1 0x00007f0d88cdbeb8 in PyThread_acquire_lock ()
#2 0x00007f0d88cdf602 in lock_PyThread_acquire_lock ()
#3 0x00007f0d88cb19cb in PyEval_EvalFrameEx ()
#4 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#5 0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#6 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#7 0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#8 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#9 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#10 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#11 0x00007f0d88c457e8 in function_call ()
#12 0x00007f0d88c2189d in PyObject_Call ()
#13 0x00007f0d88c30c14 in instancemethod_call ()
#14 0x00007f0d88c2189d in PyObject_Call ()
#15 0x00007f0d88cac3e1 in PyEval_CallObjectWithKeywords ()
#16 0x00007f0d88cdff81 in t_bootstrap ()
#17 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0
#18 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6
#19 0x0000000000000000 in ?? ()
Thread 2 (Thread 1244223840 (LWP 8281)):
#0 0x0000003f0b90a65f in sem_wait () from /lib64/tls/libpthread.so.0
#1 0x00007f0d88cdbeb8 in PyThread_acquire_lock ()
#2 0x00007f0d88cdf602 in lock_PyThread_acquire_lock ()
#3 0x00007f0d88cb19cb in PyEval_EvalFrameEx ()
#4 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#5 0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#6 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#7 0x00007f0d88cb1eee in PyEval_EvalFrameEx ()
#8 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#9 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#10 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#11 0x00007f0d88c457e8 in function_call ()
#12 0x00007f0d88c2189d in PyObject_Call ()
#13 0x00007f0d88c30c14 in instancemethod_call ()
#14 0x00007f0d88c2189d in PyObject_Call ()
#15 0x00007f0d88cac3e1 in PyEval_CallObjectWithKeywords ()
#16 0x00007f0d88cdff81 in t_bootstrap ()
#17 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0
#18 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6
#19 0x0000000000000000 in ?? ()
Thread 1 (Thread 139696105322208 (LWP 7916)):================================》主线程
#0 0x0000003f0b0bd0c2 in poll () from /lib64/tls/libc.so.6
#1 0x00007f0d853c5730 in internal_select ()
#2 0x00007f0d853c61ca in sock_accept ()
#3 0x00007f0d88cafa21 in PyEval_EvalFrameEx ()
#4 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#5 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#6 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx ()
#7 0x00007f0d88cb3624 in PyEval_EvalCodeEx ()
#8 0x00007f0d88cb3822 in PyEval_EvalCode ()
#9 0x00007f0d88ccc49c in run_mod ()
#10 0x00007f0d88cccafc in PyRun_FileExFlags ()
#11 0x00007f0d88ccda11 in PyRun_SimpleFileExFlags ()
#12 0x00007f0d88cddd26 in Py_Main ()
#13 0x0000003f0b01c4bb in __libc_start_main () from /lib64/tls/libc.so.6
#14 0x00000000004006da in _start ()
#15 0x00007fff62a773d8 in ?? ()
#16 0x000000000000001c in ?? ()
#17 0x0000000000000003 in ?? ()
#18 0x00007fff62a776db in ?? ()
#19 0x00007fff62a776e5 in ?? ()
#20 0x00007fff62a7771e in ?? ()
#21 0x0000000000000000 in ?? ()1.2.4 查看 Butterfly stack trace
此功能在 handler 中进行实现，代码见 /handler/demo_stackdump
使用
$ curl "http://127.0.0.1:8585/demo_stackdump/stackdump"
stack trace 日志记录在 logs/stack.log例子 (butterfly/logs/stack.log 内容）
----------------------------------------------------------------
dump_time=2021-04-09 12:19:55 reqid=2A5BE75641F1BC5A
1 ActiveThreadCount: 18
2 KnownActiveThreadNames:
MainThread-------------------------------- 主线程
APScheduler------------------------------- 任务调度线程
CP Server Thread-1------------------------Butterfly handler 处理线程，以下类似
CP Server Thread-11
CP Server Thread-5
CP Server Thread-9
CP Server Thread-14
CP Server Thread-7
CP Server Thread-13
CP Server Thread-15
CP Server Thread-3
CP Server Thread-2
CP Server Thread-4
CP Server Thread-6
CP Server Thread-12
CP Server Thread-8
CP Server Thread-10
CP Server Thread-16
3 ThreadInfo:
Thread: 1082784096 CP Server Thread-16
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap
self.__bootstrap_inner()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner
self.run()
File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1643, in run
conn = self.server.requests.get()
File: "/home/users/meetbill/.jumbo/lib/python2.7/Queue.py", line 168, in get
self.not_empty.wait()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 243, in wait
waiter.acquire()
Thread: 1093196128 CP Server Thread-15
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap
self.__bootstrap_inner()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner
self.run()
File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1643, in run
conn = self.server.requests.get()
File: "/home/users/meetbill/.jumbo/lib/python2.7/Queue.py", line 168, in get
self.not_empty.wait()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 243, in wait
waiter.acquire()
Thread: 1089005920 CP Server Thread-3
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap
self.__bootstrap_inner()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner
self.run()
File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1643, in run
conn = self.server.requests.get()
File: "/home/users/meetbill/.jumbo/lib/python2.7/Queue.py", line 168, in get
self.not_empty.wait()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 243, in wait
waiter.acquire()
Thread: 1076640096 CP Server Thread-13
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap
self.__bootstrap_inner()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner
self.run()
File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1643, in run
conn = self.server.requests.get()
File: "/home/users/meetbill/.jumbo/lib/python2.7/Queue.py", line 168, in get
self.not_empty.wait()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 243, in wait
waiter.acquire()
Thread: 1103497568 CP Server Thread-6
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap
self.__bootstrap_inner()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner
self.run()
File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1643, in run
conn = self.server.requests.get()
File: "/home/users/meetbill/.jumbo/lib/python2.7/Queue.py", line 168, in get
self.not_empty.wait()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 243, in wait
waiter.acquire()
Thread: 1094404448 CP Server Thread-5
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap
self.__bootstrap_inner()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner
self.run()
File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1643, in run
conn = self.server.requests.get()
File: "/home/users/meetbill/.jumbo/lib/python2.7/Queue.py", line 168, in get
self.not_empty.wait()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 243, in wait
waiter.acquire()
Thread: 1100007776 CP Server Thread-2
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap
self.__bootstrap_inner()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner
self.run()
File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1643, in run
conn = self.server.requests.get()
File: "/home/users/meetbill/.jumbo/lib/python2.7/Queue.py", line 168, in get
self.not_empty.wait()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 243, in wait
waiter.acquire()
Thread: 140004998178528 MainThread
File: "/home/users/meetbill/butterfly/main.py", line 30, in <module>
server.start()
File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 2104, in start
self.tick()
File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 2158, in tick
s, addr = self.socket.accept()
File: "/home/users/meetbill/.jumbo/lib/python2.7/socket.py", line 202, in accept
sock, addr = self._sock.accept()
Thread: 1102469472 CP Server Thread-7
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap
self.__bootstrap_inner()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner
self.run()
File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1643, in run
conn = self.server.requests.get()
File: "/home/users/meetbill/.jumbo/lib/python2.7/Queue.py", line 168, in get
self.not_empty.wait()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 243, in wait
waiter.acquire()
Thread: 1095801184 CP Server Thread-8
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap
self.__bootstrap_inner()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner
self.run()
File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1643, in run
conn = self.server.requests.get()
File: "/home/users/meetbill/.jumbo/lib/python2.7/Queue.py", line 168, in get
self.not_empty.wait()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 243, in wait
waiter.acquire()
Thread: 1100343648 CP Server Thread-11
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap
self.__bootstrap_inner()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner
self.run()
File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1643, in run
conn = self.server.requests.get()
File: "/home/users/meetbill/.jumbo/lib/python2.7/Queue.py", line 168, in get
self.not_empty.wait()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 243, in wait
waiter.acquire()
Thread: 1075272032 CP Server Thread-14
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap
self.__bootstrap_inner()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner
self.run()
File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1643, in run
conn = self.server.requests.get()
File: "/home/users/meetbill/.jumbo/lib/python2.7/Queue.py", line 168, in get
self.not_empty.wait()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 243, in wait
waiter.acquire()
Thread: 1100863840 APScheduler
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap
self.__bootstrap_inner()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner
self.run()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 504, in run
self.__target(*self.__args, **self.__kwargs)
File: "/home/users/meetbill/butterfly/xlib/apscheduler/schedulers/blocking.py", line 42, in _main_loop
self._event.wait(wait_seconds)
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 403, in wait
self.__cond.wait(timeout)
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 262, in wait
_sleep(delay)
Thread: 1084520800 CP Server Thread-4
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap
self.__bootstrap_inner()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner
self.run()
File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1643, in run
conn = self.server.requests.get()
File: "/home/users/meetbill/.jumbo/lib/python2.7/Queue.py", line 168, in get
self.not_empty.wait()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 243, in wait
waiter.acquire()
Thread: 1097337184 CP Server Thread-9
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap
self.__bootstrap_inner()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner
self.run()
File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1643, in run
conn = self.server.requests.get()
File: "/home/users/meetbill/.jumbo/lib/python2.7/Queue.py", line 168, in get
self.not_empty.wait()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 243, in wait
waiter.acquire()
Thread: 1091410272 CP Server Thread-12
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap
self.__bootstrap_inner()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner
self.run()
File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1643, in run
conn = self.server.requests.get()
File: "/home/users/meetbill/.jumbo/lib/python2.7/Queue.py", line 168, in get
self.not_empty.wait()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 243, in wait
waiter.acquire()
Thread: 1087252832 CP Server Thread-10
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap
self.__bootstrap_inner()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner
self.run()
File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1643, in run
conn = self.server.requests.get()
File: "/home/users/meetbill/.jumbo/lib/python2.7/Queue.py", line 168, in get
self.not_empty.wait()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 243, in wait
waiter.acquire()
Thread: 1099741536 CP Server Thread-1
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap
self.__bootstrap_inner()
File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner
self.run()
File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1651, in run
conn.communicate()
File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1460, in communicate
req.respond()
File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 901, in respond
self.server.gateway(self).respond()
File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 2426, in respond
response = self.req.server.wsgi_app(self.env, self.start_response)
File: "/home/users/meetbill/butterfly/wsgiapp.py", line 161, in application
status, headders, content = wsgigw.process(environ)
File: "/home/users/meetbill/butterfly/xlib/httpgateway.py", line 195, in process
httpstatus, headers, content = protocol(req)
File: "/home/users/meetbill/butterfly/xlib/protocol_json.py", line 171, in __call__
ret = self._func(**params)
File: "/home/users/meetbill/butterfly/handlers/demo_stackdump/__init__.py", line 79, in stackdump
for filename, lineno, name, line in traceback.extract_stack(stack):2 gdb
2.1 使用方法
gdb python <pid of running process>2.1.1 调试进程
查看 C 调用栈
(gdb) bt查看 Python 调用栈
(gdb) py-bt2.2 官方文档
https://wiki.python.org/moin/DebuggingWithGdb
2.3 查看 core 保存路径
# ulimit -c
unlimited
# cat /proc/sys/kernel/core_pattern
/home/work/core.%e.%p.%t
ulimit -c 是查看创建的核心转储的最大大小
0: 不保存
unlimited: 不限制大小
cat /proc/sys/kernel/core_pattern 生成的文件路径
2.4 查看 core 文件例子
#gdb `which python ` core_path例子
# gdb `which python ` core.python2.7.25133.1617075053
...
gdb info
...
[New LWP 25192]
[New LWP 25193]
[New LWP 25195]
[New LWP 25196]
[New LWP 25197]
[New LWP 25194]
[New LWP 25198]
[New LWP 25199]
[New LWP 25201]
[New LWP 25200]
[New LWP 25202]
[New LWP 25204]
[New LWP 25206]
[New LWP 25203]
[New LWP 25205]
[New LWP 25207]
[New LWP 25191]
[New LWP 25208]
[New LWP 25133]
warning: File "/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
To enable execution of this file add
add-auto-load-safe-path /lib64/libthread_db-1.0.so
line to your configuration file "/home/work/.gdbinit".
To completely disable this security protection add
set auto-load safe-path /
line to your configuration file "/home/work/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual. E.g., run from the shell:
info "(gdb)Auto-loading safe path"
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
warning: File "/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Core was generated by `python2.7 /home/work/chunfeng/chunfeng_app2_4600/main.py 954216e9'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007f85e85265c5 in PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2995
2995 Python/ceval.c: No such file or directory.
[Current thread is 1 (LWP 25192)]
(gdb)可以看到程序是在停止的时候，在如下部分发生的问题
#0 0x00007f85e85265c5 in PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2995可以输入 bt 查看更多，其从上到下是错误从底层到最外层的顺序。3 lsof(lists openfiles)
在 Unix 中一切（包括网络套接口）都是文件。lsof 的功能是列出打开文件（lists openfiles）
3.1 使用 -p 查看指定进程 ID 已打开的文件
lsof -p {PID}
COMMAND ：进程的名称
PID ：进程标识符
USER ：进程所有者
FD ：文件描述符，应用程序通过文件描述符识别该文件。如 cwd、txt 等
(1) cwd 值表示应用程序的当前工作目录，这是该应用程序启动的目录，除非它本身对这个目录进行更改
(2) txt 类型的文件是程序代码，如应用程序二进制文件本身或共享库，其次数值表示应用程序的文件描述符，这是打开该文件时返回的一个整数。
(3) u 表示该文件被打开并处于读取 / 写入模式，而不是只读 r 或只写 (w) 模式。同时还有大写 的 W 表示该应用程序具有对整个文件的写锁。
该文件描述符用于确保每次只能打开一个应用程序实例。
初始打开每个应用程序时，都具有三个文件描述符，从 0 到 2，分别表示标准输入、输出和错误流。
所以大多数应用程序所打开的文件的 FD 都是从 3 开始。
TYPE ：文件类型，如 DIR、REG 等
(1) 文件和目录分别称为 REG 和 DIR
(2) CHR 表示字符
(3) BLK 表示块设备
(4) UNIX -- UNIX 域套接字
FIFO -- 先进先出 (FIFO) 队列
IPv4 -- 网际协议 (IP) 套接字
DEVICE ：指定磁盘的名称
SIZE ：文件的大小
NODE ：索引节点（文件在磁盘上的标识）
NAME ：打开文件的确切名称3.2 使用 @host:port 显示基于主机与端口的连接
# lsof -i@xx.xx.xx.xx:224 mem_top 排查内存泄露问题
kill -12 <pid>将会将将日志打印在 logs/common/common.log
refs 回显信息：占用内存最大的 topN 变量中的元素个数
bytes 回显信息：占用内存最大的 topN 变量每个变量所占用的 bytes
types 回显信息：占用内存最大的 topN 类型变量个数
4.1 例子
refs:
49452 <type 'dict'> {'xlib.util.pluginbase._internalspace._spef7874fe95975333c5ff649d4c981dca.vm_manager': <module 'xlib.util.pluginbase._internalspace._spef7874fe95975333c5ff649d4c981dca.vm_manager' from '/home/work/chu
5928 <type 'dict'> {'_sp349460523b2000c7ebcfd35e0025e291': <module 'xlib.util.pluginbase._internalspace._sp349460523b2000c7ebcfd35e0025e291' (built-in)>, '_spdf494a60169fd9552a956256d05ccc00': <module 'xlib.util.pluginb
954 <type 'dict'> {'TOO_MANY_KEY_PARTS': 1070, 'WRONG_PARAMCOUNT_TO_PROCEDURE': 1107, 'SP_COND_MISMATCH': 1319, 'XAER_OUTSIDE': 1400, 'KILL_DENIED_ERROR': 1095, 'TABLE_CANT_HANDLE_SPKEYS': 1464, 'DUMP_NOT_IMPLEMENTED':
570 <type 'dict'> {'SocketType': <class 'socket._socketobject'>, 'getaddrinfo': <built-in function getaddrinfo>, 'AI_NUMERICSERV': 1024, 'PACKET_OTHERHOST': 3, 'AI_NUMERICHOST': 4, 'StringIO': <built-in function String
520 <type 'dict'> {'DateField': <class 'xlib.db.peewee.DateField'>, 'ModelNamedTupleCursorWrapper': <class 'xlib.db.peewee.ModelNamedTupleCursorWrapper'>, 'ColumnMetadata': <class 'xlib.db.peewee.ColumnMetadata'>, 'Bit
496 <type 'dict'> {'restore': <function restore at 0x299cde8>, 'zrangebylex': <function zrangebylex at 0x299f7d0>, 'renamenx': <function renamenx at 0x299cd70>, 'client_id': <function client_id at 0x29999b0>, 'acl_save
490 <type 'dict'> {'SocketType': <type '_socket.socket'>, 'getaddrinfo': <built-in function getaddrinfo>, 'IPPROTO_RAW': 255, 'AI_NUMERICSERV': 1024, 'PACKET_OTHERHOST': 3, 'AI_NUMERICHOST': 4, 'IPPORT_RESERVED': 1024,
490 <type 'dict'> {'SocketType': <type '_socket.socket'>, 'getaddrinfo': <built-in function getaddrinfo>, 'AI_NUMERICSERV': 1024, 'PACKET_OTHERHOST': 3, 'AI_NUMERICHOST': 4, 'IPPORT_RESERVED': 1024, 'AF_IRDA': 23, 'MSG
462 <type 'dict'> {'tcsendbreak': <built-in function tcsendbreak>, 'B110': 3, 'tcdrain': <built-in function tcdrain>, 'CIBAUD': 269418496, 'FF1': 32768, 'FF0': 0, 'FIONCLEX': 21584, 'ONLCR': 4, 'B600': 8, 'TCSANOW': 0,
462 <type 'dict'> {'tcsendbreak': <built-in function tcsendbreak>, 'B110': 3, 'tcdrain': <built-in function tcdrain>, 'TCIOFLUSH': 2, 'FF1': 32768, 'FF0': 0, 'FIONCLEX': 21584, 'ONLCR': 4, 'B600': 8, 'TCSANOW': 0, 'TIO
448 <type 'dict'> {'WTERMSIG': <built-in function WTERMSIG>, 'lseek': <built-in function lseek>, 'EX_IOERR': 74, 'EX_NOHOST': 68, 'seteuid': <built-in function seteuid>, 'pathsep': ':', 'execle': <function execle at 0x
412 <type 'list'> [<weakref at 0x7fa50428c050; to 'type' at 0x7fa5046695e0 (type)>, <weakref at 0x7fa50428c0a8; to 'type' at 0x7fa50466b700 (weakref)>, <weakref at 0x7fa50428c100; to 'type' at 0x7fa50466ba40 (weakcalla
364 <type 'dict'> {'empty_provider': <pkg_resources.EmptyProvider instance at 0x34a23b0>, 'ExtractionError': <class 'pkg_resources.ExtractionError'>, 'functools': <module 'functools' from '/home/work/.jumbo/lib/python2
359 <class 'xlib.db.redis.client.CaseInsensitiveDict'> {u'UNWATCH': <function bool_ok at 0x2995848>, u'SLAVEOF': <function bool_ok at 0x2995848>, u'OBJECT': <function parse_object at 0x2994c80>, u'GEODIST': <function float_or_none at 0x29957d0>, u'EXPIRE'
359 <class 'xlib.db.redis.client.CaseInsensitiveDict'> {u'UNWATCH': <function bool_ok at 0x2995848>, u'SLAVEOF': <function bool_ok at 0x2995848>, u'OBJECT': <function parse_object at 0x2994c80>, u'GEODIST': <function float_or_none at 0x29957d0>, u'EXPIRE'
358 <type 'dict'> {u'UNWATCH': <function bool_ok at 0x2995848>, u'SLAVEOF': <function bool_ok at 0x2995848>, u'MOVE': <type 'bool'>, u'GEODIST': <function float_or_none at 0x29957d0>, u'TIME': <function <lambda> at 0x2
338 <type 'dict'> {'WTERMSIG': <built-in function WTERMSIG>, 'lseek': <built-in function lseek>, 'EX_IOERR': 74, 'EX_NOHOST': 68, 'seteuid': <built-in function seteuid>, 'O_RSYNC': 1052672, 'fstatvfs': <built-in functi
338 <type 'dict'> {'WTERMSIG': <built-in function WTERMSIG>, 'lseek': <built-in function lseek>, 'EX_IOERR': 74, 'EX_NOHOST': 68, 'getpgrp': <built-in function getpgrp>, 'major': <built-in function major>, 'fstatvfs':
286 <type 'dict'> {'bytearray': <type 'bytearray'>, 'IndexError': <type 'exceptions.IndexError'>, 'all': <built-in function all>, 'help': Type help() for interactive help, or help(object) for help about object., 'vars'
286 <type 'dict'> {1: Charset(id=1, name='big5', collation='big5_chinese_ci'), 2: Charset(id=2, name='latin2', collation='latin2_czech_cs'), 3: Charset(id=3, name='dec8', collation='dec8_swedish_ci'), 4: Charset(id=4,
bytes:
3146008 {'xlib.util.pluginbase._internalspace._spef7874fe95975333c5ff649d4c981dca.vm_manager': <module 'xlib.util.pluginbase._internalspace._spef7874fe95975333c5ff649d4c981dca.vm_manager' from '/home/work/chu
196888 {'_sp349460523b2000c7ebcfd35e0025e291': <module 'xlib.util.pluginbase._internalspace._sp349460523b2000c7ebcfd35e0025e291' (built-in)>, '_spdf494a60169fd9552a956256d05ccc00': <module 'xlib.util.pluginb
49432 {'TOO_MANY_KEY_PARTS': 1070, 'WRONG_PARAMCOUNT_TO_PROCEDURE': 1107, 'SP_COND_MISMATCH': 1319, 'XAER_OUTSIDE': 1400, 'KILL_DENIED_ERROR': 1095, 'TABLE_CANT_HANDLE_SPKEYS': 1464, 'DUMP_NOT_IMPLEMENTED':
12568 {'bytearray': <type 'bytearray'>, 'IndexError': <type 'exceptions.IndexError'>, 'all': <built-in function all>, 'help': Type help() for interactive help, or help(object) for help about object., 'vars'
12568 {'bytearray': <type 'bytearray'>, 'all': <built-in function all>, 'vars': <built-in function vars>, 'SyntaxError': <type 'exceptions.SyntaxError'>, 'unicode': <type 'unicode'>, 'sorted': <built-in fun
12568 {'WTERMSIG': <built-in function WTERMSIG>, 'lseek': <built-in function lseek>, 'EX_IOERR': 74, 'EX_NOHOST': 68, 'seteuid': <built-in function seteuid>, 'pathsep': ':', 'execle': <function execle at 0x
12568 {'EISDIR': 21, 'EPROTONOSUPPORT': 93, 'ETXTBSY': 26, 'ETOOMANYREFS': 109, 'EDEADLOCK': 35, 'ETIME': 62, 'ESOCKTNOSUPPORT': 94, 'ENOTTY': 25, 'ENAMETOOLONG': 36, 'ENETRESET': 102, 'EMULTIHOP': 72, 'EMS
12568 {'WTERMSIG': <built-in function WTERMSIG>, 'lseek': <built-in function lseek>, 'EX_IOERR': 74, 'EX_NOHOST': 68, 'seteuid': <built-in function seteuid>, 'O_RSYNC': 1052672, 'fstatvfs': <built-in functi
12568 {'WTERMSIG': <built-in function WTERMSIG>, 'lseek': <built-in function lseek>, 'EX_IOERR': 74, 'EX_NOHOST': 68, 'getpgrp': <built-in function getpgrp>, 'major': <built-in function major>, 'fstatvfs':
12568 {'isstring': <function isstring at 0x7fa504193aa0>, '_optimize_unicode': <function _optimize_unicode at 0x7fa504193938>, 'SRE_FLAG_VERBOSE': 64, '_identityfunction': <function _identityfunction at 0x7
12568 {'_PATTERNENDERS': set([')', '|']), 'REPEAT_ONE': 'repeat_one', 'DIGITS': set(['1', '0', '3', '2', '5', '4', '7', '6', '9', '8']), 'makedict': <function makedict at 0x7fa504200758>, 'CATEGORY_LOC_NOT_
12568 {'REPEAT_ONE': 'repeat_one', 'makedict': <function makedict at 0x7fa504200758>, 'CATEGORY_LOC_NOT_WORD': 'category_loc_not_word', 'SRE_INFO_CHARSET': 4, 'SRE_INFO_PREFIX': 1, 'SRE_FLAG_DEBUG': 128, 'M
12568 {'latin_1_encode': <built-in function latin_1_encode>, 'getreader': <function getreader at 0x7fa5041bb0c8>, 'readbuffer_encode': <built-in function readbuffer_encode>, 'BOM': '\xff\xfe', 'StreamWriter
12568 {'functools': <module 'functools' from '/home/work/.jumbo/lib/python2.7/functools.pyc'>, 'PY3': False, 'ModuleSixMovesUrllibRobotparser': <class 'xlib.util.six.ModuleSixMovesUrllibRobotparser'>, 'get_
12568 {'gt': <built-in function gt>, '__concat__': <built-in function __concat__>, 'imul': <built-in function imul>, 'ge': <built-in function ge>, '__irshift__': <built-in function __irshift__>, 'setslice':
12568 {'Union': <type '_ctypes.Union'>, 'c_wchar': <class 'ctypes.c_wchar'>, 'c_bool': <class 'ctypes.c_bool'>, 'c_double': <class 'ctypes.c_double'>, 'CFUNCTYPE': <function CFUNCTYPE at 0x7fa4fe2ef0c8>, '_
12568 {'logMultiprocessing': 1, '__path__': ['/home/work/.jumbo/lib/python2.7/logging'], 'LogRecord': <class 'logging.LogRecord'>, 'logProcesses': 1, 'addLevelName': <function addLevelName at 0x227fde8>, '_
12568 {'BadStatusLine': <class 'httplib.BadStatusLine'>, 'UnknownProtocol': <class 'httplib.UnknownProtocol'>, '_UNKNOWN': 'UNKNOWN', 'SWITCHING_PROTOCOLS': 101, 'urlsplit': <function urlsplit at 0x2328758>
12568 {'SocketType': <class 'socket._socketobject'>, 'getaddrinfo': <built-in function getaddrinfo>, 'AI_NUMERICSERV': 1024, 'PACKET_OTHERHOST': 3, 'AI_NUMERICHOST': 4, 'StringIO': <built-in function String
12568 {'SocketType': <type '_socket.socket'>, 'getaddrinfo': <built-in function getaddrinfo>, 'IPPROTO_RAW': 255, 'AI_NUMERICSERV': 1024, 'PACKET_OTHERHOST': 3, 'AI_NUMERICHOST': 4, 'IPPORT_RESERVED': 1024,
types:
44136 <type 'dict'>
30965 <type 'list'>
21183 <type 'module'>
11067 <type 'function'>
5348 <type 'tuple'>
3270 <type 'cell'>
2961 <class 'xlib.util.pluginbase.PluginSource'>
2961 <class 'xlib.util.pluginbase._PluginSourceModule'>
2961 <class 'xlib.util.pluginbase.PluginBaseState'>
1694 <type 'weakref'>
1234 <type 'wrapper_descriptor'>
1108 <type 'builtin_function_or_method'>
905 <type 'type'>
842 <type 'getset_descriptor'>
711 <type 'method_descriptor'>
449 <type 'set'>
427 <type 'property'>
381 <type 'member_descriptor'>
198 <class '_weakrefset.WeakSet'>
168 <type 'frame'>5 传送门
5.1 根据 python 代码自动生成流程图
code2flow – 把你的 Python 和 JavaScript 代码转换为流程图。
pycallgraph - 这个库可以把你的 Python 应用的流程（调用图）进行可视化。
5.2 GDB 教程
5.3 mem_top
Last updated