# 调试和测试之调试 ### 调试和测试之 debug（调试） ## 1.debug ### 1.1 对 handler 进行调试 日常我们主要是编写 handler，如果想要看 handler 运行是否符合预期，可以通过 test\_handler.py 进行调试 #### 1.1.1 输出 usage > python test\_handler.py ``` Usage: test_handler.py /x/hello 'str_info' test_handler.py /auth/ssologin 'ticket=' test_handler.py /report/log test_handler.py /auth/verification test_handler.py /main test_handler.py /tpl test_handler.py /apidemo/ping test_handler.py /apidemo/hello 'str_info' test_handler.py /x/ping ``` 执行 test\_handler.py 后会输出当前所有的 handler 及请求参数 #### 1.1.2 debug 某个 handler 实际执行的时候，是彩色输出的 > python test\_handler.py /apidemo/hello meetbill ``` Source path:... test_handler.py >>>>>|12:59:38.334264 4524121536-MainThread call 66 def main(): ------12:59:38.335328 4524121536-MainThread line 67 return func(*args) Source path:... /Users/meetbill/test/crontab/handlers/apidemo/__init__.py Starting var:.. req = Starting var:.. str_info = 'meetbill' >>>>>|12:59:38.335414 4524121536-MainThread call 40 def hello(req, str_info): ------12:59:38.341516 4524121536-MainThread line 41 isinstance(req, Request) ------12:59:38.341742 4524121536-MainThread line 42 return retstat.HTTP_OK, {"stat":"OK","str_info": str_info}, [(__info, __version)] |<<<<<12:59:38.341985 4524121536-MainThread return 42 return retstat.HTTP_OK, {"stat":"OK","str_info": str_info}, [(__info, __version)] Return value:.. (200, {'stat': 'OK', 'str_info': 'meetbill'}, [('http_demo', '1.0.1')]) Source path:... test_handler.py |<<<<<12:59:38.342153 4524121536-MainThread return 67 return func(*args) Return value:.. (200, {'stat': 'OK', 'str_info': 'meetbill'}, [('http_demo', '1.0.1')]) Elapsed time: 00:00:00.008072 ============================================================= (200, {'stat': 'OK', 'str_info': 'meetbill'}, [('http_demo', '1.0.1')]) ============================================================= ``` ### 1.2 框架相关 #### 1.2.1 查看执行目录 ``` ll /proc/${pid}/cwd ``` #### 1.2.2 系统调用 ``` $ strace -p 17553 ... poll([{fd=11, events=POLLIN}], 1, 1000) = 0 futex(0x6a1a10, FUTEX_WAKE, 1) = 0 poll([{fd=11, events=POLLIN, revents=POLLIN}], 1, 1000) = 1 accept(11, {sa_family=AF_INET, sin_port=htons(33054), sin_addr=inet_addr("127.0.0.1")}, [2198038101723447312]) = 13 futex(0x6a1a10, FUTEX_WAKE, 1) = 0 futex(0x6a1a10, FUTEX_WAKE, 1) = 0 ... ``` 通过 strace 可以看访问 ip 等信息 另外当 Linux 命令卡住时，可以通过 `strace command` 获取系统调用信息 #### 1.2.3 Linux 下多线程查看工具 (pstree、ps、pstack) > 1. pstree 以树结构显示进程 ``` $ pstree -p 7916 python2.7(7916)─┬─{python2.7}(8262) ├─{python2.7}(8263) ├─{python2.7}(8264) ├─{python2.7}(8265) ├─{python2.7}(8266) ├─{python2.7}(8267) ├─{python2.7}(8268) ├─{python2.7}(8270) ├─{python2.7}(8271) ├─{python2.7}(8273) ├─{python2.7}(8276) ├─{python2.7}(8277) ├─{python2.7}(8278) ├─{python2.7}(8279) ├─{python2.7}(8280) └─{python2.7}(8281) ``` 7916 进程共启动了 16 个子线程，加上主线程是 17 个线程 > 1. ps -Lf $ ps -Lf 7916 ``` UID PID PPID LWP C NLWP STIME TTY STAT TIME CMD 304804 7916 1 7916 0 17 13:00 pts/18 Sl 0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque 304804 7916 1 8262 0 17 13:00 pts/18 Sl 0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque 304804 7916 1 8263 0 17 13:00 pts/18 Sl 0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque 304804 7916 1 8264 0 17 13:00 pts/18 Sl 0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque 304804 7916 1 8265 0 17 13:00 pts/18 Sl 0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque 304804 7916 1 8266 0 17 13:00 pts/18 Sl 0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque 304804 7916 1 8267 0 17 13:00 pts/18 Sl 0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque 304804 7916 1 8268 0 17 13:00 pts/18 Sl 0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque 304804 7916 1 8270 0 17 13:00 pts/18 Sl 0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque 304804 7916 1 8271 0 17 13:00 pts/18 Sl 0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque 304804 7916 1 8273 0 17 13:00 pts/18 Sl 0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque 304804 7916 1 8276 0 17 13:00 pts/18 Sl 0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque 304804 7916 1 8277 0 17 13:00 pts/18 Sl 0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque 304804 7916 1 8278 0 17 13:00 pts/18 Sl 0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque 304804 7916 1 8279 0 17 13:00 pts/18 Sl 0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque 304804 7916 1 8280 0 17 13:00 pts/18 Sl 0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque 304804 7916 1 8281 0 17 13:00 pts/18 Sl 0:00 python2.7 /home/users/meetbill/butterfly/main.py bianque ``` 进程共启动了 17 个线程 > 1. pstack 显示每个进程的栈跟踪 ``` $ pstack 7916 Thread 17 (Thread 1107855712 (LWP 8262)): #0 0x0000003f0b90a65f in sem_wait () from /lib64/tls/libpthread.so.0 #1 0x00007f0d88cdbeb8 in PyThread_acquire_lock () #2 0x00007f0d88cdf602 in lock_PyThread_acquire_lock () #3 0x00007f0d88cb19cb in PyEval_EvalFrameEx () #4 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #5 0x00007f0d88cb1eee in PyEval_EvalFrameEx () #6 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #7 0x00007f0d88cb1eee in PyEval_EvalFrameEx () #8 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx () #9 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx () #10 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #11 0x00007f0d88c457e8 in function_call () #12 0x00007f0d88c2189d in PyObject_Call () #13 0x00007f0d88c30c14 in instancemethod_call () #14 0x00007f0d88c2189d in PyObject_Call () #15 0x00007f0d88cac3e1 in PyEval_CallObjectWithKeywords () #16 0x00007f0d88cdff81 in t_bootstrap () #17 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0 #18 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6 #19 0x0000000000000000 in ?? () Thread 16 (Thread 1118345568 (LWP 8263)): #0 0x0000003f0b90a65f in sem_wait () from /lib64/tls/libpthread.so.0 #1 0x00007f0d88cdbeb8 in PyThread_acquire_lock () #2 0x00007f0d88cdf602 in lock_PyThread_acquire_lock () #3 0x00007f0d88cb19cb in PyEval_EvalFrameEx () #4 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #5 0x00007f0d88cb1eee in PyEval_EvalFrameEx () #6 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #7 0x00007f0d88cb1eee in PyEval_EvalFrameEx () #8 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx () #9 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx () #10 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #11 0x00007f0d88c457e8 in function_call () #12 0x00007f0d88c2189d in PyObject_Call () #13 0x00007f0d88c30c14 in instancemethod_call () #14 0x00007f0d88c2189d in PyObject_Call () #15 0x00007f0d88cac3e1 in PyEval_CallObjectWithKeywords () #16 0x00007f0d88cdff81 in t_bootstrap () #17 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0 #18 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6 #19 0x0000000000000000 in ?? () Thread 15 (Thread 1128835424 (LWP 8264)): #0 0x0000003f0b90a65f in sem_wait () from /lib64/tls/libpthread.so.0 #1 0x00007f0d88cdbeb8 in PyThread_acquire_lock () #2 0x00007f0d88cdf602 in lock_PyThread_acquire_lock () #3 0x00007f0d88cb19cb in PyEval_EvalFrameEx () #4 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #5 0x00007f0d88cb1eee in PyEval_EvalFrameEx () #6 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #7 0x00007f0d88cb1eee in PyEval_EvalFrameEx () #8 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx () #9 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx () #10 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #11 0x00007f0d88c457e8 in function_call () #12 0x00007f0d88c2189d in PyObject_Call () #13 0x00007f0d88c30c14 in instancemethod_call () #14 0x00007f0d88c2189d in PyObject_Call () #15 0x00007f0d88cac3e1 in PyEval_CallObjectWithKeywords () #16 0x00007f0d88cdff81 in t_bootstrap () #17 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0 #18 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6 #19 0x0000000000000000 in ?? () Thread 14 (Thread 1139325280 (LWP 8265)): #0 0x0000003f0b90a65f in sem_wait () from /lib64/tls/libpthread.so.0 #1 0x00007f0d88cdbeb8 in PyThread_acquire_lock () #2 0x00007f0d88cdf602 in lock_PyThread_acquire_lock () #3 0x00007f0d88cb19cb in PyEval_EvalFrameEx () #4 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #5 0x00007f0d88cb1eee in PyEval_EvalFrameEx () #6 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #7 0x00007f0d88cb1eee in PyEval_EvalFrameEx () #8 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx () #9 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx () #10 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #11 0x00007f0d88c457e8 in function_call () #12 0x00007f0d88c2189d in PyObject_Call () #13 0x00007f0d88c30c14 in instancemethod_call () #14 0x00007f0d88c2189d in PyObject_Call () #15 0x00007f0d88cac3e1 in PyEval_CallObjectWithKeywords () #16 0x00007f0d88cdff81 in t_bootstrap () #17 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0 #18 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6 #19 0x0000000000000000 in ?? () Thread 13 (Thread 1149815136 (LWP 8266)): #0 0x0000003f0b90a65f in sem_wait () from /lib64/tls/libpthread.so.0 #1 0x00007f0d88cdbeb8 in PyThread_acquire_lock () #2 0x00007f0d88cdf602 in lock_PyThread_acquire_lock () #3 0x00007f0d88cb19cb in PyEval_EvalFrameEx () #4 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #5 0x00007f0d88cb1eee in PyEval_EvalFrameEx () #6 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #7 0x00007f0d88cb1eee in PyEval_EvalFrameEx () #8 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx () #9 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx () #10 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #11 0x00007f0d88c457e8 in function_call () #12 0x00007f0d88c2189d in PyObject_Call () #13 0x00007f0d88c30c14 in instancemethod_call () #14 0x00007f0d88c2189d in PyObject_Call () #15 0x00007f0d88cac3e1 in PyEval_CallObjectWithKeywords () #16 0x00007f0d88cdff81 in t_bootstrap () #17 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0 #18 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6 #19 0x0000000000000000 in ?? () Thread 12 (Thread 1160304992 (LWP 8267)): #0 0x0000003f0b90a65f in sem_wait () from /lib64/tls/libpthread.so.0 #1 0x00007f0d88cdbeb8 in PyThread_acquire_lock () #2 0x00007f0d88cdf602 in lock_PyThread_acquire_lock () #3 0x00007f0d88cb19cb in PyEval_EvalFrameEx () #4 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #5 0x00007f0d88cb1eee in PyEval_EvalFrameEx () #6 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #7 0x00007f0d88cb1eee in PyEval_EvalFrameEx () #8 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx () #9 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx () #10 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #11 0x00007f0d88c457e8 in function_call () #12 0x00007f0d88c2189d in PyObject_Call () #13 0x00007f0d88c30c14 in instancemethod_call () #14 0x00007f0d88c2189d in PyObject_Call () #15 0x00007f0d88cac3e1 in PyEval_CallObjectWithKeywords () #16 0x00007f0d88cdff81 in t_bootstrap () #17 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0 #18 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6 #19 0x0000000000000000 in ?? () Thread 11 (Thread 1170794848 (LWP 8268)): #0 0x0000003f0b90a65f in sem_wait () from /lib64/tls/libpthread.so.0 #1 0x00007f0d88cdbeb8 in PyThread_acquire_lock () #2 0x00007f0d88cdf602 in lock_PyThread_acquire_lock () #3 0x00007f0d88cb19cb in PyEval_EvalFrameEx () #4 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #5 0x00007f0d88cb1eee in PyEval_EvalFrameEx () #6 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #7 0x00007f0d88cb1eee in PyEval_EvalFrameEx () #8 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx () #9 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx () #10 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #11 0x00007f0d88c457e8 in function_call () #12 0x00007f0d88c2189d in PyObject_Call () #13 0x00007f0d88c30c14 in instancemethod_call () #14 0x00007f0d88c2189d in PyObject_Call () #15 0x00007f0d88cac3e1 in PyEval_CallObjectWithKeywords () #16 0x00007f0d88cdff81 in t_bootstrap () #17 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0 #18 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6 #19 0x0000000000000000 in ?? () Thread 10 (Thread 1085798752 (LWP 8270)): #0 0x0000003f0b90a65f in sem_wait () from /lib64/tls/libpthread.so.0 #1 0x00007f0d88cdbeb8 in PyThread_acquire_lock () #2 0x00007f0d88cdf602 in lock_PyThread_acquire_lock () #3 0x00007f0d88cb19cb in PyEval_EvalFrameEx () #4 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #5 0x00007f0d88cb1eee in PyEval_EvalFrameEx () #6 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #7 0x00007f0d88cb1eee in PyEval_EvalFrameEx () #8 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx () #9 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx () #10 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #11 0x00007f0d88c457e8 in function_call () #12 0x00007f0d88c2189d in PyObject_Call () #13 0x00007f0d88c30c14 in instancemethod_call () #14 0x00007f0d88c2189d in PyObject_Call () #15 0x00007f0d88cac3e1 in PyEval_CallObjectWithKeywords () #16 0x00007f0d88cdff81 in t_bootstrap () #17 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0 #18 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6 #19 0x0000000000000000 in ?? () Thread 9 (Thread 1096288608 (LWP 8271)): #0 0x0000003f0b90a65f in sem_wait () from /lib64/tls/libpthread.so.0 #1 0x00007f0d88cdbeb8 in PyThread_acquire_lock () #2 0x00007f0d88cdf602 in lock_PyThread_acquire_lock () #3 0x00007f0d88cb19cb in PyEval_EvalFrameEx () #4 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #5 0x00007f0d88cb1eee in PyEval_EvalFrameEx () #6 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #7 0x00007f0d88cb1eee in PyEval_EvalFrameEx () #8 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx () #9 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx () #10 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #11 0x00007f0d88c457e8 in function_call () #12 0x00007f0d88c2189d in PyObject_Call () #13 0x00007f0d88c30c14 in instancemethod_call () #14 0x00007f0d88c2189d in PyObject_Call () #15 0x00007f0d88cac3e1 in PyEval_CallObjectWithKeywords () #16 0x00007f0d88cdff81 in t_bootstrap () #17 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0 #18 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6 #19 0x0000000000000000 in ?? () Thread 8 (Thread 1181284704 (LWP 8273)): #0 0x0000003f0b90a65f in sem_wait () from /lib64/tls/libpthread.so.0 #1 0x00007f0d88cdbeb8 in PyThread_acquire_lock () #2 0x00007f0d88cdf602 in lock_PyThread_acquire_lock () #3 0x00007f0d88cb19cb in PyEval_EvalFrameEx () #4 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #5 0x00007f0d88cb1eee in PyEval_EvalFrameEx () #6 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #7 0x00007f0d88cb1eee in PyEval_EvalFrameEx () #8 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx () #9 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx () #10 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #11 0x00007f0d88c457e8 in function_call () #12 0x00007f0d88c2189d in PyObject_Call () #13 0x00007f0d88c30c14 in instancemethod_call () #14 0x00007f0d88c2189d in PyObject_Call () #15 0x00007f0d88cac3e1 in PyEval_CallObjectWithKeywords () #16 0x00007f0d88cdff81 in t_bootstrap () #17 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0 #18 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6 #19 0x0000000000000000 in ?? () Thread 7 (Thread 1191774560 (LWP 8276)): #0 0x0000003f0b90a65f in sem_wait () from /lib64/tls/libpthread.so.0 #1 0x00007f0d88cdbeb8 in PyThread_acquire_lock () #2 0x00007f0d88cdf602 in lock_PyThread_acquire_lock () #3 0x00007f0d88cb19cb in PyEval_EvalFrameEx () #4 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #5 0x00007f0d88cb1eee in PyEval_EvalFrameEx () #6 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #7 0x00007f0d88cb1eee in PyEval_EvalFrameEx () #8 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx () #9 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx () #10 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #11 0x00007f0d88c457e8 in function_call () #12 0x00007f0d88c2189d in PyObject_Call () #13 0x00007f0d88c30c14 in instancemethod_call () #14 0x00007f0d88c2189d in PyObject_Call () #15 0x00007f0d88cac3e1 in PyEval_CallObjectWithKeywords () #16 0x00007f0d88cdff81 in t_bootstrap () #17 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0 #18 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6 #19 0x0000000000000000 in ?? () Thread 6 (Thread 1202264416 (LWP 8277)): #0 0x0000003f0b90a65f in sem_wait () from /lib64/tls/libpthread.so.0 #1 0x00007f0d88cdbeb8 in PyThread_acquire_lock () #2 0x00007f0d88cdf602 in lock_PyThread_acquire_lock () #3 0x00007f0d88cb19cb in PyEval_EvalFrameEx () #4 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #5 0x00007f0d88cb1eee in PyEval_EvalFrameEx () #6 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #7 0x00007f0d88cb1eee in PyEval_EvalFrameEx () #8 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx () #9 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx () #10 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #11 0x00007f0d88c457e8 in function_call () #12 0x00007f0d88c2189d in PyObject_Call () #13 0x00007f0d88c30c14 in instancemethod_call () #14 0x00007f0d88c2189d in PyObject_Call () #15 0x00007f0d88cac3e1 in PyEval_CallObjectWithKeywords () #16 0x00007f0d88cdff81 in t_bootstrap () #17 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0 #18 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6 #19 0x0000000000000000 in ?? () Thread 5 (Thread 1212754272 (LWP 8278)): #0 0x0000003f0b90a65f in sem_wait () from /lib64/tls/libpthread.so.0 #1 0x00007f0d88cdbeb8 in PyThread_acquire_lock () #2 0x00007f0d88cdf602 in lock_PyThread_acquire_lock () #3 0x00007f0d88cb19cb in PyEval_EvalFrameEx () #4 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #5 0x00007f0d88cb1eee in PyEval_EvalFrameEx () #6 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #7 0x00007f0d88cb1eee in PyEval_EvalFrameEx () #8 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx () #9 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx () #10 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #11 0x00007f0d88c457e8 in function_call () #12 0x00007f0d88c2189d in PyObject_Call () #13 0x00007f0d88c30c14 in instancemethod_call () #14 0x00007f0d88c2189d in PyObject_Call () #15 0x00007f0d88cac3e1 in PyEval_CallObjectWithKeywords () #16 0x00007f0d88cdff81 in t_bootstrap () #17 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0 #18 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6 #19 0x0000000000000000 in ?? () Thread 4 (Thread 1223244128 (LWP 8279)): #0 0x0000003f0b90a65f in sem_wait () from /lib64/tls/libpthread.so.0 #1 0x00007f0d88cdbeb8 in PyThread_acquire_lock () #2 0x00007f0d88cdf602 in lock_PyThread_acquire_lock () #3 0x00007f0d88cb19cb in PyEval_EvalFrameEx () #4 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #5 0x00007f0d88cb1eee in PyEval_EvalFrameEx () #6 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #7 0x00007f0d88cb1eee in PyEval_EvalFrameEx () #8 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx () #9 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx () #10 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #11 0x00007f0d88c457e8 in function_call () #12 0x00007f0d88c2189d in PyObject_Call () #13 0x00007f0d88c30c14 in instancemethod_call () #14 0x00007f0d88c2189d in PyObject_Call () #15 0x00007f0d88cac3e1 in PyEval_CallObjectWithKeywords () #16 0x00007f0d88cdff81 in t_bootstrap () #17 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0 #18 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6 #19 0x0000000000000000 in ?? () Thread 3 (Thread 1233733984 (LWP 8280)): #0 0x0000003f0b90a65f in sem_wait () from /lib64/tls/libpthread.so.0 #1 0x00007f0d88cdbeb8 in PyThread_acquire_lock () #2 0x00007f0d88cdf602 in lock_PyThread_acquire_lock () #3 0x00007f0d88cb19cb in PyEval_EvalFrameEx () #4 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #5 0x00007f0d88cb1eee in PyEval_EvalFrameEx () #6 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #7 0x00007f0d88cb1eee in PyEval_EvalFrameEx () #8 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx () #9 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx () #10 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #11 0x00007f0d88c457e8 in function_call () #12 0x00007f0d88c2189d in PyObject_Call () #13 0x00007f0d88c30c14 in instancemethod_call () #14 0x00007f0d88c2189d in PyObject_Call () #15 0x00007f0d88cac3e1 in PyEval_CallObjectWithKeywords () #16 0x00007f0d88cdff81 in t_bootstrap () #17 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0 #18 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6 #19 0x0000000000000000 in ?? () Thread 2 (Thread 1244223840 (LWP 8281)): #0 0x0000003f0b90a65f in sem_wait () from /lib64/tls/libpthread.so.0 #1 0x00007f0d88cdbeb8 in PyThread_acquire_lock () #2 0x00007f0d88cdf602 in lock_PyThread_acquire_lock () #3 0x00007f0d88cb19cb in PyEval_EvalFrameEx () #4 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #5 0x00007f0d88cb1eee in PyEval_EvalFrameEx () #6 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #7 0x00007f0d88cb1eee in PyEval_EvalFrameEx () #8 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx () #9 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx () #10 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #11 0x00007f0d88c457e8 in function_call () #12 0x00007f0d88c2189d in PyObject_Call () #13 0x00007f0d88c30c14 in instancemethod_call () #14 0x00007f0d88c2189d in PyObject_Call () #15 0x00007f0d88cac3e1 in PyEval_CallObjectWithKeywords () #16 0x00007f0d88cdff81 in t_bootstrap () #17 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0 #18 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6 #19 0x0000000000000000 in ?? () Thread 1 (Thread 139696105322208 (LWP 7916)):================================》主线程 #0 0x0000003f0b0bd0c2 in poll () from /lib64/tls/libc.so.6 #1 0x00007f0d853c5730 in internal_select () #2 0x00007f0d853c61ca in sock_accept () #3 0x00007f0d88cafa21 in PyEval_EvalFrameEx () #4 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx () #5 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx () #6 0x00007f0d88cb2ce6 in PyEval_EvalFrameEx () #7 0x00007f0d88cb3624 in PyEval_EvalCodeEx () #8 0x00007f0d88cb3822 in PyEval_EvalCode () #9 0x00007f0d88ccc49c in run_mod () #10 0x00007f0d88cccafc in PyRun_FileExFlags () #11 0x00007f0d88ccda11 in PyRun_SimpleFileExFlags () #12 0x00007f0d88cddd26 in Py_Main () #13 0x0000003f0b01c4bb in __libc_start_main () from /lib64/tls/libc.so.6 #14 0x00000000004006da in _start () #15 0x00007fff62a773d8 in ?? () #16 0x000000000000001c in ?? () #17 0x0000000000000003 in ?? () #18 0x00007fff62a776db in ?? () #19 0x00007fff62a776e5 in ?? () #20 0x00007fff62a7771e in ?? () #21 0x0000000000000000 in ?? () ``` #### 1.2.4 查看 Butterfly stack trace 此功能在 handler 中进行实现，代码见 /handler/demo\_stackdump > 使用 ``` $ curl "http://127.0.0.1:8585/demo_stackdump/stackdump" stack trace 日志记录在 logs/stack.log ``` > 例子 (butterfly/logs/stack.log 内容） ``` ---------------------------------------------------------------- dump_time=2021-04-09 12:19:55 reqid=2A5BE75641F1BC5A 1 ActiveThreadCount: 18 2 KnownActiveThreadNames: MainThread-------------------------------- 主线程 APScheduler------------------------------- 任务调度线程 CP Server Thread-1------------------------Butterfly handler 处理线程，以下类似 CP Server Thread-11 CP Server Thread-5 CP Server Thread-9 CP Server Thread-14 CP Server Thread-7 CP Server Thread-13 CP Server Thread-15 CP Server Thread-3 CP Server Thread-2 CP Server Thread-4 CP Server Thread-6 CP Server Thread-12 CP Server Thread-8 CP Server Thread-10 CP Server Thread-16 3 ThreadInfo: Thread: 1082784096 CP Server Thread-16 File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap self.__bootstrap_inner() File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner self.run() File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1643, in run conn = self.server.requests.get() File: "/home/users/meetbill/.jumbo/lib/python2.7/Queue.py", line 168, in get self.not_empty.wait() File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 243, in wait waiter.acquire() Thread: 1093196128 CP Server Thread-15 File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap self.__bootstrap_inner() File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner self.run() File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1643, in run conn = self.server.requests.get() File: "/home/users/meetbill/.jumbo/lib/python2.7/Queue.py", line 168, in get self.not_empty.wait() File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 243, in wait waiter.acquire() Thread: 1089005920 CP Server Thread-3 File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap self.__bootstrap_inner() File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner self.run() File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1643, in run conn = self.server.requests.get() File: "/home/users/meetbill/.jumbo/lib/python2.7/Queue.py", line 168, in get self.not_empty.wait() File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 243, in wait waiter.acquire() Thread: 1076640096 CP Server Thread-13 File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap self.__bootstrap_inner() File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner self.run() File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1643, in run conn = self.server.requests.get() File: "/home/users/meetbill/.jumbo/lib/python2.7/Queue.py", line 168, in get self.not_empty.wait() File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 243, in wait waiter.acquire() Thread: 1103497568 CP Server Thread-6 File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap self.__bootstrap_inner() File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner self.run() File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1643, in run conn = self.server.requests.get() File: "/home/users/meetbill/.jumbo/lib/python2.7/Queue.py", line 168, in get self.not_empty.wait() File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 243, in wait waiter.acquire() Thread: 1094404448 CP Server Thread-5 File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap self.__bootstrap_inner() File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner self.run() File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1643, in run conn = self.server.requests.get() File: "/home/users/meetbill/.jumbo/lib/python2.7/Queue.py", line 168, in get self.not_empty.wait() File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 243, in wait waiter.acquire() Thread: 1100007776 CP Server Thread-2 File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap self.__bootstrap_inner() File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner self.run() File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1643, in run conn = self.server.requests.get() File: "/home/users/meetbill/.jumbo/lib/python2.7/Queue.py", line 168, in get self.not_empty.wait() File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 243, in wait waiter.acquire() Thread: 140004998178528 MainThread File: "/home/users/meetbill/butterfly/main.py", line 30, in server.start() File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 2104, in start self.tick() File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 2158, in tick s, addr = self.socket.accept() File: "/home/users/meetbill/.jumbo/lib/python2.7/socket.py", line 202, in accept sock, addr = self._sock.accept() Thread: 1102469472 CP Server Thread-7 File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap self.__bootstrap_inner() File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner self.run() File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1643, in run conn = self.server.requests.get() File: "/home/users/meetbill/.jumbo/lib/python2.7/Queue.py", line 168, in get self.not_empty.wait() File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 243, in wait waiter.acquire() Thread: 1095801184 CP Server Thread-8 File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap self.__bootstrap_inner() File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner self.run() File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1643, in run conn = self.server.requests.get() File: "/home/users/meetbill/.jumbo/lib/python2.7/Queue.py", line 168, in get self.not_empty.wait() File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 243, in wait waiter.acquire() Thread: 1100343648 CP Server Thread-11 File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap self.__bootstrap_inner() File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner self.run() File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1643, in run conn = self.server.requests.get() File: "/home/users/meetbill/.jumbo/lib/python2.7/Queue.py", line 168, in get self.not_empty.wait() File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 243, in wait waiter.acquire() Thread: 1075272032 CP Server Thread-14 File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap self.__bootstrap_inner() File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner self.run() File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1643, in run conn = self.server.requests.get() File: "/home/users/meetbill/.jumbo/lib/python2.7/Queue.py", line 168, in get self.not_empty.wait() File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 243, in wait waiter.acquire() Thread: 1100863840 APScheduler File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap self.__bootstrap_inner() File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner self.run() File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 504, in run self.__target(*self.__args, **self.__kwargs) File: "/home/users/meetbill/butterfly/xlib/apscheduler/schedulers/blocking.py", line 42, in _main_loop self._event.wait(wait_seconds) File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 403, in wait self.__cond.wait(timeout) File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 262, in wait _sleep(delay) Thread: 1084520800 CP Server Thread-4 File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap self.__bootstrap_inner() File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner self.run() File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1643, in run conn = self.server.requests.get() File: "/home/users/meetbill/.jumbo/lib/python2.7/Queue.py", line 168, in get self.not_empty.wait() File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 243, in wait waiter.acquire() Thread: 1097337184 CP Server Thread-9 File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap self.__bootstrap_inner() File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner self.run() File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1643, in run conn = self.server.requests.get() File: "/home/users/meetbill/.jumbo/lib/python2.7/Queue.py", line 168, in get self.not_empty.wait() File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 243, in wait waiter.acquire() Thread: 1091410272 CP Server Thread-12 File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap self.__bootstrap_inner() File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner self.run() File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1643, in run conn = self.server.requests.get() File: "/home/users/meetbill/.jumbo/lib/python2.7/Queue.py", line 168, in get self.not_empty.wait() File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 243, in wait waiter.acquire() Thread: 1087252832 CP Server Thread-10 File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap self.__bootstrap_inner() File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner self.run() File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1643, in run conn = self.server.requests.get() File: "/home/users/meetbill/.jumbo/lib/python2.7/Queue.py", line 168, in get self.not_empty.wait() File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 243, in wait waiter.acquire() Thread: 1099741536 CP Server Thread-1 File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 524, in __bootstrap self.__bootstrap_inner() File: "/home/users/meetbill/.jumbo/lib/python2.7/threading.py", line 551, in __bootstrap_inner self.run() File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1651, in run conn.communicate() File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 1460, in communicate req.respond() File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 901, in respond self.server.gateway(self).respond() File: "/home/users/meetbill/butterfly/xlib/cherrypy_wsgiserver/__init__.py", line 2426, in respond response = self.req.server.wsgi_app(self.env, self.start_response) File: "/home/users/meetbill/butterfly/wsgiapp.py", line 161, in application status, headders, content = wsgigw.process(environ) File: "/home/users/meetbill/butterfly/xlib/httpgateway.py", line 195, in process httpstatus, headers, content = protocol(req) File: "/home/users/meetbill/butterfly/xlib/protocol_json.py", line 171, in __call__ ret = self._func(**params) File: "/home/users/meetbill/butterfly/handlers/demo_stackdump/__init__.py", line 79, in stackdump for filename, lineno, name, line in traceback.extract_stack(stack): ``` ## 2 gdb ### 2.1 使用方法 ``` gdb python ``` #### 2.1.1 调试进程 > 查看 C 调用栈 ``` (gdb) bt ``` > 查看 Python 调用栈 ``` (gdb) py-bt ``` ### 2.2 官方文档 ### 2.3 查看 core 保存路径 ``` # ulimit -c unlimited # cat /proc/sys/kernel/core_pattern /home/work/core.%e.%p.%t ``` > * ulimit -c 是查看创建的核心转储的最大大小 > * 0: 不保存 > * unlimited: 不限制大小 > * cat /proc/sys/kernel/core\_pattern 生成的文件路径 ### 2.4 查看 core 文件例子 ``` #gdb `which python ` core_path ``` > 例子 ``` # gdb `which python ` core.python2.7.25133.1617075053 ... gdb info ... [New LWP 25192] [New LWP 25193] [New LWP 25195] [New LWP 25196] [New LWP 25197] [New LWP 25194] [New LWP 25198] [New LWP 25199] [New LWP 25201] [New LWP 25200] [New LWP 25202] [New LWP 25204] [New LWP 25206] [New LWP 25203] [New LWP 25205] [New LWP 25207] [New LWP 25191] [New LWP 25208] [New LWP 25133] warning: File "/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load". To enable execution of this file add add-auto-load-safe-path /lib64/libthread_db-1.0.so line to your configuration file "/home/work/.gdbinit". To completely disable this security protection add set auto-load safe-path / line to your configuration file "/home/work/.gdbinit". For more information about this security protection see the "Auto-loading safe path" section in the GDB manual. E.g., run from the shell: info "(gdb)Auto-loading safe path" warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available. warning: File "/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load". warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available. Core was generated by `python2.7 /home/work/chunfeng/chunfeng_app2_4600/main.py 954216e9'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f85e85265c5 in PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2995 2995 Python/ceval.c: No such file or directory. [Current thread is 1 (LWP 25192)] (gdb) ``` 可以看到程序是在停止的时候，在如下部分发生的问题 ``` #0 0x00007f85e85265c5 in PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2995 ``` ``` 可以输入 bt 查看更多，其从上到下是错误从底层到最外层的顺序。 ``` ## 3 lsof(lists openfiles) 在 Unix 中一切（包括网络套接口）都是文件。lsof 的功能是列出打开文件（lists openfiles） ### 3.1 使用 -p 查看指定进程 ID 已打开的文件 > lsof -p {PID} ``` COMMAND ：进程的名称 PID ：进程标识符 USER ：进程所有者 FD ：文件描述符，应用程序通过文件描述符识别该文件。如 cwd、txt 等 (1) cwd 值表示应用程序的当前工作目录，这是该应用程序启动的目录，除非它本身对这个目录进行更改 (2) txt 类型的文件是程序代码，如应用程序二进制文件本身或共享库，其次数值表示应用程序的文件描述符，这是打开该文件时返回的一个整数。 (3) u 表示该文件被打开并处于读取 / 写入模式，而不是只读 r 或只写 (w) 模式。同时还有大写 的 W 表示该应用程序具有对整个文件的写锁。 该文件描述符用于确保每次只能打开一个应用程序实例。 初始打开每个应用程序时，都具有三个文件描述符，从 0 到 2，分别表示标准输入、输出和错误流。 所以大多数应用程序所打开的文件的 FD 都是从 3 开始。 TYPE ：文件类型，如 DIR、REG 等 (1) 文件和目录分别称为 REG 和 DIR (2) CHR 表示字符 (3) BLK 表示块设备 (4) UNIX -- UNIX 域套接字 FIFO -- 先进先出 (FIFO) 队列 IPv4 -- 网际协议 (IP) 套接字 DEVICE ：指定磁盘的名称 SIZE ：文件的大小 NODE ：索引节点（文件在磁盘上的标识） NAME ：打开文件的确切名称 ``` ### 3.2 使用 @host:port 显示基于主机与端口的连接 ``` # lsof -i@xx.xx.xx.xx:22 ``` ## 4 mem\_top 排查内存泄露问题 ``` kill -12 ``` 将会将将日志打印在 logs/common/common.log > * refs 回显信息：占用内存最大的 topN 变量中的元素个数 > * bytes 回显信息：占用内存最大的 topN 变量每个变量所占用的 bytes > * types 回显信息：占用内存最大的 topN 类型变量个数 ### 4.1 例子 ``` refs: 49452 {'xlib.util.pluginbase._internalspace._spef7874fe95975333c5ff649d4c981dca.vm_manager': {'_sp349460523b2000c7ebcfd35e0025e291': , '_spdf494a60169fd9552a956256d05ccc00': {'TOO_MANY_KEY_PARTS': 1070, 'WRONG_PARAMCOUNT_TO_PROCEDURE': 1107, 'SP_COND_MISMATCH': 1319, 'XAER_OUTSIDE': 1400, 'KILL_DENIED_ERROR': 1095, 'TABLE_CANT_HANDLE_SPKEYS': 1464, 'DUMP_NOT_IMPLEMENTED': 570 {'SocketType': , 'getaddrinfo': , 'AI_NUMERICSERV': 1024, 'PACKET_OTHERHOST': 3, 'AI_NUMERICHOST': 4, 'StringIO': {'DateField': , 'ModelNamedTupleCursorWrapper': , 'ColumnMetadata': , 'Bit 496 {'restore': , 'zrangebylex': , 'renamenx': , 'client_id': , 'acl_save 490 {'SocketType': , 'getaddrinfo': , 'IPPROTO_RAW': 255, 'AI_NUMERICSERV': 1024, 'PACKET_OTHERHOST': 3, 'AI_NUMERICHOST': 4, 'IPPORT_RESERVED': 1024, 490 {'SocketType': , 'getaddrinfo': , 'AI_NUMERICSERV': 1024, 'PACKET_OTHERHOST': 3, 'AI_NUMERICHOST': 4, 'IPPORT_RESERVED': 1024, 'AF_IRDA': 23, 'MSG 462 {'tcsendbreak': , 'B110': 3, 'tcdrain': , 'CIBAUD': 269418496, 'FF1': 32768, 'FF0': 0, 'FIONCLEX': 21584, 'ONLCR': 4, 'B600': 8, 'TCSANOW': 0, 462 {'tcsendbreak': , 'B110': 3, 'tcdrain': , 'TCIOFLUSH': 2, 'FF1': 32768, 'FF0': 0, 'FIONCLEX': 21584, 'ONLCR': 4, 'B600': 8, 'TCSANOW': 0, 'TIO 448 {'WTERMSIG': , 'lseek': , 'EX_IOERR': 74, 'EX_NOHOST': 68, 'seteuid': , 'pathsep': ':', 'execle': [, , {'empty_provider': , 'ExtractionError': , 'functools': {u'UNWATCH': , u'SLAVEOF': , u'OBJECT': , u'GEODIST': , u'EXPIRE' 359 {u'UNWATCH': , u'SLAVEOF': , u'OBJECT': , u'GEODIST': , u'EXPIRE' 358 {u'UNWATCH': , u'SLAVEOF': , u'MOVE': , u'GEODIST': , u'TIME': at 0x2 338 {'WTERMSIG': , 'lseek': , 'EX_IOERR': 74, 'EX_NOHOST': 68, 'seteuid': , 'O_RSYNC': 1052672, 'fstatvfs': {'WTERMSIG': , 'lseek': , 'EX_IOERR': 74, 'EX_NOHOST': 68, 'getpgrp': , 'major': , 'fstatvfs': 286 {'bytearray': , 'IndexError': , 'all': , 'help': Type help() for interactive help, or help(object) for help about object., 'vars' 286 {1: Charset(id=1, name='big5', collation='big5_chinese_ci'), 2: Charset(id=2, name='latin2', collation='latin2_czech_cs'), 3: Charset(id=3, name='dec8', collation='dec8_swedish_ci'), 4: Charset(id=4, bytes: 3146008 {'xlib.util.pluginbase._internalspace._spef7874fe95975333c5ff649d4c981dca.vm_manager': , '_spdf494a60169fd9552a956256d05ccc00': , 'IndexError': , 'all': , 'help': Type help() for interactive help, or help(object) for help about object., 'vars' 12568 {'bytearray': , 'all': , 'vars': , 'SyntaxError': , 'unicode': , 'sorted': , 'lseek': , 'EX_IOERR': 74, 'EX_NOHOST': 68, 'seteuid': , 'pathsep': ':', 'execle': , 'lseek': , 'EX_IOERR': 74, 'EX_NOHOST': 68, 'seteuid': , 'O_RSYNC': 1052672, 'fstatvfs': , 'lseek': , 'EX_IOERR': 74, 'EX_NOHOST': 68, 'getpgrp': , 'major': , 'fstatvfs': 12568 {'isstring': , '_optimize_unicode': , 'SRE_FLAG_VERBOSE': 64, '_identityfunction': , 'CATEGORY_LOC_NOT_ 12568 {'REPEAT_ONE': 'repeat_one', 'makedict': , 'CATEGORY_LOC_NOT_WORD': 'category_loc_not_word', 'SRE_INFO_CHARSET': 4, 'SRE_INFO_PREFIX': 1, 'SRE_FLAG_DEBUG': 128, 'M 12568 {'latin_1_encode': , 'getreader': , 'readbuffer_encode': , 'BOM': '\xff\xfe', 'StreamWriter 12568 {'functools': , 'PY3': False, 'ModuleSixMovesUrllibRobotparser': , 'get_ 12568 {'gt': , '__concat__': , 'imul': , 'ge': , '__irshift__': , 'setslice': 12568 {'Union': , 'c_wchar': , 'c_bool': , 'c_double': , 'CFUNCTYPE': , '_ 12568 {'logMultiprocessing': 1, '__path__': ['/home/work/.jumbo/lib/python2.7/logging'], 'LogRecord': , 'logProcesses': 1, 'addLevelName': , '_ 12568 {'BadStatusLine': , 'UnknownProtocol': , '_UNKNOWN': 'UNKNOWN', 'SWITCHING_PROTOCOLS': 101, 'urlsplit': 12568 {'SocketType': , 'getaddrinfo': , 'AI_NUMERICSERV': 1024, 'PACKET_OTHERHOST': 3, 'AI_NUMERICHOST': 4, 'StringIO': , 'getaddrinfo': , 'IPPROTO_RAW': 255, 'AI_NUMERICSERV': 1024, 'PACKET_OTHERHOST': 3, 'AI_NUMERICHOST': 4, 'IPPORT_RESERVED': 1024, types: 44136 30965 21183 11067 5348 3270 2961 2961 2961 1694 1234 1108 905 842 711 449 427 381 198 168 ``` ## 5 传送门 ### 5.1 根据 python 代码自动生成流程图 > * [code2flow](https://github.com/scottrogowski/code2flow) – 把你的 Python 和 JavaScript 代码转换为流程图。 > * [pycallgraph](https://github.com/gak/pycallgraph) - 这个库可以把你的 Python 应用的流程（调用图）进行可视化。 ### 5.2 GDB 教程 * [Low-level Python debugging with GDB](http://grapsus.net/blog/?q=gdb) ### 5.3 mem\_top * --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://meetbill.gitbook.io/butterfly-project-doc/project-framework/how/butterfly-debug.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.