Linux內(nèi)核為了提高內(nèi)存的使用效率采用過度分配內(nèi)存(over-commit memory)的辦法,造成物理內(nèi)存過度緊張進(jìn)而觸發(fā)OOM機(jī)制來殺死一些進(jìn)程回收內(nèi)存。該機(jī)制會(huì)監(jiān)控那些占用內(nèi)存過大,尤其是瞬間很快消耗大量內(nèi)存的進(jìn)程,為了防止內(nèi)存耗盡會(huì)把該進(jìn)程殺掉。
1、oom過程
out_of_memory函數(shù)的代碼邏輯還是非常簡單清晰的,總共有兩步
- 1.先選擇一個(gè)要?dú)⑺赖倪M(jìn)程,
- 2.殺死它。
oom_kill_process函數(shù)的目的很簡單,但是實(shí)現(xiàn)過程也有點(diǎn)復(fù)雜,這里就不展開分析了,大家可以自行去看一下代碼。我們重點(diǎn)分析一下select_bad_process函數(shù)的邏輯,select_bad_process主要是依靠oom_score來進(jìn)行進(jìn)程選擇的。
我們先來看一下和oom_score有關(guān)的三個(gè)文件。
/proc//oom_score 系統(tǒng)計(jì)算出來的oom_score值,只讀文件,取值范圍0 –- 1000,0代表never kill,1000代表aways kill,值越大,進(jìn)程被選中的概率越大。
/proc//oom_score_adj 讓用戶空間調(diào)節(jié)oom_score的接口,root可讀寫,取值范圍 -1000 --- 1000,默認(rèn)為0,若為 -1000,則oom_score加上此值一定小于等于0,從而變成never kill進(jìn)程。OS可以把一些關(guān)鍵的系統(tǒng)進(jìn)程的oom_score_adj設(shè)為-1000,從而避免被oom kill。
/proc//oom_adj 舊的接口文件,為兼容而保留,root可讀寫,取值范圍 -16 — 15,會(huì)被線性映射到oom_score_adj,特殊值 -17代表 OOM_DISABLE。大家盡量不要再用此接口。
Linux內(nèi)存管理 (21)OOM:https://www.cnblogs.com/arnoldlu/p/8567559.html
2、oom配置
2.1 /proc/sys/vm/overcommit_memory
內(nèi)核參數(shù) vm.overcommit_memory 接受三種取值:
- 0 – Heuristic overcommit handling. 這是缺省值,它允許overcommit,但過于明目張膽的overcommit會(huì)被拒絕,比如malloc一次性申請(qǐng)的內(nèi)存大小就超過了系統(tǒng)總內(nèi)存。Heuristic的意思是“試探式的”,內(nèi)核利用某種算法猜測(cè)你的內(nèi)存申請(qǐng)是否合理,它認(rèn)為不合理就會(huì)拒絕overcommit。
- 1 – Always overcommit. 允許overcommit,對(duì)內(nèi)存申請(qǐng)來者不拒。內(nèi)核執(zhí)行無內(nèi)存過量使用處理。使用這個(gè)設(shè)置會(huì)增大內(nèi)存超載的可能性,但也可以增強(qiáng)大量使用內(nèi)存任務(wù)的性能。
- 2 – Don’t overcommit. 禁止overcommit。 內(nèi)存拒絕等于或者大于總可用 swap 大小以及 overcommit_ratio 指定的物理 RAM 比例的內(nèi)存請(qǐng)求。如果您希望減小內(nèi)存過度使用的風(fēng)險(xiǎn),這個(gè)設(shè)置就是最好的。
遇到問題,線程里面一直fork()執(zhí)行內(nèi)容再退出之后,會(huì)出現(xiàn)fork失敗。
由于fork的時(shí)候會(huì)拷貝一份父進(jìn)程的內(nèi)存,所以如果一開始正常fork到后面fork失敗,那就說明一定是這個(gè)進(jìn)程有內(nèi)存泄露問題,因?yàn)閮?nèi)存泄露導(dǎo)致進(jìn)程內(nèi)存變大,fork的時(shí)候拷貝的父進(jìn)程內(nèi)存也就更大,導(dǎo)致內(nèi)存不足,fork失敗。
linux - fork() 因內(nèi)存不足錯(cuò)誤而失敗:https://www.coder.work/article/167298
2.2 /proc/sys/vm/panic_on_oom
決定系統(tǒng)出現(xiàn)oom的時(shí)候,要做的操作。接受的三種取值如下:
- 0 - 默認(rèn)值,當(dāng)出現(xiàn)oom的時(shí)候,觸發(fā)oom killer
- 1 - 程序在有cpuset、memory policy、memcg的約束情況下的OOM,可以考慮不panic,而是啟動(dòng)OOM killer。其它情況觸發(fā) kernel panic,即系統(tǒng)直接重啟
- 2 - 當(dāng)出現(xiàn)oom,直接觸發(fā)kernel panic,即系統(tǒng)直接重啟
2.3 /proc/sys/vm/min_free_kbytes
- 代表系統(tǒng)所保留空閑內(nèi)存的最低限。
min_free_kbytes設(shè)的越大,watermark的線越高,同時(shí)三個(gè)線之間的buffer量也相應(yīng)會(huì)增加。這意味著會(huì)較早的啟動(dòng)kswapd進(jìn)行回收,且會(huì)回收上來較多的內(nèi)存(直至watermark[high]才會(huì)停止),這會(huì)使得系統(tǒng)預(yù)留過多的空閑內(nèi)存,從而在一定程度上降低了應(yīng)用程序可使用的內(nèi)存量。極端情況下設(shè)置min_free_kbytes接近內(nèi)存大小時(shí),留給應(yīng)用程序的內(nèi)存就會(huì)太少而可能會(huì)頻繁地導(dǎo)致OOM的發(fā)生。
min_free_kbytes設(shè)的過小,則會(huì)導(dǎo)致系統(tǒng)預(yù)留內(nèi)存過小。kswapd回收的過程中也會(huì)有少量的內(nèi)存分配行為(會(huì)設(shè)上PF_MEMALLOC)標(biāo)志,這個(gè)標(biāo)志會(huì)允許kswapd使用預(yù)留內(nèi)存;另外一種情況是被OOM選中殺死的進(jìn)程在退出過程中,如果需要申請(qǐng)內(nèi)存也可以使用預(yù)留部分。這兩種情況下讓他們使用預(yù)留內(nèi)存可以避免系統(tǒng)進(jìn)入deadlock狀態(tài)。
3.vm參數(shù)推薦配置
# vm
vm.min_free_kbytes=4096
vm.vfs_cache_pressure=200
vm.dirty_background_ratio=5
vm.dirty_ratio=10
vm.dirty_expire_centisecs=500
vm.dirty_writeback_centisecs=200
vm.extfrag_threshold=10
vm.panic_on_oom=1
/proc/sys/vm/ 文檔介紹:https://zhuanlan.zhihu.com/p/503579974?utm_id=0
五萬字 | 深入理解Linux內(nèi)存管理:https://mp.weixin.qq.com/s/nlMGEhuaDUYqV6r8A4cRlA
記一次linux oom內(nèi)存溢出排查過程:https://blog.csdn.net/hu_jinghui/article/details/81740575
4.都有內(nèi)存回收機(jī)制了,為啥還會(huì)oom
- 后臺(tái)內(nèi)存回收(kswapd)
- 直接內(nèi)存回收(direct reclaim)
- OOM機(jī)制(Out of Memory)
kswapd 是一個(gè)內(nèi)核線程,在內(nèi)存不足時(shí)負(fù)責(zé)在后臺(tái)進(jìn)行內(nèi)存回收,這個(gè)過程發(fā)生在后臺(tái),因此是異步發(fā)生,不會(huì)阻塞進(jìn)程。
- 當(dāng)內(nèi)存大于 pages_low 時(shí),表示此時(shí)系統(tǒng)內(nèi)存足夠,不會(huì)進(jìn)行內(nèi)存回收。
- 當(dāng)內(nèi)存小于 pages_low 時(shí),表示此時(shí)內(nèi)存存在壓力,會(huì)觸發(fā) kswapd0 進(jìn)行后臺(tái)內(nèi)存回收,直到 pages_high 為止。
- 當(dāng)內(nèi)存小于 pages_min 時(shí),表示此時(shí)用戶內(nèi)存耗盡,會(huì)觸發(fā)直接內(nèi)存回收,進(jìn)程被阻塞。
如果要調(diào)整 kswapd 的觸發(fā)時(shí)機(jī),需要修改 pages_low 的值,而pages_low的值由pages_min計(jì)算,因此需要修改pages_min。
相關(guān)參數(shù)調(diào)整
1、內(nèi)核參數(shù)vm.swappiness,決定回收緩存或swap機(jī)制回收內(nèi)存的傾向取值范圍是 0-100,
- 數(shù)值越大,越積極使用 Swap,也就是更傾向于回收匿名頁;
- 數(shù)值越小,越消極使用 Swap,也就是更傾向于回收文件頁。
- 0不代表不使用swap,當(dāng)剩余內(nèi)存 + 文件頁小于頁高閾值時(shí),還是會(huì)發(fā)生 Swap。
2、內(nèi)核參數(shù)vm.min_free_kbytes,調(diào)整內(nèi)存水位
- pages_min = min_free_kbytes換算為page單位,
- pages_low = pages_min*5/4
- pages_high = pages_min*3/2
所以當(dāng)內(nèi)存消耗的速度比回收的機(jī)制快的時(shí)候就會(huì)直接觸發(fā)oom,比如dhcp flood驅(qū)動(dòng)沒有限制住流量的時(shí)候?;蛘呔褪莾?nèi)存都被消耗了,實(shí)在釋放不出來了,那也只能oom。
【Linux內(nèi)核】內(nèi)存管理——內(nèi)存回收機(jī)制:https://blog.csdn.net/weixin_45636061/article/details/127184818
出現(xiàn)oom實(shí)際日志:
[940027.316884] PTK:be634a2126f098a15a84324cb8fdc92cd05288ac3566b4d14290511de834cbc682d1ff2f7b0ca76ce0e9f956265658f3a8f5e6941538e457bbd020b27733e244
[940027.330955] peer_msg4: 33781 usec
[940029.263407] ACT - SendBSS2040CoexistMgmtAction(BSSCoexist2040=0x4)
[940029.270498] hw_ctrl_flow_v2_peer_update: wdev_idx=0
[940030.898046] p1905_managerd invoked oom-killer: gfp_mask=0x2420848, order=0, oom_score_adj=0
[940030.906882] CPU: 0 PID: 14089 Comm: p1905_managerd Not tainted 4.4.198 #0
[940030.913781] Stack : 81d168c2 0000003d 00000000 00000000 81d168c2 00000000 00000000 00000000
[940030.913781] 8358ecf4 819086e3 817d4110 00000000 00003709 81d13690 00000003 00000001
[940030.913781] 819087e0 8107446c 00000000 00000004 00000006 00000000 817db220 857cb87c
[940030.913781] 00000000 8107213c 81d168c2 0000004f 00000019 0014f000 81d168c2 007cb87c
[940030.913781] 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[940030.913781] ...
[940030.949718] Call Trace:
[940030.952288] [<810188b4>] show_stack+0x54/0x88
[940030.956791] [<811e4bd4>] dump_stack+0x8c/0xc4
[940030.961275] [<810b17d8>] dump_header.isra.4+0x54/0x19c
[940030.966546] [<810b1d68>] oom_kill_process+0xf0/0x578
[940030.971643] [<810b2504>] out_of_memory+0x314/0x3a8
[940030.976555] [<810b67e0>] __alloc_pages_nodemask+0x7dc/0x824
[940030.982249] [<810ae69c>] pagecache_get_page+0x1b8/0x288
[940030.987656] [<81122114>] __getblk_slow+0x1ac/0x3e0
[940030.992635] [<81153e80>] squashfs_bio_submit+0x1e4/0x5c8
[940030.998132] [<81154938>] __squashfs_read_data+0x264/0x298
[940031.003710] [<81154a40>] squashfs_read_data_async+0x28/0x34
[940031.009470] [<81158cd0>] squashfs_readpages_block+0x330/0x378
[940031.015404] [<811565c4>] __squashfs_readpages.isra.5+0x6d4/0x914
[940031.021595] [<81156824>] squashfs_readpages+0x20/0x30
[940031.026829] [<810bab20>] __do_page_cache_readahead+0x1a8/0x288
[940031.032860] [<810b0a1c>] filemap_fault+0x1e0/0x500
[940031.037834] [<810d04fc>] __do_fault+0x64/0xd4
[940031.042371] [<810d3f60>] handle_mm_fault+0x544/0xdfc
[940031.047546] [<81021c80>] __do_page_fault+0x138/0x48c
[940031.052689] [<81005420>] ret_from_exception+0x0/0x10
[940031.057788]
[940031.060888] Mem-Info:
[940031.063598] active_anon:1071 inactive_anon:18 isolated_anon:0
[940031.063598] active_file:431 inactive_file:482 isolated_file:70
[940031.063598] unevictable:0 dirty:0 writeback:0 unstable:0
[940031.063598] slab_reclaimable:491 slab_unreclaimable:4198
[940031.063598] mapped:222 shmem:27 pagetables:110 bounce:0
[940031.063598] free:4122 free_pcp:0 free_cma:0
[940031.096924] DMA free:2744kB min:2304kB low:2880kB high:3456kB active_anon:380kB inactive_anon:16kB active_file:200kB inactive_file:236kB unevictable:0kB isolated(anon):0kB isolated(file):100kB present:16384kB managed:16380kB mlocked:0kB dirty:0kB writeback:0kB mapped:40kB shmem:20kB slab_reclaimable:396kB slab_unreclaimable:2520kB kernel_stack:104kB pagetables:44kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[940031.140948] lowmem_reserve[]: 0 97 97
[940031.145100] Normal free:13712kB min:14076kB low:17592kB high:21112kB active_anon:3904kB inactive_anon:56kB active_file:1556kB inactive_file:1488kB unevictable:0kB isolated(anon):0kB isolated(file):352kB present:114688kB managed:99936kB mlocked:0kB dirty:0kB writeback:0kB mapped:792kB shmem:88kB slab_reclaimable:1592kB slab_unreclaimable:14292kB kernel_stack:760kB pagetables:396kB unstable:0kB bounce:0kB free_pcp:100kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:376 all_unreclaimable? no
[940031.190330] lowmem_reserve[]: 0 0 0
[940031.194458] DMA: 94*4kB (U) 76*8kB (UM) 49*16kB (U) 29*32kB (U) 1*64kB (U) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2760kB
[940031.207537] Normal: 345*4kB (UME) 464*8kB (UM) 269*16kB (UM) 59*32kB (M) 28*64kB (UM) 7*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 13972kB
[940031.221806] 987 total pagecache pages
[940031.225766] 0 pages in swap cache
[940031.229374] Swap cache stats: add 0, delete 0, find 0/0
[940031.234878] Free swap = 0kB
[940031.238101] Total swap = 0kB
[940031.241108] 32768 pages RAM
[940031.244015] 0 pages HighMem/MovableOnly
[940031.248030] 3689 pages reserved
[940031.251292] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[940031.260252] [ 654] 0 654 297 10 4 0 0 0 ubusd
[940031.269171] [ 655] 0 655 225 6 3 0 0 0 askfirst
[940031.278300] [ 1277] 0 1277 548 253 4 0 0 0 logd
[940031.287161] [ 1309] 0 1309 362 13 3 0 0 0 rpcd
[940031.295852] [ 1453] 0 1453 364 41 3 0 0 0 zmsg
[940031.304628] [ 1542] 0 1542 422 34 3 0 0 0 netifd
[940031.313645] [ 1566] 0 1566 351 11 3 0 0 0 odhcpd
[940031.322644] [ 1660] 0 1660 305 22 3 0 0 0 udhcpc
[940031.331616] [ 1672] 0 1672 305 22 4 0 0 0 udhcpc
[940031.340639] [ 1684] 0 1684 256 8 3 0 0 0 odhcp6c
[940031.349639] [ 1788] 0 1788 267 7 3 0 0 0 dropbear
[940031.358853] [ 1804] 0 1804 256 8 3 0 0 0 odhcp6c
[940031.367987] [ 1891] 0 1891 381 28 3 0 0 0 uhttpd
[940031.376973] [ 1933] 0 1933 357 12 4 0 0 0 zdisplay
[940031.386896] [ 1979] 0 1979 635 11 4 0 0 0 zdtool
[940031.396188] [ 2013] 0 2013 879 62 5 0 0 0 zmqttproxy
[940031.405512] [ 2031] 0 2031 311 29 4 0 0 0 zihomescript.sh
[940031.415776] [ 2059] 0 2059 1088 79 4 0 0 0 zapclient
[940031.425109] [ 2160] 0 2160 401 14 4 0 0 0 zdetect
[940031.434347] [ 2185] 0 2185 305 8 3 0 0 0 telnetd
[940031.443588] [ 2207] 0 2207 384 36 3 0 0 0 zdetect
[940031.452739] [ 2272] 453 2272 328 10 4 0 0 0 dnsmasq
[940031.461824] [14087] 0 14087 845 66 5 0 0 0 wapp
[940031.470593] [14088] 0 14088 820 52 5 0 0 0 wapp
[940031.479403] [14089] 0 14089 523 51 4 0 0 0 p1905_managerd
[940031.489111] [14090] 0 14090 470 47 4 0 0 0 mapd
[940031.498003] [31857] 0 31857 228 9 4 0 0 0 htpdate
[940031.507119] [26573] 0 26573 311 15 4 0 0 0 zihomescript.sh
[940031.516833] [26576] 0 26576 446 12 3 0 0 0 curl
[940031.525562] Out of memory: Kill process 1277 (logd) score 8 or sacrifice child
[940031.533175] Killed process 1277 (logd) total-vm:2192kB, anon-rss:1012kB, file-rss:0kB
[940031.677537] wapp invoked oom-killer: gfp_mask=0x24201ca, order=0, oom_score_adj=0
[940031.685326] CPU: 0 PID: 14087 Comm: wapp Not tainted 4.4.198 #0
[940031.691370] Stack : 81d168c2 00000033 00000000 00000000 81d168c2 00000000 00000000 00000000
[940031.691370] 863bbdf4 819086e3 817d4110 00000000 00003707 81d13690 00000003 00000001
[940031.691370] 819087e0 8107446c 00000000 00000004 00000006 00000000 817db220 83683b6c
[940031.691370] 00000000 8107213c 81d168c2 00000045 83683b50 00000000 81d168c2 00683b6c
[940031.691370] 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[940031.691370] ...
[940031.727377] Call Trace:
[940031.729959] [<810188b4>] show_stack+0x54/0x88
[940031.734459] [<811e4bd4>] dump_stack+0x8c/0xc4
[940031.738967] [<810b17d8>] dump_header.isra.4+0x54/0x19c
[940031.744237] [<810b1d68>] oom_kill_process+0xf0/0x578
[940031.749354] [<810b2504>] out_of_memory+0x314/0x3a8
[940031.754300] [<810b66f0>] __alloc_pages_nodemask+0x6ec/0x824
[940031.760036] [<810b0b98>] filemap_fault+0x35c/0x500
[940031.764980] [<810d04fc>] __do_fault+0x64/0xd4
[940031.769475] [<810d3f60>] handle_mm_fault+0x544/0xdfc
[940031.774577] [<81021c80>] __do_page_fault+0x138/0x48c
[940031.779662] [<81005420>] ret_from_exception+0x0/0x10
[940031.784735]
[940039.905445] Mem-Info:
[940039.907867] active_anon:415 inactive_anon:18 isolated_anon:0
[940039.907867] active_file:376 inactive_file:532 isolated_file:32
[940039.907867] unevictable:0 dirty:0 writeback:0 unstable:0
[940039.907867] slab_reclaimable:478 slab_unreclaimable:4239
[940039.907867] mapped:144 shmem:27 pagetables:73 bounce:0
[940039.907867] free:3638 free_pcp:0 free_cma:0
[940039.940722] DMA free:2568kB min:2304kB low:2880kB high:3456kB active_anon:216kB inactive_anon:16kB active_file:528kB inactive_file:784kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:16384kB managed:16380kB mlocked:0kB dirty:0kB writeback:0kB mapped:252kB shmem:20kB slab_reclaimable:404kB slab_unreclaimable:2516kB kernel_stack:96kB pagetables:32kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:156 all_unreclaimable? no
[940039.984232] lowmem_reserve[]: 0 97 97
[940039.988439] Normal free:11836kB min:14076kB low:17592kB high:21112kB active_anon:1444kB inactive_anon:56kB active_file:976kB inactive_file:1516kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:114688kB managed:99936kB mlocked:0kB dirty:0kB writeback:0kB mapped:324kB shmem:88kB slab_reclaimable:1508kB slab_unreclaimable:14440kB kernel_stack:696kB pagetables:260kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1840 all_unreclaimable? no
[940040.033091] lowmem_reserve[]: 0 0 0
[940040.036744] DMA: 143*4kB (UM) 116*8kB (U) 74*16kB (UM) 3*32kB (U) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2780kB
[940040.050026] Normal: 569*4kB (UME) 439*8kB (UM) 286*16kB (UM) 8*32kB (UM) 15*64kB (UM) 2*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 11836kB
[940040.064766] 909 total pagecache pages
[940040.068584] 0 pages in swap cache
[940040.072334] Swap cache stats: add 0, delete 0, find 0/0
[940040.078136] Free swap = 0kB
[940040.081589] Total swap = 0kB
[940040.084587] 32768 pages RAM
[940040.087964] 0 pages HighMem/MovableOnly
[940040.092312] 3689 pages reserved
[940040.096087] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[940040.105230] [ 654] 0 654 297 10 4 0 0 0 ubusd
[940040.114380] [ 655] 0 655 225 6 3 0 0 0 askfirst
[940040.123882] [ 1309] 0 1309 362 13 3 0 0 0 rpcd
[940040.132961] [ 1453] 0 1453 364 27 3 0 0 0 zmsg
[940040.141654] [ 1542] 0 1542 422 34 3 0 0 0 netifd
[940040.150504] [ 1566] 0 1566 351 45 3 0 0 0 odhcpd
[940040.160038] [ 1660] 0 1660 305 22 3 0 0 0 udhcpc
[940040.169236] [ 1672] 0 1672 305 22 4 0 0 0 udhcpc
[940040.179054] [ 1684] 0 1684 256 8 3 0 0 0 odhcp6c
[940040.188399] [ 1788] 0 1788 267 7 3 0 0 0 dropbear
[940040.197855] [ 1804] 0 1804 256 8 3 0 0 0 odhcp6c
[940040.207217] [ 1891] 0 1891 381 28 3 0 0 0 uhttpd
[940040.216402] [ 1933] 0 1933 357 21 4 0 0 0 zdisplay
[940040.225873] [ 1979] 0 1979 635 11 4 0 0 0 zdtool
[940040.235113] [ 2031] 0 2031 311 29 4 0 0 0 zihomescript.sh
[940040.245101] [ 2185] 0 2185 305 8 3 0 0 0 telnetd
[940040.254072] [ 2272] 453 2272 328 10 4 0 0 0 dnsmasq
[940040.263456] [14090] 0 14090 470 52 4 0 0 0 mapd
[940040.272473] [31857] 0 31857 228 9 4 0 0 0 htpdate
[940040.281790] [26573] 0 26573 311 15 4 0 0 0 zihomescript.sh
[940040.291889] Out of memory: Kill process 14090 (mapd) score 1 or sacrifice child
[940040.299716] Killed process 14090 (mapd) total-vm:1880kB, anon-rss:208kB, file-rss:0kB
[940047.227817] MacTableDeleteEntry(): wcid 8 =====
[940047.238415] hw_ctrl_flow_v2_disconnt_act: wdev_idx=3
[940085.994632] ACT - SendBSS2040CoexistMgmtAction(BSSCoexist2040=0x6)
[940086.001068] hw_ctrl_flow_v2_peer_update: wdev_idx=0
[940224.529243] ACT - SendBSS2040CoexistMgmtAction(BSSCoexist2040=0x4)
[940224.536168] hw_ctrl_flow_v2_peer_update: wdev_idx=0
[940276.725992] peer_auth_req: 44 usec
[940276.732087] IE_WLAN_EXTENSION: no handler for extension_id:35
[940276.738337] @@@ ap_cmm_peer_assoc_req_action(): (wcid=24), HTC_ICVErrCnt(0), HTC_AAD_OM_Freeze(0), HTC_AAD_OM_CountDown(0), HTC_AAD_OM_Freeze(1) is in Asso. stage!
[940276.753598] add he assoc_rsp, len=59