oom-killer syslog 메시지를 읽는 방법?

19

우분투 12.04 서버가 있는데 때로는 SSH, 핑, 물리적으로 재부팅 될 때까지 아무것도 완전히 죽지 않습니다.

재부팅 후, syslog에서 oom-killer가 거의 모든 것을 죽였다는 것을 알 수 있습니다.

여기에는 자세한 메모리 사용 정보가 많이 있습니다. OOM 문제의 원인을 확인하려면이 로그를 어떻게 읽습니까? 서버에 필요한 것보다 훨씬 많은 메모리가 있으므로 메모리가 부족하지 않아야합니다.


Oct 25 07:28:04 nldedip4k031 kernel: [87946.529511] oom_kill_process: 9 callbacks suppressed
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529514] irqbalance invoked oom-killer: gfp_mask=0x80d0, order=0, oom_adj=0, oom_score_adj=0
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529516] irqbalance cpuset=/ mems_allowed=0
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529518] Pid: 948, comm: irqbalance Not tainted 3.2.0-55-generic-pae #85-Ubuntu
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529519] Call Trace:
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529525]  [] dump_header.isra.6+0x85/0xc0
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529528]  [] oom_kill_process+0x5c/0x80
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529530]  [] out_of_memory+0xc5/0x1c0
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529532]  [] __alloc_pages_nodemask+0x72c/0x740
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529535]  [] __get_free_pages+0x1c/0x30
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529537]  [] get_zeroed_page+0x12/0x20
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529541]  [] fill_read_buffer.isra.8+0xaa/0xd0
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529543]  [] sysfs_read_file+0x7d/0x90
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529546]  [] vfs_read+0x8c/0x160
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529548]  [] ? fill_read_buffer.isra.8+0xd0/0xd0
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529550]  [] sys_read+0x3d/0x70
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529554]  [] sysenter_do_call+0x12/0x28
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529555] Mem-Info:
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529556] DMA per-cpu:
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529557] CPU    0: hi:    0, btch:   1 usd:   0
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529558] CPU    1: hi:    0, btch:   1 usd:   0
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529560] CPU    2: hi:    0, btch:   1 usd:   0
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529561] CPU    3: hi:    0, btch:   1 usd:   0
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529562] CPU    4: hi:    0, btch:   1 usd:   0
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529563] CPU    5: hi:    0, btch:   1 usd:   0
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529564] CPU    6: hi:    0, btch:   1 usd:   0
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529565] CPU    7: hi:    0, btch:   1 usd:   0
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529566] Normal per-cpu:
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529567] CPU    0: hi:  186, btch:  31 usd: 179
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529568] CPU    1: hi:  186, btch:  31 usd: 182
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529569] CPU    2: hi:  186, btch:  31 usd: 132
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529570] CPU    3: hi:  186, btch:  31 usd: 175
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529571] CPU    4: hi:  186, btch:  31 usd:  91
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529572] CPU    5: hi:  186, btch:  31 usd: 173
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529573] CPU    6: hi:  186, btch:  31 usd: 159
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529574] CPU    7: hi:  186, btch:  31 usd: 164
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529575] HighMem per-cpu:
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529576] CPU    0: hi:  186, btch:  31 usd: 165
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529577] CPU    1: hi:  186, btch:  31 usd: 183
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529578] CPU    2: hi:  186, btch:  31 usd: 185
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529579] CPU    3: hi:  186, btch:  31 usd: 138
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529580] CPU    4: hi:  186, btch:  31 usd: 155
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529581] CPU    5: hi:  186, btch:  31 usd: 104
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529582] CPU    6: hi:  186, btch:  31 usd: 133
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529583] CPU    7: hi:  186, btch:  31 usd: 170
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529586] active_anon:5523 inactive_anon:354 isolated_anon:0
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529586]  active_file:2815 inactive_file:6849119 isolated_file:0
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529587]  unevictable:0 dirty:449 writeback:10 unstable:0
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529587]  free:1304125 slab_reclaimable:104672 slab_unreclaimable:3419
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529588]  mapped:2661 shmem:138 pagetables:313 bounce:0
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529591] DMA free:4252kB min:780kB low:972kB high:1168kB active_anon:0kB inactive_anon:0kB active_file:4kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15756kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:11564kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1 all_unreclaimable? yes
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529594] lowmem_reserve[]: 0 869 32460 32460
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529599] Normal free:44052kB min:44216kB low:55268kB high:66324kB active_anon:0kB inactive_anon:0kB active_file:616kB inactive_file:568kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:890008kB mlocked:0kB dirty:0kB writeback:0kB mapped:4kB shmem:0kB slab_reclaimable:407124kB slab_unreclaimable:13672kB kernel_stack:992kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:2083 all_unreclaimable? yes
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529602] lowmem_reserve[]: 0 0 252733 252733
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529606] HighMem free:5168196kB min:512kB low:402312kB high:804112kB active_anon:22092kB inactive_anon:1416kB active_file:10640kB inactive_file:27395920kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:32349872kB mlocked:0kB dirty:1796kB writeback:40kB mapped:10640kB shmem:552kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:1252kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529609] lowmem_reserve[]: 0 0 0 0
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529611] DMA: 6*4kB 6*8kB 6*16kB 5*32kB 5*64kB 4*128kB 2*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 4232kB
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529616] Normal: 297*4kB 180*8kB 119*16kB 73*32kB 67*64kB 47*128kB 35*256kB 13*512kB 5*1024kB 1*2048kB 1*4096kB = 44052kB
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529622] HighMem: 1*4kB 6*8kB 27*16kB 11*32kB 2*64kB 1*128kB 0*256kB 0*512kB 4*1024kB 1*2048kB 1260*4096kB = 5168196kB
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529627] 6852076 total pagecache pages
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529628] 0 pages in swap cache
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529629] Swap cache stats: add 0, delete 0, find 0/0
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529630] Free swap  = 3998716kB
Oct 25 07:28:04 nldedip4k031 kernel: [87946.529631] Total swap = 3998716kB
Oct 25 07:28:04 nldedip4k031 kernel: [87946.571914] 8437743 pages RAM
Oct 25 07:28:04 nldedip4k031 kernel: [87946.571916] 8209409 pages HighMem
Oct 25 07:28:04 nldedip4k031 kernel: [87946.571917] 159556 pages reserved
Oct 25 07:28:04 nldedip4k031 kernel: [87946.571917] 6862034 pages shared
Oct 25 07:28:04 nldedip4k031 kernel: [87946.571918] 123540 pages non-shared
Oct 25 07:28:04 nldedip4k031 kernel: [87946.571919] [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
Oct 25 07:28:04 nldedip4k031 kernel: [87946.571927] [  421]     0   421      709      152   3       0             0 upstart-udev-br
Oct 25 07:28:04 nldedip4k031 kernel: [87946.571929] [  429]     0   429      773      326   5     -17         -1000 udevd
Oct 25 07:28:04 nldedip4k031 kernel: [87946.571931] [  567]     0   567      772      224   4     -17         -1000 udevd
Oct 25 07:28:04 nldedip4k031 kernel: [87946.571932] [  568]     0   568      772      231   7     -17         -1000 udevd
Oct 25 07:28:04 nldedip4k031 kernel: [87946.571934] [  764]     0   764      712      103   1       0             0 upstart-socket-
Oct 25 07:28:04 nldedip4k031 kernel: [87946.571936] [  772]   103   772      815      164   5       0             0 dbus-daemon
Oct 25 07:28:04 nldedip4k031 kernel: [87946.571938] [  785]     0   785     1671      600   1     -17         -1000 sshd
Oct 25 07:28:04 nldedip4k031 kernel: [87946.571940] [  809]   101   809     7766      380   1       0             0 rsyslogd
Oct 25 07:28:04 nldedip4k031 kernel: [87946.571942] [  869]     0   869     1158      213   3       0             0 getty
Oct 25 07:28:04 nldedip4k031 kernel: [87946.571943] [  873]     0   873     1158      214   6       0             0 getty
Oct 25 07:28:04 nldedip4k031 kernel: [87946.571945] [  911]     0   911     1158      215   3       0             0 getty
Oct 25 07:28:04 nldedip4k031 kernel: [87946.571947] [  912]     0   912     1158      214   2       0             0 getty
Oct 25 07:28:04 nldedip4k031 kernel: [87946.571949] [  914]     0   914     1158      213   1       0             0 getty
Oct 25 07:28:04 nldedip4k031 kernel: [87946.571950] [  916]     0   916      618       86   1       0             0 atd
Oct 25 07:28:04 nldedip4k031 kernel: [87946.571952] [  917]     0   917      655      226   3       0             0 cron
Oct 25 07:28:04 nldedip4k031 kernel: [87946.571954] [  948]     0   948      902      159   3       0             0 irqbalance
Oct 25 07:28:04 nldedip4k031 kernel: [87946.571956] [  993]     0   993     1145      363   3       0             0 master
Oct 25 07:28:04 nldedip4k031 kernel: [87946.571957] [ 1002]   104  1002     1162      333   1       0             0 qmgr
Oct 25 07:28:04 nldedip4k031 kernel: [87946.571959] [ 1016]     0  1016      730      149   2       0             0 mdadm
Oct 25 07:28:04 nldedip4k031 kernel: [87946.571961] [ 1057]     0  1057     6066     2160   3       0             0 /usr/sbin/apach
Oct 25 07:28:04 nldedip4k031 kernel: [87946.571963] [ 1086]     0  1086     1158      213   3       0             0 getty
Oct 25 07:28:04 nldedip4k031 kernel: [87946.571965] [ 1088]    33  1088     6191     1517   0       0             0 /usr/sbin/apach
Oct 25 07:28:04 nldedip4k031 kernel: [87946.571967] [ 1089]    33  1089     6191     1451   1       0             0 /usr/sbin/apach
Oct 25 07:28:04 nldedip4k031 kernel: [87946.571969] [ 1090]    33  1090     6175     1451   3       0             0 /usr/sbin/apach
Oct 25 07:28:04 nldedip4k031 kernel: [87946.571971] [ 1091]    33  1091     6191     1451   1       0             0 /usr/sbin/apach
Oct 25 07:28:04 nldedip4k031 kernel: [87946.571972] [ 1092]    33  1092     6191     1451   0       0             0 /usr/sbin/apach
Oct 25 07:28:04 nldedip4k031 kernel: [87946.571974] [ 1109]    33  1109     6191     1517   0       0             0 /usr/sbin/apach
Oct 25 07:28:04 nldedip4k031 kernel: [87946.571976] [ 1151]    33  1151     6191     1451   1       0             0 /usr/sbin/apach
Oct 25 07:28:04 nldedip4k031 kernel: [87946.571978] [ 1201]   104  1201     1803      652   1       0             0 tlsmgr
Oct 25 07:28:04 nldedip4k031 kernel: [87946.571980] [ 2475]     0  2475     2435      812   0       0             0 sshd
Oct 25 07:28:04 nldedip4k031 kernel: [87946.571982] [ 2494]     0  2494     1745      839   1       0             0 bash
Oct 25 07:28:04 nldedip4k031 kernel: [87946.571984] [ 2573]     0  2573     3394     1689   0       0             0 sshd
Oct 25 07:28:04 nldedip4k031 kernel: [87946.571986] [ 2589]     0  2589     5014      457   3       0             0 rsync
Oct 25 07:28:04 nldedip4k031 kernel: [87946.571988] [ 2590]     0  2590     7970      522   1       0             0 rsync
Oct 25 07:28:04 nldedip4k031 kernel: [87946.571990] [ 2652]   104  2652     1150      326   5       0             0 pickup
Oct 25 07:28:04 nldedip4k031 kernel: [87946.571992] Out of memory: Kill process 421 (upstart-udev-br) score 1 or sacrifice child
Oct 25 07:28:04 nldedip4k031 kernel: [87946.572407] Killed process 421 (upstart-udev-br) total-vm:2836kB, anon-rss:156kB, file-rss:452kB
Oct 25 07:28:04 nldedip4k031 kernel: [87946.573107] init: upstart-udev-bridge main process (421) killed by KILL signal
Oct 25 07:28:04 nldedip4k031 kernel: [87946.573126] init: upstart-udev-bridge main process ended, respawning
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461570] irqbalance invoked oom-killer: gfp_mask=0x80d0, order=0, oom_adj=0, oom_score_adj=0
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461573] irqbalance cpuset=/ mems_allowed=0
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461576] Pid: 948, comm: irqbalance Not tainted 3.2.0-55-generic-pae #85-Ubuntu
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461578] Call Trace:
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461585]  [] dump_header.isra.6+0x85/0xc0
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461588]  [] oom_kill_process+0x5c/0x80
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461591]  [] out_of_memory+0xc5/0x1c0
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461595]  [] __alloc_pages_nodemask+0x72c/0x740
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461599]  [] __get_free_pages+0x1c/0x30
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461602]  [] get_zeroed_page+0x12/0x20
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461606]  [] fill_read_buffer.isra.8+0xaa/0xd0
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461609]  [] sysfs_read_file+0x7d/0x90
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461613]  [] vfs_read+0x8c/0x160
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461616]  [] ? fill_read_buffer.isra.8+0xd0/0xd0
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461619]  [] sys_read+0x3d/0x70
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461624]  [] sysenter_do_call+0x12/0x28
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461626] Mem-Info:
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461628] DMA per-cpu:
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461629] CPU    0: hi:    0, btch:   1 usd:   0
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461631] CPU    1: hi:    0, btch:   1 usd:   0
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461633] CPU    2: hi:    0, btch:   1 usd:   0
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461634] CPU    3: hi:    0, btch:   1 usd:   0
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461636] CPU    4: hi:    0, btch:   1 usd:   0
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461638] CPU    5: hi:    0, btch:   1 usd:   0
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461639] CPU    6: hi:    0, btch:   1 usd:   0
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461641] CPU    7: hi:    0, btch:   1 usd:   0
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461642] Normal per-cpu:
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461644] CPU    0: hi:  186, btch:  31 usd:  61
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461646] CPU    1: hi:  186, btch:  31 usd:  49
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461647] CPU    2: hi:  186, btch:  31 usd:   8
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461649] CPU    3: hi:  186, btch:  31 usd:   0
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461651] CPU    4: hi:  186, btch:  31 usd:   0
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461652] CPU    5: hi:  186, btch:  31 usd:   0
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461654] CPU    6: hi:  186, btch:  31 usd:   0
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461656] CPU    7: hi:  186, btch:  31 usd:  30
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461657] HighMem per-cpu:
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461658] CPU    0: hi:  186, btch:  31 usd:   4
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461660] CPU    1: hi:  186, btch:  31 usd: 204
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461662] CPU    2: hi:  186, btch:  31 usd:   0
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461663] CPU    3: hi:  186, btch:  31 usd:   0
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461665] CPU    4: hi:  186, btch:  31 usd:   0
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461667] CPU    5: hi:  186, btch:  31 usd:  31
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461668] CPU    6: hi:  186, btch:  31 usd:   0
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461670] CPU    7: hi:  186, btch:  31 usd:   0
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461674] active_anon:5441 inactive_anon:412 isolated_anon:0
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461674]  active_file:2668 inactive_file:6922842 isolated_file:0
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461675]  unevictable:0 dirty:836 writeback:0 unstable:0
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461676]  free:1231664 slab_reclaimable:105781 slab_unreclaimable:3399
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461677]  mapped:2649 shmem:138 pagetables:313 bounce:0
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461682] DMA free:4248kB min:780kB low:972kB high:1168kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:4kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15756kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:11560kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:5687 all_unreclaimable? yes
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461686] lowmem_reserve[]: 0 869 32460 32460
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461693] Normal free:44184kB min:44216kB low:55268kB high:66324kB active_anon:0kB inactive_anon:0kB active_file:20kB inactive_file:1096kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:890008kB mlocked:0kB dirty:4kB writeback:0kB mapped:4kB shmem:0kB slab_reclaimable:411564kB slab_unreclaimable:13592kB kernel_stack:992kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1816 all_unreclaimable? yes
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461697] lowmem_reserve[]: 0 0 252733 252733
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461703] HighMem free:4878224kB min:512kB low:402312kB high:804112kB active_anon:21764kB inactive_anon:1648kB active_file:10652kB inactive_file:27690268kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:32349872kB mlocked:0kB dirty:3340kB writeback:0kB mapped:10592kB shmem:552kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:1252kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461708] lowmem_reserve[]: 0 0 0 0
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461711] DMA: 8*4kB 7*8kB 6*16kB 5*32kB 5*64kB 4*128kB 2*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 4248kB
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461719] Normal: 272*4kB 178*8kB 76*16kB 52*32kB 42*64kB 36*128kB 23*256kB 20*512kB 7*1024kB 2*2048kB 1*4096kB = 44176kB
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461727] HighMem: 1*4kB 45*8kB 31*16kB 24*32kB 5*64kB 3*128kB 1*256kB 2*512kB 4*1024kB 2*2048kB 1188*4096kB = 4877852kB
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461736] 6925679 total pagecache pages
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461737] 0 pages in swap cache
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461739] Swap cache stats: add 0, delete 0, find 0/0
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461740] Free swap  = 3998716kB
Oct 25 07:28:34 nldedip4k031 kernel: [87976.461741] Total swap = 3998716kB
Oct 25 07:28:34 nldedip4k031 kernel: [87976.524951] 8437743 pages RAM
Oct 25 07:28:34 nldedip4k031 kernel: [87976.524953] 8209409 pages HighMem
Oct 25 07:28:34 nldedip4k031 kernel: [87976.524954] 159556 pages reserved
Oct 25 07:28:34 nldedip4k031 kernel: [87976.524955] 6936141 pages shared
Oct 25 07:28:34 nldedip4k031 kernel: [87976.524956] 124602 pages non-shared
Oct 25 07:28:34 nldedip4k031 kernel: [87976.524957] [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
Oct 25 07:28:34 nldedip4k031 kernel: [87976.524966] [  429]     0   429      773      326   5     -17         -1000 udevd
Oct 25 07:28:34 nldedip4k031 kernel: [87976.524968] [  567]     0   567      772      224   4     -17         -1000 udevd
Oct 25 07:28:34 nldedip4k031 kernel: [87976.524971] [  568]     0   568      772      231   7     -17         -1000 udevd
Oct 25 07:28:34 nldedip4k031 kernel: [87976.524973] [  764]     0   764      712      103   3       0             0 upstart-socket-
Oct 25 07:28:34 nldedip4k031 kernel: [87976.524976] [  772]   103   772      815      164   2       0             0 dbus-daemon
Oct 25 07:28:34 nldedip4k031 kernel: [87976.524979] [  785]     0   785     1671      600   1     -17         -1000 sshd
Oct 25 07:28:34 nldedip4k031 kernel: [87976.524981] [  809]   101   809     7766      380   1       0             0 rsyslogd
Oct 25 07:28:34 nldedip4k031 kernel: [87976.524983] [  869]     0   869     1158      213   3       0             0 getty
Oct 25 07:28:34 nldedip4k031 kernel: [87976.524986] [  873]     0   873     1158      214   6       0             0 getty
Oct 25 07:28:34 nldedip4k031 kernel: [87976.524988] [  911]     0   911     1158      215   3       0             0 getty
Oct 25 07:28:34 nldedip4k031 kernel: [87976.524990] [  912]     0   912     1158      214   2       0             0 getty
Oct 25 07:28:34 nldedip4k031 kernel: [87976.524992] [  914]     0   914     1158      213   1       0             0 getty
Oct 25 07:28:34 nldedip4k031 kernel: [87976.524995] [  916]     0   916      618       86   1       0             0 atd
Oct 25 07:28:34 nldedip4k031 kernel: [87976.524997] [  917]     0   917      655      226   3       0             0 cron
Oct 25 07:28:34 nldedip4k031 kernel: [87976.524999] [  948]     0   948      902      159   5       0             0 irqbalance
Oct 25 07:28:34 nldedip4k031 kernel: [87976.525002] [  993]     0   993     1145      363   3       0             0 master
Oct 25 07:28:34 nldedip4k031 kernel: [87976.525004] [ 1002]   104  1002     1162      333   1       0             0 qmgr
Oct 25 07:28:34 nldedip4k031 kernel: [87976.525007] [ 1016]     0  1016      730      149   2       0             0 mdadm
Oct 25 07:28:34 nldedip4k031 kernel: [87976.525009] [ 1057]     0  1057     6066     2160   3       0             0 /usr/sbin/apach
Oct 25 07:28:34 nldedip4k031 kernel: [87976.525012] [ 1086]     0  1086     1158      213   3       0             0 getty
Oct 25 07:28:34 nldedip4k031 kernel: [87976.525014] [ 1088]    33  1088     6191     1517   0       0             0 /usr/sbin/apach
Oct 25 07:28:34 nldedip4k031 kernel: [87976.525017] [ 1089]    33  1089     6191     1451   1       0             0 /usr/sbin/apach
Oct 25 07:28:34 nldedip4k031 kernel: [87976.525019] [ 1090]    33  1090     6175     1451   1       0             0 /usr/sbin/apach
Oct 25 07:28:34 nldedip4k031 kernel: [87976.525021] [ 1091]    33  1091     6191     1451   1       0             0 /usr/sbin/apach
Oct 25 07:28:34 nldedip4k031 kernel: [87976.525024] [ 1092]    33  1092     6191     1451   0       0             0 /usr/sbin/apach
Oct 25 07:28:34 nldedip4k031 kernel: [87976.525026] [ 1109]    33  1109     6191     1517   0       0             0 /usr/sbin/apach
Oct 25 07:28:34 nldedip4k031 kernel: [87976.525029] [ 1151]    33  1151     6191     1451   1       0             0 /usr/sbin/apach
Oct 25 07:28:34 nldedip4k031 kernel: [87976.525031] [ 1201]   104  1201     1803      652   1       0             0 tlsmgr
Oct 25 07:28:34 nldedip4k031 kernel: [87976.525033] [ 2475]     0  2475     2435      812   0       0             0 sshd
Oct 25 07:28:34 nldedip4k031 kernel: [87976.525036] [ 2494]     0  2494     1745      839   1       0             0 bash
Oct 25 07:28:34 nldedip4k031 kernel: [87976.525038] [ 2573]     0  2573     3394     1689   3       0             0 sshd
Oct 25 07:28:34 nldedip4k031 kernel: [87976.525040] [ 2589]     0  2589     5014      457   3       0             0 rsync
Oct 25 07:28:34 nldedip4k031 kernel: [87976.525043] [ 2590]     0  2590     7970      522   1       0             0 rsync
Oct 25 07:28:34 nldedip4k031 kernel: [87976.525045] [ 2652]   104  2652     1150      326   5       0             0 pickup
Oct 25 07:28:34 nldedip4k031 kernel: [87976.525048] [ 2847]     0  2847      709       89   0       0             0 upstart-udev-br
Oct 25 07:28:34 nldedip4k031 kernel: [87976.525050] Out of memory: Kill process 764 (upstart-socket-) score 1 or sacrifice child
Oct 25 07:28:34 nldedip4k031 kernel: [87976.525484] Killed process 764 (upstart-socket-) total-vm:2848kB, anon-rss:204kB, file-rss:208kB
Oct 25 07:28:34 nldedip4k031 kernel: [87976.526161] init: upstart-socket-bridge main process (764) killed by KILL signal
Oct 25 07:28:34 nldedip4k031 kernel: [87976.526180] init: upstart-socket-bridge main process ended, respawning
Oct 25 07:28:44 nldedip4k031 kernel: [87986.439671] irqbalance invoked oom-killer: gfp_mask=0x80d0, order=0, oom_adj=0, oom_score_adj=0
Oct 25 07:28:44 nldedip4k031 kernel: [87986.439674] irqbalance cpuset=/ mems_allowed=0
Oct 25 07:28:44 nldedip4k031 kernel: [87986.439676] Pid: 948, comm: irqbalance Not tainted 3.2.0-55-generic-pae #85-Ubuntu
Oct 25 07:28:44 nldedip4k031 kernel: [87986.439678] Call Trace:
Oct 25 07:28:44 nldedip4k031 kernel: [87986.439684]  [] dump_header.isra.6+0x85/0xc0
Oct 25 07:28:44 nldedip4k031 kernel: [87986.439686]  [] oom_kill_process+0x5c/0x80
Oct 25 07:28:44 nldedip4k031 kernel: [87986.439688]  [] out_of_memory+0xc5/0x1c0
Oct 25 07:28:44 nldedip4k031 kernel: [87986.439691]  [] __alloc_pages_nodemask+0x72c/0x740
Oct 25 07:28:44 nldedip4k031 kernel: [87986.439694]  [] __get_free_pages+0x1c/0x30
Oct 25 07:28:44 nldedip4k031 kernel: [87986.439696]  [] get_zeroed_page+0x12/0x20
Oct 25 07:28:44 nldedip4k031 kernel: [87986.439699]  [] fill_read_buffer.isra.8+0xaa/0xd0
Oct 25 07:28:44 nldedip4k031 kernel: [87986.439702]  [] sysfs_read_file+0x7d/0x90
Oct 25 07:28:44 nldedip4k031 kernel: [87986.439704]  [] vfs_read+0x8c/0x160
Oct 25 07:28:44 nldedip4k031 kernel: [87986.439707]  [] ? fill_read_buffer.isra.8+0xd0/0xd0
Oct 25 07:28:44 nldedip4k031 kernel: [87986.439709]  [] sys_read+0x3d/0x70
Oct 25 07:28:44 nldedip4k031 kernel: [87986.439712]  [] sysenter_do_call+0x12/0x28
Oct 25 07:28:44 nldedip4k031 kernel: [87986.439714] Mem-Info:
Oct 25 07:28:44 nldedip4k031 kernel: [87986.439714] DMA per-cpu:
Oct 25 07:28:44 nldedip4k031 kernel: [87986.439716] CPU    0: hi:    0, btch:   1 usd:   0
Oct 25 07:28:44 nldedip4k031 kernel: [87986.439717] CPU    1: hi:    0, btch:   1 usd:   0
Oct 25 07:28:44 nldedip4k031 kernel: [87986.439718] CPU    2: hi:    0, btch:   1 usd:   0
Oct 25 07:28:44 nldedip4k031 kernel: [87986.439719] CPU    3: hi:    0, btch:   1 usd:   0
Oct 25 07:28:44 nldedip4k031 kernel: [87986.439720] CPU    4: hi:    0, btch:   1 usd:   0
Oct 25 07:28:44 nldedip4k031 kernel: [87986.439721] CPU    5: hi:    0, btch:   1 usd:   0
Oct 25 07:28:44 nldedip4k031 kernel: [87986.439722] CPU    6: hi:    0, btch:   1 usd:   0
Oct 25 07:28:44 nldedip4k031 kernel: [87986.439723] CPU    7: hi:    0, btch:   1 usd:   0
Oct 25 07:28:44 nldedip4k031 kernel: [87986.439724] Normal per-cpu:
Oct 25 07:28:44 nldedip4k031 kernel: [87986.439725] CPU    0: hi:  186, btch:  31 usd:   0
Oct 25 07:28:44 nldedip4k031 kernel: [87986.439726] CPU    1: hi:  186, btch:  31 usd:   0
Oct 25 07:28:44 nldedip4k031 kernel: [87986.439727] CPU    2: hi:  186, btch:  31 usd:   0
Oct 25 07:28:44 nldedip4k031 kernel: [87986.439728] CPU    3: hi:  186, btch:  31 usd:   0
Oct 25 07:28:44 nldedip4k031 kernel: [87986.439729] CPU    4: hi:  186, btch:  31 usd:   0
Oct 25 07:33:48 nldedip4k031 kernel: imklog 5.8.6, log source = /proc/kmsg started.
Oct 25 07:33:48 nldedip4k031 rsyslogd: [origin software="rsyslogd" swVersion="5.8.6" x-pid="2880" x-info="http://www.rsyslog.com"] start
Oct 25 07:33:48 nldedip4k031 rsyslogd: rsyslogd's groupid changed to 103
Oct 25 07:33:48 nldedip4k031 rsyslogd: rsyslogd's userid changed to 101
Oct 25 07:33:48 nldedip4k031 rsyslogd-2039: Could not open output pipe '/dev/xconsole' [try http://www.rsyslog.com/e/2039 ]
linux  outofmemoryerror 
— 부여
소스
어딘가에 메모리가 부족 하지만 어디 있는지 모릅니다 (커널 출력을 읽는 것이 좋지 않습니다). 솔직히 이상한 커널 버그에 부딪쳤다 고 생각합니다.
— derobert
serverfault.com/questions/564068/linux-oom-situation을 확인하십시오 . 정답이 있습니다.
— seaquest
— Ciro Santilli 新疆 改造 中心 法轮功 六四 事件

답변:

18

OOM 킬러는 실제로 메모리가 부족하다고 제안합니다.

필요한 것보다 더 많은 메모리가 있다고 가정하면 일부 시스템 이벤트가 어딘가에서 메모리 누수를 생성하고 있지만 OOM 킬러는 왜 메모리 누수가 있는지 알려주지 않고 메모리가 부족하다는 것을 알지 못합니다. 중요한 것들 ( oom_score 기반 ).

그리고 메모리 누수가 발생하는 경우 oom-killer는 procs 만 죽여서 루즈가 더 많은 메모리를 할당 할 수 있습니다.

그래서 내가 할 경우는

  1. 커널 패닉 후 크래시 덤프 vmcore를 생성하는 kdump를 구성하십시오 . (자세한 설명은 여기에 있습니다 )
  2. vm.panic_on_oom=1커널 매개 변수 설정 머신에 메모리가 부족한 경우 커널 패닉이 발생합니다.
  3. 다음에 패닉이 발생하면 kdump가 생성 한 vmcore 파일을 열고 프로세스 테이블을 보면 범인이 드러납니다.
— 피터 H
소스
거기에 어떤 줄이 실제로 메모리가 부족하다는 것을 알려줍니까? 미래의 메시지를 직접 해석하는 방법을 알고 싶을뿐입니다.
— Grant
3
OOM-killer가 호출되었다는 사실이 그 사실을 알려줍니다.
— MadHatter는 Monica
1
oom-killer가 메모리 부족없이 실행되는 문제는 다음과 같습니다. bl0rg.krunch.be/oom-frag.html 내가 모르는 것은 실제 OOM 상태를 말하는 방법입니다 그런 것들로부터. 나는 그것이 로그의 어딘가에 쓰여졌다 고 확신하지만, 그것을 해석하는 방법을 모른다.
— Grant
아직도 나는 당신을 위해 최선의 선택은 여전히 ​​kdump와 vmcore를 조사하여 실제 오류 시간에 시스템이 어떻게 보이는지에 대한 스냅 샷을 얻는 것입니다. "ps"만이 사용 가능한 도구는 아닙니다. 또한 우분투에서 지원 티켓을 만들거나 문제를 해결하기 위해 소싱 티켓을 만들면 근본 원인 분석을 위해 vmcore를 원할 것입니다. 이 페이지를 확인하십시오 : dedoimedo.com/computers/crash-analyze.html
— Petter H
당사 사이트를 사용함과 동시에 당사의 쿠키 정책과 개인정보 보호정책을 읽고 이해하였음을 인정하는 것으로 간주합니다.
Licensed under cc by-sa 3.0 with attribution required.