6.13. Out-of-memory profiling

The files oom_profile.log* are populated every thirty seconds with the output of the commands cat /proc/meminfo and ps.

Figure 6.87. Contents of the file oom_profile.log

================================================================================
Fri Sep 14 15:40:28 CEST 2012
================================================================================

cat /proc/meminfo
MemTotal:      8162944 kB
MemFree:       2683204 kB
Buffers:         67880 kB
Cached:        4221328 kB
SwapCached:          0 kB
Active:        4292324 kB
Inactive:       944588 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:      8162944 kB
LowFree:       2683204 kB
SwapTotal:     6297336 kB
SwapFree:      6297336 kB
Dirty:           32504 kB
Writeback:           0 kB
Mapped:        4696692 kB
Slab:           178840 kB
Committed_AS:  4968392 kB
PageTables:      21556 kB
VmallocTotal: 536870911 kB
VmallocUsed:    280440 kB
VmallocChunk: 536588807 kB
HugePages_Total:     0
HugePages_Free:      0
Hugepagesize:     2048 kB

ps -Aww -o pid,ppid,pcpu,vsize,rss,majflt,tty,stat,wchan,command
  PID  PPID %CPU   VSZ  RSS MAJFLT TT       STAT WCHAN  COMMAND
    1     0  0.0  4768  628     14 ?        S    -      init [3]  
    2     1  0.0     0    0      0 ?        S    migrat [migration/0]
    3     1  0.0     0    0      0 ?        SN   ksofti [ksoftirqd/0]
    4     1  0.0     0    0      0 ?        S    watchd [watchdog/0]
    5     1  0.0     0    0      0 ?        S    migrat [migration/1]
    6     1  0.0     0    0      0 ?        SN   ksofti [ksoftirqd/1]
    7     1  0.0     0    0      0 ?        S    watchd [watchdog/1]
    8     1  0.0     0    0      0 ?        S    migrat [migration/2]
    9     1  0.0     0    0      0 ?        SN   ksofti [ksoftirqd/2]
   10     1  0.0     0    0      0 ?        S    watchd [watchdog/2]
   11     1  0.0     0    0      0 ?        S    migrat [migration/3]
[...]
 8010     1  0.0 28716 7676      0 ?        Ss   -      /opt/tms/bin/pm
 8109     1  0.0  2396  320      0 ?        S<   -      /opt/tms/bin/csoftwatch
 8138     1  0.0  4188  588      3 ?        S<   wait   sh /sbin/softwatch.sh   
          
 8156     1  0.0  2540  468      0 tty1     Ss+  -      /sbin/mingetty tty1
 8157     1  0.0  2540  468      0 tty2     Ss+  -      /sbin/mingetty tty2
10554     1  0.1 20696 3520      0 ?        Ss   fuse_d /usr/local/bin/encfs -S 
/var/opt/rbt/encrypted/ /var/opt/rbt/decrypted
10907  8010  0.0  3640  660      0 ?        Ss   -      /usr/sbin/crond -n
10910  8010  0.0 17040 2008      7 ?        Ss   -      /usr/sbin/ntpd -n -u ntp
 -g
10912  8010  0.0 99480 28788    90 ?        Ssl  -      /opt/rbt/bin/rcud --logr
c /etc/rcud.logrc
11011  8010  0.0 15172 1876      3 ?        Ss   -      /usr/sbin/sshd -D
11012  8010  0.0 18536 8328      4 ?        S<Lsl -     /opt/tms/bin/wdt
11560  8010  0.0 38732 5176     33 ?        Ss   -      /usr/sbin/winbindd -F
17859 11560  0.0 38692 5128      0 ?        S    fcntl_ /usr/sbin/winbindd -F
19591  8010  4.2 4294084 4283876 0 ?        S<Lsl -     /opt/rbt/bin/sport /etc/opt/tms/ou \
    tput/configfile.xml --logrc /etc/sport.logrc --logfile /var/log/sport.log -t /etc/opt/ \
    tms/output/tds_config.xml

With running this every 30 seconds, memory usage related issues are tracked.