Analyzing the Core Dump

To analyze the vmcore dump file, you must have the crash and kernel-debuginfo packages installed. To do so, type the following at a shell prompt:

~]# yum install --enablerepo=rhel-debuginfo crash kernel-debuginfo

Refer to Part II, "Package Management" for more information on how to install new packages in Community Enterprise Linux.

To determine the cause of the system crash, you can use the crash utility. This utility allows you to interactively analyze a running Linux system as well as a core dump created by netdump, diskdump, xendump, or kdump. When started, it presents you with an interactive prompt very similar to the GNU Debugger (GDB).

To start the utility, type the command in the following form at a shell prompt:

crash /var/crash/timestamp/vmcore /usr/lib/debug/lib/modules/kernel/vmlinux

Note that the kernel version should be the same as the one that was captured by kdump. To find out which kernel you are currently running, use the uname -r command.

Example 44.2. Running the `crash` utility

~]# crash /var/crash/2010-08-04-17\:55/vmcore \
/usr/lib/debug/lib/modules/2.6.18-194.8.1.el5/vmlinux
crash 4.1.2-4.el5_5.1
Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009  CentOS, Inc.
Copyright (C) 2004, 2005, 2006  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
GNU gdb 6.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...
      KERNEL: /usr/lib/debug/lib/modules/2.6.18-194.8.1.el5/vmlinux
    DUMPFILE: /var/crash/2010-08-04-17:55/vmcore
        CPUS: 1
        DATE: Wed Aug  4 17:50:41 2010
      UPTIME: 00:56:53
LOAD AVERAGE: 0.47, 0.47, 0.55
       TASKS: 128
    NODENAME: localhost.localdomain
     RELEASE: 2.6.18-194.el5
     VERSION: #1 SMP Tue Mar 16 21:52:43 EDT 2010
     MACHINE: i686  (2702 Mhz)
      MEMORY: 1 GB
       PANIC: "SysRq : Trigger a crashdump"
         PID: 6042
     COMMAND: "bash"
        TASK: f09c7000  [THREAD_INFO: e1ba9000]
         CPU: 0
       STATE: TASK_RUNNING (SYSRQ)
crash>

To exit the interactive prompt and terminate crash, type exit.

Displaying the Message Buffer

To display the kernel message buffer, type the log command at the interactive prompt.

Example 44.3. Displaying the kernel message buffer

crash> log
Linux version 2.6.18-194.el5 (mockbuild@x86-007.build.bos.redhat.com) (gcc version 4.1.2 20080704 (CentOS 4.1.2-48)) #1 SMP Tue Mar 16 21:52:43 EDT 2010
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000010000 - 000000000009fc00 (usable)
 BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 000000003fff0000 (usable)
 BIOS-e820: 000000003fff0000 - 0000000040000000 (ACPI data)
 BIOS-e820: 00000000fffc0000 - 0000000100000000 (reserved)
127MB HIGHMEM available.
896MB LOWMEM available.
Using x86 segment limits to approximate NX protection
On node 0 totalpages: 262128
  DMA zone: 4096 pages, LIFO batch:0
  Normal zone: 225280 pages, LIFO batch:31
  HighMem zone: 32752 pages, LIFO batch:7
DMI 2.5 present.
Using APIC driver default
... several lines omitted ...
SysRq : Trigger a crashdump

Type help log for more information on the command usage.

Displaying a Backtrace

To display the kernel stack trace, type the bt command at the interactive prompt. You can use bt pid to display the backtrace of the selected process.

Example 44.4. Displaying the kernel stack trace

crash> bt
PID: 6042   TASK: f09c7000  CPU: 0   COMMAND: "bash"
 #0 [e1ba9d10] schedule at c061c738
 #1 [e1ba9d28] netlink_getsockopt at c05d50bb
 #2 [e1ba9d34] netlink_queue_skip at c05d40d5
 #3 [e1ba9d40] netlink_sock_destruct at c05d506d
 #4 [e1ba9d84] sock_recvmsg at c05b6cc8
 #5 [e1ba9dd4] enqueue_task at c041eed5
 #6 [e1ba9dec] try_to_wake_up at c041f798
 #7 [e1ba9e10] vsnprintf at c04efef2
 #8 [e1ba9ec0] machine_kexec at c0419bf0
 #9 [e1ba9f04] sys_kexec_load at c04448a1
#10 [e1ba9f4c] tty_audit_exit at c0549f06
#11 [e1ba9f50] tty_audit_add_data at c0549d5d
#12 [e1ba9f84] do_readv_writev at c0476055
#13 [e1ba9fb8] system_call at c0404f10
    EAX: ffffffda  EBX: 00000001  ECX: b7f7f000  EDX: 00000002 
    DS:  007b      ESI: 00000002  ES:  007b      EDI: b7f7f000
    SS:  007b      ESP: bf83f478  EBP: bf83f498
    CS:  0073      EIP: 009ac402  ERR: 00000004  EFLAGS: 00000246

Type help bt for more information on the command usage.

Displaying a Process Status

To display a status of processes in the system, type the ps command at the interactive prompt. You can use ps pid to display the status of the selected process.

Example 44.5. Displaying status of processes in the system

crash> ps
   PID    PPID  CPU   TASK    ST  %MEM     VSZ    RSS  COMM
      0      0   0  c068a3c0  RU   0.0       0      0  [swapper]
      1      0   0  f7c81aa0  IN   0.1    2152    616  init
... several lines omitted ...
   6017      1   0  e39f6550  IN   1.2   40200  13000  gnome-terminal
   6019   6017   0  e39f6000  IN   0.1    2568    708  gnome-pty-helpe
   6020   6017   0  f0421550  IN   0.1    4620   1480  bash
   6021      1   0  f7f69aa0  ??   1.2   40200  13000  gnome-terminal
   6039   6020   0  e7e84aa0  IN   0.1    5004   1300  su
>  6042   6039   0  f09c7000  RU   0.1    4620   1464  bash

Type help ps for more information on the command usage.

Displaying Virtual Memory Information

To display basic virtual memory information, type the vm command at the interactive prompt. You can use vm pid to display information on the selected process.

Example 44.6. Displaying virtual memory information of the current context

crash> vm
PID: 6042   TASK: f09c7000  CPU: 0   COMMAND: "bash"
   MM       PGD      RSS    TOTAL_VM
e275ee40  e2b08000  1464k    4620k  
  VMA       START      END    FLAGS  FILE
e315d764    1fe000    201000     75  /lib/libtermcap.so.2.0.8
e315de9c    201000    202000 100073  /lib/libtermcap.so.2.0.8
c9b040d4    318000    46a000     75  /lib/libc-2.5.so
e315da04    46a000    46c000 100071  /lib/libc-2.5.so
e315d7b8    46c000    46d000 100073  /lib/libc-2.5.so
e315de48    46d000    470000 100073  
e315dba8    9ac000    9ad000 8040075  
c9b04a04    a2f000    a4a000    875  /lib/ld-2.5.so
c9b04374    a4a000    a4b000 100871  /lib/ld-2.5.so
e315d6bc    a4b000    a4c000 100873  /lib/ld-2.5.so
e315d908    fa1000    fa4000     75  /lib/libdl-2.5.so
e315db00    fa4000    fa5000 100071  /lib/libdl-2.5.so
e315df44    fa5000    fa6000 100073  /lib/libdl-2.5.so
e315d320    ff0000    ffa000     75  /lib/libnss_files-2.5.so
e315d668    ffa000    ffb000 100071  /lib/libnss_files-2.5.so
e315def0    ffb000    ffc000 100073  /lib/libnss_files-2.5.so
e315d374   8048000   80f5000   1875  /bin/bash
c9b045c0   80f5000   80fa000 101873  /bin/bash
... several lines omitted ...

Type help vm for more information on the command usage.

Displaying Open Files

To display information about open files, type the files command at the interactive prompt. You can use files pid to display files opened by the selected process.

Example 44.7. Displaying information about open files of the current context

crash> files
PID: 6042   TASK: f09c7000  CPU: 0   COMMAND: "bash"
ROOT: /    CWD: /root
 FD    FILE     DENTRY    INODE    TYPE  PATH
  0  e33be480  e609bf70  f0e1d880  CHR   /dev/pts/1
  1  e424d8c0  d637add8  f7809b78  REG   /proc/sysrq-trigger
  2  e33be480  e609bf70  f0e1d880  CHR   /dev/pts/1
 10  e33be480  e609bf70  f0e1d880  CHR   /dev/pts/1
255  e33be480  e609bf70  f0e1d880  CHR   /dev/pts/1

Type help files for more information on the command usage.

Example 44.2. Running the crash utility