Debugging after kernel panic
Description:
If a user experiences system crashes, when using the censhare Clients with the VFS enabled, this may be an indication of an error in the kernel extension (not necessarily censhare however). Because such errors are difficult to analyze, Mac OS X provides the ability to intercept the panic to generate so-called core dumps as soon as such a system crash occurs.
A core dump provides an image of the machine state at the time of the crash. It collects information about system extensions loaded (such as the VFS) and values from the memory and the register content of the processor. Based on such a dump it can be traced, at which position in the program code the system crash occurred.
Since these core dumps are produced at a time when the operating system is already in an unstable state, the dump files are not stored on the computer itself. This is a precautionary measure from the operating system, because among other things, it can not be guaranteed that such a write operation would not damage the local hard drive.
For this reason it is necessary to setup a dedicated core dump server to collect the dump files of affected computers (the so-called core dump clients).
OS X Kernel Panic
Note: Use a core dump server only in trusted networks and only on servers where you can trust all users Core dumps are transmitted unencrypted from a core dump client to the core dump server Since core dumps can contain sensitive data, make sure that they are protected against unauthorized access
Core dump servic
The core dump service consists of a core dump server, the censhare crash server and the core dump clients running the "censhare Crash Reporter" app. Using the collected data sent via these services, it is possible for us to analyze the problem of the crash closely and isolate the cause in more detail.
This article provides all necessary information needed on how you can set up such a system in your network.
System Requirements
For the core dump server, you need a Mac with the following characteristics:
At least Mac OS X 10.5
Enough memory (a single core dump is several hundred MB in size)
The server must be accessible on a 24/7 basis and own a static IPv4 address.
This IP address must be visible to all core dump clients no NAT and or firewall (between client and server, etc.) is allowed.
For the core dump Cient Macs the following conditions apply:
At least OS X 10.7 (Lion)
A recent censhare client (at least 4.7) with VFS enabled
The clients are in the same network as the server (no NAT and no firewall between client and server, etc.).
Setting Up the Server
The server consists of two parts. The core dump server and the censhare Crash Server app.
Setting up a Core Dump Server
To set up a core dump server, proceed as follows:
Create the directory "/PanicDumps". In this directory, the received core dumps are stored.
sudo mkdir /PanicDumps sudo chown root:wheel /PanicDumps sudo chmod 1777 /PanicDumps
Enable the Core Dump Server using the tool "launchctl". With the argument '-w' the core dump server is restarted automatically if you reboot the computer.
sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.kdumpd.plist
You can check also whether the core dump server is running using a launchctl command:
sudo launchctl list | grep kdump
As longa as com.apple.kdumpd is displayed in the console output, the server is up and running. If, however, there is no output, the server has stopped.
Setting up the censhare Crash Server App
Censhare crash server app must be running on the same computer on which the core dump server is set up.
Download the censhare Crash Server
Start the server by double clicking the icon
The server has not to be configured and has no graphical user interface. Its task is to receive panic reports of core dump clients and to store them in the "/ PanicDumps" directory. Please make sure that the app gets launched automatically each time after a system reboot.
Setting up the Clients
To enable the Macs on which the censhare Client is running to submit core dumps to the server, they must be configured accordingly. Furthermore, the app "censhare Crash Reporter" should be installed on all your computers.
Setting up the core dump clients
Download "Kernel Core Dump Service" from the link at the bottom.
Open the AppleScript Editor (located in Applications/Utilities/).
Open the Kernel Core Dump Service.app in the Editor.
Replace the found IP 192.168.213.23 entry of the attribute panicd_ip with the IP of your own core dump server.
[...] do shell script "nvram boot-args="debug=0x0544 _panicd_ip=192.168.213.23"" with administrator privileges [...]
Save the modified app.
Copy the app on each computer that is to be used as a Core Dump client.
Run the script, click on "Enable" and then restart the Mac.
Setting Up and Using the censhare Crash Reporter App
This app should be executed each time after a system crash has occurred and the system has rebooted again.. The app will ask you several questions and sends the response with additional information to the censhare crash server. However, first the reporter must be adjusted to the IP of the censhare Crash Server.
Download the app censhare Crash Reporter (see bottom link).
Now right-click on the app in the Finder and select "Show Package Contents" from the context menu.
Open the file "Info.plist" from inside the directory "Contents" with a text editor.
Replace the IP 192.168.213.23 in the server-ip attribute with the IP of your Core Dump server.
JVMArguments --server-ip=192.168.213.23 --server-port=9999 --question=Your name --question=How many (render-)clients ran in parallel --question=Which client version(s) did you use --question=What have you done just before the system crashed
The values of the attributes reflect the questions asked by the app. You can add any number of additional attributes of this value, as more questions are desired or needed
Save the modified app.
Install the app on each computer that is to be used as a Dore Dump client.
Run the app every time when there was a system crash and the system has rebooted.
Testing the Core Dump Services
To test if the individual parts of the Core Dump Services operate all as expected, proceed as follows. The test is performed on a computer which is configured as a Core Dump client.
Quit all running applications, save open data.
Run the following command in the terminal. Attention, this will lead to a system crash:
sudo dtrace -w -n "BEGIN{ panic();}"Your screen should now appear similar to the following output (white text on black lines):
Please wait until the end of data transmission which is done when the screen shows "Waiting for remote debugger connection".
After that restart your computer.
After restart now run the app censhare Crash Reporter Please do also answer all questions (for testing functionality) and then press "Send Report".
Then connect to the Core Cump server.
Open the directory /PanicDumps. Here should now appear two files. One file is the core dump, the other file is the panic report that was generated by the crash reporter.
If one of the two files does not exist, please check if all servers are running and whether the write permissions are set correctly on the directory PanicDump (try using the command "touch /PanicDumps/test.file" to create an empty file. If this fails, the permissions are not correct.
Note: The transmission of core dumps by a core dump client is only possibly over the installed Ethernet interface of your Mac. Wireless transfer via the AirPort interface is not supported by Apple. That means device units without a wired Ethernet interface (the MacBook Air) can not send core dumps unless you try a Thunderbolt / Gibabit Ethernet adapter (not tested). The use of the "censhare Crash Reporter 'is not affected by this limitation and should therefore be used in any case.
For additional information about Apple's core dump client / server, see
For more information, see this Technical Note TN2118 - Kernel Core Dumps