Workaround for Crashing Xen Kernel on Ubuntu Hardy High Memory Systems

(updated: the problem still exists with kernels on systems with > 4Gb of memory, see this xen kernel bug for Ubuntu which has been ignored for months along with many similar bugs
We really get to taking Linux kernel stability for granted sometimes, but knowing that the hardware was solid after plenty of testing we found that the problem was surprisingly coming from the kernels distributed with Ubuntu.
The Xen kernel 2.6.24-17-xen which is currently available with the standard distribution of Ubuntu Hardy is a crashing fool on the Dell SC1425, but it holds up under light IO and therefore does not get crashing until you put it on a machine that’s chugging away full force, and even then it doesn’t always crash.
In my case it ended up giving kernel oops, cpu soft locks, and other problems for other people. This happened only a few times a day in my case and was somewhat manageable with scripts to monitor dmesg and reboot when trouble was found in addition to scripts running on the (usually stable due to low IO) dom0 to check services.
The problem doesn’t seem to have a high priority with Ubuntu at the moment, they are probably very busy with numerous other bugs that probably should have been fixed pre-release but due to their strict timeline of releases I guess an unfinished product is inevitable. Though it has been very impressive how they have managed to keep the vast majority of their software stable, playing nice and sound.
So back to my fix, that’s likely what you’re here for anyway.
I tried the -19 revision kernels put out by Hirano Takahito and they crashed a lot less often.
But the best solution was to go with Xen’s stock 2.6.18 kernel. Get the source at Xen.org. Here are my configs for dom0 and for domU kernels for the Dell SC1425 server in particular.
The documentation at Gentoo on Xen was very helpful in ensuring all the options were correct for each kernel type and I got a working kernel on the first try.
Since using the Xen recommended kernel all crashes have gone. Ubuntu tried to graft the Xen code onto a kernel that has too many changes to make it a stable choice, at least so far. Hopefully the Xen team will adopt a newer kernel but for now we’ll be glad that there’s a solution to this problem at all.