IBM XL Compilers in Gentoo Linux
From UMaine Supercomputer
Rumor has it that the IBM XL compilers are far faster for HPC applications then gcc on the G5 chips. In our previous generation cluster, we found this to be true in the OS X 10.4 series. We also tested a small Linux cluster during this time running Debian, but never got the IBM XL compilers to install as IBM pretty much demands RedHat, Suse or Yellow Dog, all of which we’ve had problems with on the Xserves.
This being the case, we first followed the steps mentioned in this mailing list, to get the compilers installed on Debian. Sadly, we ran into future problems because Debian likes 32-bits, whereas we need the preference to be 64. This lead to the last post about getting Gentoo Linux up and going with multilib enabled. Now, how do we trick the IBM compilers to install on Gentoo?
Well first, we sign our lives away and download the evaluation copies from the IBM website. The key word being evaluation. We’re going to benchmark the applications we commonly run using gcc-4.1.0 and the XL compilers. If we see the performance increase we expect with the XL compilers, we’ll be buying more licences for Linux and not just for OS X.
Anyways, here’s the links.
Extract these files somewhere, noting that you’ll probably want to use the -C flag with tar because the files extract to your current working directory. The only files we’re really concerned with are the rpms in RHEL4/rpms/. We’re sadly going to need to install rpm as well, because it’s my belief that the compilers use rpm internally to make sure all the components are installed. Luckily, they don’t check for prerequisites, so we’ll force rpm to install everything.
emerge '>=rpm-4.2-r1'
The following steps are essentially the same (barring directory names) for both xlf and xlc. First we get into the RHEL4/rpms directory and execute:
for i in `ls`; do rpm --nodeps -i $i;done
This should install everything you need into /opt/ibmcmp/. So now we have the compilers installed, but they are not configured. I refuse to find a video card, install X, and install Java on our Xserves so I could click a little “I Accept” button about the license. In fact, attempting to get this working on my workstation didn’t work either. So read the license then edit /opt/ibmcmp/xlf/10.0/bin/new_install and add the line “return 1;”
sub click_accept {
++ return 1;
#***************************************************
#* Runs license accept process..
#*
We’ll now generate the configuration file, I found it silly that the file is installed into /etc/opt/ibmcmp/xlf/10.1, when I have a perfectly good directory tree already in /opt. So I made a link. The installer also checks for the Suse, RedHat or YellowDog version. We’ll pretend that we’re RedHat, which means that we have a certain file in /etc. (There may be other small differences I haven’t found yet. For instance in Debian, one has to link /bin/awk to /usr/bin/awk as that is where the compilers expect it to be).
echo "RedHat release 4.3 (Final)" >> /etc/redhat-release ln -s /opt /etc/opt /opt/ibmcmp/xlf/10.1/bin/new_install
We’ve now generated a configuration file. Of course that’s not everything, sadly the installer did not find the 32-bit libraries correctly, instead linking everything with the 64-bit ones. This is fairly easy to fix. Open up /opt/ibmcmp/xlf/10.0/xlf.cfg and pretty much find all the variables that are declared as:
foo = foo_64 =
And fix them so that foo is correct. If your system looks the same as mine, then the files linked in the references should work. If not, you can browse through them and it’ll become quite apparent what I had to edit.
Again a simple test to check that things worked.
# cd /opt/ibmcmp/xlf/10.1/samples/floating_point/ # xlf -q32 -c fhandler.F xlf: 1501-274 An incompatible level of gcc has been specified. ** fhandler === End of Compilation 1 === 1501-510 Compilation successful for file fhandler.F. # file fhandler.o fhandler.o: ELF 32-bit MSB relocatable, PowerPC or cisco 4500, version 1 (SYSV), not stripped # xlf -q64 -c fhandler.F xlf: 1501-274 An incompatible level of gcc has been specified. ** fhandler === End of Compilation 1 === 1501-510 Compilation successful for file fhandler.F. # file fhandler.o fhandler.o: ELF 64-bit MSB relocatable, cisco 7500, version 1 (SYSV), not stripped
Once again, I’m still working on testing things out and do not have any reasonable benchmarks to post at this time. Ideally, I’ll post some soon as opposed to posting that I screwed up and none of this works.
You’ll note the one worrying thing here is the error that the XL compilers are spitting out:
xlf: 1501-274 An incompatible level of gcc has been specified.
I can’t find anything on google or in the documents about this error. I can confirm that it did not occur with glibc-2.3.4 and gcc-3.4.6. It seems benign but I’d love it if anyone had an explanation or potential fix. We’ll see what happens in the future…
Errata
- By linking against an earlier version of gcc, specifically 3.4.6, I was able to remove the 1501-274 error. Probably just like rpm is hardcoded into XL, a maximum gcc version is as well. I also found that for some reason /lib64/libgcc_s.so.1 kept getting installed as a 32-bit library. I’ve copied the correct one in, but it is strange and worth noting. Currently I’ll continue with these slight changes. We can easily use gcc4 when running user applications, and keep XL linked against the earlier version of gcc.
- If you’ll be using any code that is threaded, XL apparently likes to have both NPTL and linuxthreads enabled glibc’s. At least it doesn’t like it when I don’t have the linuxthreads one around, even though it still seems fine linking against the NPTL one by default.
- Limited testing with HPL hasn't shown much difference between XL and gcc-4.1.1. This being the case, I'll be ignoring XL for a while until we finish other projects such as the OS Provisioning via Moab. Hopefully I'll have time to visit this later.
