LSB Distro Testing

I’ve seen several requests for a simple set of instructions that test a distribution against the LSB. I wrote some Debian-specific instructions in a mailing list post back in October, and thought they’d do better as more general instructions.

Before you start, you have to find out two things about the distribution you want to test:

  • How to install the LSB on your distribution. At minimum, your distro should provide something like a “lsb” package. Make sure that’s installed.
  • How to install LSB packages on your distribution. RPM-based distros have it easy here, since the procedure is likely to be the same as installing any other package on your system. Debian users (and users of Debian derivatives) need to use alien; I’ve found that you get the best results with alien -ick [package]. Other distributions will have their own ways, possibly involving alien as well.

Go to the LSB download page and start downloading everything under the Runtime Tests for the LSB version and architecture you want to test. Make sure you don’t get the betas unless that’s what you want to test. You will also need the lsb-python package from the Application Battery list. Install all of these packages.

Once all of the packages are installed, run the tests:

  • /opt/lsb/bin/lsblibchk
  • /opt/lsb/bin/lsbcmdchk
  • Add /usr/opt/lsb/appbat/bin to your PATH, run gcc -dumpmachine to get your machine triplet, and run /opt/lsb/share/test/qmtest_libstdcpp_[version]/run_tests. Answer the questions as appropriate.
  • For the X test suites, install enough of XFree86 or X.org on your system for proper client support, as well as the Xvfb X server. Debian users can install the x-window-system-core and xvfb packages, but see below about problems with Xvfb. After this is done, go to the directory /opt/lsb/test/vsw4 and run the run_vsw4.sh script found there.
  • For the main runtime test, make sure the loop driver is loaded. (This is mostly a problem on systems using udev.) After the test package is installed, set passwords for the “vsx0”, “vsx1”, and “vsx2” users, log in as vsx0 at a system console, and run run_tests. Logging in is required; su or sudo won’t work. The default answers to the questions asked are mostly OK, but a few don’t have good defaults. In particular, you’ll have to tell the script the right passwords for the vsx users, and you should make sure the test block device is /home/tet/test_sets/nonexistb. Be prepared to type the root password several times to set up various things the tests need.

Most of the tests are quick, taking less than 15 minutes usually per test. The runtime test takes somewhere in the neighborhood of six hours, and often looks like it has hung. Don’t assume the runtime test has hung until it’s run overnight. If you’re using an emulator, don’t give up on runtime until it’s run for at least 24 hours.

The tests create two kinds of files: journal files, and the official runtime report (created by the runtime test). Where you ran the tests, look for files with names starting with “journal” in the current directory, or for paths like results/0001e/journal. The C++ tests create a directory called “qmtest_libstdcpp_[version]”. There’s a handy utility called tjreport in /opt/lsb/bin; run that on the journal to get a quick summary of the results. If you want to post results, use tjreport. The official runtime report (in /home/tet/test_sets/results) has some additional information: a list of FIP (Further Information Provided) results.

What to do with failures: First, make sure it’s really a failure of the environment, and not a failure of the tests. The LSB publishes a list of official waivers for test failures; check that your failures aren’t on that list. For failures not on that list, Google is your best resource; most likely, someone else has experienced the same failure, and will have more information about it. If you really can’t figure it out, come over to one of the LSB’s mailing lists and ask around.

On Xvfb: The Xvfb versions shipped with most distributions have bugs that can cause problems. Anything running XFree86 will likely not be able to complete the X tests, and anything running X.org may see a number of failures. Because Xvfb is not required by the LSB, you can replace Xvfb with one that doesn’t have these bugs. Recent Debian xvfb packages for etch don’t have these bugs, so Debian and derivatives can use them for testing. (The sarge backport of Xvfb can be installed on vanilla sarge without upgrading any of the other XFree86 packages; missing packages can also be pulled from the sarge backport without affecting the packages that do ship.)

Please post questions, problems, criticisms, etc. in the comments. A version of this might end up on the LSB site someday, so any improvements would be appreciated.

One thought on “LSB Distro Testing

Comments are closed.