Portable Software

Much of the software out there in sourcecode form is already "portable": readily altered to suit different computing circumstances. Often, though, the default configuration settings and the "quick start" instructions make assumptions which aren't reasonable, valid, or even safe in a production environment such as ours.

This article highlights some of the more important and immediately useful tricks of the trade, which will help you install and use programs and libraries without conflicting with software which may already be present on the system, and enable you to build and run the same programs on multiple platforms (MacOS X, standalone Linux systems, or one or other of the compute clusters) with minimal change.

I've tried to keep the following discussion fairly general; the examples are based on MacOS X, but that's simply because that's where I've been doing this sort of thing most recently. Readers working under Linux and/or using the compute clusters should have little difficulty modifying the advice to suit their own environments.

Please see also:

The Problem

You have just found (or been presented with) an inscrutable lump of sourcecode which you wish to use, and the instructions say "do this". But when you try this, you get complaints about missing libraries, missing tools, or inadequate permissions.

When this happens to you, try the following, in the order given. Those attempting to build Python libraries should bear in mind that some of this may be going on under the covers: please see also Python and SciSoft under MacOS X (Astrophysics Systems), notably the section on third-party libraries.

  • If the this in question is ./configure, and the complaints concern missing libraries, you're in luck. Please see the section below on automatic configuration.

  • If this is make install or sudo make install, and an earlier step was ./configure, you're in luck again. Please see the section below on automatic configuration.

  • If this is make or make install, and you were provided with a Makefile, you'll need to edit said file in order to specify any required libraries, and to change the destination directory; similarly if you were provided with a shell script. Please see the companion piece (currently in preparation) for more detailed advice.

Automatic configuration

Increasingly, much of the software you may wish to install uses the "Autotools" build system (GNU autoconf, automake, libtool etc), or one or more hand-crafted shell scripts which emulate it. If your software's "quick start" installation instructions tell you to do something like this:

make install

.... then it follows the Autotools pattern: configure builds a Makefile, which is used to build, and to then install, the software.

  • The Good News is that configure relieves you of the burden of dealing with much of the operating-system administrivia, such as the fine details of how to build or use a library.

  • The Bad News is that it will need guidance to avoid conflicting with stuff which is already on the system, and may need help to find libraries that the software being built may need to be linked to.

To help configure to help you, you'll need to give it some command-line options, to educate it about the fine details of the environment. Please note:

  • These options are discussed and demonstrated individually, but can and should be used in whatever combination is necessary.

  • Often (but not always), ./configure will ignore options that aren't relevant to the software it builds. It's thus reasonable, and common, to use the "kitchen sink" approach when building multiple Autotools packages to work together: use every option that's relevant to something on all of them, unless there's objections.

Installation directories:  --prefix=

By default, configure will arrange for the code being built to be installed into /usr/local/bin, /usr/local/lib etc. This is seriously Bad News on a production system: for one, you won't be able to write into it; for another, there's already stuff there, with which it can conflict. (As a recent example from Astrophysics, the gfortran from HPC, and the binary build of the obsolete g77, would both go into /usr/local, and included files of the same name in the same directories which were not identical.)

To change the destination directory, add a prefix option. For example:

./configure --prefix=/Data/myspace

.... tells configure to tell make install to put your executables and libraries in /Data/myspace/bin,
/Data/myspace/lib, and so forth. (Please note that these directories won't actually be created until you say make install.)

Libraries:  --with-LIB=

Libraries are looked for in certain standard places, but sometimes the net is not cast wide enough, or you may wish to use your own versions; and sometimes the use of a particular library is optional, and needs to be enabled at configuration time. In either case, you'll need to use one or more of the with-LIB options, for suitable values of LIB. Real-life examples from a recent installation exercise include:

./configure --with-fftw=$(FFTW)/lib \
./configure --with-jpeg-libdir=$(JPEG)/lib

.... where $(FFTW) and $(JPEG) are Makefile variables which contain the pathnames of the "prefix directories" under which the respective libraries live (in this case, they both happened to be set to the MacPorts default of /opt/local). I've also used a backslash to fold the first line to fit into this column.

Problems:  --help

Not all software suites are fully well-behaved, or completely debugged under every possible combination of circumstances. One thing that's always worth trying if you get stuck, or wish to list the available options (to see what's possible, what's necessary, and what's optional) is:

./configure --help

This is usually sufficiently long and detailed that it's worth redirecting it to a file for printing-off, or using a pager to view it:

./configure --help >help.out
./configure --help | less

Some of the more interesting problems I've encountered have included undeclared dependencies, incomplete configuration information transfer, and finding it desirable or necessary to run configure from somewhere other than the sourcecode directory. I'll go into some of the deeper magic required in these circumstances in the companion piece to this article.

.... but I can't remember all that

At this point, you're probably getting weary of getting multiple arguments bit-perfect, and dreading having to do the same thing for every other library and program you need to build. So do I; and that's where batch files come in. My typical modus operandi is:

  • Run ./configure --help to obtain a first guess at the required configuration options.

  • Write an initial batch file, containing a first cut at the configure incantation, then:

    • Run said file, and check for problems.

    • Do the make, and the make install, and check for problems.

    • If there's problems, seek help from ./configure --help if necessary, edit the batch file, and loop back.

  • Once everything's installing correctly, commit the batch file(s) (and optionally the original source tarball(s)), but nothing else, to git.

    • If it's something you think other users would benefit from, please send us said files and tarballs, so that we can build a package for more general distribution.

    • Bonus marks for persuading the batch file to download the latest version of each tarball from its distribution site, hands-free, for sufficiently general values of "latest".

Categories: Apple | Astro software | Astrophysics | Development | HOWTO | Linux | Mac | OS X