swpkg - Software Packaging

Table of contents

• History
• Design requirements and goals
• Essential concepts
  • The basics
  • The source directory
  • The target directory: to link or not link?
• Advanced concepts and special cases
  • Handling dependencies
  • The target directory: which path to use?
  • "Local" software
  • Handling additions to packages already installed
  • Handling man pages and Info documents
  • Static versus dynamic libraries
• In short

The concept of installing additional software on a system under /usr/local is (relatively) old. Typically, the hierarchy found under /usr/local replicates the one found under /usr where most of the base system is installed. This can get confusing, as conventions have changed with time. For example, under SunOS (e.g. in the pre Solaris days), binaries were often located under the "etc" directory, something that many people now regard as wrong or even evil. Also, while a particular vendor has complete control of its system, third party software comes in various forms and shapes, following what sometimes feels like random conventions. And this actually brings us to the problem with the usual /usr/local approach: it is not centrally controlled nor managed. Not only this, but most vendors have software package tools aimed and managing the installation, updates, removal, and dependencies between the different software components they ship. They have a fairly easy job, as they have complete control over things.

Of course, each vendor uses their own proprietary package tools, so if it would be a real headache for software maintainers to ship their product for multiple OS flavors as they'd need to understand and master all of these tools, not to mention the fact that third party software is most often distributed in source form, not binary form ready to be installed and used.

Because of the lack of management, it is often difficult to upgrade or remove a particular software, meaning that /usr/local is almost never cleaned up, and piles up. It is also practically impossible to know or understand the many dependencies existing between different packages.

Finally, as there are now thousands of freely available software applications available out there, and almost as many maintainers, it is highly unlikely for any package management tool to ever become universaly used. In other words, any system we develop should assume as little as possible and work with software as installed following the maintainer's specifications.

The "swpkg philosophy" was born several years ago and first implemented in an early form at Bell Laboratories. Although it has kept evolving, little has changed. The management tools part of the swpkg were written with the following requirements in mind:

• place as few requirements as possible on the software to be managed (ideally none)
• allow easy transition of an existing installed base of software
• be transparent to users

Also, these are the goals swpkg aims to solve:

• addition as well as removal of software packages should be easy (in particular upgrades and downgrades)
• installed software should be easy to identify and/or inventory
• dependencies between packages should be apparent

In particular one should note that swpkg makes no attempt to address the distribution of installed software packages across systems. It is believed that that by keeping the swpkg tools simple and flexible, it will be possible to build a distribution mechanism on top of it.

To avoid the chaos which is inherant to installing many very different software packages under the same subtree, a simple solution is used: each software package is installed under its very own directory. (You're probably familiar with "GNU configure" scripts, this amounts to using the "--prefix" option.) This directory is named after the software and its version.

To keep this transparent to users, symbolic links are used to populate a common directory. The subtree located in this directory is what gets used and referenced to access the installed software.

All packages should be installed under a common directory called the source directory, and the directory under which symbolic links are created is called the target directory. These default to /usr/local/pkgs and /usr/local, respectively.

Here's an example of what one may find under a source directory:

drwxr-sr-x   6 swpkg    swpkg        512 Jun  4 08:57 amanda-2.4.2p2
drwxr-sr-x   6 swpkg    swpkg        512 Mar 27  2001 apache-1.3.12
drwxr-sr-x  10 swpkg    swpkg        512 Aug 15 09:51 bind-9.1.3
drwxr-sr-x   6 swpkg    swpkg        512 Sep 25  2000 bzip2-1.0.1
drwxr-sr-x   7 swpkg    swpkg        512 Feb 13  2001 clearcase-4.0
drwxr-sr-x   8 swpkg    swpkg        512 Dec 12  2000 gcc-2.95.2
drwxr-sr-x   6 swpkg    swpkg        512 Sep 26  2000 gdb-5.0
drwxr-sr-x   3 swpkg    swpkg        512 Jul 25  2000 memconf-1.69
drwxr-sr-x   8 swpkg    swpkg        512 Aug  2 17:26 net-snmp-4.2.1
drwxr-sr-x   5 swpkg    swpkg        512 Jan 13  2000 perl-5.005_03
drwxr-sr-x   4 swpkg    swpkg        512 Jan 21  2001 rsync-2.4.6
drwxr-sr-x   4 swpkg    swpkg        512 Apr 11  2000 screen-3.9.4
drwxr-sr-x   7 swpkg    swpkg        512 Sep 20 11:22 tar-1.13.23

drwxrwsr-x   2 swpkg    swpkg       9728 Sep  1 08:00 /usr/local/bin
lrwxrwxrwx   1 swpkg    swpkg         39 Jun 20 13:59 /usr/local/bin/bunzip2
                                                      -> /usr/local/pkgs/bzip2-1.0.1/bin/bunzip2
lrwxrwxrwx   1 swpkg    swpkg         37 Jun 20 13:59 /usr/local/bin/bzcat
                                                      -> /usr/local/pkgs/bzip2-1.0.1/bin/bzcat
lrwxrwxrwx   1 swpkg    swpkg         37 Jun 20 13:59 /usr/local/bin/bzip2
                                                      -> /usr/local/pkgs/bzip2-1.0.1/bin/bzip2
lrwxrwxrwx   1 swpkg    swpkg         44 Jun 20 13:59 /usr/local/bin/bzip2recover
                                                      -> /usr/local/pkgs/bzip2-1.0.1/bin/bzip2recover
drwxrwsr-x  10 swpkg    swpkg       1024 Aug 20 16:31 /usr/local/include
lrwxrwxrwx   1 swpkg    swpkg         43 Jun 20 13:59 /usr/local/include/bzlib.h
                                                      -> /usr/local/pkgs/bzip2-1.0.1/include/bzlib.h
drwxrwsr-x  29 swpkg    swpkg       3072 Aug 20 16:31 /usr/local/lib
lrwxrwxrwx   1 swpkg    swpkg         40 Jun 20 14:04 /usr/local/lib/libbz2.a
                                                      -> /usr/local/pkgs/bzip2-1.0.1/lib/libbz2.a
drwxrwsr-x  15 swpkg    swpkg        512 Sep 22 02:01 /usr/local/man
drwxrwsr-x  15 swpkg    swpkg        512 Sep 22 02:01 /usr/local/man/man1
lrwxrwxrwx   1 swpkg    swpkg         44 Jun 20 14:04 /usr/local/man/man1/bzip2.1
                                                      -> /usr/local/pkgs/bzip2-1.0.1/man/man1/bzip2.1

Each package is entirely installed and self-contained under its own directory in the source directory. While it may seem reasonnable to directly access installed files using their real path, it is very important to understand that this must never be done as it will only lead to problems.

For example, if you write a shell script using bzip2, and use its real PATH "/usr/local/pkgs/bzip2-1.0.1/bin/bzip2". This works fine and may seem alright, up to the day when the system administrator installs the latest version of bzip2, and removes the old one. When this happens, "/usr/local/bin/bzip2" will still exist and point to the newer version, while /usr/local/pkgs/bzip2-1.0.1/bin/bzip2 no longer will.

The above holds true for binaries, but also and even more for other files, such as dynamic libraries and so on.

At first, it seems that the target directory should be the union of all the packages found in the source directory, with every single file found in these packages represented by a symbolic link in the target directory. While collisions between different packages will occasionally occur, there is nothing really wrong with doing this, but there is also little sense in doing so, and it may also be wrong/misleading.

Let's take the example of a perl installation. As you may know, perl comes with a number of base modules which may be used in scripts, and more modules can be added to the installation. Such extra modules are "site" specific, and installed under a "site_perl" directory:

drwxr-sr-x   3 swpkg    swpkg        512 Jan 13  2000 /usr/local/pkgs/perl-5.005_03/lib/perl5/site_perl/
drwxr-sr-x  32 swpkg    swpkg       1024 Sep 13 10:01 /usr/local/pkgs/perl-5.005_03/lib/perl5/site_perl/5.005
-r--r--r--   1 swpkg    swpkg      36021 Sep  3  1998 /usr/local/pkgs/perl-5.005_03/lib/perl5/site_perl/5.005/Expect.pm

drwxr-sr-x   3 swpkg    swpkg        512 Jan 13  2000 /usr/local/lib/perl5/site_perl
drwxr-sr-x  23 swpkg    swpkg       1024 Nov 13  2000 /usr/local/lib/perl5/site_perl/5.005
lrwxrwxrwx   1 swpkg    swpkg         65 Jun 20 14:04 /usr/local/lib/perl5/site_perl/5.005/Expect.pm
                                                      -> /usr/local/pkgs/perl-5.005_03/lib/perl5/site_perl/5.005/Expect.pm

$ perl -V
Summary of my perl5 (5.0 patchlevel 5 subversion 3) configuration:
[...]
Characteristics of this binary (from libperl):
  Built under solaris
  Compiled at Jan 13 2000 15:23:07
  @INC:
    /usr/local/pkgs/perl-5.005_03/lib/perl5/5.00503/sun4-solaris
    /usr/local/pkgs/perl-5.005_03/lib/perl5/5.00503
    /usr/local/pkgs/perl-5.005_03/lib/perl5/site_perl/5.005/sun4-solaris
    /usr/local/pkgs/perl-5.005_03/lib/perl5/site_perl/5.005

Each installation and site should decide how much to link in the target directory. Ask yourself what is really needed, and only link the minimum. Typically, this means that only binaries found under the bin directory and some others should be linked. Unless some users are developers needing some of the libraries installed, C header files (typically found under the include directory), static and dynamic libraries (typically found under the lib directory) don't need to be linked. Some other files may be linked simply for ease of access, for example configuration files (typically found under the etc directory) often accessed by system administrators.

By default, it is recommended to only link files found below one of the following directories:

• bin and sbin which are referenced by the PATH environment variable.
• man which is referenced by the MANPATH environment variable.
• info (without any included dir catalog) which is referenced by info tools.
• lib/app-defaults which is typically referenced by the XUSERFILESEARCHPATH environment variable.

Many software packages depend on others. There are two ways to approach the problem:

• the easy way: don't worry, be happy!

  To save time and make things easier, it is possible to mostly ignore dependencies and not worry about them. Of course, while it will be alright most of the time and for a large number of software packages, problems will eventually arise.

  To go this way, software packages are simply built using the resources found in the target directory, and therefore end up referencing other packages found there at the time they are built. It is then difficult (if not impossible) to keep track of existing dependencies.

• the detailed way: nothing left to chance.

  To keep track of all dependencies, and leave nothing to chance, software packages should be built without using the resources of the target directory; rather, each package directly consumes other packages found under the source directory. For this to work, required software packages are linked inside the new package directory, turning this directory into a small "target directory" of its own.

  For example, here we have the ost-1.3 package which depends on gcc-4.1.2:

  lrwxrwxrwx   1 root     root          25 May 14 21:20 /usr/local/pkgs/ost-1.3/bin/gcc
                                                        -> /usr/local/pkgs/gcc-4.1.2/bin/gcc
  lrwxrwxrwx   1 root     root          25 Mar  6  2007 /usr/local/bin/gcc
                                                        -> /usr/local/pkgs/gcc-4.1.2/bin/gcc
  

  It is then possible to have several packages depending on the same package X, and have them use different versions of package X, providing extra flexibility. In particular, this means that you will never be forced into upgrades by dependencies: if you need to upgrade package X to install a new one, you don't have to worry about:

  • breaking already installed packages depending on package X
  • having to recompile many already installed packages (which can easily turn into dozens and dozens as there is a cascading effect here)
  • making sure that all packages depending on package X support the new version you are upgrading to

As mentioned above, the default is /usr/local. It is important not to read to much into this, and to understand that this was chosen more so that it would speak to people and out of the need to have a default than anything else. It is also what makes the most sense if you want to migrate an existing base of installed software (which for most organizations is installed under /usr/local.

The truth is that /usr/local is a poor choice, especially if you plan to handle dependencies the "detailed way". Too many scripts (and in particular configuration ones) are too smart for their own good and will try very hard to find everything they need anywhere on your system. This means that they will probe a number of "standard" directories searching for things, and will use anything they find, often without even making it clear where the goods where found. /usr/local is too standard and common a location to escape such probing, and it makes installing new software without dependencies to /usr/local harder than it should be.

In other words, it is strongly recommended to use an obscure path as the target directory. However, this is not always practical, especially when migrating. It is also a good idea to choose a source directory that is not under the target directory (and vice-versa).

What if you are stuck with /usr/local?
Well, as mentioned above, it will make things harder, but there are ways you can help yourself:

• Play tricks with permissions: set the group of the target directory (or all the directories beneath it if the source directory is under it as well) to a special group and then remove permissions to access the directories for the group. Then use an account belonging to the group for software installations.
• Dedicate a system where installations will be done: If you look around hard enough, you'll probably find an unused system, or one on the brink of retirement for the job. (Too old may be problematic as it'll be slow which isn't a desirable characteristic to do a lot of compilations.) Such system could be setup without a /usr/local directory, or a minimal one.
• Setup a chrooted environment with no /usr/local: If you can't dedicate a system but have disk space, this might work well for you. It may prove difficult to do as you need the OS part to be the real thing.
• Got a better idea? let me know about it!

Most environments have locally developed software installed on their systems, ranging from simple shell scripts to extensive C programs. No, there isn't anything wrong with these, but where do they fit in all this?

While they could be installed in the target directory, it doesn't make sense to mess up a nice and clean setup. Instead, they should be grouped under one or more packages in the source directory and linked in the target directory as any other package. Name your package in a way that will not conflict with real packages, use the version number in the name or not depending whether you actively maintain different versions. Don't necessarily create a package for each little piece of code locally written and maintained, but group these into a bigger package. In short, do what makes sense, and if you can't decide on a name consider using "local".

For most packages, once you've installed them, it's all over, and nothing else will need to be done with them. Some packages aren't this way, and will need some special attention and care. The following gives some examples along with recommendations to handle such packages:

• Perl

  Perl is actually a relatively easy case because it has its own mechanism to handle module additions post-installation. These modules will get installed under the base of the Perl installation, and everything will be fine. Whenever this is done, there isn't even much need to update the links in the target directory because in all likelihood, the directories where modules are installed aren't linked in the first place, although man page directories probably are. Then it's just a matter of having the right procedure in place so that when perl is upgraded, all modules are also reinstalled. This isn't a big deal especially if you properly keep track of what software is installed in the source directory. (Reinstalling modules for new Perl installations is a sane thing to do anyways :-)

  The bigger challenge with Perl comes with packages that depend on Perl and want to install modules. You may well end up with a circular inter-dependency between the two packages. While this may seem problematic, it really isn't although it may be a bit tricky to handle.

• NET-SNMP

  NET-SNMP is a collection of SNMP tools. It comes with a large number of MIB files which are used to convert numeric OIDs (long ugly strings) to human readable OIDs. (If you must, think of working with IP addresses as opposed to hostnames.) New standard and proprietary MIBs are defined all the time, and NET-SNMP (rightly) does not even try to keep up with all of them. MIB files can be added at any time in a specific directory, and the tools will pick these up automatically. Let's go over the reasonnable options here:

  • Files are added directly under the NET-SNMP package in the source directory: While not a terrible way to handle this, it is fraught with problems. All these MIB files are unmanaged, so when you upgrade NET-SNMP, it may be a hassle to find out which ones where added on later. Remembering this fact alone isn't guaranteed. (So we have two fairly critical differences between NET-SNMP and Perl.)
  • Files are installed as an independent (pseudo) package under the source directory: The NET-SNMP package is made to depend on the pseudo package. This is probably the cleanest way to handle this particular case.
  • Files are installed under the "local" package (see above) under the source directory: The NET-SNMP package is NOT made to depend on the "local" package, instead it is built to know to look for the files directly in that package. This will cause less hassle when installing additional MIB files (as no links will need to be added/updated), but isn't absolutely clean (and breaks the "direct reference rule" existing for files under the source directory), although it is reasonnable. It is important to understand why breaking that important rule is deemed reasonnable in this case: the "local" package is very special as it will always be there (since the name won't change, and it is usually not versioned).

Manual pages and Info documents are just like any other file, they can (and should be linked in the target directory without any adverse effect. What gets them to be mentioned here is that in both cases, there is an extra file than needs to be updated as files are added or removed:

• database file for whatis(1), apropos(1), man(1), ..: This file (which name varies per OS) is built by a special command (makewhatis(8), catman(1M), ..) by scanning for manual pages found under the man directory. Such tool needs to be run regularly to keep the database up to date, missing entries for man pages are not reported by the above user tools but still accessible via man(1).
• directory file for info(1), emacs(1), ..: This file (named "dir") lists all the Info documents available under the info directory. It is critical to have this up to date as the documents are found using these entries. Normally the directory file is updated as part of the installation, rather that by a regular cron job.

swpkg offers no new insight to the often heated debate over static versus dynamic libraries. It is however important to realize that when building a package from source, you should almost never remove the dependencies used during the build once you have completed it, even for static libraries which code may contain direct references to files under its own source directory. If you were to remove the dependency once the package is built, you may later mistakenly delete that package thinking nothing depends on it.

The philosophy at the base of swpkg allows for a great deal of flexibility. The following items are a summary of its key points, and the way things should be done:

• Each software package is installed under its own unique base directory named after the package and its version. For example: screen-3.9.4
• All base directories are found under a common directory called the source directory. By default this is /usr/local/pkgs.
• Installed software packages are accessed through the target directory where the software packages are presented using symbolic links. By default, this is /usr/local.
• Installed software packages should NOT have any direct reference to anything found under the target directory.
• Nothing should never directly reference what is found under the source directory. (Files under the source directory are accessed indirectly via symbolic links, either from the target directory, or within the source directory for inter-package dependencies.)

© 2001-2008 - Christophe Kalt