ost - OS Templates

Getting Started

Introduction
It's all about the files
Setting up the initial templates
Establishing a baseline
Ongoing maintenance
Advanced usage

Introduction

ost is a simple yet powerful tool providing mechanisms to automatically audit, maintain and backup systems customizations. It is particularly well suited to environments where most systems are similarly configured, but will operate just as well if each system is unique.

It is important to realize that ost is a fairly generic tool and that while it greatly simplifies implementing OS configuration policies, it does nothing to help you define such policies. This document is a guide to help you understand the concepts behind ost and explains the most common steps that need to be taken to get started with ost.

OS configuration policies can vary greatly, and you will need to decide for yourself how you want to use ost. For example, ost allows administration to be done using either of two extreme methods:

System administrators configure everything on the fly, directly on the system. (Not something that is usually recommended.) In such a scenario, ost can then be used to capture and backup all of the manual customizations.
System administrators never actually log on to the systems they configure. Instead, all customization work is done on a central server in the "OS Templates" and ost is used to push these to the actual systems being administered.

ost can be used to achieve various goals, but at the core, the ultimate goal is to clearly define and efficiently capture every bit of data that is custom to a system. The benefits of this should be obvious: there is no unknown or uncertainty about the system's state or configuration.

It's (almost) all about the files

From an administrative point of view, there are three basic categories of files on a live system:

unmodified vendor provided files: These are the uninteresting files. They can be recovered directly from the vendor.

missing vendor provided files: These should be correctly identified to avoid uncertainty.

all other files: These are automatically identified by ost and may be backed up (depending on ost's configuration). Unsurprisingly, most of the work involved in getting started with ost has to do with this category of files which actually includes the following two sub-categories:

modified vendor provided files
additional files

More important is what causes a file to be in this category:

initial system setup/configuration
manual administration
running daemons on the system

The initial work that should be done to get ost setup mostly consists of deciding how to handle each file included in the last category. The following approaches are recommended:

files modified/maintained by running daemons or files resulting from user activity (such as crontab files for example) should silently be ignored by ost (but should usually be selected for backup by ost).
files that represent a customization or configuration aspect of the system should probably be part of an "OS Template" applied by ost.

How much work does this really involve? Not all that much:

A fairly complete installation Solaris 10 includes well over 100000 files
Less that 500 hundred of these are considered modifiable by the Solaris 10 catalog.
Most systems can be configured with only a few dozen files.

The way ost looks at a system is by looking at all the filesystems on the system (which is typically sufficient on UNIX). ost uses three data sources:

The vendor provided system catalog (if it exists)
The user defined "OS Templates"
The live filesystem

The combination of the first two data sources should ideally be very close to the last data source. For convenience, ost allows any number of "OS Templates" to be applied to a particular system. This is entirely configurable, allowing various methods to be used to suit the defined policies.

Setting up the initial templates

To be able to do anything with ost you need to setup the "OS template" area, and define at least one template. The default template area is /etc/ost:

# ost -VV
ost version 0.7 protocol version 3

        TEMPLATE AREA = "/etc/ost"
        GNU DIFF      = "diff"
        RSH           = "ssh"
        TAR           = "/usr/sfw/bin/gtar"

        LFS Support   = yes

The system you will be working on needs to be accessible from the system where the templates are defined. By default, ost uses ssh, although this can be overriden (with the OST_RSH environment variable, for example). Whether you use ssh, rsh or something else, the target system needs to trust the system on which you launch ost. The only exception is if both systems are the same, in which case you can use "localhost".

Assuming you want to work on a system called "basehost", you can find out the possible template names with the following command:

# ost -S basehost
Solaris [undefined]
Solaris/i386 [undefined]
Solaris/i386/basehost [undefined]
Solaris-5.10 [undefined]
Solaris-5.10/i386 [undefined]
Solaris-5.10/i386/basehost [undefined]
[driver.error] At least one template needs to be defined!

To create the "Solaris" (for example) template, simply create the directory under /etc/ost:

# mkdir /etc/ost/Solaris
# mkdir /etc/ost/Solaris/ost

The second directory created above is an empty "template", data for this template would be found under /etc/ost/Solaris/template. The last directory created above is where the configuration directives for that particular template are located. At a minimum, you need to define which parts of the filesystem are off-limits. This should include any user data areas (home directories, ...), for example:

# cat /etc/ost/Solaris/ost/trek
+ /
- /boot
- /cdrom
- /etc/ost
- /home
- /lost\+found
- /mnt
- /var/sadm
- /var/tmp
- /vol

The first line tells ost to look at the entire filesystem with subsequent lines excluding specific subdirectories. This is all that is really needed to get started. If you run the above command again, you will see that ost now considers the "Solaris" template to be defined:

# ost -S basehost
Solaris
Solaris/i386 [undefined]
Solaris/i386/basehost [undefined]
Solaris-5.10 [undefined]
Solaris-5.10/i386 [undefined]
Solaris-5.10/i386/basehost [undefined]

Establishing a baseline

To do this, it is recommended to use a freshly built system with as few customizations as possible. You will need ost installed both on the system where you have defined the templates and the system you are using establish the baseline. By default, ost uses ssh to access systems from the server which holds the templates. (This work can also all be done on a single system in which case no special access is needed.)

The process of establishing the baseline is iterative, with the same few steps repeated until a satisfactory setup is achieved. These steps are:

run ost
analyze report from ost
update configuration and/or template(s) to resolve divergences

Without any preliminary work, the initial ost report on a Solaris 10 system is about 500 lines (without counting /dev entries). It is not a lot, especially compared to the 100000 files that a typical Solaris 10 installation includes), yet it still is more than it should. Here's a rough breakdown:

~100 under /etc
~200 under /usr
~150 under /var

The above is a good way to explain why "so many" files are flagged by ost on a freshly built system:

The expected list of files used for configuration (under /etc).
The ones resulting from sloppy (Sun) packages (mostly under /usr) which do not correctly update the system catalog.
The ones which are expected to vary on a running system (under /var).

The good news is that it really does not take much work to clean up this initial ost report. This is usually done with a combination of exclusions in the "trek" configuration and inclusions in the "mutable" configuration.

First, run ost:

# ost -xvC basehost
## Processing host basehost...
- X /etc/coreadm.conf                                                 [unknown]
  X /etc/default/init                                                  [vendor]
- X /etc/defaultrouter                                                [unknown]
[...]

Most of the report is made of two types of entries (as shown above):

Files missing from the system catalog: "/etc/coreadm.conf" and "/etc/defaultrouter" above
Files that have been modified: "/etc/default/init" above

There are usually two choices when it comes to cleaning up the above ost report:

Tell ost that the file may vary (and that you basically do not care to know about it). This is done by including the file in the "mutable" configuration.
Define the file in a template which will prevent it from being modified (without your knowledge). This is done by creating the file in a template.

For example:

# cd /etc/ost/Solaris/ost
# echo "+ /etc/coreadm.conf" >> mutable
# cd /etc/ost/Solaris
# mkdir template
# mkdir template/etc
# chgrp sys template/etc 
# scp -p basehost:/etc/defaultrouter template/etc

At this point, running the above command again would show the following:

# ost -xvC basehost
## Processing host basehost...
  X /etc/coreadm.conf                                                 [unknown]
  X /etc/default/init                                                  [vendor]
[...]

It is worthwhile to note that using the -v option is important at this stage: the /etc/default/init file is marked as mutable in the system catalog and is normally excluded from ost reports because it is "expected" to vary from the system catalog. While excluded from regular reports, such files are flagged for backup by ost (depending on the "preserve" configuration). It is important to identify such files as you may or may not want to define them in a template: just because Solaris expects these files to vary does not mean that you should, this entirely depends on your own environment. By default, ost will not backup anything. To change this, edit the "preserve" configuration:

# cd /etc/ost/Solaris/ost
# echo "+ /" >> preserve

Now, running the above command once again shows the following:

# ost -xvC basehost
## Processing host basehost...
  b /etc/coreadm.conf                                                 [unknown]
  b /etc/default/init                                                  [vendor]
[...]

For clarity and simplicity, you may want to omit the -v flag when first working on your baseline, as it produces more verbose reports which may be a bit confusing to the first time user.

Once you have a clean report, you have a baseline that can be used for many systems.

Ongoing maintenance

Once your templates are setup, the amount of work required to maintain them is proportional to the amount of changes your systems experience. The recommended approach is to update the templates, and then use ost to push the changes because of the following benefits:

The templates don't fall out of date
Any system can be down while the change is made
There is no need to figure out the list of systems affected by the change (as ost will know based on the "OS template" you updated)
The changes can be pushed manually, or automatically by a (for example) nightly cronjob

At the very least, you should run ost on a regular basis for backup purposes (unless you don't use it for this purpose), but this can also be done to automatically push updates that are not critical or time sensitive.

Advanced usage

The above document is a good introduction to ost but falls far short of covering all that can be done with it. The following sections explore a few thoughts and possibilities.

Template design
The way you design and define your templates is critical to how useful and scalable ost will be in your environment. The names of templates applied to a system is controlled by two shell scripts:
- ost-sysinfo runs on the client
- ost-tmpldef runs on a server, using the information provided by ost-sysinfo to define the list of templates applicable to the client
The default scripts only define 6 templates per system. These are enough to administer a set of homogenous systems, allowing for different versions of Solaris on multiple architectures. While this is sufficient for some small environments, it is fairly limited.

In bigger more complex environments, systems typically belong to one or more "groups" of machines, and will likely want to define one or more template for each such "group". For example:
- class of machine: desktop, server, cluster node, ..
- localization: site, building, ..
- services provided: NIS master, mail server, web server, ..
Automated template maintenance
Some of the templates content "never" changes, they are the customizations that are so central to your environment, so needed, that they're always there. However, to really benefit from ost, you need to put it at the core of your system administration practices. In particular, the content of your templates should be dynamically/automatically generated and updated, ost is then only used to push configuration changes, and audit configuration settings.

One of the advantages of this approach is that while you are effectively using a "push" model for your configuration settings, your updates never fail, no matter the state of the "remote" system. For non time-sensitive updates, it is also possible to simply update the templates without pushing the changes, simply waiting for the weekly, daily or hourly ost cron job to kick in.
Automated actions
As explained in the document above, ost is (almost) all about the files. This is a serious limitation for some updates. What is the point of updating /etc/syslog.conf for example, if you don't send a SIGHUP to the running syslogd? Each template may define a "post" script which is run by ost after it is done making changes to the system. The use of such script(s) allows one to enable, refresh/reload and disable services.
Dealing with dynamic system state
Another limitation of the fact that ost is (almost) all about the files is apparent when dealing with dynamic system state which may or may not be consistent with system files. For example:
- Is /etc/vfstab consistent with the currently mounted filesystems?
- How does one check the state of the SMF configuration?
Such questions and problems can be solved by a combination of scripts, cron jobs, and the use of "post" scripts mentioned above.
Rapid zone provisioning
ost clearly has a place in your build environment, whether it is a manual process, JumpStart driven or other. Once you have defined all the templates needed to capture the configuration of a system, ost can be used to simply "apply" all these templates on a freshly built system. This allows to greatly minimize the amount of manual work needed to build a new system.

It becomes especially worthwhile and useful with Solaris zones. These are very quick and easy to setup and bring to life, completing their configuration with ost only takes seconds. All in all, the process of setting up a zone can be entirely automated with a few lines of shell scripting.