rPath Makes Appliances Easy
Michael K Johnson
September 29, 2007
rPath Makes Appliances Easy
- What is an Appliance?
- Conary and rBuilder Make This Possible
- Developers: Easy to develop and maintain
- Users: Easy to deploy and use
- Building Packages with rMake
What is an Appliance?
- A software application
- Combined with just enough operating system (JeOS) for it to run optimally
- On industry standard hardware (typically a server)
- In a virtual machine
- Simplify server applications by minimizing installation, configuration, and maintenance tasks.
What does that mean?
- Only what you need is included
- Maintain only what you need
- Simplified testing: precisely known software stack
- No extraneous security exposure
- No extraneous software updates
- Less data to download
- Less oportunity for conflicts
Focus on application, not OS
- Forget the OS
- Don't worry about dependencies
Maintainable Development
- Developer can change almost anything
- Developer rarely touches OS
- Designed for easy long-term maintenance: No one-offs
Reliable, Predictable updates
- Entire appliance is updated, not individual packages
- End result of an update is predictable
- Designed for reliable rollback if necessary
Building Appliances: Overview
Products and Technologies
Building Images in rBuilder
- Everything in an image comes out of Conary repositories
- Distributed source code control comes to system management
- One repository contains rPath Linux
- Many other repositories exist
- rBuilder Online hosts many repositories
Images and Groups
- A group determines what to fetch from repositories
- List the things you want installed on a system or included in an image
- Conary turns this into a list of exact versions
Simple Group Code Example
class Dist(GroupRecipe):
name = 'group-openfiler'
version = '0.2'
autoResolve = True
def setup(r):
r.setLabelPath('openfiler.rpath.org@rpl:devel',
'conary.rpath.com@rpl:1')
r.add('aoe-cec')
r.add('aoetools')
...
# replace group-core troves with openfiler-patched troves.
r.replace('authconfig')
...
# stuff we don't want in a default install
r.remove('iptables')
Group Cooked: Versions
group-openfiler=0.2-19-1
acl=2.2.28-2-0.1
aoe-cec=4-1-1
...
aoetools=10-1-1
...
attr=2.4.20-3-0.1
authconfig=4.6.12-8.2.2-1
...
Group Cooked: Labels
group-openfiler=openfiler.rpath.org@rpl:devel/0.2-19-1
acl=conary.rpath.com@rpl:1/2.2.28-2-0.1
aoe-cec=openfiler.rpath.org@rpl:devel/4-1-1
...
aoetools=openfiler.rpath.org@rpl:devel/10-1-1
...
attr=conary.rpath.com@rpl:1/2.4.20-3-0.1
authconfig=openfiler.rpath.org@rpl:devel/4.6.12-8.2.2-1
...
Group Versions Are Slices
- Each time you cook a group, Conary chooses the current best match for each item
- The set of versions of all selected contents (including all included files) are recorded in the repository
- Entire systems are up- or downgraded as a consistent set of packages, tested together
- When things break, slices make testing sane
Groups To Images
- rBuilder creates images based on cooked groups
- Installable images (anaconda)
- Installed images
- Virtual: VMware, qemu, Parallels, Xen, VHD, and more
- Raw: hard drive images, file system images, tar archives
Groups To Images
- Manage Builds -> Create a new build
- Name and describe the build
- Choose the precise version to build
- Select the image type and configure it
- Build and test
- Add to a release (collection of builds) and publish the release
Managing Updates
- Version control concepts make update stream maintainable
- Recooking a group makes updates available
- Conary updates top-down, keeping dependencies consistent
- Updates are not available to users until you publish them: more on release management later
- Updating systems: more than one way
- Command line: conary updateall
- rPath Appliance Platform Agent web framework includes system update and other tasks
- Standard system configuration and management with no shell
- Makes appliance easy to deploy and use
- Framework and a set of core plugins for that framework
- Plugin-writing documentation is available
- Optional customized "branding": images, CSS, and templates
rMake
- Building on fully-installed system is
- good for discovering build requirements
- bad for reproducible builds
- Minimal chroot is
- bad for discovering build requirements
- good for reproducible builds
- very good for avoiding unwanted change in a stable, maintained OS
rMake
- rMake builds in minimal chroot environments
- Uses the buildRequires lists to populate the chroot environments
- Builds the software into a private repository
- Commits by promoting those shadows from the private repository to the source repository
rMake
- rMake can build multiple packages together in one job
- The entire job is committed to the parent branch as a single operation
- Each build within the job is built in a chroot with only its requirements installed
- rMake caches the data used to build the chroot to make installing chroots efficient
$ rmake build --poll lzo zebra zile
Added Job 1
lzo:source=jdoe.rpath.org@rpl:1/2.02-1
zebra:source=jdoe.rpath.org@rpl:1/0.95a-1
zile:source=jdoe.rpath.org@rpl:1/2.2.2-1
...
[19:54:47] [1] - build job finished successfully
$ rmake q
1 Built 0 minutes ago
(3 troves) lzo, zebra, zile
$ rmake changeset 1 job1.ccs
$ conary scs --troves job1.ccs
zile=2.2.2-1-0.1
zile:data=2.2.2-1-0.1
zile:doc=2.2.2-1-0.1
zile:runtime=2.2.2-1-0.1
lzo=2.02-1-0.1
lzo:devel=2.02-1-0.1
lzo:devellib=2.02-1-0.1
lzo:doc=2.02-1-0.1
lzo:lib=2.02-1-0.1
zebra=0.95a-1-0.1
zebra:doc=0.95a-1-0.1
zebra:runtime=0.95a-1-0.1
$ rmake commit 1
Committed:
lzo=/jdoe.rpath.org@rpl:1/2.02-1-2[is: x86]
zebra=/jdoe.rpath.org@rpl:1/0.95a-1-2[is: x86]
zile=/jdoe.rpath.org@rpl:1/2.2.2-1-2[is: x86]
Policy for Packages
- Generic actions, specific exceptions
- Easy exceptions allow powerful defaults
- Fix common errors
- Avoids boilerplate rotting in .spec files
Release Management
- Base your work on "upstream" work with a shadow:
- $ cvc shadow jdoe.rpath.org@jdoe:1-devel foo:source=conary.rpath.com@rpl:1
- foo:source=jdoe.rpath.org@jdoe:1-devel is under your control
- Create stages for development, qa, and so forth as
labels:
- jdoe.rpath.org@jdoe:1-devel
- jdoe.rpath.org@jdoe:1-qa
- jdoe.rpath.org@jdoe:1
Release Management: Promotion
- Promote work from one stage to the next (development to QA,
QA to released) with the cvc promote command:
- $ cvc promote group-jdoe=jdoe.rpath.org@jdoe:1-devel :1-devel :1-qa
- $ cvc promote group-jdoe=jdoe.rpath.org@jdoe:1-qa :1-qa :1
- Normally done using entire groups, not just individual packages
Example Release Management
- Build development packages
- Build development groups, test included software
- Promote to QA
- Build QA images, do full tests of installs and updates
- Promote to full release
- Build release images (if necessary)
- Prerequisites
- Building some example packages
- Making it easier
- Importing other packages
Contexts
$ cvc context jdoe
$ cat CONARY
context jdoe
$ cvc context
[jdoe]
buildFlavor
buildLabel jdoe.rpath.org@rpl:1
...
Recipes
- Build instructions for packages
- Concise, easy to read
- Cooking metaphor: Conary reads the recipe to know how to "cook" the package
- Object Oriented: inherit most behavior
Policy for Packages
- Generic actions, specific exceptions
- Easy exceptions allow powerful defaults
- Fix common errors
- Avoids boilerplate rotting in .spec files
AutoPackageRecipe
- Packages that use automake and autoconf with generally-accepted practice are usually simple
- The AutoPackageRecipe class does almost everything for you
AutoPackageRecipe: zile
$ cvc newpkg zile
$ cd zile
$ cat > zile.recipe <<EOF
class zile(AutoPackageRecipe):
name = 'zile'
version = '2.2.2'
def unpack(r):
r.addArchive('mirror://sourceforge/zile/')
EOF
$ cvc add zile.recipe
$ cvc commit
AutoPackageRecipe: zile
$ cvc cook zile.recipe
warning: Suggested buildRequires additions: [
'install-info:runtime', 'libtermcap:devel',
'texinfo:runtime' ]
This means: add to the recipe, after the version, the lines:
buildRequires = [ 'install-info:runtime',
'libtermcap:devel', 'texinfo:runtime',
]
$ cvc cook zile.recipe
...
This time, no complaints about missing buildRequires
Other Recipe Classes
- CPackageRecipe makes no assumptions about how to build
- BinaryPackageRecipe helps pull in already-compiled tar archives, RPMs, or even ISO images
- Several templates exist on the rPath web site
Packages and Components
- A package is a collection of components
- A component is a collection of files
- zile was automatically partitioned:
- zile:data is contents of /usr/share
- zile:doc is man pages, info pages, /usr/share/doc
- zile:runtime is anything not otherwise specified; here /usr/bin/zile
Changesets
$ conary scs --troves zile-2.2.2.ccs
zile=2.2.2-1-0.1
zile:data=2.2.2-1-0.1
zile:doc=2.2.2-1-0.1
zile:runtime=2.2.2-1-0.1
These are the components that would be installed by default.
Build Directory Contents
- ~/conary/build/zile/
- conary-build-info
- Records conary build progress
- Lets you restart local builds (not into repository)
- _ROOT_
- autotools DESTDIR, proxy for installed system / directory
- Files here are included in the package
- No manifest, directories not packaged by default
Build Directory Contents
- zile-2.2.2
- Directory into which source was unpacked
- This is where the binaries were built
- zile-2.2.2-log.bz2
- Complete build log, including environment
- Packaged in zile:debuginfo
Restarting Builds
- $ cvc cook zile.recipe --resume=policy
- Does not rebuild, just re-packs from _ROOT_
- $ cvc cook zile.recipe --resume
- Resumes after the final previously completed command
- Allows you to hit control-c during a build, then pick up later (good for debugging builds)
- --resume not allowed for building into repository
Dependencies
$ conary scs --deps zile-2.2.2.ccs
...
zile:runtime=2.2.2-1-0.1
...
Requires:
...
soname: ELF32/libc.so.6(... GLIBC_2.3 SysV x86)
soname: ELF32/libtermcap.so.2(SysV x86)
...
$ conary scs --deps --file-deps zile-2.2.2.ccs
...
/usr/bin/zile
Requires:
...
soname: ELF32/libc.so.6(... GLIBC_2.3 SysV x86)
soname: ELF32/libtermcap.so.2(SysV x86)
...
Dependency Scope
- Dependency elements:
- soname: Shared libraries
- trove: Other packages
- user/groupinfo: Creating system users and groups
- python, java, perl, cil (Mono), and ruby: What you expect
- Each lives in its own namespace
- More can be added
- Automatic component assignment prevents kudzu-like dependency closure
Flavors
- Flavor elements:
- use: Build configuration
- is: What CPU the code runs on
- target: What CPU the tools build for (cross-compiling)
Repository View: Component
Repository View: Files
Building into the Repository
$ cvc cook zile
+ Building zile
+ found zile-2.2.2.tar.gz in repository
...
Changeset committed to the repository.
$ conary rq --buildreqs zile
...
binutils:devellib=2.15.90.0.3-7.5-1
binutils:lib=2.15.90.0.3-7.5-1
binutils:runtime=2.15.90.0.3-7.5-1
bzip2:runtime=1.0.3-4-0.1
...
tar:runtime=1.15.1-7-0.1
texinfo:runtime=4.8-6-0.1
Repository View: Binaries
Source and Binary Components
- A :source component:
- Contains files needed to assemble a package
- They may contain source code and objects
- They are checked out and processed into binary components ("cooked")
- All other components are binary components:
- They are installed on a system
- They may contain source code and objects
Browsing the Repository
- You can download any file in the package from the web interface
- Downloading the buildlog is particularly useful when debugging
- No directories are listed, Conary handles them automatically
- Everything is owned by root.root automatically
Browsing the Repository
Shadows
- Like a branch in source code control
- Except that it makes it easy to track upstream changes:
- Designed to allow moving in parallel
- Has automated merge capability
- Can be explicitly reverted to contents of parent branch (start over)
- Can be in a different repository from the parent branch (distributed development)
- Can rebuild on shadow or just modify relative to parent