Conary Packaging: Simply Powerful

Michael K Johnson

FOSDEM 2008

February 24, 2008

http://people.rpath.com/~johnsonm/fosdem2008/

Simply Powerful

• Simple Process
• Powerful Features
• Easy Overrides
• Fighting Bitrot: Just Enough Packaging
• Dependency Heaven: Just Enough OS

Only Packaging Here! Not:

• System Management with Conary
• Conary Repository Management
• Building System Images with rBuilder

Why Conary?

• System management inspired by source code control
• Extensive legacy packaging experience: good and bad

Legacy Packaging: Good

• Pristine sources plus patches
• Some dependencies automatically discovered

Legacy Packaging: Bad

• Verbosity-generated bitrot in control files
• Too manual, too complicated
• Dependencies insufficiently complete
• Dependencies too broad, insufficiently specific
• Scripts often buggy

Conary Goals

• Provide only necessary information, only once
• Do the best thing by default, but easy to override
• Avoid copying errors
• Branch without forking
• Reproducible builds:
• Store everything necessary in the repository
• Specify clearly and completely what to do
• Record what was really done (no magic bullet)

Recipes (Control Files)

• Do the right thing 90% of the time, easy to override in the 10% case
• Are python, but that's not important right now
• Infrastructure intentionally inhibits complex code
• Conditionals are rare: no "if file in build directory" function
• Conditionals normally expressed as configuration items

Recipe Stages

• Cooking a recipe is done in three distinct stages:
  • Preparing sources
  • Building/installing
  • Checking results with policy

Preparing Sources

• Use partial URL, archive name guessed:
  • r.addArchive('http://sqlite.org/')
  • r.addArchive(​'mirror://sourceforge/inkscape/')
  • r.addPatch('CVE-2007-0104.patch')
• Add archives (tar, cpio, rpm, iso...), patches, individual files, or even pull directly from source control systems:
  • r.addMercurialSnapshot(​'http://hg.rpath.com/conary')

Building

• Sane automatic defaults (arguments and environment) for consistent builds, for example:
  • r.Configure() includes consistent set of default command-line arguments
  • r.Make() sets environment variables like CFLAGS
  • r.MakeInstall() sets DESTDIR to the proxy root
• Most common build and install commands encapsulated

Policy Keeps You Sane

• Over 100 primarily automatic operations
• Replaces most manual work in spec/control files, e.g.
  • Generally transparent multi-arch support
  • r.NonBinariesInBindirs()
• Easily overridden in recipes for unusual cases
• Easily enhanced in recipes providing additional information
• Easily augmented (plug-in model)

Recipe Example

class MyApplication(AutoPackageRecipe):
    name = 'myapplication'
    version = '1.0'
    buildRequires = [ 'libfoo:devel', 'bar:runtime' ]

    def unpack(r):
        r.addSource('http://example.com/download')

Real Recipe Example

class zile(AutoPackageRecipe):
    name = 'zile'
    version = '2.2.2'
    buildRequires = [ 'install-info:runtime',
        'libtermcap:devel', 'texinfo:runtime',
    ]

    def unpack(r):
        r.addArchive('mirror://sourceforge/zile/')

Packaging Process Characteristics

• Transparent (easy to see what happens)
• Repeatable
• Packages automatically subdivided into components
• Rich file-level dependencies
• Robust system script design

Transparent and Repeatable Builds

• Sources stored in Conary repository
• Build requirements automatically discovered
• Packages that satisfy build requirements recorded
• Build logs and debugging information stored
• Build logs display actions taken during build

Components

• Files automatically assigned to components
  • No need to decide whether to create the foo-devel package; you will get foo:devel and foo:devellib if appropriate
  • Based on regular expressions plus file modes
  • Easy to override for exceptional cases
• Lots of components (6-8 per package are common) not a problem
  • Repository model avoids management problem of downloading 10,000 files via FTP
  • Group-based updates means that users see the big picture

Dependencies

• Semantically rich (e.g. express symbol versions)
• Per-file (rolled up into components as well)
• Resolved in the context of consistent sets of packages
• Components avoid dependency creep ("dependency hell")

Tag Handlers

• Each file has zero or more "tags" defining type
  • Each file type is handled by a single script
  • When adding or removing files, Conary runs scripts for each affected type, telling script which files are affected
• Robust:
  • No transcription of errors or errors in transcription
  • Only runs when there are relevant changes
  • Enables robust rollback mechanism

Conary Build Process

• Package Building with rMake
• Shadows
• Groups
• Package Inspection
• Testing

Build Packages with rMake

• Builds in clean chroot() environments constructed to include only required components (not packages)
• Builds into temporary local Conary repository
• Allows you to inspect/test before committing to permanent repository
• Build for multiple architectures from one console

Shadows

• Branch without forking
• Track changes relative to some upstream version
• Typical uses:
  • Add a patch or a configuration file
  • Build separate binaries against different base OS versions
• Can avoid rebuilding binaries: Derived packages

Groups

• Collections of packages
  • Built from source just like packages
  • Same query, display, and inspection tools available
• Consistent sets of packages
• Used to drive QA/release process (cvc promote command)

Package Inspection

• File versions: What files are different in this version from previous versions?
• File dependencies: Which file added that requirement?
• The conary rdiff command: what is different between these two versions?
• Repository browser: web-based access to all flavors of all versions, sources and binaries

Testing

• System rollbacks make testing easy
• System updates are relative:
  • Like a merge in a source control system
  • Do not assume all files are overwritten on update
    • In packaging: do not assume that files will be "fixed" on next update
    • In testing: do not make random change to files and assume that next update operation will revert the changes

More information

• http://wiki.rpath.com/wiki/Conary
• http://wiki.rpath.com/wiki/Conary:Packaging
• http://wiki.rpath.com/wiki/Application_to_Appliance