4. Advanced

Packaging Guide: advanced topics¶

This section of the packaging guide covers a few advanced topics.

Multiple build systems¶

Packages may use different build systems over time or across platforms. Spack is designed to handle this seamlessly within a single package.py file.

Let’s assume we work with curl and that the package is built using Autotools so far:

from spack_repo.builtin.build_systems.autotools import AutotoolsPackage

class Curl(AutotoolsPackage):

    depends_on("zlib-api")

    def configure_args(self):
        return [f"--with-zlib={self.spec['zlib-api'].prefix}"]

To add CMake as a further build system we need to:

Add another base to the Curl package class (in our case cmake.CMakePackage),
Explicitly declare which build systems are supported using the build_system directive,
Move the build instructions in separate builder classes.

from spack_repo.builtin.build_systems import autotools, cmake

class Curl(cmake.CMakePackage, autotools.AutotoolsPackage):

    build_system("autotools", "cmake", default="cmake")

    depends_on("zlib-api")

class AutotoolsBuilder(autotools.AutotoolsBuilder):
    def configure_args(self):
        return [f"--with-zlib={self.spec['zlib-api'].prefix}"]

class CMakeBuilder(cmake.CMakeBuilder):
    def cmake_args(self):
        return [self.define_from_variant("USE_NGHTTP2", "nghttp2")]

In general, with multiple build systems there is a clear split between the package metadata and the build instructions:

The directives such as depends_on, variant, patch go into the package class
The build phase functions like configure, build and install, and helper functions such as cmake_args or configure_args go into the builder classes

When curl is concretized, we can select its build system using the build_system variant, which is available for every package:

$ spack install curl build_system=cmake

Override “phases” of a build system¶

Sometimes package recipes need to override entire phases of a build system. Let’s assume this happens for cp2k:

from spack.package import *
from spack_repo.builtin.build_systems import autotools


class Cp2k(autotools.AutotoolsPackage):
    def install(self, spec: Spec, prefix: str) -> None:
        # ...existing code...
        pass

If we want to add CMake as another build system we need to remember that the signature of phases changes when moving from the Package to the Builder class:

from spack.package import *
from spack_repo.builtin.build_systems import autotools, cmake


class Cp2k(autotools.AutotoolsPackage, cmake.CMakePackage):
    build_system("autotools", "cmake", default="cmake")


class AutotoolsBuilder(autotools.AutotoolsBuilder):
    def install(self, pkg: Cp2k, spec: Spec, prefix: str) -> None:
        # ...existing code...
        pass

The install method now takes the Package instance as the first argument, since self refers to the builder class.

Add dependencies conditional on a build system¶

Many build dependencies are conditional on which build system is chosen. An effective way to handle this is to use a with when("build_system=...") block to specify dependencies that are only relevant for a specific build system:

from spack.package import *
from spack_repo.builtin.build_systems import cmake, autotools


class Cp2k(cmake.CMakePackage, autotools.AutotoolsPackage):

    build_system("cmake", "autotools", default="cmake")

    # Runtime dependencies
    depends_on("ncurses")
    depends_on("libxml2")

    # Lowerbounds for cmake only apply when using cmake as the build system
    with when("build_system=cmake"):
        depends_on("cmake@3.18:", when="@2.0:", type="build")
        depends_on("cmake@3:", type="build")

    # Specify extra build dependencies used only in the configure script
    with when("build_system=autotools"):
        depends_on("perl", type="build")
        depends_on("pkgconfig", type="build")

Transition from one build system to another¶

Packages that transition from one build system to another can be modeled using conditional variant values:

from spack.package import *
from spack_repo.builtin.build_systems import cmake, autotools


class Cp2k(cmake.CMakePackage, autotools.AutotoolsPackage):

    build_system(
        conditional("cmake", when="@0.64:"),
        conditional("autotools", when="@:0.63"),
        default="cmake",
    )

In the example, the directive imposes a change from Autotools to CMake going from v0.63 to v0.64.

Inherit from a package with multiple build systems¶

Customizing a package supporting multiple build systems is straightforward. If we need to only customize the metadata, we can just define the derived package class.

For instance, let’s assume we want to add a new version to the silo package:

from spack_repo.builtin.packages.silo.package import Silo as BuiltinSilo

class Silo(BuiltinSilo):
    # Version not in builtin.silo
    version("special_version")

If we don’t define any builder, Spack will reuse the custom builder from builtin.silo by default. If we need to customize the builder too, we just have to inherit from it, like any other Python class:

from spack_repo.builtin.packages.silo.package import CMakeBuilder as SiloCMakeBuilder

class CMakeBuilder(SiloCMakeBuilder):
    def cmake_args(self):
        return [self.define_from_variant("USE_NGHTTP2", "nghttp2")]

Making a package discoverable with `spack external find`¶

The simplest way to make a package discoverable with spack external find is to:

Define the executables associated with the package.
Implement a method to determine the versions of these executables.

Minimal detection¶

The first step is fairly simple, as it requires only to specify a package-level executables attribute:

class Foo(Package):
    # Each string provided here is treated as a regular expression, and
    # would match for example "foo", "foobar", and "bazfoo".
    executables = ["foo"]

This attribute must be a list of strings. Each string is a regular expression (e.g. “gcc” would match “gcc”, “gcc-8.3”, “my-weird-gcc”, etc.) to determine a set of system executables that might be part of this package. Note that to match only executables named “gcc” the regular expression "^gcc$" must be used.

Finally, to determine the version of each executable the determine_version method must be implemented:

@classmethod
def determine_version(cls, exe):
    """Return either the version of the executable passed as argument
    or ``None`` if the version cannot be determined.

    Args:
        exe (str): absolute path to the executable being examined
    """

This method receives as input the path to a single executable and must return as output its version as a string. If the version cannot be determined, or if the executable turns out to be a false positive, the value None must be returned, which ensures that the executable is discarded as a candidate. Implementing the two steps above is mandatory, and gives the package the basic ability to detect if a spec is present on the system at a given version.

Note

Any executable for which the determine_version method returns None will be discarded and won’t appear in later stages of the workflow described below.

Additional functionality¶

Besides the two mandatory steps described above, there are also optional methods that can be implemented to either increase the amount of details being detected or improve the robustness of the detection logic in a package.

Variants and custom attributes¶

The determine_variants method can be optionally implemented in a package to detect additional details of the spec:

@classmethod
def determine_variants(cls, exes, version_str):
    """Return either a variant string, a tuple of a variant string
    and a dictionary of extra attributes that will be recorded in
    packages.yaml or a list of those items.

    Args:
        exes (list of str): list of executables (absolute paths) that
            live in the same prefix and share the same version
        version_str (str): version associated with the list of
            executables, as detected by ``determine_version``
    """

This method takes as input a list of executables that live in the same prefix and share the same version string, and returns either:

A variant string
A tuple of a variant string and a dictionary of extra attributes
A list of items matching either 1 or 2 (if multiple specs are detected from the set of executables)

If extra attributes are returned, they will be recorded in packages.yaml and be available for later reuse. As an example, the gcc package will record by default the different compilers found and an entry in packages.yaml would look like:

packages:
  gcc:
    externals:
    - spec: "gcc@9.0.1 languages=c,c++,fortran"
      prefix: /usr
      extra_attributes:
        compilers:
          c: /usr/bin/x86_64-linux-gnu-gcc-9
          c++: /usr/bin/x86_64-linux-gnu-g++-9
          fortran: /usr/bin/x86_64-linux-gnu-gfortran-9

This allows us, for instance, to keep track of executables that would be named differently if built by Spack (e.g. x86_64-linux-gnu-gcc-9 instead of just gcc).

Filter matching executables¶

Sometimes defining the appropriate regex for the executables attribute might prove to be difficult, especially if one has to deal with corner cases or exclude “red herrings”. To help keep the regular expressions as simple as possible, each package can optionally implement a filter_detected_exes method:

@classmethod
def filter_detected_exes(cls, prefix, exes_in_prefix):
    """Return a filtered list of the executables in prefix"""

which takes as input a prefix and a list of matching executables and returns a filtered list of said executables.

Using this method has the advantage of allowing custom logic for filtering, and does not restrict the user to regular expressions only. Consider the case of detecting the GNU C++ compiler. If we try to search for executables that match g++, that would have the unwanted side effect of selecting also clang++ - which is a C++ compiler provided by another package - if present on the system. Trying to select executables that contain g++ but not clang would be quite complicated to do using only regular expressions. Employing the filter_detected_exes method it becomes:

class Gcc(Package):
    executables = ["g++"]

    @classmethod
    def filter_detected_exes(cls, prefix, exes_in_prefix):
        return [x for x in exes_in_prefix if "clang" not in x]

Another possibility that this method opens is to apply certain filtering logic when specific conditions are met (e.g. take some decisions on an OS and not on another).

Validate detection¶

To increase detection robustness, packagers may also implement a method to validate the detected Spec objects:

@classmethod
def validate_detected_spec(cls, spec, extra_attributes):
    """Validate a detected spec. Raise an exception if validation fails."""

This method receives a detected spec along with its extra attributes and can be used to check that certain conditions are met by the spec. Packagers can either use assertions or raise an InvalidSpecDetected exception when the check fails. If the conditions are not honored the spec will be discarded and any message associated with the assertion or the exception will be logged as the reason for discarding it.

As an example, a package that wants to check that the compilers attribute is in the extra attributes can implement this method like this:

@classmethod
def validate_detected_spec(cls, spec, extra_attributes):
    """Check that "compilers" is in the extra attributes."""
    msg = "the extra attribute 'compilers' must be set for the detected spec '{0}'".format(spec)
    assert "compilers" in extra_attributes, msg

or like this:

@classmethod
def validate_detected_spec(cls, spec, extra_attributes):
    """Check that "compilers" is in the extra attributes."""
    if "compilers" not in extra_attributes:
        msg = "the extra attribute 'compilers' must be set for the detected spec '{0}'".format(
            spec
        )
        raise InvalidSpecDetected(msg)

Custom detection workflow¶

In the rare case when the mechanisms described so far don’t fit the detection of a package, the implementation of all the methods above can be disregarded and instead a custom determine_spec_details method can be implemented directly in the package class (note that the definition of the executables attribute is still required):

@classmethod
def determine_spec_details(cls, prefix, exes_in_prefix):
    # exes_in_prefix = a set of paths, each path is an executable
    # prefix = a prefix that is common to each path in exes_in_prefix

    # return None or [] if none of the exes represent an instance of
    # the package. Return one or more Specs for each instance of the
    # package which is thought to be installed in the provided prefix
    ...

This method takes as input a set of discovered executables (which match those specified by the user) as well as a common prefix shared by all of those executables. The function must return one or more spack.package.Spec associated with the executables (it can also return None to indicate that no provided executables are associated with the package).

As an example, consider a made-up package called foo-package which builds an executable called foo. FooPackage would appear as follows:

class FooPackage(Package):
    homepage = "..."
    url = "..."

    version(...)

    # Each string provided here is treated as a regular expression, and
    # would match for example "foo", "foobar", and "bazfoo".
    executables = ["foo"]

    @classmethod
    def determine_spec_details(cls, prefix, exes_in_prefix):
        candidates = [x for x in exes_in_prefix if os.path.basename(x) == "foo"]
        if not candidates:
            return
        # This implementation is lazy and only checks the first candidate
        exe_path = candidates[0]
        exe = Executable(exe_path)
        output = exe("--version", output=str, error=str)
        version_str = ...  # parse output for version string
        return Spec.from_detection("foo-package@{0}".format(version_str))

Add detection tests to packages¶

To ensure that software is detected correctly for multiple configurations and on different systems users can write a detection_test.yaml file and put it in the package directory alongside the package.py file. This YAML file contains enough information for Spack to mock an environment and try to check if the detection logic yields the results that are expected.

As a general rule, attributes at the top-level of detection_test.yaml represent search mechanisms and they each map to a list of tests that should confirm the validity of the package’s detection logic.

The detection tests can be run with the following command:

$ spack audit externals

Errors that have been detected are reported to screen.

Tests for PATH inspections¶

Detection tests insisting on PATH inspections are listed under the paths attribute:

paths:
- layout:
  - executables:
    - "bin/clang-3.9"
    - "bin/clang++-3.9"
    script: |
      echo "clang version 3.9.1-19ubuntu1 (tags/RELEASE_391/rc2)"
      echo "Target: x86_64-pc-linux-gnu"
      echo "Thread model: posix"
      echo "InstalledDir: /usr/bin"
  platforms: ["linux", "darwin"]
  results:
  - spec: "llvm@3.9.1 +clang~lld~lldb"

If the platforms attribute is present, tests are run only if the current host matches one of the listed platforms. Each test is performed by first creating a temporary directory structure as specified in the corresponding layout and by then running package detection and checking that the outcome matches the expected results. The exact details on how to specify both the layout and the results are reported in the table below:

Test based on PATH inspections¶
Option Name	Description	Allowed Values	Required Field
`layout`	Specifies the filesystem tree used for the test	List of objects	Yes
`layout:[0]:executables`	Relative paths for the mock executables to be created	List of strings	Yes
`layout:[0]:script`	Mock logic for the executable	Any valid shell script	Yes
`results`	List of expected results	List of objects (empty if no result is expected)	Yes
`results:[0]:spec`	A spec that is expected from detection	Any valid spec	Yes
`results:[0]:extra_attributes`	Extra attributes expected on the associated Spec	Nested dictionary with string as keys, and regular expressions as leaf values	No

Reuse tests from other packages¶

When using a custom repository, it is possible to customize a package that already exists in builtin and reuse its external tests. To do so, just write a detection_test.yaml alongside the customized package.py with an includes attribute. For instance the detection_test.yaml for myrepo.llvm might look like:

includes:
- "builtin.llvm"

This YAML file instructs Spack to run the detection tests defined in builtin.llvm in addition to those locally defined in the file.

Specifying ABI Compatibility¶

Warning

The can_splice directive is experimental, and may be replaced by a higher-level interface in future versions of Spack.

Packages can include ABI-compatibility information using the can_splice directive. For example, if Foo version 1.1 can always replace version 1.0, then the package could have:

can_splice("foo@1.0", when="@1.1")

For virtual packages, packages can also specify ABI compatibility with other packages providing the same virtual. For example, zlib-ng could specify:

can_splice("zlib@1.3.1", when="@2.2+compat")

Some packages have ABI-compatibility that is dependent on matching variant values, either for all variants or for some set of ABI-relevant variants. In those cases, it is not necessary to specify the full combinatorial explosion. The match_variants keyword can cover all single-value variants.

# any value for bar as long as they're the same
can_splice("foo@1.1", when="@1.2", match_variants=["bar"])

# any variant values if all single-value variants match
can_splice("foo@1.2", when="@1.3", match_variants="*")

The concretizer will use ABI compatibility to determine automatic splices when automatic splicing is enabled.

Customizing Views¶

Warning

This is advanced functionality documented for completeness, and rarely needs customization.

Spack environments manage a view of their packages, which is a single directory that merges all installed packages through symlinks, so users can easily access them. The methods of PackageViewMixin can be overridden to customize how packages are added to views. Sometimes it’s impossible to get an application to work just through symlinking its executables, and patching is necessary. For example, Python scripts in a bin directory may have a shebang that points to the Python interpreter in Python’s install prefix and not to the Python interpreter in the view. However, it’s more convenient to have the shebang point to the Python interpreter in the view, since that interpreter can locate other Python packages in the view without PYTHONPATH being set. Therefore, Python extension packages (those inheriting from PythonPackage) override add_files_to_view in order to rewrite shebang lines.

Packaging Guide: advanced topics¶

Multiple build systems¶

Override “phases” of a build system¶

Add dependencies conditional on a build system¶

Transition from one build system to another¶

Inherit from a package with multiple build systems¶

Making a package discoverable with spack external find¶

Minimal detection¶

Additional functionality¶

Variants and custom attributes¶

Filter matching executables¶

Validate detection¶

Custom detection workflow¶

Add detection tests to packages¶

Tests for PATH inspections¶

Reuse tests from other packages¶

Specifying ABI Compatibility¶

Customizing Views¶

Making a package discoverable with `spack external find`¶