Package Settings (packages.yaml)¶

Spack allows you to customize how your software is built through the packages.yaml file. Using it, you can make Spack prefer particular implementations of virtual dependencies (e.g., MPI or BLAS/LAPACK), or you can make it prefer to build with particular compilers. You can also tell Spack to use external software installations already present on your system.

At a high level, the packages.yaml file is structured like this:

packages:
  package1:
    # settings for package1
  package2:
    # settings for package2
  # ...
  all:
    # settings that apply to all packages.

You can either set build preferences specifically for one package, or you can specify that certain settings should apply to all packages. The types of settings you can customize are described in detail below.

Spack’s build defaults are in the default etc/spack/defaults/packages.yaml file. You can override them in ~/.spack/packages.yaml or etc/spack/packages.yaml. For more details on how this works, see Configuration Scopes.

External packages¶

Spack can be configured to use externally-installed packages rather than building its own packages. This may be desirable if machines ship with system packages, such as a customized MPI, which should be used instead of Spack building its own MPI.

External packages are configured through the packages.yaml file. Here’s an example of an external configuration:

packages:
  openmpi:
    externals:
    - spec: "openmpi@1.4.3~debug"
      prefix: /opt/openmpi-1.4.3
    - spec: "openmpi@1.4.3+debug"
      prefix: /opt/openmpi-1.4.3-debug

This example lists two installations of OpenMPI, one with debug information, and one without. If Spack is asked to build a package that uses one of these MPIs as a dependency, it will use the pre-installed OpenMPI in the given directory. Note that the specified path is the top-level install prefix, not the bin subdirectory.

packages.yaml can also be used to specify modules to load instead of the installation prefixes. The following example says that module CMake/3.7.2 provides CMake version 3.7.2.

cmake:
  externals:
  - spec: cmake@3.7.2
    modules:
    - CMake/3.7.2

Each packages.yaml begins with a packages: attribute, followed by a list of package names. To specify externals, add an externals: attribute under the package name, which lists externals. Each external should specify a spec: string that should be as well-defined as reasonably possible. If a package lacks a spec component, such as missing a compiler or package version, then Spack will guess the missing component based on its most-favored packages, and it may guess incorrectly.

Automatically find external packages¶

You can run the spack external find command to search for system-provided packages and add them to packages.yaml. After running this command your packages.yaml may include new entries:

packages:
  cmake:
    externals:
    - spec: cmake@3.17.2
      prefix: /usr

Generally this is useful for detecting a small set of commonly-used packages; for now this is generally limited to finding build-only dependencies. Specific limitations include:

Packages are not discoverable by default: For a package to be discoverable with spack external find, it needs to add special logic. See here for more details.
The logic does not search through module files, it can only detect packages with executables defined in PATH; you can help Spack locate externals which use module files by loading any associated modules for packages that you want Spack to know about before running spack external find.
Spack does not overwrite existing entries in the package configuration: If there is an external defined for a spec at any configuration scope, then Spack will not add a new external entry (spack config blame packages can help locate all external entries).

Prevent packages from being built from sources¶

Adding an external spec in packages.yaml allows Spack to use an external location, but it does not prevent Spack from building packages from sources. In the above example, Spack might choose for many valid reasons to start building and linking with the latest version of OpenMPI rather than continue using the pre-installed OpenMPI versions.

To prevent this, the packages.yaml configuration also allows packages to be flagged as non-buildable. The previous example could be modified to be:

packages:
  openmpi:
    externals:
    - spec: "openmpi@1.4.3~debug"
      prefix: /opt/openmpi-1.4.3
    - spec: "openmpi@1.4.3+debug"
      prefix: /opt/openmpi-1.4.3-debug
    buildable: false

The addition of the buildable flag tells Spack that it should never build its own version of OpenMPI from sources, and it will instead always rely on a pre-built OpenMPI.

Note

If concretizer:reuse is on (see Concretization Settings (concretizer.yaml) for more information on that flag) pre-built specs are taken from: the local store, an upstream store, a registered buildcache and externals in packages.yaml. If concretizer:reuse is off, only external specs in packages.yaml are included in the list of pre-built specs.

If an external module is specified as not buildable, then Spack will load the external module into the build environment which can be used for linking.

The buildable attribute does not need to be paired with external packages. It could also be used alone to forbid packages that may be buggy or otherwise undesirable.

Non-buildable virtual packages¶

Virtual packages in Spack can also be specified as not buildable, and external implementations can be provided. In the example above, OpenMPI is configured as not buildable, but Spack will often prefer other MPI implementations over the externally available OpenMPI. Spack can be configured with every MPI provider not buildable individually, but more conveniently:

packages:
  mpi:
    buildable: false
  openmpi:
    externals:
    - spec: "openmpi@1.4.3~debug"
      prefix: /opt/openmpi-1.4.3
    - spec: "openmpi@1.4.3+debug"
      prefix: /opt/openmpi-1.4.3-debug

Spack can then use any of the listed external implementations of MPI to satisfy a dependency, and will choose among them depending on the compiler and architecture.

In cases where the concretizer is configured to reuse specs, and other mpi providers (available via stores or build caches) are not desirable, Spack can be configured to require specs matching only the available externals:

packages:
  mpi:
    buildable: false
    require:
    - one_of:
      - "openmpi@1.4.3~debug"
      - "openmpi@1.4.3+debug"
  openmpi:
    externals:
    - spec: "openmpi@1.4.3~debug"
      prefix: /opt/openmpi-1.4.3
    - spec: "openmpi@1.4.3+debug"
      prefix: /opt/openmpi-1.4.3-debug

This configuration prevents any spec using MPI and originating from stores or build caches to be reused, unless it matches the requirements under packages:mpi:require. For more information on requirements see Package Requirements.

Specifying dependencies among external packages¶

External packages frequently have dependencies on other software components. Explicitly modeling these relationships provides Spack with a more complete representation of the software stack. This ensures that:

Runtime environments include all necessary components.
Build-time dependencies are accurately represented.

This comprehensive view, in turn, enables Spack to more reliably build software that depends on these externals.

Spack provides two methods for configuring dependency relationships among externals, each offering different trade-offs between conciseness and explicit control:

An “inline” spec syntax.
A structured YAML configuration that is more verbose but also more explicit.

The following sections will detail both approaches.

Dependencies using inline spec syntax¶

Spack allows you to define external package dependencies using the standard spec syntax directly within your package configuration. This approach is concise and leverages the familiar spec syntax that you already use elsewhere in Spack.

When configuring an external package with dependencies using the spec syntax, you can include dependency specifications directly in the main spec: field:

# Specification for the following DAG:
#
# o mpileaks@2.3
# |\
# | o callpath@1.0
# |/
# o mpich@3.0.4
packages:
  mpileaks:
    externals:
    - spec: "mpileaks@2.3~debug+opt %mpich@3 %callpath"
      prefix: /user/path
  callpath:
    externals:
    - spec: "callpath@1.0 %mpi=mpich"
      prefix: /user/path
  mpich:
    externals:
    - spec: "mpich@3.0.4"
      prefix: /user/path

In this example mpileaks depends on both mpich and callpath. Spack will parse the mpileaks spec string, and create the appropriate dependency relationships automatically.

Users need to ensure that each dependency maps exactly to a single other external package. In case multiple externals can satisfy the same dependency, or in case no external can satisfy a dependency, Spack will error and point to the configuration line causing the issue.

Whenever no information is given about the dependency type, Spack will infer it from the current package recipe. For instance, the dependencies in the configuration above are inferred to be of build,link type from the recipe of mpileaks and callpath:

$ spack -m spec --types -l --cover edges mpileaks
[e]  oelprl6  [    ]  mpileaks@2.3~debug+opt+shared+static build_system=generic platform=linux os=ubuntu20.04 target=icelake
[e]  jdhzy2t  [bl  ]      ^callpath@1.0 build_system=generic platform=linux os=ubuntu20.04 target=icelake
[e]  pgem3yp  [bl  ]          ^mpich@3.0.4~debug build_system=generic platform=linux os=ubuntu20.04 target=icelake
[e]  pgem3yp  [bl  ]      ^mpich@3.0.4~debug build_system=generic platform=linux os=ubuntu20.04 target=icelake

When inferring the dependency types, Spack will also infer virtuals if they are not already specified.

This method’s conciseness comes with a strict requirement: each dependency must resolve to a single, unambiguous external package. This makes the approach suitable for simple or temporary configurations. In larger, more dynamic environments, however, it can become a maintenance challenge, as adding new external packages over time may require frequent updates to existing specs to preserve their uniqueness.

Dependencies using YAML configuration¶

While the spec syntax offers a concise way to specify dependencies, Spack’s YAML-based explicit dependency configuration provides more control and clarity, especially for complex dependency relationships. This approach uses the dependencies: field to precisely define each dependency relationship. The example in the previous section, written using the YAML configuration, becomes:

# Specification for the following DAG:
#
# o mpileaks@2.3
# |\
# | o callpath@1.0
# |/
# o mpich@3.0.4
packages:
  mpileaks:
    externals:
    - spec: "mpileaks@2.3~debug+opt"
      prefix: /user/path
      dependencies:
      - id: callpath_id
        deptypes: link
      - spec: mpich
        deptypes:
        - "build"
        - "link"
        virtuals: "mpi"
  callpath:
    externals:
    - spec: "callpath@1.0"
      prefix: /user/path
      id: callpath_id
      dependencies:
      - spec: mpich
        deptypes:
        - "build"
        - "link"
        virtuals: "mpi"
  mpich:
    externals:
    - spec: "mpich@3.0.4"
      prefix: /user/path

Each dependency can be specified either by:

A spec: that matches an available external package, like in the previous case, or by
An id that explicitly references another external package.

Using the id provides an unambiguous reference to a specific external package, which is essential for differentiating between externals that have similar specs but differ, for example, only by their installation prefix.

The dependency types can be specified in the optional deptypes field, while virtuals can be specified in the optional virtuals field. As before, when the dependency types are not specified, Spack will infer them from the package recipe.

Extra attributes for external packages¶

Sometimes external packages require additional attributes to be used effectively. This information can be defined on a per-package basis and stored in the extra_attributes section of the external package configuration. In addition to per-package information, this section can be used to define environment modifications to be performed whenever the package is used. For example, if an external package is built without rpath support, it may require LD_LIBRARY_PATH settings to find its dependencies. This could be configured as follows:

packages:
  mpich:
    externals:
    - spec: mpich@3.3 +hwloc
      prefix: /path/to/mpich
      extra_attributes:
        environment:
          prepend_path:
            LD_LIBRARY_PATH: /path/to/hwloc/lib64

See Environment Modifications for more information on how to configure environment modifications in Spack config files.

Configuring system compilers as external packages¶

In Spack, compilers are treated as packages like any other. This means that you can also configure system compilers as external packages and use them in Spack.

Spack automatically detects system compilers and configures them in packages.yaml for you. You can also run spack compiler find to find and configure new system compilers.

When configuring compilers as external packages, you need to set a few extra attributes for them to work properly. The compilers extra attribute field is required to clarify which paths within the compiler prefix are used for which languages:

packages:
  gcc:
    externals:
    - spec: gcc@10.5.0 languages='c,c++,fortran'
      prefix: /usr
      extra_attributes:
        compilers:
          c: /usr/bin/gcc-10
          cxx: /usr/bin/g++-10
          fortran: /usr/bin/gfortran-10

Other fields accepted by compilers under extra_attributes are flags, environment, extra_rpaths, and implicit_rpaths.

packages:
  gcc:
    externals:
    - spec: gcc@10.5.0 languages='c,c++,fortran'
      prefix: /usr
      extra_attributes:
        compilers:
          c: /usr/bin/gcc-10
          cxx: /usr/bin/g++-10
          fortran: /usr/bin/gfortran-10
        flags:
          cflags: -O3
          fflags: -g -O2
        environment:
          set:
            GCC_ROOT: /usr
          prepend_path:
            PATH: /usr/unusual_path_for_ld/bin
        implicit_rpaths:
        - /usr/lib/gcc
        extra_rpaths:
        - /usr/lib/unusual_gcc_path

The flags attribute specifies compiler flags to apply to every spec that depends on this compiler. The accepted flag types are cflags, cxxflags, fflags, cppflags, ldflags, and ldlibs. In the example above, every spec compiled with this compiler will pass the flags -g -O2 to /usr/bin/gfortran-10 and will pass the flag -O3 to /usr/bin/gcc-10.

The environment attribute specifies user environment modifications to apply before every time the compiler is invoked. The available operations are set, unset, prepend_path, append_path, and remove_path. In the example above, Spack will set GCC_ROOT=/usr and set PATH=/usr/unusual_path_for_ld/bin:$PATH before handing control to the build system that will use this compiler.

The extra_rpaths and implicit_rpaths fields specify additional paths to pass as rpaths to the linker when using this compiler. The implicit_rpaths field is filled in automatically by Spack when detecting compilers, and the extra_rpaths field is available for users to configure necessary rpaths that have not been detected by Spack. In addition, paths from extra_rpaths are added as library search paths for the linker. In the example above, both /usr/lib/gcc and /usr/lib/unusual_gcc_path would be added as rpaths to the linker, and -L/usr/lib/unusual_gcc_path would be added as well.

Package Requirements¶

Spack can be configured to always use certain compilers, package versions, and variants during concretization through package requirements.

Package requirements are useful when you find yourself repeatedly specifying the same constraints on the command line, and wish that Spack respects these constraints whether you mention them explicitly or not. Another use case is specifying constraints that should apply to all root specs in an environment, without having to repeat the constraint everywhere.

Apart from that, requirements config is more flexible than constraints on the command line, because it can specify constraints on packages when they occur as a dependency. In contrast, on the command line it is not possible to specify constraints on dependencies while also keeping those dependencies optional.

Requirements syntax¶

The package requirements configuration is specified in packages.yaml, keyed by package name and expressed using the Spec syntax. In the simplest case you can specify attributes that you always want the package to have by providing a single spec string to require:

packages:
  libfabric:
    require: "@1.13.2"

In the above example, libfabric will always build with version 1.13.2. If you need to compose multiple configuration scopes require accepts a list of strings:

packages:
  libfabric:
    require:
    - "@1.13.2"
    - "%gcc"

In this case libfabric will always build with version 1.13.2 and using GCC as a compiler.

For more complex use cases, require accepts also a list of objects. These objects must have either a any_of or a one_of field, containing a list of spec strings, and they can optionally have a when and a message attribute:

packages:
  openmpi:
    require:
    - any_of: ["@4.1.5", "%c,cxx,fortran=gcc"]
      message: "in this example only 4.1.5 can build with other compilers"

any_of is a list of specs. One of those specs must be satisfied and it is also allowed for the concretized spec to match more than one. In the above example, that means you could build openmpi@4.1.5%gcc, openmpi@4.1.5%clang or openmpi@3.9%gcc, but not openmpi@3.9%clang.

If a custom message is provided, and the requirement is not satisfiable, Spack will print the custom error message:

$ spack spec openmpi@3.9%clang
==> Error: in this example only 4.1.5 can build with other compilers

We could express a similar requirement using the when attribute:

packages:
  openmpi:
    require:
    - any_of: ["%c,cxx,fortran=gcc"]
      when: "@:4.1.4"
      message: "in this example only 4.1.5 can build with other compilers"

In the example above, if the version turns out to be 4.1.4 or less, we require the compiler to be GCC. For readability, Spack also allows a spec key accepting a string when there is only a single constraint:

packages:
  openmpi:
    require:
    - spec: "%c,cxx,fortran=gcc"
      when: "@:4.1.4"
      message: "in this example only 4.1.5 can build with other compilers"

This code snippet and the one before it are semantically equivalent.

Finally, instead of any_of you can use one_of which also takes a list of specs. The final concretized spec must match one and only one of them:

packages:
  mpich:
    require:
    - one_of: ["+cuda", "+rocm"]

In the example above, that means you could build mpich+cuda or mpich+rocm but not mpich+cuda+rocm.

Note

For any_of and one_of, the order of specs indicates a preference: items that appear earlier in the list are preferred (note that these preferences can be ignored in favor of others).

Note

When using a conditional requirement, Spack is allowed to actively avoid the triggering condition (the when=... spec) if that leads to a concrete spec with better scores in the optimization criteria. To check the current optimization criteria and their priorities you can run spack solve zlib.

Setting default requirements¶

You can also set default requirements for all packages under all like this:

packages:
  all:
    require: "%[when=%c]c=clang %[when=%cxx]cxx=clang"

which means every spec will be required to use clang as the compiler for C and C++ code.

Warning

The simpler config require: %clang will fail to build any package that does not include compiled code, because those packages cannot depend on clang (alias for llvm+clang). In most contexts, default requirements must use either conditional dependencies or a toolchain that combines conditional dependencies.

Requirements on variants for all packages are possible too, but note that they are only enforced for those packages that define these variants, otherwise they are disregarded. For example:

packages:
  all:
    require:
    - "+shared"
    - "+cuda"

will just enforce +shared on zlib, which has a boolean shared variant but no cuda variant.

Constraints in a single spec literal are always considered as a whole, so in a case like:

packages:
  all:
    require: "+shared +cuda"

the default requirement will be enforced only if a package has both a cuda and a shared variant, and will never be partially enforced.

Finally, all represents a default set of requirements - if there are specific package requirements, then the default requirements under all are disregarded. For example, with a configuration like this:

packages:
  all:
    require:
    - "build_type=Debug"
    - "%[when=%c]c=clang %[when=%cxx]cxx=clang"
  cmake:
    require:
    - "build_type=Debug"
    - "%c,cxx=gcc"

Spack requires cmake to use gcc and all other nodes (including cmake dependencies) to use clang. If enforcing build_type=Debug is needed also on cmake, it must be repeated in the specific cmake requirements.

Setting requirements on virtual specs¶

A requirement on a virtual spec applies whenever that virtual is present in the DAG. This can be useful for fixing which virtual provider you want to use:

packages:
  mpi:
    require: "mvapich2 %c,cxx,fortran=gcc"

With the configuration above the only allowed mpi provider is mvapich2 built with gcc/g++/gfortran.

Requirements on the virtual spec and on the specific provider are both applied, if present. For instance with a configuration like:

packages:
  mpi:
    require: "mvapich2 %c,cxx,fortran=gcc"
  mvapich2:
    require: "~cuda"

you will use mvapich2~cuda %c,cxx,fortran=gcc as an mpi provider.

Conflicts and strong preferences¶

If the semantic of requirements is too strong, you can also express “strong preferences” and “conflicts” from configuration files:

packages:
  all:
    prefer:
    - "%c,cxx=clang"
    conflict:
    - "+shared"

The prefer and conflict sections can be used whenever a require section is allowed. The argument is always a list of constraints, and each constraint can be either a simple string, or a more complex object:

packages:
  all:
    conflict:
    - spec: "%c,cxx=clang"
      when: "target=x86_64_v3"
      message: "reason why clang cannot be used"

The spec attribute is mandatory, while both when and message are optional.

Note

Requirements allow for expressing both “strong preferences” and “conflicts”. The syntax for doing so, though, may not be immediately clear. For instance, if we want to prevent any package from using %clang, we can set:

packages:
  all:
    require:
    - one_of: ["%clang", "@:"]

Since only one of the requirements must hold, and @: is always true, the rule above is equivalent to a conflict. For “strong preferences” the same construction works, with the any_of policy instead of the one_of policy.

Package Preferences¶

In some cases package requirements can be too strong, and package preferences are the better option. Package preferences do not impose constraints on packages for particular versions or variants values, they rather only set defaults. The concretizer is free to change them if it must, due to other constraints, and also prefers reusing installed packages over building new ones that are a better match for preferences.

Package Permissions¶

Spack can be configured to assign permissions to the files installed by a package.

In the packages.yaml file under permissions, the attributes read, write, and group control the package permissions. These attributes can be set per-package, or for all packages under all. If permissions are set under all and for a specific package, the package-specific settings take precedence.

The read and write attributes take one of user, group, and world.

packages:
  all:
    permissions:
      write: group
      group: spack
  my_app:
    permissions:
      read: group
      group: my_team

The permissions settings describe the broadest level of access to installations of the specified packages. The execute permissions of the file are set to the same level as read permissions for those files that are executable. The default setting for read is world, and for write is user. In the example above, installations of my_app will be installed with user and group permissions but no world permissions, and owned by the group my_team. All other packages will be installed with user and group write privileges, and world read privileges. Those packages will be owned by the group spack.

The group attribute assigns a Unix-style group to a package. All files installed by the package will be owned by the assigned group, and the sticky group bit will be set on the install prefix and all directories inside the install prefix. This will ensure that even manually placed files within the install prefix are owned by the assigned group. If no group is assigned, Spack will allow the OS default behavior to go as expected.

Assigning Package Attributes¶

You can assign class-level attributes in the configuration:

packages:
  mpileaks:
    package_attributes:
      # Override existing attributes
      url: http://www.somewhereelse.com/mpileaks-1.0.tar.gz
      # ... or add new ones
      x: 1

Attributes set this way will be accessible to any method executed in the package.py file (e.g. the install() method). Values for these attributes may be any value parseable by yaml.

These can only be applied to specific packages, not “all” or virtual packages.