Environments (spack.yaml, spack.lock)¶

An environment is used to group a set of specs intended for some purpose to be built, rebuilt, and deployed in a coherent fashion. Environments define aspects of the installation of the software, such as:

which specs to install;
how those specs are configured; and
where the concretized software will be installed.

Aggregating this information into an environment for processing has advantages over the à la carte approach of building and loading individual Spack modules.

With environments, you concretize, install, or load (activate) all of the specs with a single command. Concretization fully configures the specs and dependencies of the environment in preparation for installing the software. This is a more robust solution than ad-hoc installation scripts. And you can share an environment or even re-use it on a different computer.

Environment definitions, especially how specs are configured, allow the software to remain stable and repeatable even when Spack packages are upgraded. Changes are only picked up when the environment is explicitly re-concretized.

Defining where specs are installed supports a filesystem view of the environment. Yet Spack maintains a single installation of the software that can be re-used across multiple environments.

Activating an environment determines when all of the associated (and installed) specs are loaded so limits the software loaded to those specs actually needed by the environment. Spack can even generate a script to load all modules related to an environment.

Other packaging systems also provide environments that are similar in some ways to Spack environments; for example, Conda environments or Python Virtual Environments. Spack environments provide some distinctive features though:

A spec installed “in” an environment is no different from the same spec installed anywhere else in Spack.
Spack environments may contain more than one spec of the same package.

Spack uses a “manifest and lock” model similar to Bundler gemfiles and other package managers. The environment’s user input file (or manifest), is named spack.yaml. The lock file, which contains the fully configured and concretized specs, is named spack.lock.

Using Environments¶

Here we follow a typical use case of creating, concretizing, installing and loading an environment.

Creating a managed Environment¶

An environment is created by:

$ spack env create myenv

The directory $SPACK_ROOT/var/spack/environments/myenv is created to manage the environment.

Note

By default, all managed environments are stored in the $SPACK_ROOT/var/spack/environments folder. This location can be changed by setting the environments_root variable in config.yaml.

Spack creates the file spack.yaml, hidden directory .spack-env, and spack.lock file under $SPACK_ROOT/var/spack/environments/myenv. User interaction occurs through the spack.yaml file and the Spack commands that affect it. Metadata and, by default, the view are stored in the .spack-env directory. When the environment is concretized, Spack creates the spack.lock file with the fully configured specs and dependencies for the environment.

The .spack-env subdirectory also contains:

repo/: A subdirectory acting as the repo consisting of the Spack packages used in the environment. It allows the environment to build the same, in theory, even on different versions of Spack with different packages!
logs/: A subdirectory containing the build logs for the packages in this environment.

Spack Environments can also be created from another environment. Environments can be created from the manifest file (the user input), the lockfile, or the entire environment at once. Create an environment from a manifest using:

$ spack env create myenv spack.yaml

The resulting environment is guaranteed to have the same root specs as the original but may concretize differently in the presence of different explicit or default configuration settings (e.g., a different version of Spack or for a different user account).

Environments created from a manifest will copy any included configs from relative paths inside the environment. Relative paths from outside the environment will cause errors, and absolute paths will be kept absolute. For example, if spack.yaml includes:

spack:
  include: [./config.yaml]

then the created environment will have its own copy of the file config.yaml copied from the location in the original environment.

Create an environment from a spack.lock file using:

$ spack env create myenv spack.lock

The resulting environment, when on the same or a compatible machine, is guaranteed to initially have the same concrete specs as the original.

Create an environment from an entire environment using either the environment name or path:

$ spack env create myenv /path/to/env
$ spack env create myenv2 myenv

The resulting environment will include the concrete specs from the original if the original is concretized (as when created from a lockfile) and all of the config options and abstract specs specified in the original (as when created from a manifest file). It will also include any other files included in the environment directory, such as repos or source code, as they could be referenced in the environment by relative path.

Note

Environment creation also accepts a full path to the file.

If the path is not under the $SPACK_ROOT/var/spack/environments directory then the source is referred to as an independent environment.

The name of an environment can be a nested path to help organize environments via subdirectories.

$ spack env create projectA/configA/myenv

This will create a managed environment under $environments_root/projectA/configA/myenv. Changing environment_root can therefore also be used to make a whole group of nested environments available.

Activating an Environment¶

To activate an environment, use the following command:

$ spack env activate myenv

By default, the spack env activate will load the view associated with the environment into the user environment. The -v, --with-view argument ensures this behavior, and the -V, --without-view argument activates the environment without changing the user environment variables.

The -p option to the spack env activate command modifies the user’s prompt to begin with the environment name in brackets.

$ spack env activate -p myenv
[myenv] $ ...

The activate command can also be used to create a new environment, if it is not already defined, by adding the --create flag. Managed and independent environments can both be created using the same flags that spack env create accepts. If an environment already exists then Spack will simply activate it and ignore the create-specific flags.

$ spack env activate --create -p myenv
# ...
# [creates if myenv does not exist yet]
# ...
[myenv] $ ...

To deactivate an environment, use the command:

$ spack env deactivate

or the shortcut alias

$ despacktivate

If the environment was activated with its view, deactivating the environment will remove the view from the user environment.

Independent Environments¶

Independent environments can be located in any directory outside of Spack.

Note

When uninstalling packages, Spack asks the user to confirm the removal of packages that are still used in a managed environment. This is not the case for independent environments.

To create an independent environment, use one of the following commands:

$ spack env create --dir my_env
$ spack env create ./my_env

As a shorthand, you can also create an independent environment upon activation if it does not already exist:

$ spack env activate --create ./my_env

For convenience, Spack can also place an independent environment in a temporary directory for you:

$ spack env activate --temp

Environment-Aware Commands¶

Spack commands are environment-aware. For example, the find command shows only the specs in the active environment if an environment has been activated. Otherwise it shows all specs in the Spack instance. The same rule applies to the install and uninstall commands.

$ spack find
==> 0 installed packages

$ spack install zlib@1.2.11
[+] q6cqrdt zlib@1.2.11 ~/spack/opt/spack/linux-rhel7-broadwell/gcc-8.1.0/zlib-1.2.11-q6cqrdto4iktfg6qyqcc5u4vmfmwb7iv (12s)

$ spack env activate myenv

$ spack find
==> In environment myenv
==> No root specs
==> 0 installed packages

$ spack install zlib@1.2.8
[+] yfc7epf zlib@1.2.8 ~/spack/opt/spack/linux-rhel7-broadwell/gcc-8.1.0/zlib-1.2.8-yfc7epf57nsfn2gn4notccaiyxha6z7x (12s)
==> Updating view at ~/spack/var/spack/environments/myenv/.spack-env/view

$ spack find
==> In environment myenv
==> Root specs
zlib@1.2.8

==> 1 installed package
-- linux-rhel7-broadwell / gcc@8.1.0 ----------------------------
zlib@1.2.8

$ despacktivate

$ spack find
==> 2 installed packages
-- linux-rhel7-broadwell / gcc@8.1.0 ----------------------------
zlib@1.2.8  zlib@1.2.11

Note that when we installed the abstract spec zlib@1.2.8, it was presented as a root of the environment. All explicitly installed packages will be listed as roots of the environment.

All of the Spack commands that act on the list of installed specs are environment-aware in this way, including install, uninstall, find, extensions, etc. In the Configuring Environments section we will discuss environment-aware commands further.

Adding Abstract Specs¶

An abstract spec is the user-specified spec before Spack applies defaults or dependency information.

You can add abstract specs to an environment using the spack add command. This adds the abstract spec as a root of the environment in the spack.yaml file. The most important component of an environment is a list of abstract specs.

Adding abstract specs does not immediately install anything, nor does it affect the spack.lock file. To update the lockfile, the environment must be re-concretized, and to update any installations, the environment must be (re)installed.

The spack add command is environment-aware. It adds the spec to the currently active environment. An error is generated if there isn’t an active environment.

$ spack env activate myenv
$ spack add mpileaks

$ spack -e myenv add python

Note

All environment-aware commands can also be called using the spack -e flag to specify the environment.

Concretizing¶

Once user specs have been added to an environment, they can be concretized. There are three different modes of operation to concretize an environment, explained in detail in Spec concretization. Regardless of which mode of operation is chosen, the following command will ensure all of the root specs are concretized according to the constraints that are prescribed in the configuration:

[myenv]$ spack concretize

In the case of specs that are not concretized together, the command above will concretize only the specs that were added and not yet concretized. Forcing a re-concretization of all of the specs can be done by adding the -f option:

[myenv]$ spack concretize -f

Without the option, Spack guarantees that already concretized specs are unchanged in the environment.

The concretize command does not install any packages. For packages that have already been installed outside of the environment, the process of adding the spec and concretizing is identical to installing the spec assuming it concretizes to the exact spec that was installed outside of the environment.

The spack find command can show concretized specs separately from installed specs using the -c (--concretized) flag.

[myenv]$ spack add zlib
[myenv]$ spack concretize
[myenv]$ spack find -c
==> In environment myenv
==> Root specs
zlib

==> Concretized roots
-- linux-rhel7-x86_64 / gcc@4.9.3 -------------------------------
zlib@1.2.11

==> 0 installed packages

Installing an Environment¶

In addition to adding individual specs to an environment, one can install the entire environment at once using the command

[myenv]$ spack install

If the environment has been concretized, Spack will install the concretized specs. Otherwise, spack install will concretize the environment before installing the concretized specs.

Note

Every spack install process builds one package at a time with multiple build jobs, controlled by the -j flag and the config:build_jobs option (see build_jobs). To speed up environment builds further, independent packages can be installed in parallel by launching more Spack instances. For example, the following will build at most four packages in parallel using three background jobs:

[myenv]$ spack install & spack install & spack install & spack install

Another option is to generate a Makefile and run make -j<N> to control the number of parallel install processes. See Generating Depfiles from Environments for details.

As it installs, spack install creates symbolic links in the logs/ directory in the environment, allowing for easy inspection of build logs related to that environment. The spack install command also stores a Spack repo containing the package.py file used at install time for each package in the repos/ directory in the environment.

The --no-add option can be used in a concrete environment to tell Spack to install specs already present in the environment but not to add any new root specs to the environment. For root specs provided to spack install on the command line, --no-add is the default, while for dependency specs, it is optional. In other words, if there is an unambiguous match in the active concrete environment for a root spec provided to spack install on the command line, Spack does not require you to specify the --no-add option to prevent the spec from being added again. At the same time, a spec that already exists in the environment, but only as a dependency, will be added to the environment as a root spec without the --no-add option.

Developing Packages in a Spack Environment¶

The spack develop command allows one to develop Spack packages in an environment. It will configure Spack to install the package from local source. By default, spack develop will also clone the package to a subdirectory in the environment for the local source. These choices can be overridden with the --path argument, and the --no-clone argument. Relative paths provided to the --path argument will be resolved relative to the environment directory. All of these options are recorded in the environment manifest, although default values may be left implied.

$ spack develop --path src/foo foo@develop
$ cat `spack location -e`/spack.yaml
spack:
  ...
  develop
    foo:
      spec: foo@develop
      path: src/foo

When spack develop is run in a concretized environment, Spack will modify the concrete specs in the environment to reflect the modified provenance. Any package built from local source will have a dev_path variant, and the hash of any dependent of those packages will be modified to reflect the change. The value of the dev_path variant will be the absolute path to the package source directory. If the develop spec conflicts with the concrete specs in the environment, Spack will raise an exception and require the spack develop --no-modify-concrete-specs option, followed by a spack concretize --force to apply the dev_path variant and constraints from the develop spec.

When concretizing an environment with develop specs, the version, variants, and other attributes of the spec provided to the spack develop command will be treated as constraints by the concretizer (in addition to any constraints from the packages specs list). If the develop configuration for the package does not include a spec version, Spack will choose the highest version of the package. This means that any “infinity” versions (develop, main, etc.) will be preferred for specs marked with the spack develop command, which is different from the standard Spack behavior to prefer the highest numeric version. These packages will have an automatic dev_path variant added by the concretizer, with a value of the absolute path to the local source Spack is building from.

Spack will ensure the package and its dependents are rebuilt any time the environment is installed if the package’s local source code has been modified. Spack’s native implementation is to check if mtime is newer than the installation. A custom check can be created by overriding the detect_dev_src_change method in your package class. This is particularly useful for projects using custom Spack repos to drive development and want to optimize performance.

When spack develop is run without any arguments, Spack will clone any develop specs in the environment for which the specified path does not exist.

When working deep in the graph it is often desirable to have multiple specs marked as develop so you don’t have to restage and/or do full rebuilds each time you call spack install. The --recursive flag can be used in these scenarios to ensure that all the dependents of the initial spec you provide are also marked as develop specs. The --recursive flag requires a pre-concretized environment so the graph can be traversed from the supplied spec all the way to the root specs.

For packages with git attributes, git branches, tags, and commits can also be used as valid concrete versions (see Version specifier). This means that for a package foo, spack develop foo@git.main will clone the main branch of the package, and spack install will install from that git clone if foo is in the environment. Further development on foo can be tested by re-installing the environment, and eventually committed and pushed to the upstream git repo.

If the package being developed supports out-of-source builds then users can use the --build_directory flag to control the location and name of the build directory. This is a shortcut to set the package_attributes:build_directory in the packages configuration (see Assigning Package Attributes). The supplied location will become the build-directory for that package in all future builds.

Potential pitfalls of setting the build directory

Spack does not check for out-of-source build compatibility with the packages and so the onus of making sure the package supports out-of-source builds is on the user. For example, most autotool and makefile packages do not support out-of-source builds while all CMake packages do. Understanding these nuances is up to the software developers and we strongly encourage developers to only redirect the build directory if they understand their package’s build-system.

Modifying Specs in an Environment¶

The spack change command allows the user to change individual specs in a Spack environment.

By default, spack change operates on the abstract specs of an environment. The command a list of spec arguments. For each argument, the root spec with the same name as the provided spec is modified to satisfy the provided spec. For example, in an environment with the root spec hdf5+mpi+fortran, then

spack change hdf5~mpi+cxx

will change the root spec to hdf5~mpi+cxx+fortran.

When more complex matching semantics are necessary, the --match-spec argument replaces the spec name as the selection criterion. When using the --match-spec argument, the spec name is not required. In the same environment,

spack change --match-spec "+fortran" +hl

will constrain the hdf5 spec to +hl.

By default, the spack change command will result in an error and no change to the environment if it will modify more than one abstract spec. Use the --all option to allow spack change to modify multiple abstract specs.

The --concrete option allows spack change to modify the concrete specs of an environment as well as the abstract specs. Multiple concrete specs may be modified, even for a change that modifies only a single abstract spec. The --all option does not affect how many concrete specs may be modified.

Warning

Concrete specs are modified without any constraints from the packages. The spack change --concrete command may create invalid specs that will not build properly if applied without caution.

The --concrete-only option allows for modifying concrete specs without modifying abstract specs. It allows changes to be applied to non-root nodes in the environment, and other changes that do not modify any root specs.

Loading¶

Once an environment has been installed, the following creates a load script for it:

$ spack env loads -r

This creates a file called loads in the environment directory. Sourcing that file in Bash will make the environment available to the user, and can be included in .bashrc files, etc. The loads file may also be copied out of the environment, renamed, etc.

Including Concrete Environments¶

Spack can create an environment that includes information from already concretized environments. You can think of the new environment as a combination of existing environments. It uses information from the existing environments’ spack.lock files in the creation of the new environment. When such an environment is concretized it will generate its own spack.lock file that contains relevant information from the included environments.

Creating combined concrete environments¶

To create a combined concrete environment, you must have at least one existing concrete environment. You will use the command spack env create with the argument --include-concrete followed by the name or path of the environment you’d like to include. Here is an example of how to create a combined environment from the command line:

$ spack env create myenv
$ spack -e myenv add python
$ spack -e myenv concretize
$ spack env create --include-concrete myenv combined_env

You can also include concrete environments directly in the spack.yaml file. It involves adding the absolute paths to the concrete environments spack.lock under the new environment’s include heading. Spack-specific configuration variables, such as $spack, and environment variables can be used in the include paths as long as the expression expands to an absolute path. (See Config File Variables for more information.)

For example,

spack:
  include:
  - /absolute/path/to/environment1/spack.lock
  - $spack/../path/to/environment2/spack.lock
  specs: []
  concretizer:
    unify: true

will include the specs from environment1 and environment2 where the second environment’s path is the absolute path of the directory that is relative to the spack root.

Note

Once the spack.yaml file is updated you must concretize the new environment to get the concrete specs from the included environments. This will produce the combined spack.lock file.

Updating a combined environment¶

If you want changes made to one of the included environments reflected in the combined environment, then you will need to re-concretize the included environment then the combined environment for the change to be incorporated. For example:

$ spack env create myenv
$ spack -e myenv add python
$ spack -e myenv concretize
$ spack env create --include-concrete myenv combined_env

$ spack -e myenv find
==> In environment myenv
==> Root specs
python

==> 0 installed packages

$ spack -e combined_env find
==> In environment combined_env
==> No root specs
==> Included specs
python

==> 0 installed packages

Here we see that combined_env contains the python package from myenv environment. But if we were to add another spec to myenv, combined_env will not know about the other spec.

$ spack -e myenv add perl
$ spack -e myenv concretize
$ spack -e myenv find
==> In environment myenv
==> Root specs
perl  python

==> 0 installed packages

$ spack -e combined_env find
==> In environment combined_env
==> No root specs
==> Included specs
python

==> 0 installed packages

It isn’t until you run the spack concretize command that the combined environment will get the updated information from the re-concretized myenv.

$ spack -e combined_env concretize
$ spack -e combined_env find
==> In environment combined_env
==> No root specs
==> Included specs
perl  python

==> 0 installed packages

Configuring Environments¶

A variety of Spack behaviors are changed through Spack configuration files, covered in more detail in the Configuration Files section.

Spack Environments provide an additional level of configuration scope between the custom scope and the user scope discussed in the configuration documentation.

There are two ways to include configuration information in a Spack Environment:

Inline in the spack.yaml file
Included in the spack.yaml file from another file.

Many Spack commands also affect configuration information in files automatically. Those commands take a --scope argument, and the environment can be specified by env:NAME (to affect environment foo, set --scope env:foo). These commands will automatically manipulate configuration inline in the spack.yaml file.

Inline configurations¶

Inline environment-scope configuration is done using the same yaml format as standard Spack configuration scopes, covered in the Configuration Files section. Each section is contained under a top-level yaml object with its name. For example, a spack.yaml manifest file containing some package preference configuration (as in a packages.yaml file) could contain:

spack:
  # ...
  packages:
    all:
      providers:
        mpi: [openmpi]
  # ...

This configuration sets the default mpi provider to be openmpi.

Included configurations¶

Spack environments allow an include heading in their yaml schema. This heading pulls in external configuration files and applies them to the environment.

spack:
  include:
  - environment/relative/path/to/config.yaml
  - path: https://github.com/path/to/raw/config/compilers.yaml
    sha256: 26e871804a92cd07bb3d611b31b4156ae93d35b6a6d6e0ef3a67871fcb1d258b
  - /absolute/path/to/packages.yaml
  - path: /path/to/$os/$target/environment
    optional: true
  - path: /path/to/os-specific/config-dir
    when: os == "ventura"

Included configuration files are required unless they are explicitly optional or the entry’s condition evaluates to false. Optional includes are specified with the optional clause and conditional with the when clause. (See Include Settings (include.yaml) for more information on optional and conditional entries.)

Files are listed using paths to individual files or directories containing them. Path entries may be absolute or relative to the environment or specified as URLs. URLs to individual files must link to the raw form of the file’s contents (e.g., GitHub or GitLab) and include a valid sha256 for the file. Only the file, ftp, http and https protocols (or schemes) are supported. Spack-specific, environment and user path variables can be used. (See Config File Variables for more information.)

Warning

Recursive includes are not currently processed in a breadth-first manner so the value of a configuration option that is altered by multiple included files may not be what you expect. This will be addressed in a future update.

Configuration precedence¶

Inline configurations take precedence over included configurations, so you don’t have to change shared configuration files to make small changes to an individual environment. Included configurations listed earlier will have higher precedence, as the included configs are applied in reverse order.

Manually Editing the Specs List¶

The list of abstract/root specs in the environment is maintained in the spack.yaml manifest under the heading specs.

spack:
  specs:
  - ncview
  - netcdf
  - nco
  - py-sphinx

Appending to this list in the yaml is identical to using the spack add command from the command line. However, there is more power available from the yaml file.

Spec concretization¶

An environment can be concretized in three different modes and the behavior active under any environment is determined by the concretizer:unify configuration option.

The default mode is to unify all specs:

spack:
  specs:
  - hdf5+mpi
  - zlib@1.2.8
  concretizer:
    unify: true

This means that any package in the environment corresponds to a single concrete spec. In the above example, when hdf5 depends down the line of zlib, it is required to take zlib@1.2.8 instead of a newer version. This mode of concretization is particularly useful when environment views are used: if every package occurs in only one flavor, it is usually possible to merge all install directories into a view.

A downside of unified concretization is that it can be overly strict. For example, a concretization error would happen when both hdf5+mpi and hdf5~mpi are specified in an environment.

The second mode is to unify when possible: this makes concretization of root specs more independent. Instead of requiring reuse of dependencies across different root specs, it is only maximized:

spack:
  specs:
  - hdf5~mpi
  - hdf5+mpi
  - zlib@1.2.8
  concretizer:
    unify: when_possible

This means that both hdf5 installations will use zlib@1.2.8 as a dependency even if newer versions of that library are available.

The third mode of operation is to concretize root specs entirely independently by disabling unified concretization:

spack:
  specs:
  - hdf5~mpi
  - hdf5+mpi
  - zlib@1.2.8
  concretizer:
    unify: false

In this example hdf5 is concretized separately, and does not consider zlib@1.2.8 as a constraint or preference. Instead, it will take the latest possible version.

The last two concretization options are typically useful for system administrators and user support groups providing a large software stack for their HPC center.

Note

The concretizer:unify config option was introduced in Spack 0.18 to replace the concretization property. For reference, concretization: together is replaced by concretizer:unify:true, and concretization: separately is replaced by concretizer:unify:false.

Re-concretization of user specs

The spack concretize command without additional arguments will not change any previously concretized specs. This may prevent it from finding a solution when using unify: true, and it may prevent it from finding a minimal solution when using unify: when_possible. You can force Spack to ignore the existing concrete environment with spack concretize -f.

Spec Matrices¶

Entries in the specs list can be individual abstract specs or a spec matrix.

A spec matrix is a yaml object containing multiple lists of specs, and evaluates to the cross-product of those specs. Spec matrices also contain an excludes directive, which eliminates certain combinations from the evaluated result.

The following two environment manifests are identical:

spack:
  specs:
  - zlib %gcc@7.1.0
  - zlib %gcc@4.9.3
  - libelf %gcc@7.1.0
  - libelf %gcc@4.9.3
  - libdwarf %gcc@7.1.0
  - cmake

spack:
  specs:
  - matrix:
    - [zlib, libelf, libdwarf]
    - ["%gcc@7.1.0", "%gcc@4.9.3"]
    exclude:
    - libdwarf%gcc@4.9.3
  - cmake

Spec matrices can be used to install swaths of software across various toolchains.

Spec List References¶

The last type of possible entry in the specs list is a reference.

The Spack Environment manifest yaml schema contains an additional heading definitions. Under definitions is an array of yaml objects. Each object has one or two fields. The one required field is a name, and the optional field is a when clause.

The named field is a spec list. The spec list uses the same syntax as the specs entry. Each entry in the spec list can be a spec, a spec matrix, or a reference to an earlier named list. References are specified using the $ sigil, and are “splatted” into place (i.e. the elements of the referent are at the same level as the elements listed separately). As an example, the following two manifest files are identical.

spack:
  definitions:
  - first: [libelf, libdwarf]
  - compilers: ["%gcc", "%intel"]
  - second:
    - $first
    - matrix:
      - [zlib]
      - [$compilers]
  specs:
  - $second
  - cmake

spack:
  specs:
  - libelf
  - libdwarf
  - zlib%gcc
  - zlib%intel
  - cmake

Note

Named spec lists in the definitions section may only refer to a named list defined above itself. Order matters.

In short files like the example, it may be easier to simply list the included specs. However for more complicated examples involving many packages across many toolchains, separately factored lists make environments substantially more manageable.

Additionally, the -l option to the spack add command allows one to add to named lists in the definitions section of the manifest file directly from the command line.

The when directive can be used to conditionally add specs to a named list. The when directive takes a string of Python code referring to a restricted set of variables, and evaluates to a boolean. The specs listed are appended to the named list if the when string evaluates to True. In the following snippet, the named list compilers is ["%gcc", "%clang", "%intel"] on x86_64 systems and ["%gcc", "%clang"] on all other systems.

spack:
  definitions:
  - compilers: ["%gcc", "%clang"]
  - when: arch.satisfies("target=x86_64:")
    compilers: ["%intel"]

Note

Any definitions with the same named list with true when clauses (or absent when clauses) will be appended together

The valid variables for a when clause are:

platform. The platform string of the default Spack architecture on the system.
os. The OS string of the default Spack architecture on the system.
target. The target string of the default Spack architecture on the system.
architecture or arch. A Spack spec satisfying the default Spack architecture on the system. This supports querying via the satisfies method, as shown above.
arch_str. The architecture string of the default Spack architecture on the system.
re. The standard regex module in Python.
env. The user environment (usually os.environ in Python).
hostname. The hostname of the system (if hostname is an executable in the user’s PATH).

SpecLists as Constraints¶

Dependencies and compilers in Spack can be both packages in an environment and constraints on other packages. References to SpecLists allow a shorthand to treat packages in a list as either a compiler or a dependency using the $% or $^ syntax respectively.

For example, the following environment has three root packages: gcc@8.1.0, mvapich2@2.3.1 %gcc@8.1.0, and hdf5+mpi %gcc@8.1.0 ^mvapich2@2.3.1.

spack:
  definitions:
  - compilers: [gcc@8.1.0]
  - mpis: [mvapich2@2.3.1]
  - packages: [hdf5+mpi]

  specs:
  - $compilers
  - matrix:
    - [$mpis]
    - [$%compilers]
  - matrix:
    - [$packages]
    - [$^mpis]
    - [$%compilers]

This allows for a much-needed reduction in redundancy between packages and constraints.

Spec Groups¶

Added in version 1.2.

Environments can be organized with named spec groups, enabling you to apply localized configuration overrides and establish concretization dependencies. This is extremely useful in a couple of common scenarios, as detailed below.

Building and using a compiler in a single environment¶

A common use case is to build a recent compiler on top of an existing system and then compile a stack of software with it. For instance, assume we are interested in building hdf5 and libtree with gcc@15.2. The following manifest file would do exactly that:

spack:
  specs:
  - group: compiler
    specs:
    - gcc@15.2

  - group: apps
    needs: [compiler]
    specs:
    - hdf5 %gcc@15.2
    - libtree %gcc@15.2

The group: attribute allows to name a group of specs, which are then listed under the specs: attribute in the same object. The simplest example is the compiler group composed of just the gcc@15.2 spec.

To express dependencies among groups of specs the needs: attribute is used, which is a list of names corresponding to the groups we depend on. The way this works is that group dependencies are always concretized before the current group, and their specs are always available for reuse when the current group is concretized.

Configuring a group of specs¶

Another common scenario is the deployment of different configurations (e.g. CUDA enabled vs. ROCm enabled) of the same set of software. As an example, assume we want to install gromacs and quantum-espresso for both target=x86_64_v3 and target=x86_64_v4. That can be done with the following manifest file:

spack:
- group: apps-x86_64_v3
  specs:
  - gromacs
  - quantum-espresso
  override:
    packages:
      all:
        prefer:
        - target=x86_64_v3

- group: apps-x86_64_v4
  specs:
  - gromacs
  - quantum-espresso
  override:
    packages:
      all:
        prefer:
        - target=x86_64_v4

The override: attribute allows us to override the configuration for a single group of specs. The overridden part is always added as the topmost scope when the current group is concretized. This ensures the override always takes precedence over other sources of configuration.

Controlling garbage collection with `explicit: false`¶

By default every spec group is treated as a set of explicit roots. This means its specs are preserved by spack gc even when nothing else depends on them. Setting explicit: false on a group marks its specs as implicit, making them eligible for garbage collection once no other installed spec depends on them:

spack:
  specs:
  - group: compiler
    explicit: false
    specs:
    - gcc@15.2

  - group: apps
    needs: [compiler]
    specs:
    - hdf5 %gcc@15.2
    - libtree %gcc@15.2

After the apps are installed, spack gc will remove the compiler once no installed spec has a link or run dependency on it.

Note

Flipping explicit: false on a group that has already been installed does not retroactively update the database record for the already-installed specs. The flag takes effect only for specs installed, or re-installed, after the change. To immediately mark an existing spec as implicit, use spack mark -i <spec>.

Modifying Environment Variables¶

Spack Environments can modify the active shell’s environment variables when activated. The environment can be configured to set, unset, prepend, or append using env_vars configuration in spack.yaml:

spack:
  env_vars:
    set:
      ENVAR_TO_SET_IN_ENV_LOAD: "FOO"
    unset:
    - ENVAR_TO_UNSET_IN_ENV_LOAD
    prepend_path:
      PATH_LIST: "path/to/prepend"
    append_path:
      PATH_LIST: "path/to/append"
    remove_path:
      PATH_LIST: "path/to/remove"

Environment Views¶

Spack Environments can have an associated filesystem view, which is a directory with a more traditional structure <view>/bin, <view>/lib, <view>/include in which all files of the installed packages are linked.

By default a view is created for each environment, thanks to the view: true option in the spack.yaml manifest file:

spack:
  specs: [perl, python]
  view: true

The view is created in a hidden directory .spack-env/view relative to the environment. If you’ve used spack env activate, you may have already interacted with this view. Spack prepends its <view>/bin dir to PATH when the environment is activated, so that you can directly run executables from all installed packages in the environment.

Views are highly customizable: you can control where they are put, modify their structure, include and exclude specs, change how files are linked, and you can even generate multiple views for a single environment.

Minimal view configuration¶

The minimal configuration

spack:
  # ...
  view: true

lets Spack generate a single view with default settings under the .spack-env/view directory of the environment.

Another short way to configure a view is to specify just where to put it:

spack:
  # ...
  view: /path/to/view

Views can also be disabled by setting view: false.

Advanced view configuration¶

One or more view descriptors can be defined under view, keyed by a name. The example from the previous section with view: /path/to/view is equivalent to defining a view descriptor named default with a root attribute:

spack:
  # ...
  view:
    default:  # name of the view
      root: /path/to/view  # view descriptor attribute

The default view descriptor name is special: when you spack env activate your environment, this view will be used to update (among other things) your PATH variable.

View descriptors must contain the root of the view, and optionally projections, select and exclude lists and link information via link and link_type.

As a more advanced example, in the following manifest file snippet we define a view named mpis, rooted at /path/to/view in which all projections use the package name, version, and compiler name to determine the path for a given package. This view selects all packages that depend on MPI, and excludes those built with the GCC compiler at version 8.5. The root specs with their (transitive) link and run type dependencies will be put in the view due to the link: all option, and the files in the view will be symlinks to the Spack install directories.

spack:
  # ...
  view:
    mpis:
      root: /path/to/view
      select: [^mpi]
      exclude: ["%gcc@8.5"]
      projections:
        all: "{name}/{version}-{compiler.name}"
      link: all
      link_type: symlink
      link_dirs: true

The default for the select and exclude values is to select everything and exclude nothing. The default projection is the default view projection ({}). The link attribute allows the following values:

link: all include root specs with their transitive run and link type dependencies (default);
link: run include root specs with their transitive run type dependencies;
link: roots include root specs without their dependencies.

The link_type defaults to symlink but can also take the value of hardlink or copy.

Added in version 1.2: The link_dirs option controls whether directories are symlinked. This is the default behavior in Spack v1.2 and later. This is an optimization that significantly reduces the time to create views, and reduces the inode usage of the view. It only applies when link_type is set to symlink. If you want to link only non-directory files, set link_dirs: false.

Tip

The option link: run can be used to create small environment views for Python packages. Python will be able to import packages inside of the view even when the environment is not activated, and linked libraries will be located outside of the view thanks to rpaths.

From the command line, the spack env create command takes an argument --with-view [PATH] that sets the path for a single, default view. If no path is specified, the default path is used (view: true). The argument --without-view can be used to create an environment without any view configured.

The spack env view command can be used to manage views of an environment. The subcommand spack env view enable will add a view named default to an environment. It takes an optional argument to specify the path for the new default view. The subcommand spack env view disable will remove the view named default from an environment if one exists. The subcommand spack env view regenerate will regenerate the views for the environment. This will apply any updates in the environment configuration that have not yet been applied.

View Projections¶

The default projection into a view is to link every package into the root of the view. The projections attribute is a mapping of partial specs to spec format strings, defined by the format() function, as shown in the example below:

projections:
  zlib: "{name}-{version}"
  ^mpi: "{name}-{version}/{^mpi.name}-{^mpi.version}-{compiler.name}-{compiler.version}"
  all: "{name}-{version}/{compiler.name}-{compiler.version}"

Projections also permit environment and Spack configuration variable expansions as shown below:

projections:
  all: "{name}-{version}/{compiler.name}-{compiler.version}/$date/$SYSTEM_ENV_VARIABLE"

where $date is the Spack configuration variable that will expand with the YYYY-MM-DD format and $SYSTEM_ENV_VARIABLE is an environment variable defined in the shell.

The entries in the projections configuration file must all be either specs or the keyword all. For each spec, the projection used will be the first non-all entry that the spec satisfies, or all if there is an entry for all and no other entry is satisfied by the spec. Where the keyword all appears in the file does not matter.

Given the example above, the spec zlib@1.2.8 will be linked into /my/view/zlib-1.2.8/, the spec hdf5@1.8.10+mpi %gcc@4.9.3 ^mvapich2@2.2 will be linked into /my/view/hdf5-1.8.10/mvapich2-2.2-gcc-4.9.3, and the spec hdf5@1.8.10~mpi %gcc@4.9.3 will be linked into /my/view/hdf5-1.8.10/gcc-4.9.3.

If the keyword all does not appear in the projections configuration file, any spec that does not satisfy any entry in the file will be linked into the root of the view as in a single-prefix view. Any entries that appear below the keyword all in the projections configuration file will not be used, as all specs will use the projection under all before reaching those entries.

Group of Specs¶

Views can also be applied to a selected list of spec groups. This can be done by specifying the group: attribute in the view configuration. For instance, with the following manifest:

spack:
  concretizer:
    unify: true

  packages:
    all:
      require:
      - target=x86_64_v4

  specs:
  - group: compiler
    specs:
    - gcc@15.2

  - group: apps
    needs: [compiler]
    specs:
    - hdf5~mpi %gcc@15.2
    - libtree %gcc@15.2

  view:
    apps:
      root: ./views/apps
      group: apps

The view will only contain entries from the apps group, and will not include specs from the compiler group.

Activating environment views¶

The spack env activate <env> command has two effects:

It activates the environment so that further Spack commands such as spack install will run in the context of the environment.
It activates the view so that environment variables such as PATH are updated to include the view.

Without further arguments, the default view of the environment is activated. If a view with a different name has to be activated, spack env activate --with-view <name> <env> can be used instead. You can also activate the environment without modifying further environment variables using --without-view.

The environment variables affected by the spack env activate command and the paths that are used to update them are determined by the prefix inspections defined in your modules configuration; the defaults are summarized in the following table.

Variable	Paths
PATH	bin
MANPATH	man, share/man
ACLOCAL_PATH	share/aclocal
PKG_CONFIG_PATH	lib/pkgconfig, lib64/pkgconfig, share/pkgconfig
CMAKE_PREFIX_PATH	.

Each of these paths are appended to the view root, and added to the relevant variable if the path exists. For this reason, it is not recommended to use non-default projections with the default view of an environment.

The spack env deactivate command will remove the active view of the Spack environment from the user’s environment variables.

Generating Depfiles from Environments¶

Spack can generate Makefiles to make it easier to build multiple packages in an environment in parallel.

Note

Since Spack v1.1, there is a new experimental installer that supports package-level parallelism out of the box with POSIX jobserver support. You can enable it with spack config add config:installer:new. This new installer may provide a simpler alternative to the spack env depfile workflow described in this section for users primarily interested in speeding up environment installations.

Generated Makefiles expose targets that can be included in existing Makefiles, to allow other targets to depend on the environment installation.

A typical workflow is as follows:

$ spack env create -d .
$ spack -e . add perl
$ spack -e . concretize
$ spack -e . env depfile -o Makefile
$ make -j64

This generates a Makefile from a concretized environment in the current working directory, and make -j64 installs the environment, exploiting parallelism across packages as much as possible. Spack respects the Make jobserver and forwards it to the build environment of packages, meaning that a single -j flag is enough to control the load, even when packages are built in parallel.

By default the following phony convenience targets are available:

make all: installs the environment (default target);
make clean: cleans files used by make, but does not uninstall packages.

Tip

GNU Make version 4.3 and above have great support for output synchronization through the -O and --output-sync flags, which ensure that output is printed orderly per package install. To get synchronized output with colors, use make -j<N> SPACK_COLOR=always --output-sync=recurse.

Specifying dependencies on generated `make` targets¶

An interesting question is how to include generated Makefiles in your own Makefiles. This comes up when you want to install an environment that provides executables required in a command for a make target of your own.

The example below shows how to accomplish this: the env target specifies the generated spack/env target as a prerequisite, meaning that the environment gets installed and is available for use in the env target.

SPACK ?= spack

.PHONY: all clean env

all: env

spack.lock: spack.yaml
	$(SPACK) -e . concretize -f

env.mk: spack.lock
	$(SPACK) -e . env depfile -o $@ --make-prefix spack

env: spack/env
	$(info environment installed!)

clean:
	rm -rf spack.lock env.mk spack/

ifeq (,$(filter clean,$(MAKECMDGOALS)))
include env.mk
endif

This works as follows: when make is invoked, it first “remakes” the missing include env.mk as there is a target for it. This triggers concretization of the environment and makes Spack output env.mk. At that point the generated target spack/env becomes available through include env.mk.

As it is typically undesirable to remake env.mk as part of make clean, the include is conditional.

Note

When including generated Makefiles, it is important to use the --make-prefix flag and use the non-phony target <prefix>/env as prerequisite, instead of the phony target <prefix>/all.

Building a subset of the environment¶

The generated Makefiles contain install targets for each spec, identified by <name>-<version>-<hash>. This allows you to install only a subset of the packages in the environment. When packages are unique in the environment, it’s enough to know the name and let tab-completion fill out the version and hash.

The following phony targets are available: install/<spec> to install the spec with its dependencies, and install-deps/<spec> to only install its dependencies. This can be useful when certain flags should only apply to dependencies. Below we show a use case where a spec is installed with verbose output (spack install --verbose) while its dependencies are installed silently:

$ spack env depfile -o Makefile

# Install dependencies in parallel, only show a log on error.
$ make -j16 install-deps/python-3.11.0-<hash> SPACK_INSTALL_FLAGS=--show-log-on-error

# Install the root spec with verbose output.
$ make -j16 install/python-3.11.0-<hash> SPACK_INSTALL_FLAGS=--verbose

Adding post-install hooks¶

Another advanced use-case of generated Makefiles is running a post-install command for each package. These “hooks” could be anything from printing a post-install message, running tests, or pushing just-built binaries to a build cache.

This can be accomplished through the generated [<prefix>/]SPACK_PACKAGE_IDS variable. Assuming we have an active and concrete environment, we generate the associated Makefile with a prefix example:

$ spack env depfile -o env.mk --make-prefix example

And we now include it in a different Makefile, in which we create a target example/push/% with % referring to a package identifier. This target depends on the particular package installation. In this target we automatically have the target-specific HASH and SPEC variables at our disposal. They are respectively the spec hash (excluding leading /), and a human-readable spec. Finally, we have an entry point target push that will update the build cache index once every package is pushed. Note how this target uses the generated example/SPACK_PACKAGE_IDS variable to define its prerequisites.

SPACK ?= spack
BUILDCACHE_DIR = $(CURDIR)/tarballs

.PHONY: all

all: push

include env.mk

example/push/%: example/install/%
	@mkdir -p $(dir $@)
	$(info About to push $(SPEC) to a buildcache)
	$(SPACK) -e . buildcache push --only=package $(BUILDCACHE_DIR) /$(HASH)
	@touch $@

push: $(addprefix example/push/,$(example/SPACK_PACKAGE_IDS))
	$(info Updating the buildcache index)
	$(SPACK) -e . buildcache update-index $(BUILDCACHE_DIR)
	$(info Done!)
	@touch $@

Environments (spack.yaml, spack.lock)¶

Using Environments¶

Creating a managed Environment¶

Activating an Environment¶

Independent Environments¶

Environment-Aware Commands¶

Adding Abstract Specs¶

Concretizing¶

Installing an Environment¶

Developing Packages in a Spack Environment¶

Modifying Specs in an Environment¶

Loading¶

Including Concrete Environments¶

Creating combined concrete environments¶

Updating a combined environment¶

Configuring Environments¶

Inline configurations¶

Included configurations¶

Configuration precedence¶

Manually Editing the Specs List¶

Spec concretization¶

Spec Matrices¶

Spec List References¶

SpecLists as Constraints¶

Spec Groups¶

Building and using a compiler in a single environment¶

Configuring a group of specs¶

Controlling garbage collection with explicit: false¶

Modifying Environment Variables¶

Environment Views¶

Minimal view configuration¶

Advanced view configuration¶

View Projections¶

Group of Specs¶

Activating environment views¶

Generating Depfiles from Environments¶

Specifying dependencies on generated make targets¶

Building a subset of the environment¶

Adding post-install hooks¶

Controlling garbage collection with `explicit: false`¶

Specifying dependencies on generated `make` targets¶