spack.ci package

class spack.ci.PushResult(success, url)

Bases: tuple

success

Alias for field number 0

url

Alias for field number 1

class spack.ci.RebuildDecision(rebuild: bool = True, reason: str = '')

Bases: object

spack.ci.can_sign_binaries()

Utility method to determine if this spack instance is capable of signing binary packages. This is currently only possible if the spack gpg keystore contains exactly one secret key.

spack.ci.can_verify_binaries()

Utility method to determine if this spack instance is capable (at least in theory) of verifying signed binaries.

spack.ci.check_for_broken_specs(pipeline_specs: List[Spec], broken_specs_url: str) → bool

Check the pipeline specs against the list of known broken specs and return True if there were any matches, False otherwise.

spack.ci.collect_pipeline_options(env: Environment, args) → PipelineOptions

Gather pipeline options from cli args, spack environment, and os environment variables

spack.ci.compute_affected_packages(repo: Repo, rev1: str = 'HEAD^', rev2: str = 'HEAD') → Set[str]

Determine which packages were added, removed or changed between rev1 and rev2, and return the names as a set

spack.ci.copy_stage_logs_to_artifacts(job_spec: Spec, job_log_dir: str) → None

Copy selected build stage file(s) to the given artifacts directory

Looks for build logs in the stage directory of the given job_spec, and attempts to copy the files into the directory given by job_log_dir.

Parameters:

• job_spec – spec associated with spack install log

• job_log_dir – path into which build log should be copied

spack.ci.copy_test_logs_to_artifacts(test_stage, job_test_dir)

Copy test log file(s) to the given artifacts directory

Parameters:

• test_stage (str) – test stage path

• job_test_dir (str) – the destination artifacts test directory

spack.ci.create_already_built_pruner(check_index_only: bool = True) → Callable[[Spec], RebuildDecision]

Return a filter that prunes specs already present on any configured mirrors

spack.ci.create_buildcache(input_spec: Spec, *, destination_mirror_urls: List[str], sign_binaries: bool = False) → List[PushResult]

Create the buildcache at the provided mirror(s).

Parameters:

• input_spec – Installed spec to package and push

• destination_mirror_urls – List of urls to push to

• sign_binaries – Whether or not to sign buildcache entry

Returns: A list of PushResults, indicating success or failure.

spack.ci.create_external_pruner() → Callable[[Spec], RebuildDecision]

Return a filter that prunes external specs

spack.ci.create_unaffected_pruner(affected_specs: Set[Spec]) → Callable[[Spec], RebuildDecision]

Given a set of “affected” specs, return a filter that prunes specs not in the set.

spack.ci.display_broken_spec_messages(base_url, hashes)

Fetch the broken spec file for each of the hashes under the base_url and print a message with some details about each one.

spack.ci.download_and_extract_artifacts(url: str, work_dir: str) → str

Look for gitlab artifacts.zip at the given url, and attempt to download and extract the contents into the given work_dir

Parameters:

• url – Complete url to artifacts.zip file

• work_dir – Path to destination where artifacts should be extracted

Returns:

Artifacts root path relative to the archive root

spack.ci.filter_added_checksums(checksums: Iterable[str], path: str, from_ref: str = 'HEAD~1', to_ref: str = 'HEAD') → List[str]

Get a list of the version checksums added between from_ref and to_ref.

Parameters:

• checksums – an iterable of checksums to look for in the diff

• path – path to the package.py

• from_ref – oldest git ref, defaults to HEAD~1

• to_ref – newer git ref, defaults to HEAD

Returns: list of version checksums added between refs

spack.ci.generate_pipeline(env: Environment, args) → None

Given an environment and the command-line args, generate a pipeline.

Parameters:

• env (spack.environment.Environment) – Activated environment object which must contain a ci section describing attributes for all jobs and a target which should specify an existing pipeline generator.

• args – (spack.main.SpackArgumentParser): Parsed arguments from the command line.

spack.ci.get_change_revisions(path: str) → Tuple[str | None, str | None]

If this is a git repo get the revisions to use when checking for changed packages and spack core modules.

spack.ci.get_git_root(path: str) → str | None

spack.ci.get_spack_info()

If spack is running from a git repo, return the most recent git log entry, otherwise, return a string containing the spack version.

spack.ci.get_spec_filter_list(env: Environment, affected_pkgs: Set[str], dependent_traverse_depth: int | None = None) → Set[Spec]

Given a list of package names and an active/concretized environment, return the set of all concrete specs from the environment that could have been affected by changing the list of packages.

If a dependent_traverse_depth is given, it is used to limit upward (in the parent direction) traversal of specs of touched packages. E.g. if 1 is provided, then only direct dependents of touched package specs are traversed to produce specs that could have been affected by changing the package, while if 0 is provided, only the changed specs themselves are traversed. If None is given, upward traversal of touched package specs is done all the way to the environment roots. Providing a negative number results in no traversals at all, yielding an empty set.

Parameters:

• env – Active concrete environment

• affected_pkgs – Affected package names

• dependent_traverse_depth – Integer to limit dependent traversal, None means no limit

Returns:

A set of concrete specs from the active environment including those associated with affected packages, their dependencies and dependents, as well as their dependents dependencies.

spack.ci.get_unaffected_pruners(env: Environment, untouched_pruning_dependent_depth: int | None) → Callable[[Spec], RebuildDecision] | None

spack.ci.import_signing_key(base64_signing_key: str) → None

Given Base64-encoded gpg key, decode and import it to use for signing packages.

Parameters:

base64_signing_key – A gpg key including the secret key, armor-exported and base64 encoded, so it can be stored in a gitlab CI variable. For an example of how to generate such a key, see https://github.com/spack/spack-infrastructure/blob/main/gitlab-docker/files/gen-key.

spack.ci.process_command(name, commands, repro_dir, run=True, exit_on_failure=True)

Create a script for and run the command. Copy the script to the reproducibility directory.

Parameters:

• name (str) – name of the command being processed

• commands (list) – list of arguments for single command or list of lists of arguments for multiple commands. No shell escape is performed.

• repro_dir (str) – Job reproducibility directory

• run (bool) – Run the script and return the exit code if True

Returns: the exit code from processing the command

spack.ci.prune_pipeline(pipeline: PipelineDag, pruning_filters: List[Callable[[Spec], RebuildDecision]], print_summary: bool = False) → None

Given a PipelineDag and a list of pruning filters, return a modified PipelineDag containing only the nodes that survive pruning by all of the filters.

spack.ci.push_to_build_cache(spec: Spec, mirror_url: str, sign_binaries: bool) → bool

Push one or more binary packages to the mirror.

Parameters:

• spec – Installed spec to push

• mirror_url – URL of target mirror

• sign_binaries – If True, spack will attempt to sign binary package before pushing.

spack.ci.read_broken_spec(broken_spec_url)

Read data from broken specs file located at the url, return as a yaml object.

spack.ci.reproduce_ci_job(url, work_dir, autostart, gpg_url, runtime, use_local_head)

Given a url to gitlab artifacts.zip from a failed spack ci rebuild job, attempt to setup an environment in which the failure can be reproduced locally. This entails the following:

First download and extract artifacts. Then look through those artifacts to glean some information needed for the reproduer (e.g. one of the artifacts contains information about the version of spack tested by gitlab, another is the generated pipeline yaml containing details of the job like the docker image used to run it). The output of this function is a set of printed instructions for running docker and then commands to run to reproduce the build once inside the container.

spack.ci.run_standalone_tests(*, cdash: CDashHandler | None = None, fail_fast: bool = False, log_file: str | None = None, job_spec: Spec | None = None, repro_dir: str | None = None, timeout: int | None = None)

Run stand-alone tests on the current spec.

Parameters:

• cdash – cdash handler instance

• fail_fast – terminate tests after the first failure

• log_file – test log file name if NOT CDash reporting

• job_spec – spec that was built

• repro_dir – reproduction directory

• timeout – maximum time (in seconds) that tests are allowed to run

spack.ci.setup_spack_repro_version(repro_dir: str, checkout_commit: str, merge_commit: str | None = None) → bool

Look in the local spack clone to find the checkout_commit, and if provided, the merge_commit given as arguments. If those commits can be found locally, then clone spack and attempt to recreate a merge commit with the same parent commits as tested in gitlab. This looks something like

1. git clone repo && cd repo

2. git checkout <checkout_commit>

3. git merge <merge_commit>

If there is no merge_commit provided, then skip step (3).

Parameters:

• repro_dir – Location where spack should be cloned

• checkout_commit – SHA of PR branch commit

• merge_commit – SHA of target branch parent

Returns: True iff the git repo state was successfully recreated

spack.ci.stack_changed(env_path: str) → bool

Given an environment manifest path, return whether or not the stack was changed. Returns True iff the environment manifest changed between the provided revisions (or additionally if the .gitlab-ci.yml file itself changed).

spack.ci.write_broken_spec(url, pkg_name, stack_name, job_url, pipeline_url, spec_dict)

Given a url to write to and the details of the failed job, write an entry in the broken specs list.

Submodules

spack.ci.common module

class spack.ci.common.CDashHandler(ci_cdash)

Bases: object

Class for managing CDash data and processing.

args()

build_name(spec: Spec | None = None) → str | None

Returns the CDash build name.

A name will be generated if the spec is provided, otherwise, the value will be retrieved from the environment through the SPACK_CDASH_BUILD_NAME variable.

Returns: (str) given spec’s CDash build name.

property build_stamp

Returns the CDash build stamp.

The one defined by SPACK_CDASH_BUILD_STAMP environment variable is preferred due to the representation of timestamps; otherwise, one will be built.

Returns: (str) current CDash build stamp

copy_test_results(source, dest)

Copy test results to artifacts directory.

create_buildgroup()

Create the CDash buildgroup if it does not already exist.

property project_enc

report_skipped(spec: Spec, report_dir: str, reason: str | None)

Explicitly report skipping testing of a spec (e.g., it’s CI configuration identifies it as known to have broken tests or the CI installation failed).

Parameters:

• spec – spec being tested

• report_dir – directory where the report will be written

• reason – reason the test is being skipped

property upload_url

class spack.ci.common.PipelineDag(specs: List[Spec])

Bases: object

Turn a list of specs into a simple directed graph, that doesn’t keep track of edge types.

get_dependencies(node: PipelineNode) → List[PipelineNode]

Returns a list of nodes corresponding to the direct dependencies of the given node.

classmethod key(spec: Spec) → str

prune(node_key: str)

Remove a node from the graph, and reconnect its parents and children

traverse_nodes(direction: str = 'children') → Generator[Tuple[int, PipelineNode], None, None]

Yields (depth, node) from the pipeline graph. Traversal is topologically ordered from the roots if direction is children, or from the leaves if direction is parents. The yielded depth is the length of the longest path from the starting point to the yielded node.

class spack.ci.common.PipelineNode(spec: Spec)

Bases: object

children: Set[str]

property key

Return key of the stored spec

parents: Set[str]

spec: Spec

class spack.ci.common.PipelineOptions(env: Environment, buildcache_destination: Mirror, artifacts_root: str = 'jobs_scratch_dir', print_summary: bool = True, output_file: str | None = None, check_index_only: bool = False, broken_specs_url: str | None = None, rebuild_index: bool = True, untouched_pruning_dependent_depth: int | None = None, prune_untouched: bool = False, prune_up_to_date: bool = True, prune_unaffected: bool = True, prune_external: bool = True, stack_name: str | None = None, pipeline_type: PipelineType | None = None, require_signing: bool = False, cdash_handler: CDashHandler | None = None)

Bases: object

A container for all pipeline options that can be specified (whether via cli, config/yaml, or environment variables)

forward_variables: List[str]

class spack.ci.common.PipelineType(*values)

Bases: Enum

COPY_ONLY

PROTECTED_BRANCH

PULL_REQUEST

spack_copy_only

spack_protected_branch

spack_pull_request

class spack.ci.common.SpackCIConfig(ci_config)

Bases: object

Spack CI object used to generate intermediate representation used by the CI generator(s).

generate_ir()

Generate the IR from the Spack CI configurations.

init_pipeline_jobs(pipeline: PipelineDag)

exception spack.ci.common.SpackCIError(msg)

Bases: SpackError

spack.ci.common.copy_files_to_artifacts(src: str, artifacts_dir: str, *, compress_artifacts: bool = False) → None

Copy file(s) to the given artifacts directory

Parameters:

• src (str) – the glob-friendly path expression for the file(s) to copy

• artifacts_dir (str) – the destination directory

• compress_artifacts (bool) – option to compress copied artifacts using Gzip

spack.ci.common.copy_gzipped(glob_or_path: str, dest: str) → None

Copy all of the files in the source glob/path to the destination.

Parameters:

• glob_or_path – path to file to test

• dest – destination path to copy to

spack.ci.common.ensure_expected_target_path(path: str) → str

Returns passed paths with all Windows path separators exchanged for posix separators

TODO (johnwparent): Refactor config + cli read/write to deal only in posix style paths

spack.ci.common.unpack_script(script_section, op=_noop)

spack.ci.common.win_quote(quote_str: str) → str

spack.ci.common.write_pipeline_manifest(specs, src_prefix, dest_prefix, output_file)

Write out the file describing specs that should be copied

spack.ci.generator_registry module

Generators that support writing out pipelines for various CI platforms, using a common pipeline graph definition.

exception spack.ci.generator_registry.UnknownGeneratorException(generator_name)

Bases: SpackError

spack.ci.generator_registry.generator(name)

Decorator to register a pipeline generator method. A generator method should take PipelineDag, SpackCIConfig, and PipelineOptions arguments, and should produce a pipeline file.

spack.ci.generator_registry.get_generator(name)

spack.ci.gitlab module

spack.ci.gitlab.generate_gitlab_yaml(pipeline: PipelineDag, spack_ci: SpackCIConfig, options: PipelineOptions)

Given a pipeline graph, job attributes, and pipeline options, write a pipeline that can be consumed by GitLab to the given output file.

Parameters:

• pipeline – An already pruned graph of jobs representing all the specs to build

• spack_ci – An object containing the configured attributes of all jobs in the pipeline

• options – An object containing all the pipeline options gathered from yaml, env, etc…

spack.ci.gitlab.get_job_name(spec: Spec, build_group: str | None = None) → str

Given a spec and possibly a build group, return the job name. If the resulting name is longer than 255 characters, it will be truncated.

Parameters:

• spec – Spec job will build

• build_group – Name of build group this job belongs to (a CDash notion)

Returns: The job name

spack.ci.gitlab.maybe_generate_manifest(pipeline: PipelineDag, options: PipelineOptions, manifest_path)