Spec File Format¶
This document describes the spec file format, which stores the DAG of a spec as JSON.
The current version of the format is v6, defined by spack.spec.SPECFILE_FORMAT_VERSION.
Spec files are found in the following places:
In
<prefix>/.spack/spec.json, written when a spec is installed.spack reindexrebuilds the Database from these files.In build caches, where a few keys specific to build caches are added to the spec file.
In the output of
spack spec --json.
The Database and the lockfile store the nodes of a spec file without the enclosing spec and _meta keys, as described in Versioned File Formats.
All of these files hold concrete specs, and the rest of this document describes concrete specs.
Abstract specs
The format can also represent abstract specs, which are written only in the entries of the concretization cache, for troubleshooting, and are not read back.
The nodes of an abstract spec have "concrete": false, and may have the keys abstract_hash, compiler_flags, propagate, abstract, propagated_parameters, and propagated_abstract; their dependencies may have when and propagation.
None of these keys is written for concrete specs, so they do not affect the DAG hash.
Structure¶
This section describes the current version of the format, v6.
A spec file has a single top-level spec key, which holds the version of the format and the list of nodes:
{
"spec": {
"_meta": {"version": 6},
"nodes": [
{
"name": "zlib-ng",
"version": "2.3.3",
"arch": {"platform": "linux", "platform_os": "ubuntu24.04", "target": {"name": "alderlake"}},
"namespace": "builtin",
"parameters": {"build_system": "autotools", "shared": true, "cflags": []},
"package_hash": "bnbx7ezlmmjgjyubhncgeel7lpbjrj6l4aeuhyewdeyz56l7tqsq====",
"provided_virtuals": ["zlib-api"],
"dependencies": [
{
"name": "gcc",
"hash": "26zpexhcroz5n23vbzm5ypre7tlfus2b",
"parameters": {"deptypes": ["build"], "virtuals": ["c", "cxx"]}
},
{
"name": "glibc",
"hash": "cyucshxqmd4krmsp7nco25trnybrdbam",
"parameters": {"deptypes": ["link"], "virtuals": ["libc"]}
}
],
"annotations": {"original_specfile_version": 6},
"hash": "nic37xsihlqze3xyg3xd6pthcgqxoy6p"
}
]
}
}
To keep the example short, the features of the target, most variants, the remaining dependencies, and the remaining nodes are not shown.
The first node of the list is the root of the DAG. The order of the remaining nodes is not significant, since nodes refer to each other by hash. Each node is listed once, even if it can be reached through several paths. For a spliced spec, the list also includes the nodes of the build spec.
Each node is a dictionary with the following keys. The keys of previous versions are described in Version History.
name,version,arch,namespaceThe identity of the node.
parametersThe values of the variants, and the compiler flags.
patchesThe hashes of the patches applied to the node, in the order in which they are applied. It is present only in nodes that have patches.
package_hashThe hash of the content of the
package.pythat the node was concretized with.provided_virtualsThe virtuals that a concrete node provides, possibly with a version, for instance
libgfortran@5. It is written only when the node provides at least one virtual.dependenciesOne entry for each edge, which refers to the dependency by name and DAG hash. The
parametersof the entry hold the dependency types, the virtuals that the dependency provides on that edge, anddirect: truefor direct dependencies.build_specThe name and DAG hash of the spec that was built. It is present only in spliced specs.
externalThe path or the modules of an external, and its extra attributes. It is present only in externals.
annotationsoriginal_specfile_versionis the version of the spec file from which the node was first read. Nodes first read from a spec file before v5 also havecompiler, the compiler that those versions stored as a node attribute.hashThe DAG hash of the node.
A JSON schema of the format is in lib/spack/spack/schema/spec.py.
How the DAG Hash Is Computed¶
The DAG hash of a concrete node is computed in two steps:
The node dictionary, without the
hashkey, is serialized as JSON with no whitespace, and with the keys in the order in which they are written.The SHA-1 digest of the serialization is encoded in base32, in lowercase.
This definition has the following consequences:
The node dictionary contains the hashes of the dependencies, so the DAG hash depends on the entire sub-DAG.
Any change to what Spack writes for a concrete node, including the order of its keys, gives a different hash to every spec concretized afterwards.
The hash stored in a spec file is never recomputed when the file is read. Spack can therefore read a spec written with an older definition of the hash, and still find the installation it refers to. The changes to the definition are listed in Version History.
Spliced specs
The hash of a spliced spec is computed in the same way, except that its last 7 characters are replaced by those of the hash of its build spec. A spliced spec is installed by relocating the binaries of its build spec to a new prefix, and relocation has to leave the end of the prefix unchanged. The reason is that compilers save space by storing a string constant as the suffix of a longer one, so rewriting the end of a path in a binary could alter an unrelated string that shares it. With the default layout the prefix ends with the hash, hence the two hashes have the same ending.
Compatibility¶
The properties below are defined in Versioned File Formats.
- Backward compatibility
Every version of Spack reads all the versions of the format up to its own. When reading an older version, Spack reconstructs the information that the version does not store from the package repositories.
- Forward compatibility
None. Reading a version newer than the current one is an error.
- Rewrite policy
Spec files are never rewritten. The
spec.jsonfile of an installation prefix remains in the version it was written in. When a node read from an older version is written to a new file, its original version is recorded inannotations.original_specfile_version.
Version History¶
The version is recorded in spec._meta.version.
Spec files in version 1 have no _meta key, and are recognized because their spec key holds a list.
- Version 6 (Spack v1.3)
Concrete nodes record the virtuals they provide in
provided_virtuals, which is part of the DAG hash (#53012).- Version 5 (Spack v1.0)
Compilers become build dependencies (#45189). The
compilerattribute of nodes is removed,annotationsis added, andparameters.directmarks direct dependencies. From Spack v1.0 on, the DAG hash also includes test dependencies (#49505). Theabstractkey was added for abstract specs before the release of Spack v1.0 (#49756). The keysabstract_hash,compiler_flags,when, andpropagationwere added later for abstract specs, without a new version (#52788). The keyspropagated_parametersandpropagated_abstractwere added in the same way (#52963).- Version 4 (Spack v0.21)
Virtuals are recorded on the edges of the DAG (#34821). The dependency types of each entry move to
parameters.deptypes, next to the newparameters.virtuals. Thepropagatekey was added in Spack v0.23 for abstract specs, without a new version (#47351).- Version 3 (Spack v0.18)
The build hash and the full hash are removed (#28504). The structure is the same as in version 2, and
hashbecomes the DAG hash that includes build dependencies and the package hash. In versions 1 and 2 the DAG hash includes only link and run dependencies.- Version 2 (Spack v0.17)
Installation prefixes move from
spec.yamltospec.json(#22845). The_meta.versionkey is added,nodesbecomes a list of dictionaries with anamekey,dependenciesbecomes a list, andbuild_specis added.- Version 1 (Spack v0.16 and earlier)
Spec files are YAML (
spec.yaml), or JSON with the same structure. Thespeckey holds a list with one{<name>: <node>}entry for each node. Dependencies are a dictionary keyed by name, with the hash of the dependency and itstype. The hash of the dependency is stored underhash,full_hash, orbuild_hash, depending on the version of Spack that wrote the file.