..
   Copyright Spack Project Developers. See COPYRIGHT file for details.

   SPDX-License-Identifier: (Apache-2.0 OR MIT)

.. meta::
   :description lang=en:
      Advanced topics in Spack packaging, covering packages with multiple build systems, making packages discoverable with spack external find, and specifying ABI compatibility.

.. list-table::
   :widths: 25 25 25 25
   :header-rows: 0
   :width: 100%

   * - :doc:`1. Creation <packaging_guide_creation>`
     - :doc:`2. Build <packaging_guide_build>`
     - :doc:`3. Testing <packaging_guide_testing>`
     - **4. Advanced**

Packaging Guide: advanced topics
================================

This section of the packaging guide covers a few advanced topics.

.. _multiple_build_systems:

Multiple build systems
----------------------

Packages may use different build systems over time or across platforms.
Spack is designed to handle this seamlessly within a single ``package.py`` file.

Let's assume we work with ``curl`` and that the package is built using Autotools so far:

.. code-block:: python

   from spack_repo.builtin.build_systems.autotools import AutotoolsPackage


   class Curl(AutotoolsPackage):

       depends_on("zlib-api")

       def configure_args(self):
           return [f"--with-zlib={self.spec['zlib-api'].prefix}"]

To add CMake as a further build system we need to:

1. Add another base to the ``Curl`` package class (in our case ``cmake.CMakePackage``),
2. Explicitly declare which build systems are supported using the ``build_system`` directive,
3. Move the :doc:`build instructions <packaging_guide_build>` in *separate builder classes*.

.. code-block:: python

   from spack_repo.builtin.build_systems import autotools, cmake


   class Curl(cmake.CMakePackage, autotools.AutotoolsPackage):

       build_system("autotools", "cmake", default="cmake")

       depends_on("zlib-api")


   class AutotoolsBuilder(autotools.AutotoolsBuilder):
       def configure_args(self):
           return [f"--with-zlib={self.spec['zlib-api'].prefix}"]


   class CMakeBuilder(cmake.CMakeBuilder):
       def cmake_args(self):
           return [self.define_from_variant("USE_NGHTTP2", "nghttp2")]

In general, with multiple build systems there is a clear split between the :doc:`package metadata <packaging_guide_creation>` and the :doc:`build instructions <packaging_guide_build>`:

1. The directives such as ``depends_on``, ``variant``, ``patch`` go into the package class
2. The build phase functions like ``configure``, ``build`` and ``install``, and helper functions such as ``cmake_args`` or ``configure_args`` go into the builder classes

When ``curl`` is concretized, we can select its build system using the ``build_system`` variant, which is available for every package:

.. code-block:: spec

   $ spack install curl build_system=cmake

Override "phases" of a build system
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Sometimes package recipes need to override entire :ref:`phases <overriding-phases>` of a build system.
Let's assume this happens for ``cp2k``:

.. code-block:: python

   from spack.package import *
   from spack_repo.builtin.build_systems import autotools


   class Cp2k(autotools.AutotoolsPackage):
       def install(self, spec: Spec, prefix: str) -> None:
           # ...existing code...
           pass

If we want to add CMake as another build system we need to remember that the signature of phases changes when moving from the ``Package`` to the ``Builder`` class:

.. code-block:: python

   from spack.package import *
   from spack_repo.builtin.build_systems import autotools, cmake


   class Cp2k(autotools.AutotoolsPackage, cmake.CMakePackage):
       build_system("autotools", "cmake", default="cmake")


   class AutotoolsBuilder(autotools.AutotoolsBuilder):
       def install(self, pkg: Cp2k, spec: Spec, prefix: str) -> None:
           # ...existing code...
           pass

The ``install`` method now takes the ``Package`` instance as the first argument, since ``self`` refers to the builder class.

Add dependencies conditional on a build system
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Many build dependencies are conditional on which build system is chosen.
An effective way to handle this is to use a ``with when("build_system=...")`` block to specify dependencies that are only relevant for a specific build system:

.. code-block:: python

   from spack.package import *
   from spack_repo.builtin.build_systems import cmake, autotools


   class Cp2k(cmake.CMakePackage, autotools.AutotoolsPackage):

       build_system("cmake", "autotools", default="cmake")

       # Runtime dependencies
       depends_on("ncurses")
       depends_on("libxml2")

       # Lowerbounds for cmake only apply when using cmake as the build system
       with when("build_system=cmake"):
           depends_on("cmake@3.18:", when="@2.0:", type="build")
           depends_on("cmake@3:", type="build")

       # Specify extra build dependencies used only in the configure script
       with when("build_system=autotools"):
           depends_on("perl", type="build")
           depends_on("pkgconfig", type="build")

Transition from one build system to another
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Packages that transition from one build system to another can be modeled using :ref:`conditional variant values <variant-conditional-values>`:

.. code-block:: python

   from spack.package import *
   from spack_repo.builtin.build_systems import cmake, autotools


   class Cp2k(cmake.CMakePackage, autotools.AutotoolsPackage):

       build_system(
           conditional("cmake", when="@0.64:"),
           conditional("autotools", when="@:0.63"),
           default="cmake",
       )

In the example, the directive imposes a change from ``Autotools`` to ``CMake`` going from ``v0.63`` to ``v0.64``.

Inherit from a package with multiple build systems
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Customizing a package supporting multiple build systems is straightforward.
If we need to only customize the metadata, we can just define the derived package class.

For instance, let's assume we want to add a new version to the ``silo`` package:

.. code-block:: python

   from spack_repo.builtin.packages.silo.package import Silo as BuiltinSilo

   class Silo(BuiltinSilo):
       # Version not in builtin.silo
       version("special_version")

If we don't define any builder, Spack will reuse the custom builder from ``builtin.silo`` by default.
If we need to customize the builder too, we just have to inherit from it, like any other Python class:

.. code-block:: python

   from spack_repo.builtin.packages.silo.package import CMakeBuilder as SiloCMakeBuilder

   class CMakeBuilder(SiloCMakeBuilder):
       def cmake_args(self):
           return [self.define_from_variant("USE_NGHTTP2", "nghttp2")]

.. _make-package-findable:

Making a package discoverable with ``spack external find``
----------------------------------------------------------

The simplest way to make a package discoverable with :ref:`spack external find <cmd-spack-external-find>` is to:

1. Define the executables associated with the package.
2. Implement a method to determine the versions of these executables.

Minimal detection
^^^^^^^^^^^^^^^^^

The first step is fairly simple, as it requires only to specify a package-level ``executables`` attribute:

.. code-block:: python

   class Foo(Package):
       # Each string provided here is treated as a regular expression, and
       # would match for example "foo", "foobar", and "bazfoo".
       executables = ["foo"]

This attribute must be a list of strings.
Each string is a regular expression (e.g. "gcc" would match "gcc", "gcc-8.3", "my-weird-gcc", etc.) to determine a set of system executables that might be part of this package.
Note that to match only executables named "gcc" the regular expression ``"^gcc$"`` must be used.

Finally, to determine the version of each executable the ``determine_version`` method must be implemented:

.. code-block:: python

   @classmethod
   def determine_version(cls, exe):
       """Return either the version of the executable passed as argument
       or ``None`` if the version cannot be determined.

       Args:
           exe (str): absolute path to the executable being examined
       """

This method receives as input the path to a single executable and must return as output its version as a string.
If the version cannot be determined, or if the executable turns out to be a false positive, the value ``None`` must be returned, which ensures that the executable is discarded as a candidate.
Implementing the two steps above is mandatory, and gives the package the basic ability to detect if a spec is present on the system at a given version.

.. note::
   Any executable for which the ``determine_version`` method returns ``None`` will be discarded and won't appear in later stages of the workflow described below.

Additional functionality
^^^^^^^^^^^^^^^^^^^^^^^^

Besides the two mandatory steps described above, there are also optional methods that can be implemented to either increase the amount of details being detected or improve the robustness of the detection logic in a package.

Variants and custom attributes
""""""""""""""""""""""""""""""

The ``determine_variants`` method can be optionally implemented in a package to detect additional details of the spec:

.. code-block:: python

   @classmethod
   def determine_variants(cls, exes, version_str):
       """Return either a variant string, a tuple of a variant string
       and a dictionary of extra attributes that will be recorded in
       packages.yaml or a list of those items.

       Args:
           exes (list of str): list of executables (absolute paths) that
               live in the same prefix and share the same version
           version_str (str): version associated with the list of
               executables, as detected by ``determine_version``
       """

This method takes as input a list of executables that live in the same prefix and share the same version string, and returns either:

1. A variant string
2. A tuple of a variant string and a dictionary of extra attributes
3. A list of items matching either 1 or 2 (if multiple specs are detected from the set of executables)

If extra attributes are returned, they will be recorded in ``packages.yaml`` and be available for later reuse.
As an example, the ``gcc`` package will record by default the different compilers found and an entry in ``packages.yaml`` would look like:

.. code-block:: yaml

   packages:
     gcc:
       externals:
       - spec: "gcc@9.0.1 languages=c,c++,fortran"
         prefix: /usr
         extra_attributes:
           compilers:
             c: /usr/bin/x86_64-linux-gnu-gcc-9
             c++: /usr/bin/x86_64-linux-gnu-g++-9
             fortran: /usr/bin/x86_64-linux-gnu-gfortran-9

This allows us, for instance, to keep track of executables that would be named differently if built by Spack (e.g. ``x86_64-linux-gnu-gcc-9`` instead of just ``gcc``).

.. TODO: we need to gather some more experience on overriding "prefix"
   and other special keywords in extra attributes, but as soon as we are
   confident that this is the way to go we should document the process.
   See https://github.com/spack/spack/pull/16526#issuecomment-653783204

Filter matching executables
"""""""""""""""""""""""""""

Sometimes defining the appropriate regex for the ``executables`` attribute might prove to be difficult, especially if one has to deal with corner cases or exclude "red herrings".
To help keep the regular expressions as simple as possible, each package can optionally implement a ``filter_detected_exes`` method:

.. code-block:: python

    @classmethod
    def filter_detected_exes(cls, prefix, exes_in_prefix):
        """Return a filtered list of the executables in prefix"""

which takes as input a prefix and a list of matching executables and returns a filtered list of said executables.

Using this method has the advantage of allowing custom logic for filtering, and does not restrict the user to regular expressions only.
Consider the case of detecting the GNU C++ compiler.
If we try to search for executables that match ``g++``, that would have the unwanted side effect of selecting also ``clang++`` - which is a C++ compiler provided by another package - if present on the system.
Trying to select executables that contain ``g++`` but not ``clang`` would be quite complicated to do using only regular expressions.
Employing the ``filter_detected_exes`` method it becomes:

.. code-block:: python

   class Gcc(Package):
       executables = ["g++"]

       @classmethod
       def filter_detected_exes(cls, prefix, exes_in_prefix):
           return [x for x in exes_in_prefix if "clang" not in x]

Another possibility that this method opens is to apply certain filtering logic when specific conditions are met (e.g. take some decisions on an OS and not on another).

Validate detection
^^^^^^^^^^^^^^^^^^

To increase detection robustness, packagers may also implement a method to validate the detected Spec objects:

.. code-block:: python

   @classmethod
   def validate_detected_spec(cls, spec, extra_attributes):
       """Validate a detected spec. Raise an exception if validation fails."""

This method receives a detected spec along with its extra attributes and can be used to check that certain conditions are met by the spec.
Packagers can either use assertions or raise an ``InvalidSpecDetected`` exception when the check fails.
If the conditions are not honored the spec will be discarded and any message associated with the assertion or the exception will be logged as the reason for discarding it.

As an example, a package that wants to check that the ``compilers`` attribute is in the extra attributes can implement this method like this:

.. code-block:: python

   @classmethod
   def validate_detected_spec(cls, spec, extra_attributes):
       """Check that "compilers" is in the extra attributes."""
       msg = "the extra attribute 'compilers' must be set for the detected spec '{0}'".format(spec)
       assert "compilers" in extra_attributes, msg

or like this:

.. code-block:: python

   @classmethod
   def validate_detected_spec(cls, spec, extra_attributes):
       """Check that "compilers" is in the extra attributes."""
       if "compilers" not in extra_attributes:
           msg = "the extra attribute 'compilers' must be set for the detected spec '{0}'".format(
               spec
           )
           raise InvalidSpecDetected(msg)

.. _determine_spec_details:

Custom detection workflow
^^^^^^^^^^^^^^^^^^^^^^^^^

In the rare case when the mechanisms described so far don't fit the detection of a package, the implementation of all the methods above can be disregarded and instead a custom ``determine_spec_details`` method can be implemented directly in the package class (note that the definition of the ``executables`` attribute is still required):

.. code-block:: python

   @classmethod
   def determine_spec_details(cls, prefix, exes_in_prefix):
       # exes_in_prefix = a set of paths, each path is an executable
       # prefix = a prefix that is common to each path in exes_in_prefix

       # return None or [] if none of the exes represent an instance of
       # the package. Return one or more Specs for each instance of the
       # package which is thought to be installed in the provided prefix
       ...

This method takes as input a set of discovered executables (which match those specified by the user) as well as a common prefix shared by all of those executables.
The function must return one or more :py:class:`spack.package.Spec` associated with the executables (it can also return ``None`` to indicate that no provided executables are associated with the package).

As an example, consider a made-up package called ``foo-package`` which builds an executable called ``foo``.
``FooPackage`` would appear as follows:

.. code-block:: python

   class FooPackage(Package):
       homepage = "..."
       url = "..."

       version(...)

       # Each string provided here is treated as a regular expression, and
       # would match for example "foo", "foobar", and "bazfoo".
       executables = ["foo"]

       @classmethod
       def determine_spec_details(cls, prefix, exes_in_prefix):
           candidates = [x for x in exes_in_prefix if os.path.basename(x) == "foo"]
           if not candidates:
               return
           # This implementation is lazy and only checks the first candidate
           exe_path = candidates[0]
           exe = Executable(exe_path)
           output = exe("--version", output=str, error=str)
           version_str = ...  # parse output for version string
           return Spec.from_detection("foo-package@{0}".format(version_str))

Add detection tests to packages
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

To ensure that software is detected correctly for multiple configurations and on different systems users can write a ``detection_test.yaml`` file and put it in the package directory alongside the ``package.py`` file.
This YAML file contains enough information for Spack to mock an environment and try to check if the detection logic yields the results that are expected.

As a general rule, attributes at the top-level of ``detection_test.yaml`` represent search mechanisms and they each map to a list of tests that should confirm the validity of the package's detection logic.

The detection tests can be run with the following command:

.. code-block:: console

   $ spack audit externals

Errors that have been detected are reported to screen.

Tests for PATH inspections
""""""""""""""""""""""""""

Detection tests insisting on ``PATH`` inspections are listed under the ``paths`` attribute:

.. code-block:: yaml

   paths:
   - layout:
     - executables:
       - "bin/clang-3.9"
       - "bin/clang++-3.9"
       script: |
         echo "clang version 3.9.1-19ubuntu1 (tags/RELEASE_391/rc2)"
         echo "Target: x86_64-pc-linux-gnu"
         echo "Thread model: posix"
         echo "InstalledDir: /usr/bin"
     platforms: ["linux", "darwin"]
     results:
     - spec: "llvm@3.9.1 +clang~lld~lldb"

If the ``platforms`` attribute is present, tests are run only if the current host matches one of the listed platforms.
Each test is performed by first creating a temporary directory structure as specified in the corresponding ``layout`` and by then running package detection and checking that the outcome matches the expected ``results``.
The exact details on how to specify both the ``layout`` and the ``results`` are reported in the table below:

.. list-table:: Test based on PATH inspections
   :header-rows: 1

   * - Option Name
     - Description
     - Allowed Values
     - Required Field
   * - ``layout``
     - Specifies the filesystem tree used for the test
     - List of objects
     - Yes
   * - ``layout:[0]:executables``
     - Relative paths for the mock executables to be created
     - List of strings
     - Yes
   * - ``layout:[0]:script``
     - Mock logic for the executable
     - Any valid shell script
     - Yes
   * - ``results``
     - List of expected results
     - List of objects (empty if no result is expected)
     - Yes
   * - ``results:[0]:spec``
     - A spec that is expected from detection
     - Any valid spec
     - Yes
   * - ``results:[0]:extra_attributes``
     - Extra attributes expected on the associated Spec
     - Nested dictionary with string as keys, and regular expressions as leaf values
     - No

Reuse tests from other packages
"""""""""""""""""""""""""""""""

When using a custom repository, it is possible to customize a package that already exists in ``builtin`` and reuse its external tests.
To do so, just write a ``detection_test.yaml`` alongside the customized ``package.py`` with an ``includes`` attribute.
For instance the ``detection_test.yaml`` for ``myrepo.llvm`` might look like:

.. code-block:: yaml

   includes:
   - "builtin.llvm"

This YAML file instructs Spack to run the detection tests defined in ``builtin.llvm`` in addition to those locally defined in the file.

.. _abi_compatibility:

Specifying ABI Compatibility
----------------------------

.. warning::

   The ``can_splice`` directive is experimental, and may be replaced by a higher-level interface in future versions of Spack.

Packages can include ABI-compatibility information using the ``can_splice`` directive.
For example, if ``Foo`` version 1.1 can always replace version 1.0, then the package could have:

.. code-block:: python

   can_splice("foo@1.0", when="@1.1")

For virtual packages, packages can also specify ABI compatibility with other packages providing the same virtual.
For example, ``zlib-ng`` could specify:

.. code-block:: python

   can_splice("zlib@1.3.1", when="@2.2+compat")

Some packages have ABI-compatibility that is dependent on matching variant values, either for all variants or for some set of ABI-relevant variants.
In those cases, it is not necessary to specify the full combinatorial explosion.
The ``match_variants`` keyword can cover all single-value variants.

.. code-block:: python

   # any value for bar as long as they're the same
   can_splice("foo@1.1", when="@1.2", match_variants=["bar"])

   # any variant values if all single-value variants match
   can_splice("foo@1.2", when="@1.3", match_variants="*")

The concretizer will use ABI compatibility to determine automatic splices when :ref:`automatic splicing<automatic_splicing>` is enabled.

Customizing Views
-----------------

.. warning::

   This is advanced functionality documented for completeness, and rarely needs customization.

Spack environments manage a view of their packages, which is a single directory that merges all installed packages through symlinks, so users can easily access them.
The methods of ``PackageViewMixin`` can be overridden to customize how packages are added to views.
Sometimes it's impossible to get an application to work just through symlinking its executables, and patching is necessary.
For example, Python scripts in a ``bin`` directory may have a shebang that points to the Python interpreter in Python's install prefix and not to the Python interpreter in the view.
However, it's more convenient to have the shebang point to the Python interpreter in the view, since that interpreter can locate other Python packages in the view without ``PYTHONPATH`` being set.
Therefore, Python extension packages (those inheriting from ``PythonPackage``) override ``add_files_to_view`` in order to rewrite shebang lines.