Apptainer Process

Apptainer Process#

class b2luigi.batch.processes.apptainer.ApptainerProcess(*args, **kwargs)[source]#

Bases: BatchProcess

Simple implementation of a batch process for running jobs in an Apptainer container. Strictly speaking, this is not a batch process, but it is a simple way to run jobs in a container environment.

This process inherits the basic properties from b2luigi.batch.processes.BatchProcess but does not need to be executed in the batch context. However, running in batch mode is possible for the lsf and the htcondor batch systems. Although, for the latter batch system it is not recommended to use apptainer images since HTCondor is already running in a container environment.

The core principle of this process is to run the task in an Apptainer container. To achieve the execution of tasks, an apptainer exec command is build within this class and executed in a subprocess. To steer the execution, one can use the following settings:

  • apptainer_image: The image to use for the Apptainer container.s

    This parameter is mandatory and needs to be set if the task should be executed in an Apptainer container. The image needs to be accessible from the machine where the task is executed. There are no further checks if the image is available or valid. When using custom images, it may be helpful to first check the image with apptainer inspect. For people with access to the Belle II own /cvmfs directory, images are provided in the /cvmfs/belle.cern.ch/images directory. The description of the images (the repository contains the docker images which are transformed to Apptainer images) and instructions on how to create them can be found in https://gitlab.desy.de/belle2/software/docker-images.

  • apptainer_mounts: A list of directories to mount into the Apptainer container.

    This parameter is optional and can be used to mount directories into the Apptainer container. The directories need to be accessible from the machine where the task is executed. The directories are mounted under the exact same path as they are provided/on the host machine. For most usecases mounts need to be provided to access software or data locations. For people using for example basf2 software in the Apptainer container, the /cvmfs directory needs to be mounted. Caution is required when system specific directories are mounted.

  • apptainer_mount_defaults: Boolean parameter to mount log_dir and result_dir by default.

    The default value is True meaning the result_dir and log_dir are automatically created and mounted if they are not accessible from the execution location. When using custom targets with non local output directories, this parameter should be set to False to avoid mounting non-existing directories.

  • apptainer_additional_params: Additional parameters to pass to the apptainer exec command.

    This parameter should be a string and will be directly appended to the apptainer exec command. It can be used to pass additional parameters to the apptainer exec command as they would be added in the CLI. A very useful parameter is the --cleanenv parameter which will clean the environment before executing the task in the Apptainer container. This can be useful to avoid conflicts with the environment in the container. A prominent usecase is the usage of software which depends on the operating system.

A simple example of how an Apptainer based task can be defined is shown below:

class MyApptainerTask(luigi.Task):
    apptainer_image = "/cvmfs/belle.cern.ch/images/belle2-base-el9"
    apptainer_mounts = ["/cvmfs"]
    apptainer_mount_defaults = True
    apptainer_additional_params = "--cleanenv"

    <rest of the task definition>
get_job_status()[source]#

Determine the current status of the job associated with this process.

Returns:

The current status of the job. Possible values are:

  • JobStatus.aborted: If the process is not initialized or has a non-zero return code.

  • JobStatus.running: If the process is still running.

  • JobStatus.running: If the process has finished successfully (return code is 0).

Return type:

JobStatus

start_job()[source]#

Starts a job by constructing and executing an Apptainer command.

This method generates the necessary command to execute a task using Apptainer, starts the job as a subprocess, and captures the process information for further management.

The command is constructed by combining the task-specific command (see create_cmd_from_task()) and the Apptainer execution wrapper (see create_apptainer_command).

self.task#

The task containing the details required to construct the command.

terminate_job()[source]#

Terminates the currently running process if it exists.

This method checks if a process is associated with the instance and terminates it by calling the terminate method on the process.

_write_output()[source]#

Writes the captured standard output and standard error of the task to separate log files in the task’s log directory (see get_log_file_dir).

The method creates two files, “stdout” and “stderr”, in the log directory obtained from the task. If the standard output or standard error is not None, their contents are decoded and written to the respective files.

Contents