The b2luigi CLI

b2luigi ships a dedicated command-line tool, b2luigi, that is installed automatically with the package. It provides a structured interface for running, inspecting, and managing your task graphs and replaces the legacy python file.py --flag invocation style.

Note

If you are currently using python file.py --batch or python file.py --dry-run, the equivalent new commands are b2luigi run and b2luigi run --dry. The legacy flags are still supported; see Run Modes.

Subcommands overview

Command	Description
b2luigi run	Run one or more tasks (local, batch, or test mode)
b2luigi tasks	List all task classes available in tasks.py and show their parameters
b2luigi show	Display output file status for the dependency tree
b2luigi graph	Render the task dependency graph as a Rich terminal tree or Graphviz DOT
b2luigi remove	Delete output files for one or more tasks
b2luigi test	Wrap an arbitrary script as a b2luigi task for debugging
b2luigi about	Print environment and installation information
b2luigi init	Scaffold a starter tasks.py and parameters.py in the current directory
b2luigi version	Print the installed version string
b2luigi self-update	Upgrade b2luigi via pip in the current environment

Project layout

The b2luigi CLI resolves your task definitions and parameter configurations from two files in the working directory:

my_project/
├── tasks.py          # task definitions (required)
└── parameters.py     # parameter config (optional)

parameters.py is optional. For a single run pass parameters directly via --param key=value; parameters.py is only needed when using ParameterGenerator for multi-value sweeps.

tasks.py is the entry point of task discovery, not a boundary. Two tiers of task classes are addressable by name:

• Manifest tasks — every b2luigi.Task subclass in tasks.py’s namespace, defined there or imported into it. Their bare class name always resolves, and takes precedence over any other class with the same name.

• Project tasks — every b2luigi.Task subclass that importing tasks.py loads from a module inside your project directory (the directory containing the task file). These are addressable by bare name when unique, and by their qualified module.ClassName (e.g. analysis.skim.SkimTask) always. If two project modules define the same class name, the bare name is rejected as ambiguous and the error lists the qualified candidates.

# tasks.py
from analysis.skim import SkimTask
from analysis.reco import RecoTask

Your task code stays where it lives; importing a class into tasks.py is what makes it addressable as a manifest task by b2luigi run, tasks, show, graph and remove. Task classes reachable only transitively (for example a task that SkimTask.requires() depends on, in a module never imported into tasks.py) are still project tasks, addressable by their qualified name. Classes belonging to b2luigi or luigi themselves (e.g. from b2luigi import Task) are never treated as runnable tasks.

Task classes from installed libraries (outside the project directory) are never addressable by name and never listed — they still appear in b2luigi graph and full-tree b2luigi show output when they are part of the dependency graph. b2luigi tasks shows each task’s module of origin.

Batch submission encodes each task’s module, so a project task that is a dependency of a submitted task executes correctly on the worker even if it was never imported into tasks.py.

You can override these paths with flags or environment variables:

CLI flag	Environment variable	Description
--task-file	B2LUIGI_TASK_FILE	Path to the task definitions file
--params-file	B2LUIGI_PARAMS_FILE	Path to the parameters file

b2luigi run

Run one or more tasks. With explicit parameter values:

b2luigi run MyTask --my-parameter 3

When a parameters.py file is present, you can omit all parameter flags — b2luigi run MyTask will read the configuration automatically:

b2luigi run MyTask

Using ParameterGenerator

ParameterGenerator lets you fan out over many parameter combinations directly inside parameters.py:

# parameters.py
from b2luigi import ParameterGenerator

config = {
    "my_parameter": ParameterGenerator([1, 2, 3]),
}

A single b2luigi run then schedules one task instance per value. ZippedParameterGenerator pairs values across multiple parameters positionally instead of taking their Cartesian product:

from b2luigi import ZippedParameterGenerator

config = ZippedParameterGenerator(
    my_parameter=[1, 2, 3],
    other_parameter=["a", "b", "c"],
)

See CLI Reference for the full class reference.

Precedence: --param overrides parameters.py

When a parameter is defined in both places, the command line wins. The config dict from parameters.py is loaded first and each --param value is merged on top of it, key by key. This applies identically to b2luigi run, show, remove and graph.

Only the keys you actually pass are affected — everything else in parameters.py is left untouched:

# parameters.py
config = {
    "my_parameter": 1,
    "other_parameter": "a",
}

b2luigi run MyTask --param my_parameter=2

runs with my_parameter=2 and other_parameter="a".

Values are parsed as JSON when possible, so --param n=5 yields the integer 5 and --param items=[1,2,3] a list (useful for a luigi.ListParameter). Anything that is not valid JSON is kept as a plain string.

Warning

Overriding a ParameterGenerator from the command line collapses the sweep to a single value. Given

config = {"my_parameter": ParameterGenerator([1, 2, 3])}

then b2luigi run MyTask schedules three task instances, but

b2luigi run MyTask --param my_parameter=2

schedules exactly one. This is the intended way to re-run a single point of a sweep — for example to reproduce one failed job — but it is easy to do by accident.

Note that this holds for any --param value, including a JSON list. Only ParameterGenerator and ZippedParameterGenerator objects expand into multiple task instances, and those can only be constructed in parameters.py. Passing --param my_parameter=[1,2,3] therefore schedules one task whose my_parameter is the list [1, 2, 3], not three tasks. To change the sweep itself, edit parameters.py.

Parameters that do not apply

parameters.py is shared across every task in a project, so it may hold keys that a given task does not declare. Those keys are filtered out rather than treated as errors, and the affected command prints a single warning naming them:

Warning: ignoring parameters not declared by TaskB: number

A --param override is different. It is aimed at one invocation, so an override that applies to no task is a mistake rather than a normal consequence of sharing a config:

b2luigi run TaskA --param numbr=99

Error: TaskA has no parameter 'numbr'. Did you mean 'number'?

run always names a task, so an inapplicable override is always an error there. show and graph error the same way when you name a task, and warn instead when you do not, because a whole-tree listing legitimately spans tasks that declare different parameters.

remove is the deliberate exception to that rule: it errors on an inapplicable override whether or not you name a task. It deletes files, and quietly ignoring an override that was meant to narrow what gets deleted is the dangerous direction.

The parameters.py warning above fires whenever a command errors on overrides — that is, for run, for remove, and for show/graph with a task named. With no task named, show and graph drop inapplicable parameters.py keys silently: a whole-tree listing has nothing specific to warn about, since it legitimately spans tasks with different parameters.

Warnings are written to standard error, so show --paths and graph --format dot stay pipeable.

b2luigi show

With no arguments, b2luigi show renders the full dependency tree with colour-coded output status for every task:

b2luigi show

To inspect the outputs of a specific task, pass its class name as a positional argument:

b2luigi show MyTask

Use --with-requirements to traverse the full requirement tree downward from the named task (showing all tasks it transitively depends on):

b2luigi show MyTask --with-requirements

Output is colour-coded: green means the file exists, red means it is missing.

Long output paths are never truncated: the Location column folds across lines so every character stays on screen and can be selected or piped.

For a bare listing suitable for shell substitution, use --paths:

b2luigi show MyTask --paths

This prints one absolute output path per line with no table, styling, or status column, so it composes with other tools:

ls -lh $(b2luigi show MyTask --paths)

--links additionally makes local paths clickable in terminals that support hyperlinks:

b2luigi show MyTask --links

Links are emitted only for local files, never for remote (XRootD or WebDAV) targets. Be aware that a link resolves against the machine your terminal runs on: when you run b2luigi show over SSH on a cluster, the link points at a path on your local machine and will not find the file. This is why --links is opt-in rather than the default.

b2luigi graph

Render the full task dependency graph in the terminal:

b2luigi graph

Scope the graph to a specific task and its requirements:

b2luigi graph MyTask

Add --params to show parameter values on each node, and --status to show a completion indicator (✓ / ✗) for every task:

b2luigi graph --params --status

Use --summary to replace the tree with per-class completion counts. On a parameter sweep the tree is hundreds of nodes of a handful of classes; the summary answers “how far along am I, and which stage is stuck” in a few lines:

$ b2luigi graph --summary
Task Graph Summary
 ────────────────────────────────────────────
  Task                 Complete   Status
 ────────────────────────────────────────────
  SummaryTask               0/1   incomplete
  SquareTask               7/10   incomplete
  GenerateNumberTask      10/10   complete
 ────────────────────────────────────────────
  total                   17/21   (80%)
 ────────────────────────────────────────────

Rows are printed in first-seen traversal order, roots first — here SummaryTask requires SquareTask, which requires GenerateNumberTask. The percentage is a floored integer (17 * 100 // 21 is 80, never rounded up).

Counts are task instances. A task counts as complete when all of its outputs exist — the same rule --status uses for its ✓ marker — and tasks declaring no outputs (wrapper tasks) are excluded entirely. A dependency shared by several parents is counted once.

--summary cannot be combined with --format dot (a summary is not a graph serialization) or with --params (parameter values are per-instance); either combination is an error. --status is accepted but redundant, since the summary always checks output existence.

Note

The summary checks every output of every task in the graph. Against remote targets that is one network round-trip per output, so a large sweep on remote storage will take a while.

Export to Graphviz DOT format and pipe it to dot to produce an image:

b2luigi graph --format dot | dot -Tpng -o graph.png
b2luigi graph --format dot > graph.dot   # save first, render later

Note

DOT export requires Graphviz to be installed (brew install graphviz on macOS, apt install graphviz on Debian/Ubuntu).

b2luigi remove

Remove the output files of one or more named tasks:

b2luigi remove MyTask

Add -y to skip the confirmation prompt:

b2luigi remove MyTask -y

Use --with-requirements to also remove all tasks that MyTask transitively depends on:

b2luigi remove MyTask --with-requirements -y

b2luigi test

Wrap an arbitrary script as a b2luigi task for local debugging:

b2luigi test -s my_script.py -o output_file.txt

The -s flag specifies the script to run; -o names the output file the task is expected to produce.

Any extra arguments are forwarded verbatim to the script, and the generated command is <interpreter> <script> -o <output> [-i <input>] [extra args].

Note

The extra arguments are appended after -o/-i. Earlier versions placed them before, so a script that reads its arguments positionally out of sys.argv rather than through argparse may need adjusting. This applies whether or not --executable is used.

Choosing a batch system

--batch alone detects a batch system by probing PATH in a fixed order (bsub → condor_submit → sbatch, else local). Use --batch-system to choose explicitly instead:

b2luigi test -s my_script.py -o output_file.txt --batch-system slurm

--batch-system implies --batch. It is needed on a machine with more than one scheduler installed, and to reach gbasf2, which the PATH probe can never detect. An unknown name is rejected with a did-you-mean rather than ignored.

The flag is sugar over an ordinary setting: it is equivalent to --setting batch_system=<name> or a batch_system entry in settings.json.

Note

Earlier versions ignored batch_system from settings.json and --setting for b2luigi test, because FastTask pinned the value as a class attribute, which outranks both in the settings cascade. They now take effect. If you have a settings.json naming one scheduler while relying on auto-detection to pick a different one, the setting now wins.

Relative -s/-o/-i paths are resolved against the directory you invoke b2luigi test from, and may carry directory components (-i data/input.txt). Under --batch they are resolved on the submission host before being sent to the worker, so the job reads and writes the same files regardless of the working_dir its wrapper changes into. This is deliberate, and differs from b2luigi run, whose task file is forwarded as given so that a relative one resolves against a relocated copy of the project. b2luigi test runs a single script rather than a project, so it has nothing to relocate.

By default the script runs under the current Python interpreter. Use --executable to run it under something else — most importantly basf2, whose -o/-i options are provided by the basf2 wrapper binary rather than by the steering file, and therefore only take effect when the script is invoked as basf2 steer.py:

b2luigi test -s steering_file.py -o output_file.root --executable basf2

The value is split on shell rules, so multi-token commands work: --executable "apptainer exec image.sif basf2". Whenever --executable is given — even to name a plain Python interpreter — -- is inserted before any extra arguments so that the script’s own arguments are passed through to it rather than consumed by the wrapper:

basf2 steering_file.py -o output_file.root -- --my-script-flag

Note

--executable is not the executable setting. The flag chooses what runs your script; the setting chooses what launches the b2luigi batch worker on the cluster. Both can appear in one --batch run.

When submitting to a real batch system via --batch, use --env-script to source an environment setup script before the job runs (only takes effect combined with --batch; a no-op otherwise):

b2luigi test -s my_script.py -o output_file.txt --batch --env-script setup.sh

Use --setting key=value (repeatable, JSON-aware) to override any other b2luigi setting for this run without creating a settings.json, e.g. --setting apptainer_image=my_image.sif. An apptainer/container run always needs an environment setup script, so apptainer_image must be combined with --env-script; setting it alone raises ValueError: Apptainer execution requires an environment setup script..

Unlike settings.json, which is re-read fresh by the batch worker, --setting overrides live only in the submitting process’s in-memory settings and are never propagated to the batch worker (the worker reconstructs the task via batch-runner --script with no knowledge of --setting values). Use --setting for submission-side-only settings such as apptainer_image, env, env_script, or working_dir. For settings that both the submission host and the worker must agree on (result_dir, log_dir), use settings.json instead — otherwise the submission side and the worker will resolve output paths differently under --batch.

--setting cannot override batch_system or env_script: both are already set as class attributes on the generated task (via --batch and --env-script respectively), and get_setting() checks task attributes before global settings. Use --batch/--env-script for those two.

Utility commands

b2luigi about

Print version, Python, platform, working directory, and current environment variable overrides.

b2luigi init

Scaffold a minimal tasks.py and parameters.py in the current working directory. Use --force to overwrite existing files.

b2luigi version

Print the installed version string and exit.

b2luigi self-update

Upgrade b2luigi to the latest available version via pip.

Shell auto-completion

b2luigi ships with shell auto-completion for subcommand names and flag names. To install it for your current shell:

b2luigi --install-completion

After sourcing your shell’s rc file (~/.bashrc, ~/.zshrc, etc.), pressing Tab after b2luigi will complete subcommand names and flag names automatically.

To print the completion script without installing it:

b2luigi --show-completion

Supported shells: bash, zsh, fish, and PowerShell.

Migration from the legacy interface

If you are currently calling python file.py --batch or similar legacy flags, the equivalent b2luigi commands are:

Legacy command	New CLI equivalent
python tasks.py	b2luigi run <CLASSNAME>
python tasks.py --batch	b2luigi run <CLASSNAME> --batch
python tasks.py --dry-run	b2luigi run <CLASSNAME> --dry
python tasks.py --show-output	b2luigi show
python tasks.py --remove	b2luigi remove
python tasks.py --test	b2luigi test

The legacy flags are still supported when calling b2luigi.process() directly in a script. See Run Modes for the full reference.

For a full worked walkthrough of moving an existing project over — what carries over unchanged, how to adopt parameters.py and ParameterGenerator incrementally, and how the two interfaces coexist during a gradual migration — see Migrating to the New CLI.