Run Modes

b2luigi.cli.runner.stderr_console = <console width=80 None>

Diagnostics that must not pollute machine-readable stdout. show --paths is designed to be piped and graph --format dot to be fed to Graphviz, so anything printed alongside them belongs on stderr.

b2luigi.cli.runner.test_task(exec_script: str, output: str, input_file: str | None, force: bool, batch: bool, extra_args: list[str], env_script: str | None = None, settings: list[str] | None = None, literal_path: bool = True, executable: str | None = None, batch_system: str | None = None) → None

Run a one-off b2luigi task that executes exec_script as a subprocess.

Builds FastTask (and optionally a prerequisite FastReqTask) via _build_fast_task() and _build_fast_req_task(), then runs them via run_luigi(). Arms the __batch_runner_use_cli setting so that, if batch=True and the task is submitted to a real cluster, create_cmd_from_task() emits the new batch-runner --script reconstruction command instead of the legacy argparse invocation. Exits non-zero when any task fails so that the CLI binary propagates the failure to the shell.

Parameters:

• exec_script (str) – Path to the Python script to run.

• output (str) – Output filename for the task target.

• input_file (str | None) – Optional input filename; if set, a prerequisite task is created so Luigi waits for the input before running the main task.

• force (bool) – When True, the task always runs regardless of whether the output already exists.

• batch (bool) – When True, submit via batch system (batch_system="auto").

• extra_args (list[str]) – Extra CLI arguments forwarded verbatim to the subprocess.

• env_script (str | None) – Optional path to an environment setup script, forwarded to _build_fast_task(). Only takes effect combined with batch=True; see _build_fast_task() for details.

• settings (list[str] | None) – Optional list of "key=value" strings (JSON-aware, parsed via parse_kv_params()), applied via set_setting() before the task is built and run. Unlike settings.json (re-read fresh by the batch worker), these overrides live only in the submitting process’s in-memory settings and are never forwarded to the batch worker — the worker reconstructs FastTask via batch-runner --script with no knowledge of --setting values. Safe for submission-side-only settings (apptainer_image, env, env_script, working_dir); for settings both sides must agree on (result_dir, log_dir), use settings.json instead. batch_system can be set this way (equivalently to --batch-system); env_script cannot, since _build_fast_task() sets it as a class attribute on FastTask, which get_setting() checks before global settings.

• literal_path (bool) – Forwarded to _build_fast_task(). See there for details.

• executable (str | None) – Forwarded to _build_fast_task(). See there for details.

• batch_system (str | None) – Optional batch system to submit to, validated by _resolve_batch_system(). Implies batch=True and is applied as an ordinary setting, so it is exactly equivalent to --setting batch_system=... or a settings.json entry — discoverable sugar over the one mechanism, not a second one. Needed to reach a system PATH probing cannot detect (gbasf2), and to choose between several that are installed.

Raises:

• SystemExit – With exit code 1 when any task in the build fails.

• CliUserError – If batch_system names no known batch system.

b2luigi.cli.runner.run_batch_worker(task)

Executes a single task directly as a batch worker.

This is the new-CLI counterpart to run_as_batch_worker. Instead of searching a task graph by ID, the caller is responsible for passing the already-reconstructed task instance. This avoids the need for cli_args and a full dependency-tree traversal.

Parameters:

task – The task instance to execute.

Raises:

BaseException – If execution fails, the exception is re-raised after calling the task’s failure handler.

b2luigi.cli.runner.run_as_batch_worker(task_list, cli_args)

Executes a specific task from a list of tasks as a batch worker.

This function iterates through a list of root tasks and their dependencies to find and execute the task specified by the cli_args.task_id. If the task is found, it sets up the environment, runs the task, and handles success or failure events. If the task is not found, an error is raised.

Parameters:

• task_list (list) – A list of tasks to search for the specified task.

• cli_args – Command-line arguments containing the task_id of the task to execute.

Raises:

• ValueError – If the specified task_id does not exist in the task graph.

• BaseException – If the task execution fails, the exception is raised after invoking the task’s failure handler.

b2luigi.cli.runner.run_batched(task_list, kwargs)

Executes a batch of Luigi tasks with the provided keyword arguments.

Parameters:

• task_list (list) – A list of task instances to be executed.

• kwargs (dict) – A dictionary of additional keyword arguments to pass to the Luigi runner.

b2luigi.cli.runner.run_local(task_list, kwargs)

Executes a list of Luigi tasks locally by setting the batch system to local.

Parameters:

• task_list (list) – A list of Luigi task instances to be executed.

• kwargs (dict) – Additional keyword arguments for task execution.

b2luigi.cli.runner.run_with_tui(task_list: list, kwargs: dict, batch=False)

Run tasks with a live Textual progress TUI.

Requires the ‘tui’ optional dependency: pip install b2luigi[tui]

b2luigi.cli.runner.run_luigi(task_list: list, kwargs: dict, worker_scheduler_factory=None)

Executes Luigi tasks with the specified configuration.

This function sets up the luigi scheduler and worker configurations based on the provided keyword arguments, then runs the specified list of tasks.

Parameters:

• task_list (list) – A list of task instances to be executed.

• kwargs (dict) – Additional keyword arguments to configure luigi.build. Supported keys: scheduler_host, scheduler_port, workers, and any argument accepted by luigi.build().

Returns:

True if all tasks completed successfully, False otherwise.

Return type:

bool

b2luigi.cli.runner.run_test_mode(task_list, kwargs)

Executes the given tasks in test mode with local execution enabled.

This function sets _dispatch_local_execution (see dispatch) to enable local execution and then builds the provided list of tasks using the local scheduler.

Parameters:

• task_list (list) – A list of task instances to be executed.

• kwargs (dict) – Additional keyword arguments to be passed to luigi.build.

b2luigi.cli.runner.get_task_outputs(task)

Return the same dict structure as get_all_output_files_in_tree but for a single task without traversing its dependency tree.

Each output-entry dict carries an is_local key alongside exists, parameters and file_name, recording whether the underlying target is a luigi.LocalTarget.

Parameters:

task – The task instance whose outputs should be collected.

Returns:

Mapping of output key to list of output-entry dicts.

Return type:

collections.defaultdict

b2luigi.cli.runner.render_graph_tree(task_list: list, show_params: bool = False, show_status: bool = False) → None

Render the task dependency graph as a Rich terminal tree.

Each task is a node in the tree; edges follow task.requires(). Shared nodes (a task required by multiple parents) are shown in full under their first occurrence and as a reference marker (↳ ClassName (already shown above)) under subsequent parents.

Parameters:

• task_list (list) – Root task instances to render.

• show_params (bool) – If True, include parameter values on each node label.

• show_status (bool) – If True, check output existence and append a completion indicator (✓ / ✗) to each node label.

b2luigi.cli.runner.render_graph_dot(task_list: list, show_params: bool = False, show_status: bool = False) → None

Render the task dependency graph as Graphviz DOT output to stdout.

Emits valid DOT syntax via print() (not the Rich console) so the output can be piped directly to dot -Tpng -o graph.png. Each task instance is a node keyed by its task_id; edges follow task.requires(). Shared nodes (required by multiple parents) are represented as a single DOT node with multiple incoming edges.

Parameters:

• task_list (list) – Root task instances to render.

• show_params (bool) – If True, include parameter values in each node’s label.

• show_status (bool) – If True, check output existence and colour each node green (complete) or red (incomplete).

b2luigi.cli.runner.render_graph_summary(task_list: list) → None

Render per-class completion counts for the task graph.

Walks the same graph render_graph_tree() walks and prints one row per task class — <name>  <complete>/<total>  <verdict> — followed by a rule and a total row carrying the floored integer percentage. Counts are task instances, and a class’s verdict is complete only when every one of its instances is complete.

A task counts as complete when every one of its outputs exists, identical to the rule render_graph_tree() uses for its ✓ marker. Tasks declaring no outputs are excluded entirely, matching the tree, which gives them no marker.

Instances are deduped on (type(task), task.task_id) rather than task_id alone. luigi derives task_id from the task family — the class name — plus a parameter hash, with no module component, so two same-named classes from different modules with equal parameters share a task_id. The tuple key protects the counts against this collision across separate roots in task_list, where each root gets its own call to task_iterator(). It cannot protect a collision within a single root’s tree: task_iterator() dedups its own walk on bare task_id (see b2luigi/core/utils.py’s already_seen_tasks), so if two same-named colliding classes both appear as dependencies of one root, the second is dropped before this function ever sees it. That is a pre-existing limitation of task_iterator() shared by every consumer of this traversal, not something this renderer can fix locally.

Rows display cls.__name__, falling back to module.Class for every class sharing a __name__ with another class in the same graph.

Parameters:

task_list (list) – Root task instances to summarise.

b2luigi.cli.runner.show_task_outputs(task_list: list, details: bool = False, paths_only: bool = False, links: bool = False) → None

Show output files for the given tasks only — no dependency-tree traversal.

Parameters:

• task_list (list) – Task instances whose outputs should be displayed.

• details (bool) – If True, show the Output key-name column (and, for multi-instance classes, the Params column). Hidden by default.

• paths_only (bool) – If True, print one bare output path per line instead of rendering a table. Suitable for piping.

• links (bool) – If True, wrap local output paths in clickable terminal hyperlinks. Ignored under paths_only.

b2luigi.cli.runner.show_all_outputs(task_list: list, show_required_by: bool = False, details: bool = False, paths_only: bool = False, links: bool = False) → None

Show output files for all tasks in the dependency trees rooted at task_list.

Parameters:

• task_list (list) – Root task instances; the full dependency tree is traversed.

• show_required_by (bool) – If True, annotate each requirement panel with a required by: subtitle listing immediate parent class names.

• details (bool) – If True, show the Output key-name column (and, for multi-instance classes, the Params column). Hidden by default.

• paths_only (bool) – If True, print one bare output path per line instead of rendering a table. Suitable for piping.

• links (bool) – If True, wrap local output paths in clickable terminal hyperlinks. Ignored under paths_only.

b2luigi.cli.runner.dry_run(task_list)

Perform a dry run of the given tasks, simulating their execution without actually running them. This function iterates through the provided task list, identifies tasks that are not yet complete, and executes their dry_run method.

Parameters:

task_list (list) – A list of tasks to be processed. Each task is expected to be iterable and may contain subtasks.

b2luigi.cli.runner.remove_outputs(task_list: list, target_tasks: list[str], auto_confirm: bool = False, keep_tasks: list[str] | None = None) → None

Remove the outputs of the specified tasks (new b2luigi remove CLI).

Removes only the explicitly named tasks — no dependents cascade. For downward removal of a task’s requirements, see remove_requirement_outputs() (--with-requirements). For the legacy dependents-cascade behaviour of python tasks.py --remove, see legacy_remove_outputs().

Parameters:

• task_list (list) – A list of root tasks to traverse.

• target_tasks (list[str]) – Task class names whose outputs should be removed.

• auto_confirm (bool) – If True, skip confirmation prompt.

• keep_tasks (list[str] | None) – List of task class names to KEEP outputs for.

Raises:

SystemExit – Always raises SystemExit(0) on completion.

b2luigi.cli.runner.legacy_remove_outputs(task_list: list, target_tasks: list[str], only: bool = False, auto_confirm: bool = False, keep_tasks: list[str] | None = None) → None

Remove the outputs of specified tasks (legacy python tasks.py --remove path).

Backs the legacy --remove/--remove-only CLI flags parsed by get_cli_arguments(), and the remove=/ remove_only= kwargs to b2luigi.process(), for users who call process() directly from their own scripts instead of using the new b2luigi remove CLI (see remove_outputs()). Kept only for backward compatibility — a candidate for removal in a future major version.

Unlike remove_outputs(), only has real effect here: when False (the default, matching --remove), removal cascades to every task that transitively depends on (requires, directly or indirectly) each named target task. When True (--remove-only), only the named task(s) are removed.

Parameters:

• task_list (list) – A list of root tasks to traverse.

• target_tasks (list[str]) – Task class names whose outputs should be removed.

• only (bool) – If True, remove only the named tasks. If False, also remove outputs of every task that depends on them.

• auto_confirm (bool) – If True, skip confirmation prompt.

• keep_tasks (list[str] | None) – List of task class names to KEEP outputs for.

Raises:

SystemExit – Always raises SystemExit(0) on completion.

b2luigi.cli.runner.remove_requirement_outputs(task_list: list, auto_confirm: bool = False, keep_tasks: list[str] | None = None) → None

Remove outputs for the given tasks and all tasks they transitively require.

Traverses the full dependency tree downward from each task in task_list via task_iterator() and removes outputs for every discovered task.

Parameters:

• task_list (list) – Task instances to start traversal from.

• auto_confirm (bool) – If True, skip confirmation prompt.

• keep_tasks (list[str] | None) – List of task class names to KEEP outputs for.

Returns:

None

Return type:

None

b2luigi.cli.runner.render_task_list(entries: list[tuple[type, str]]) → None

Render a Rich table listing all available task classes.

Wraps the table in a b2luigi-branded panel and prints it to stdout.

Parameters:

entries (list[tuple[type, str]]) – (task class, module label) pairs to display.

b2luigi.cli.runner.render_task_help(cls) → None

Render a Rich panel with the docstring and parameter table for a single task class.

Parameters:

cls – The task class whose documentation should be displayed.