Run Modes

Run Modes#

b2luigi.cli.runner.run_as_batch_worker(task_list, cli_args, kwargs)[source]#

Executes a specific task from a list of tasks as a batch worker.

This function iterates through a list of root tasks and their dependencies to find and execute the task specified by the cli_args.task_id. If the task is found, it sets up the environment, runs the task, and handles success or failure events. If the task is not found, an error is raised.

Parameters:

  • task_list (list) – A list of tasks to search for the specified task.

  • cli_args (Namespace) – Command-line arguments containing the task_id of the task to execute.

  • kwargs (dict) – Additional keyword arguments (currently unused).

Raises:

  • ValueError – If the specified task_id does not exist in the task graph.

  • BaseException – If the task execution fails, the exception is raised after invoking the task’s failure handler.

Notes

  • The function sets the _dispatch_local_execution setting to True before running the task. See dispatch.

  • The task’s output directories are created before execution. See create_output_dirs.

b2luigi.cli.runner.run_batched(task_list, cli_args, kwargs)[source]#

Executes a batch of Luigi tasks with the provided command-line arguments and keyword arguments.

Parameters:

  • task_list (list) – A list of task instances to be executed.

  • cli_args (list) – A list of command-line arguments to be passed to run_luigi.

  • kwargs (dict) – A dictionary of additional keyword arguments to pass to the Luigi runner.

b2luigi.cli.runner.run_local(task_list, cli_args, kwargs)[source]#

Executes a list of Luigi tasks locally by setting the batch system to local.

Parameters:

  • task_list (list) – A list of Luigi task instances to be executed.

  • cli_args (list) – Command-line arguments to be passed to run_luigi.

  • kwargs (dict) – Additional keyword arguments for task execution.

b2luigi.cli.runner.run_luigi(task_list, cli_args, kwargs)[source]#

Executes Luigi tasks with the specified configuration.

This function sets up the luigi scheduler and worker configurations based on the provided command-line arguments and keyword arguments, then runs the specified list of tasks.

Parameters:

  • task_list (list) – A list of task instances to be executed.

  • cli_args (Namespace) – Parsed command-line arguments containing scheduler host and port information.

  • kwargs (dict) – Additional keyword arguments to configure luigi.build.

Behavior:

  • If scheduler_host or scheduler_port is provided in cli_args, they are used to configure the scheduler. Otherwise, a local scheduler is used.

  • A custom worker-scheduler factory (SendJobWorkerSchedulerFactory) is set in the configuration.

b2luigi.cli.runner.run_test_mode(task_list, cli_args, kwargs)[source]#

Executes the given tasks in test mode with local execution enabled.

This function sets _dispatch_local_execution (see dispatch) to enable local execution and then builds the provided list of tasks using the local scheduler.

Parameters:

  • task_list (list) – A list of task instances to be executed.

  • cli_args (list) – Command-line arguments passed to the CLI (not used in this function).

  • kwargs (dict) – Additional keyword arguments to be passed to luigi.build.

b2luigi.cli.runner.show_all_outputs(task_list, *args, **kwargs)[source]#

Displays all output files for a list of tasks, grouped by their respective keys.

This function iterates through a list of tasks, collects all output files from their dependency trees, and prints them grouped by their keys. The existence of each file is visually indicated using colored output (green for existing files, red for non-existing files).

Parameters:

task_list (list) – A list of tasks to process.

Notes

  • The function uses get_all_output_files_in_tree to retrieve output files for each task.

  • The existence of files is determined but the task status is not checked.

b2luigi.cli.runner.dry_run(task_list)[source]#

Perform a dry run of the given tasks, simulating their execution without actually running them. This function iterates through the provided task list, identifies tasks that are not yet complete, and executes their dry_run method.

Parameters:

task_list (list) – A list of tasks to be processed. Each task is expected to be iterable and may contain subtasks.

b2luigi.cli.runner.remove_outputs(task_list, target_tasks, only=False, auto_confirm=False)[source]#

Removes the outputs of specified tasks and their dependent tasks.

Parameters:

  • task_list (list) – A list of tasks to iterate over.

  • target_tasks (list) – A list of target task class names whose outputs should be removed.

  • only (bool, optional) – If True, removes only the outputs of the specified target tasks. If False, removes the outputs of the target tasks and all their dependent tasks. Defaults to False.

  • auto_confirm (bool, optional) – If True, skips the confirmation prompt and proceeds with removal. Defaults to False.

Notes

  • The find_dependents function is used to identify dependent tasks for a given target task. In large task graphs, this can be time-consuming.

Contents