Target

Target#

class b2luigi.core.target.FileSystemTarget(*args, **kwargs)[source]#

Bases: FileSystemTarget

A local file system target that extends luigi.LocalTarget with temporary file handling.

This target provides additional functionality for handling temporary files during task execution, including support for temporary input and output paths with proper cleanup and atomic file operations.

scratch_dir#

Directory path where temporary files will be created

Type:

str

Parameters:

  • _scratch_dir (str) – Directory path where temporary files will be created

  • *args – Additional positional arguments passed to LocalTarget

  • **kwargs – Additional keyword arguments passed to LocalTarget

property tmp_name: str#

Generate a temporary filename based on the target’s path.

Creates a temporary filename by adding a random suffix to the original filename while preserving the file extension.

Returns:

A temporary filename in the format:

"{original_name}-luigi-tmp-{random_number}.{extension}"

Return type:

str

Warning

This method assumes the file has an extension. Files without extensions will raise an :obj:IndexError. This limitation should be addressed in future versions.

Example

If self.path is “data.txt”, might return “data-luigi-tmp-1234567890.txt”

get_temporary_input(task: Task | None = None, **tmp_file_kwargs) → Generator[str, None, None][source]#

Create a temporary copy of the input file for processing.

Creates a temporary copy of the input file in the scratch directory, allowing safe concurrent access to the same input file by multiple tasks.

Parameters:

**tmp_file_kwargs – Keyword arguments passed to tempfile.TemporaryDirectory

Yields:

str – Absolute path to the temporary copy of the input file

Note

The temporary file and its parent directory are automatically cleaned up when exiting the context manager, regardless of whether an exception occurred.

Example

with target.get_temporary_input() as tmp_input:
    process_file(tmp_input)
temporary_path(task: Task | None = None, **tmp_file_kwargs) → Generator[str, None, None][source]#

Create a temporary file for output that will be moved to the final location.

Implements atomic write operations by first writing to a temporary location and then moving the file to its final destination only after successful completion.

Parameters:

**tmp_file_kwargs – Keyword arguments passed to tempfile.TemporaryDirectory()

Yields:

str – Absolute path to the temporary file for writing

Note

  • The temporary directory and its contents are automatically cleaned up after the file is copied to its final location

  • The final copy operation is atomic, preventing partial writes from corrupting the output file

  • Parent directories of the target path will be created if they don’t exist

Example

with target.temporary_path() as tmp_output:
    with open(tmp_output, 'w') as f:
        f.write('result data')
# File is automatically moved to target location after context exit
class b2luigi.core.target.LocalTarget(*args, **kwargs)[source]#

Bases: FileSystemTarget, LocalTarget

A local file system target that extends luigi.LocalTarget with temporary file handling.

This target provides additional functionality for handling temporary files during task execution, including support for temporary input and output paths with proper cleanup and atomic file operations.

scratch_dir#

Directory path where temporary files will be created

Type:

str

Parameters:

  • _scratch_dir (str) – Directory path where temporary files will be created

  • *args – Additional positional arguments passed to luigi.LocalTarget

  • **kwargs – Additional keyword arguments passed to luigi.LocalTarget

Contents