Remote Systems and Targets

Remote Systems and Targets#

See Remote Targets for an educational explanation.

class b2luigi.core.remote_target.RemoteFileSystem(server_path: str)[source]#

Bases: FileSystem

Base remote file system for b2luigi Targets. Inspiration taken from RHofsaess. It implements some standard file system operations, which can be used by the RemoteTarget.

exists(path: str) → bool[source]#

Implementation of the exists function for the RemoteFileSystem. Will return True if the file or directory exists and False if it can not be found. This might also include cases, where the server is not reachable.

Parameters:

path (str) – The path to check for existence.

Returns:

True if the path exists, False otherwise..

Return type:

bool

copy_file_to_remote(local_path: str, remote_path: str, force: bool = False) → None[source]#

Function to copy a file from the local file system to the remote file system.

Parameters:

  • local_path (str) – The path to the file on the local file system.

  • remote_path (str) – The destination path for the file on the remote file system.

  • force (bool, optional) – If True, overwrites the file on the remote system if it already exists. Defaults to False.

copy_file_from_remote(remote_path: str, local_path: str, force: bool = False) → None[source]#

Function to copy a file from the remote file system to the local file system.

Parameters:

  • remote_path – Path to the file on the remote file system.

  • local_path – Path to the file on the local file system.

  • force – If True, the file will be overwritten if it already exists. Default is False.

copy_dir_from_remote(remote_path: str, local_path: str, force: bool = False) → None[source]#

Copies a directory and its files from the remote file system to the local file system.

Parameters:

  • remote_path (str) – Path to the directory on the remote file system.

  • local_path (str) – Path to the directory on the local file system.

  • force (bool, optional) – If True, overwrites files on the local system if they already exist. Defaults to False.

move(source_path: str, dest_path: str) → None[source]#

A function to move a file from one location to another on the XRootD server.

Parameters:

  • source_path (str) – The current path of the file on the remote file system.

  • dest_path (str) – The target path for the file on the remote file system.

mkdir(path: str, parents: bool = True, raise_if_exists: bool = False) → None[source]#

A function to create a directory on the remote file system.

Parameters:

path (str) – The path of the directory to create.

locate(path: str) → bool[source]#

Checks the location of a file on the remote file system.

This method queries the remote file system to locate the specified file.

Parameters:

path (str) – The path to the file on the remote file system.

Returns:

True if the file location is successfully determined.

Return type:

bool

remove(path: str, recursive: bool = True, skip_trash: bool = True) → None[source]#

This method deletes a single file and does not support directory removal.

Parameters:

path (str) – The path to the file on the remote file system.

listdir(path: str, print_entries: bool = False) → Tuple[Dict[str, int], Any] | List[str][source]#

Lists the contents of a directory on the remote file system.

This method retrieves the directory listing and categorizes entries as files or directories.

Parameters:

path (str) – Path to the directory on the remote file system.

remove_dir(path: str) → None[source]#

A function to iteratively remove a directory and all its content from the remote file system.

Parameters:

path (str) – Path to the directory on the remote file system.

rename_dont_move(path: str, dest: str) → None[source]#

Overwriting a the luigi function used to handle the atomic write problem. (See spotify/luigi) In this case it is just an alias for copy_file_to_remote with force=True.

Parameters:

  • local_path – Path to the file on the local file system.

  • remote_path – Path to the file on the remote file system.

class b2luigi.RemoteTarget(path: str, file_system: RemoteFileSystem)[source]#

Bases: FileSystemTarget

Luigi target implementation for the remote file system.

property base_name: str#

Get the base name of the target path.

property fs: RemoteFileSystem#

Get the associated file system.

makedirs() → None[source]#

Create the target’s directory on the remote file system.

get(path: str = '~') → str[source]#

A function to copy the file from the remote file system to the local file system.

Parameters:

path – Path to copy the file to.

Returns:

Path to the copied file.

open(mode: str) → None[source]#

Raise NotImplementedError as open is not supported.

get_temporary_input(task: Task | None = None, **tmp_file_kwargs) → Generator[str, None, None][source]#

Create a temporary local copy of a remote input file.

Downloads the remote file to a temporary local directory for processing, allowing safe concurrent access to the same remote input file by multiple tasks.

Parameters:

task (Optional[Task]) – Task instance used to determine scratch directory settings. If None, uses default settings.

Yields:

str – Absolute path to the temporary local copy of the remote input file.

Note

  • The temporary file and its parent directory are automatically cleaned up when exiting the context manager.

  • Files are downloaded to the scratch directory specified in settings (defaults to the path returned by tempfile.gettempdir() if not set).

Example

target = RemoteTarget("remote://server/path/data.root", fs)
with target.get_temporary_input() as tmp_input:
    process_local_file(tmp_input)
# Temporary file is automatically cleaned up.
class b2luigi.XRootDSystem(server_path: str)[source]#

Bases: RemoteFileSystem

XRootD file system for b2luigi Targets. Inspiration taken from RHofsaess. It implements some standard file system operations, which can be used by the XRootDTarget. The error handling is done by assertions, since XRootD does not raise exceptions.

exists(path: str) → bool[source]#

Implementation of the exists function for the XRootDSystem. Will return True if the file or directory exists and False if it can not be found. This might also include cases, where the server is not reachable.

Parameters:

path (str) – The path to check for existence.

Returns:

True if the path exists, False otherwise..

Return type:

bool

copy_file_to_remote(local_path: str, remote_path: str, force: bool = False) → None[source]#

Function to copy a file from the local file system to the remote file system. In case the copy fails, a warning will be printed and a assertion will fail.

Parameters:

  • local_path (str) – The path to the file on the local file system.

  • remote_path (str) – The destination path for the file on the remote file system.

  • force (bool, optional) – If True, overwrites the file on the remote system if it already exists. Defaults to False.

Raises:

AssertionError – If the file copy operation fails.

copy_file_from_remote(remote_path: str, local_path: str, force: bool = False) → None[source]#

This method uses the client to perform the file transfer. If the copy operation fails, a warning message is logged, and an assertion error is raised.

Parameters:

  • remote_path (str) – The path to the file on the remote file system.

  • local_path (str) – The destination path for the file on the local file system.

  • force (bool, optional) – If True, overwrites the file on the local system if it already exists. Defaults to False.

Raises:

AssertionError – If the file copy operation fails.

copy_dir_from_remote(remote_path: str, local_path: str, force: bool = False) → None[source]#

Copies a directory and its files from the remote file system to the local file system.

This method does not support nested directories. It iterates through the contents of the remote directory and copies each file to the specified local directory.

Parameters:

  • remote_path (str) – Path to the directory on the remote file system.

  • local_path (str) – Path to the directory on the local file system.

  • force (bool, optional) – If True, overwrites files on the local system if they already exist. Defaults to False.

move(source_path: str, dest_path: str) → None[source]#

A function to move a file from one location to another on the XRootD server. In case the move fails, a warning will be printed and a assertion will fail.

Parameters:

  • source_path (str) – The current path of the file on the remote file system.

  • dest_path (str) – The target path for the file on the remote file system.

Raises:

AssertionError – If the move operation fails.

mkdir(path: str, parents: bool = True, raise_if_exists: bool = False) → None[source]#

A function to create a directory on the remote file system. In case the creation fails, a warning will be printed and a assertion will fail.

Parameters:

path (str) – The path of the directory to create.

Raises:

AssertionError – If the directory creation operation fails.

locate(path: str) → bool[source]#

Checks the location of a file on the remote file system.

This method queries the remote file system to locate the specified file. If the operation fails, a warning is logged, and an assertion error is raised.

Parameters:

path (str) – The path to the file on the remote file system.

Returns:

True if the file location is successfully determined.

Return type:

bool

Raises:

AssertionError – If the locate operation fails.

remove(path: str, recursive: bool = True, skip_trash: bool = True) → None[source]#

A function to remove a file from the remote file system. This function can not remove directories. Use remove_dir for that. In case the removal fails, a warning will be printed and a assertion will fail.

Parameters:

path (str) – The path to the file on the remote file system.

Raises:

AssertionError – If the file removal operation fails.

listdir(path: str, print_entries: bool = False) → Tuple[Dict[str, int], Any] | List[str][source]#

Lists the contents of a directory on the remote file system.

This method retrieves the directory listing and categorizes entries as files or directories. If the operation fails, a warning is logged, and an assertion error is raised.

Parameters:

path (str) – Path to the directory on the remote file system.

Returns:

A dictionary mapping paths to their type (1 for directories, 0 for files), and the raw listing object.

Return type:

Tuple[Dict[str, int], Any]

Raises:

AssertionError – If the directory listing operation fails.

remove_dir(path: str) → None[source]#

A function to iteratively remove a directory and all its content from the remote file system. In case the removal fails, a warning will be printed and a assertion will fail.

Parameters:

path (str) – Path to the directory on the remote file system.

Raises:

AssertionError – If the directory removal operation fails.

class b2luigi.WebDAVSystem(server_path: str)[source]#

Bases: RemoteFileSystem

WebDAV file system for b2luigi Targets. The davs protocol is not supported, but https.

exists(path: str) → bool[source]#

Implementation of the exists function for the WebDAVSystem. Will return True if the file or directory exists and False if it can not be found. This might also include cases, where the server is not reachable.

Parameters:

path (str) – The path to check for existence.

Returns:

True if the path exists, False otherwise..

Return type:

bool

copy_file_to_remote(local_path: str, remote_path: str, force: bool = False) → None[source]#

Function to copy a file from the local file system to the remote file system.

Parameters:

  • local_path (str) – The path to the file on the local file system.

  • remote_path (str) – The destination path for the file on the remote file system.

  • force (bool, optional) – If True, overwrites the file on the remote system if it already exists. Defaults to False.

copy_file_from_remote(remote_path: str, local_path: str, force: bool = False) → None[source]#

This method uses the client to perform the file transfer.

Parameters:

  • remote_path (str) – The path to the file on the remote file system.

  • local_path (str) – The destination path for the file on the local file system.

  • force (bool, optional) – If True, overwrites the file on the local system if it already exists. Defaults to False.

copy_dir_from_remote(remote_path: str, local_path: str, force: bool = False) → None[source]#

Copies a directory and its files from the remote file system to the local file system.

This method does not support nested directories. It iterates through the contents of the remote directory and copies each file to the specified local directory.

Parameters:

  • remote_path (str) – Path to the directory on the remote file system.

  • local_path (str) – Path to the directory on the local file system.

  • force (bool, optional) – If True, overwrites files on the local system if they already exist. Defaults to False.

move(source_path: str, dest_path: str) → None[source]#

A function to move a file from one location to another on the XRootD server.

Parameters:

  • source_path (str) – The current path of the file on the remote file system.

  • dest_path (str) – The target path for the file on the remote file system.

mkdir(path: str, parents: bool = True, raise_if_exists: bool = False) → None[source]#

A function to create a directory on the remote file system. In case the creation fails, a warning will be printed and a assertion will fail.

Parameters:

path (str) – The path of the directory to create.

locate(path: str) → bool[source]#

This method queries the remote file system to locate the specified file.

Parameters:

path (str) – The path to the file on the remote file system.

Returns:

True if the file location is successfully determined.

Return type:

bool

remove(path: str, recursive: bool = True, skip_trash: bool = True) → None[source]#

A function to remove a file from the remote file system. This function can not remove directories. Use remove_dir for that.

Parameters:

path (str) – The path to the file on the remote file system.

listdir(path: str, print_entries: bool = False) → Tuple[Dict[str, int], Any] | List[str][source]#

Lists the contents of a directory on the remote file system.

This method retrieves the directory listing and categorizes entries as files or directories.

Parameters:

  • path (str) – Path to the directory on the remote file system.

  • print_entries (bool, optional) – Unused

Returns:

A list of paths to the entries in the directory.

Return type:

List[str]

remove_dir(path: str) → None[source]#

A function to iteratively remove a directory and all its content from the remote file system.

Parameters:

path (str) – Path to the directory on the remote file system.

Contents