XRootDTargets#

To ease the work with files stored on the grid, b2luigi provides an Target implementation using XRootD. In the background, this relies on the Python bindings of the XRootD client and of course requires a working XRootD client installation.

Hint

The XRootD client is already installed in basf2 enviroments on cvmfs.

Hint

To access XRootD storage you will need a valid VOMS proxy.

To use the targets, you will need to overwrite the _get_output_file_target function of your Task. This requires some additional setup compared to the standard LocalTarget. First you need to create a FileSystem object, which is used to connect to the XRootD server. For that you need to provide the server address like so: root://<server_address>. Optionally, one can also set a scratch_dir which will be used to store temporary files when using the temporary_path context. (https://luigi.readthedocs.io/en/stable/api/luigi.target.html) When entering this context, a temporary path will be created in the scratch_directory. At leaving the context, the file will then be copied to the final location on the XRootD storage. A full task using XRootDTargets could look like this:

from b2luigi.targets import XRootDTarget
from b2luigi.core.utils import create_output_filename
import b2luigi

class MyTask(b2luigi.Task):
    def _get_output_file_target(self, base_filename: str, **kwargs) -> b2luigi.Target:
        filename = create_output_filename(self, base_filename,  result_dir="<path on your XRootD storage>")
        fs = XRootDSystem("root://eospublic.cern.ch")
        return XRootDTarget(filename, fs, scratch_dir)

    def run(self):
        file_name = "Hello_world.txt"
        target = self._get_output_file_target(file_name)
        with target.temporary_path() as temp_path:
            with open(temp_path, "w") as f:
                f.write("Hello World")


    def output(self):
        yield self.add_to_output("Hello_world.txt")

class b2luigi.XRootDSystem(server_path: str)#

Bases: luigi.target.FileSystem

XRootDFileSystem for b2luigi Targets. Inspiration taken from rhofsaess https://github.com/RHofsaess/xrd-interactive/blob/main/XRootD_utils.py It implements some standard file system operations, which can be used by the XRootDTarget. The error handling is done by assertions, since XRootD does not raise exceptions.

copy_dir_from_remote(remote_path: str, local_path: str, force: bool = False) → None#

A function to copy a directory with all its files from the remote file system to the local file system. Nested directories are not supported.

Parameters

remote_path – Path to the directory on the remote file system.
local_path – Path to the directory on the local file system.
force – If True, the files will be overwritten if they already exist. Default is False.

copy_file_from_remote(remote_path: str, local_path: str, force: bool = False) → None#: Function to copy a file from the remote file system to the local file system. In case the copy fails, a warning will be printed and a assertion will fail. :param remote_path: Path to the file on the remote file system. :param local_path: Path to the file on the local file system. :param force: If True, the file will be overwritten if it already exists. Default is False.

copy_file_to_remote(local_path: str, remote_path: str, force: bool = False) → None#: Function to copy a file from the local file system to the remote file system. In case the copy fails, a warning will be printed and a assertion will fail. :param local_path: Path to the file on the local file system. :param remote_path: Path to the file on the remote file system. :param force: If True, the file will be overwritten if it already exists. Default is False.

exists(path: str) → bool#

Implementation of the exists function for the XRootDSystem. Will return True if the file or directory exists and Fals if it can not be found. This might also include cases, where the server is not reachable.

Parameters: path – Path to the file or directory to check.

listdir(path: str) → Tuple[Dict[str, int], Any]#: A function to list the content of a directory on the remote file system. In case the listing fails, a warning will be printed and a assertion will fail. :param path: Path to the directory on the remote file system.

locate(path: str) → bool#

mkdir(path: str) → None#: A function to create a directory on the remote file system. In case the creation fails, a warning will be printed and a assertion will fail.

move(source_path: str, dest_path: str) → None#: A function to move a file from one location to another on the XRootD server. In case the move fails, a warning will be printed and a assertion will fail. :param source_path: Path to the file on the remote file system. :param dest_path: Path to the file on the remote file system.

remove(path: str) → None#: A function to remove a file from the remote file system. This function can not remove directories. Use remove_dir for that. In case the removal fails, a warning will be printed and a assertion will fail. :param path: Path to the file on the remote file system.

remove_dir(path: str) → None#: A function to iteratively remove a directory and all its content from the remote file system. In case the removal fails, a warning will be printed and a assertion will fail. :param path: Path to the directory on the remote file system.

rename_dont_move(path: str, dest: str) → None#

Potentially rename path to dest, but don’t move it into the dest folder (if it is a folder). This relates to AtomicWrites.

This method has a reasonable but not bullet proof default implementation. It will just do move() if the file doesn’t exists() already.

b2luigi latest documentation

XRootDTargets

XRootDTargets#