Parameters

As b2luigi automatically also imports luigi, you can use all the parameters from luigi you know and love. We have just added a single new flag called hashed to the parameters constructor. Turning it to true (it is turned off by default) will make b2luigi use a hashed version of the parameters value, when constructing output or log file paths. This is especially useful if you have parameters, which may include “dangerous” characters, like “/” or “{” (e.g. when using list or dictionary parameters). If you want to exclude certain parameters from the creation of the directory structure , you can use the significant flag and set it to False.

See also one of our FAQ.

For more information on luigi.Parameters and what types there are see the Luigi Documentation

b2luigi.wrap_parameter()

Monkey patch the parameter base class (and with it all other parameters( of luigi to include three additional parameters in its constructor: hashed, hash_function and hidden.

Enabling the hashed parameter will use a hashed version of the parameter value when creating file paths our of the parameters of a task instead of the value itself. By default an md5 hash is used. A custom hash function can be provided via the hash_function parameter. This function should take one input, the value of the parameter. It is up to the user to ensure a unique string is created from the input.

This is especially useful when you have list, string or dict parameters, where the resulting file path may include “/” or “{}”.

With the hidden parameter, you can control whether the parameter should be hiddened in the task’s output directory structure when using add_to_output.

Caution

This will remove the parameter from the unique output of the task, so be sure to add it back, e.g. into the output file name:

class MyTask(b2luigi.Task):
    iddened_parameter = b2luigi.Parameter(hidden=True)

    def output(self):
        yield self.add_to_output(f"test_{self.hiddened_parameter}.txt")