Parameters

Contents

Parameters#

As b2luigi automatically also imports luigi, you can use all the parameters from luigi you know and love. We have just added a single new flag called hashed to the parameters constructor. Turning it to true (it is turned off by default) will make b2luigi use a hashed version of the parameters value, when constructing output or log file paths. This is especially useful if you have parameters, which may include “dangerous” characters, like “/” or “{” (e.g. when using list or dictionary parameters). If you want to exclude certain parameters from the creation of the directory structure , you can use the significant flag and set it to False.

See also one of our FAQ.

For more information on luigi.Parameters and what types there are see the Luigi Documentation

b2luigi.wrap_parameter() → None[source]#

Monkey patch the parameter base class (and with it all other parameters( of luigi to include three additional parameters in its constructor: hashed, hash_function, hidden, grouping and grouping_function.

Enabling the hashed parameter will use a hashed version of the parameter value when creating file paths our of the parameters of a task instead of the value itself. By default an md5 hash is used. A custom hash function can be provided via the hash_function parameter. This function should take one input, the value of the parameter. It is up to the user to ensure a unique string is created from the input.

This is especially useful when you have list, string or dict parameters, where the resulting file path may include “/” or “{}”.

With the hidden parameter, you can control whether the parameter should be hiddened in the task’s output directory structure when using add_to_output.

With the grouping parameter, you can control whether the parameter should be treated as a grouping parameter. If no grouping_function is provided, the default function will be to return a list of the input value. You still treat the parameter as a normal parameter when defining the task, but during execution, the task will be executed once for each value in the group. If you provide a custom grouping_function, it should follow the format: function(iterable[x])->x where x is the parameter you want to group over. To enable grouping, you also need to set the task property max_grouping_size to a value greater than 1. For more information on parameter grouping, see Parameter Grouping.

Caution

This will remove the parameter from the unique output of the task, so be sure to add it back, e.g. into the output file name:

class MyTask(b2luigi.Task):
    iddened_parameter = b2luigi.Parameter(hidden=True)

    def output(self):
        yield self.add_to_output(f"test_{self.hiddened_parameter}.txt")
Contents