FAQ

Can I specify my own paths for the log files for tasks running on a batch system?

b2luigi will automatically create log files for the stdout and stderr output of a task processed on a batch system. The paths of these log files are defined relative to the location of the executed python file and contain the parameter of the task. In some cases one might one to specify other paths for the log files. To achieve this, a own get_log_file_dir() method of the task class must be implemented. This method must return a directory path for the stdout and the stderr files, for example:

class MyBatchTask(b2luigi.Task):
    ...
    def get_log_file_dir(self):
        filename = os.path.realpath(sys.argv[0])
        path = os.path.join(os.path.dirname(filename), "logs")
        return path

b2luigi will use this method if it is defined and write the log output in the respective files. Be careful, though, as these log files will of course be overwritten if more than one task receive the same paths to write to!

Can I exclude one job from batch processing

The setting batch_system defines which submission method is used for scheduling your tasks when using batch=True or --batch. In most cases, you set your batch_system globally (e.g. in a settings.json) file and start all your tasks with --batch or batch=True. If you want a single task to run only locally (e.g. because of constraints in the batch farm) you can set the batch_system only for this job by adding a member to this task:

class MyLocalTask(b2luigi.Task):
    batch_system = "local"

    def run(self):
        ...

How do I handle parameter values which include “/” (or other unusual characters)?

b2luigi automatically generates the filenames for your output or log files out of the current tasks values in the form:

<result-path>/param1=value1/param2=value2/.../<output-file-name.ext>

The values are given by the serialisation of your parameter, which is basically its string representation. Sometimes, this representation may include characters not suitable for their usage as a path name, e.g. “/”. Especially when you use a DictParameter or a ListParameter, you might not want to have its value in your output. Also, if you have credentials in the parameter (what you should never do of course!), you do not want to show them to everyone.

When using a parameter in b2luigi (or any of its derivatives), they have a new flag called hashed in their constructor, which makes the path creation only using a hashed version of your parameter value.

For example will this task:

class MyTask(b2luigi.Task):
    my_parameter = b2luigi.ListParameter(hashed=True)

    def run(self):
        with open(self.get_output_file_name("test.txt"), "w") as f:
            f.write("test")

    def output(self):
        yield self.add_to_output("test.txt")


if __name__ == "__main__":
    b2luigi.process(MyTask(my_parameter=["Some", "strange", "items", "with", "bad / signs"]))

create a file called my_parameter=hashed_08928069d368e4a0f8ac02a0193e443b/test.txt in your output folder instead of using the list value.

What does the ValueError “The task id {task.task_id} to be executed…” mean?

The ValueError exception The task id <task_id> to be executed by this batch worker does not exist in the locally reproduced task graph. is thrown by b2luigi batch workers if the task that should have been executed by this batch worker does not exist in the task graph reproduced by the batch worker. This means that the task graph produced by the initial b2luigi.process call and the one reproduced in the batch job differ from each other. This can be caused by a non-deterministic behavior of your dependency graph generation, such as a random task parameter.

I do not like to have “=” in my output file names. Is there a way to not have them in the generated output paths?

Yes, you can use the setting use_parameter_name_in_output to control this behaviour. If you set it to False the paths for your outputs and logs will be generated using only the parameter values. It is then up to you to remember which parameter value belongs to which parameter name.