3.7.1. b2luigi

Overview

Teaching: 10 minutes

Prerequisites:

• access to kekcc

• grid-certificate for gbasf2

Questions:

• How do I implement my analysis in a b2luigi workflow?

• How do I automatize job submission and job steering on remote computing resources?

Objectives:

• Look at a complete Belle II workflow example.

• Understand how to modify and extend the workflow.

Luigi is a workflow management framework named after a well-known plumber and widely used in industry. It was originally developed by the Spotify group and open-sourced in 2012. In luigi, the workflow logic is decentralized and integrated with the analysis code for every processing step.

For Belle II workflows, the helper package b2luigi adds additional support for remote execution and ROOT file handling. In the following, we will provide a full example of a minimal Belle II analysis, employing gbasf2, basf2 and the LSF batch system. A full guide to b2luigi and luigi can be found elsewhere.

Let us outline the general processing steps (called tasks in b2luigi) for a generic Belle II workflow:

1. Run and download skims from the grid.

2. Reconstruct the skims on the KEKcc batch system. Employ batching for shorter computing times.

3. Merge the reconstructed batches.

4. Perform your offline analysis on the merged batches.

5. Produce some plots.

Now let us look at the directed acyclic graph for this workflow:

We read it from bottom to top: At the bottom, we see the task Plot, which is our final goal and fills some histograms for our reconstructed particles. Above it we see all of its dependencies, i.e. tasks that are required for Plot. First there are the merged reconstructed batches, which themselves require the reconstructed batches. We see that the skim is split into three batches, which are reconstructed simultaneously. The reconstructed batches build upon a batch of the skimmed files. b2luigi will build this dependency tree for your workflow and execute it from top to bottom.

Typically in Monte-Carlo simulation, you perform these steps multiple times for different event types (B mesons, quark-antiquark continuum etc.). Therefore the full dependency tree might look like this:

Now, let us go through the source code step by step. You are welcome to use this as a template for you own analysis.

• Offline Analysis
• Reconstruction on KEKcc Batch System
• Skim using gbasf2
• Full Workflow Execution

Stuck? We can help!

If you get stuck or have any questions to the online book material, the #starterkit-workshop channel in our chat is full of nice people who will provide fast help.

Refer to Collaborative Tools. for other places to get help if you have specific or detailed questions about your own analysis.

Improving things!

If you know how to do it, we recommend you to report bugs and other requests with GitLab. Make sure to use the documentation-training label of the basf2 project.

If you just want to give very quick feedback, use the last box “Quick feedback”.

Please make sure to be as precise as possible to make it easier for us to fix things! So for example:

• typos (where?)

• missing bits of information (what?)

• bugs (what did you do? what goes wrong?)

• too hard exercises (which one?)

• etc.

If you are familiar with git and want to create your first merge request for the software, take a look at How to contribute. We’d be happy to have you on the team!

Quick feedback!

Author of this lesson

Caspar Schmitt