Eligibility, Booking, and Accounting

Preparation

General Duties

Your task as software quality shifter is to help to improve the quality of the Belle II software and its documentation by detecting, reporting, and solving problems.

  1. For the duration of your shift, you will automatically receive emails addressed to software-shifter@belle2.org . You should monitor all of these. Investigate any failures as far as you are able (see the sections "Build results" and "Build infrastructure" below).
  2. You should have been issued with a shift GitLab issue. You can note down work in the comments there, or simply in your shift report slides. At the end of your shift, please add a comment linking to your report slides to the ticket, and close the ticket. If you did not get a ticket, ask the software coordinators.
  3. At the beginning of your shift attend the software developers' or data production meeting and help with taking minutes.
  4. Every day of your shift check for failing scripts or red plots on the Validation Page. Detailed instructions are below.
  5. Every day of your shift check for a significant change of resource usage or compilation/memory issues on the monitoring summary page. Detailed instructions are below.
  6. At the end of your shift attend the software developers' or data production meeting and give a report of any problems that were detected/reported/solved. You should also think about handover notes for the next shifter. What should they keep an eye on? Examples of previous reports are attached to the recent software quality shift tickets.

Tasks for new software shifter

Tasks for experienced software shifter

Detailed Instructions

Who and how to contact people

General information about the validation plots and our build system

Belle II software makes use of two separate build systems: GitLab pipelines and buildbot.

The validation plots are made as part of the buildbot validation build. The resource monitoring plots are made as part of the buildbot development build. (The data-points for these might arrive at different times of the day).

Validation and resource monitoring

  1. Monitor the validation plots for indications of a regression. First of all, please check that the validation page does not remain in the state "Loading..." and that the validation for the last nightly build, i.e., the most recent validation plots should be less than about 24 hours old, are in fact loaded and shown to you. In the left sidebar you can select which software versions to compare with the references R. It makes sense to use the prebuilt setup R+latest nightly. Red plots will also display what some of the previously opened GitLab issues were. You can check the issue discussions to see if they are statistical fluctuations or recurrence of old issues, in which case you can reopen the old one. For reporting new problems (red histograms or failed scripts), you can create a GitLab issue directly from the plot window, and in some cases update existing issues as well. There are some even more detailed instructions here. In brief: follow up with any problems (red histograms or failed scripts) on the relevant GitLab issue. The responsible people should also get automatically notidfied of both the failing script and that you added a comment to the GitLab issue. So there is no particular need to email them as well. All actions should be recorded in your shift GitLab ticket and/or shift report slides. Please also check if there are only small fluctuations that might cause the comparison to fail. This is especially true for plots in the tracking package (Charge asymmetry vs pT), which often are in error state due to tiny fluctuations in the tracking reconstruction. These plots do not need to be further addressed with a GitLab issue or by informing the tracking librarians. The validation plot/script failures are summarized in a meta-plot 😀 on the monitoring summary page so no need to manually record failures. In GitLab you can get an overview of open validation plot issues.

    1. Tip: make sure that the "Expert plots" checkbox is unticked when looking at the page.

  2. Check the Execution Time plot on the monitoring summary page. If the Clustering, Simulation, HLT_Skim_Calculation, or any of the Tracking times change significantly, try to find the module(s) responsible for it in the Resources Plots page and report it on the software mailing list. If all times change simultaneously by a similar factor this could be caused by additional load on the machine where the development build is executed. In the latter case just make a note in the shift issue and check that the values go back to the previous ones on the next day.

  3. Check the evolution of the resident and virtual memory usage on the monitoring summary page. If it changes significantly report it on the software mailing list.

    You may need to click on the plot legend to select both (only virtual is enabled by default).

  4. Check the mDST TOTAL size on the monitoring summary page. Make sure TOTAL - DST is deselected (if not click it) so that you can better see the evolution of the mDST size. If it changes significantly try to find the component responsible for it in the plots above and report it on the software mailing list. This is important because it directly effects the cost of storing data over the lifetime of Belle II.
  5. Check the execution time of Clustering, Simulation, and Prefilter_Tracking for the three scenarios Data, High Multiplicity, and Low Multiplicity against the assigned budget on the Benchmark Plots page.
  6. Missing data points on any of the plots from 1-5 (above) should be investigated and noted in the shift report slides (you should receive an automated email or browse the corresponding build).
    If there have been no points in more than three days and the reason is not understood, report this to Thomas Kuhr. (And make a note in the shift report)
    1. One possible reason for missing points on memory-related plots (computing time, memory usage, (m)DST total size, memory leaks, and memory issues) on the monitoring summary page is if there have been no recent changes to the main software branch. This is easy to check and does not indicate a problem. To check, go to the memcheck build page, click on the most recent completed build number, click on the Build Properties tab, and look at the revision number. Then go to the previous build and check the revision number; if they are the same, then this explains why points on these plots have not been updated. You can also check recent commits to the main branch and see how long ago the last merge was (i.e., the last time the branch was changed).

Build results

  1. Check the nightly development build result for compilation errors or warnings, issues reported by cppcheck, failing tests, geometry overlaps, missing documentation, and library linking issues. Click on the Geometry link and check the log file for geometry overlap errors that are not reported on the nightly build result page. If based on the history plots you have the impression that there are issues that haven't been worked on for some time, ask the librarian about their status and for which ones they would appreciate help (from you or somebody else).
    1. For those able, consider correcting the warnings found in the nightly builds. For each shifter if the warnings of a single package is focused on and corrected, globally these warnings may be greatly reduced in a matter of weeks.
    2. To do so, create a clone of the main branch to correct these warnings. Push your corrections to your branch and create a merge request of your new branch with corrections to the main. Include relevant code authors and librarians as reviewers of this merge request.
    3. If the nightly build fails, find out which package is causing the fail (grepping for the string "FAILED:" can be very useful here). Then trace it back to the latest commit in the package. (Check out a new HEAD version, and use "git log pkgName"). Issue a GitLab issue to the person responsible for the latest changes.
    4. If a particular part of the build (e.g., Intel Build Result), indicates that it has failed (red X at the top of the column), but no individual package shows any errors, probably that build was not actually run. This can be difficult to diagnose if you don't realize it is a possibility. In this case, please report the problem to Thomas Kuhr.
  2. Check the memcheck for problems appearing in the event loop (columns with header "event") and ask the librarian about the status of issues if they appear stale according to the history. If you feel capable you may offer help to resolve the issues. If the nightly memcheck failed, go to the memcheck build page, select the latest Build #, and then click stdio of the failing command (highlighted in red). If there are false positives or issues in external packages that we cannot fix, ask the framework or externals librarian to include them in the suppression file.

Build infrastructure

  1. Check the GitLab pipelines queue.
    1. If there are more than ~20 pending pipelines, please report this to b2cs@belle2.org.
    2. If all recent Belle II builds show failures (sporadic failures or failures of single branches are perfectly fine), please write an email to Giacomo De Pietro and make a note in the shift report.
  2. If one of the automatic build procedures has an issue, take a look at the buildbot. Make note of the URL which contains the builder and the builder number.
    1. Try to investigate! There should be some stdout which might help.
      The output is split between the stages of the build (you should see "git", "setup" or "set property", and "compile")
      Here is an example of a failed build...

      in this particular case there was some problem with the externals
    2. If the stdout mentions a line number of a file, check the source code and try to find where the problem was introduced.
    3. If you have a line number, you can find the author by clicking "Blame" on the GitLab source view. And if in doubt email the relevant librarian(s) of the package(s) where the problem originates. [List of librarians.]
    4. If you get stuck: you can wait for the next build of that particular flavor to be triggered and see if anything was fixed.
    5. If the buildbot server goes down, and the page is not accessible, contact Thomas Kuhr. The server is quite stable, so the issue is unlikely to be temporary, and instead requires a restart.
    6. If there is a persistent problem, contact Thomas Kuhr.

GitLab

  1. Check with the assignees of inactive software GitLab issues how to proceed with them. Read the last actual status updates and discussions, i.e., everything that is not a shifter reminding about inactivity and asking for an update. Check whether reporter and/or assignee are still in the collaboration. Commenting "Is this issue still relevant?" on a ticket with no active member as assignee or reporter will not yield any progress. Similarly, just posting "mass messages" on any issue without context of the issue likely will not result in reactions. In case neither reporter nor assignee are still in the collaboration, check which package the issue belongs to and tag the librarian of that package, or the software coordinators if in doubt.
    1. If the issue is marked as being blocked by another issue, investigate that blocker issue as well.
    2. Follow up the issue by adding a comment asking the assigned person for a short update on progress and/or plans. Make use of GitLab's mention ("@") feature as it triggers more prominent notifications and encourages people to feel concerned and reply.
    3. If previous shifters have already asked for an update and didn't get one, or if there is no assignee, ping the librarian of the corresponding package(s) so that they look into whether the corresponding person (if any) is still active in Belle II, and if not who should take care of the work. [List of librarians.]
  2. Check for inactive or unassigned issues in data production. Check with the assignee (or reporter) how to proceed with them.
  3. Occasionally you may be asked to review a merge request. See this B2Questions post: "How do I review a pull request?" This is helpful for minor changes, in particular changes by a librarian (who cannot review their own work).

Questions and user support

  1. Check questions.belle2.org for any new questions. New questions are typically answered within (at most) a couple of days.
  2. Take a look at unanswered questions that have been posted for some time.
  3. Check that appropriate tags are used for the questions, especially that the "b2bii" tag is attached to all Belle-related questions.
  4. Have a go at answering!
  5. In case you don't know the answer you can @mention relevant experts to check they noticed the question.
  6. When questions are answered make sure that one of the answers is accepted.
  7. Up-vote any helpful and correct information.

Code Documentation

  1. Browse the most recent basf2 documentation (software.belle2.org > "sphinx documentation" > development version). Point out outdated, wrong, incomplete, or misleading documentation to relevant developers and librarians. If you feel confident, you can try to fix it yourself with a merge request.
  2. Browse the descriptions of analysis variables (b2help-variables / basf2 variables.py and/or on the basf2 documentation development version). If you find any are lacking details or are wrong, please report them. If you feel confident, you can try to fix it yourself with a merge request.
  3. Check the description of modules and module parameters (with b2help-modules / basf2 -m) and ask the maintainer of a module to fix meaningless, incomplete or wrong descriptions.
  4. Take a look at the "Doxygen" and "Sphinx" columns on the nightly build page (these are the automated documentation warnings for missing or incomplete documentation). Try to fix anything you can with a merge request.

Miscellaneous tips for a good and helpful shift (and report)

Don't be shy in reporting problems. It's better to have problems reported multiple times than to have problems overlooked or forgotten. People in the software group are friendly and will appreciate you checking things for them even if they know about it.

  1. Avoid just reporting information from the nightly builds. This information is stored already.
  2. No need to screenshot the green and yellow status of the validation plots. This information is now plotted in history trend plots (screenshot the plot if there was some noteworthy change)
  3. Software librarians are automatically notified about warnings in the build. There is no particular need to send another email.
  4. If you cannot attend the software developes' or data production meetings (due to timezone or similar reasons), please upload the report to the Indico agenda and mail the link to software@belle2.org before the meeting.