ValidationRoot Class Reference

Public Member Functions
def	__init__ (self, working_folder, gitlab_object, gitlab_config, gitlab_map)
def	create_comparison (self)
def	index (self)
def	plots (self, *args)
def	check_comparison_status (self)
def	revisions (self, revision_label=None)
def	comparisons (self, comparison_label=None)
def	system_info (self)
def	retrieve_file_metadata (self, filename)
def	create_issue (self, title, description)
def	issue (self, file_path, rev_label, contact)
def	issue_redirect (self, iid)
def	update_issue (self, id, file_path, rev_label)

Public Attributes
	working_folder
	html folder that contains plots etc.
	last_restart
	Date when this object was instantiated.
	version
	Git version.
	gitlab_object
	Gitlab object.
	gitlab_config
	Gitlab config.
	gitlab_map
	Gitlab usermap.
	file_path
	placeholder variable for path
	revision_label
	placeholder variable for revision label
	contact
	placeholder variable for contact

Detailed Description

Root Validation class to handle non-static HTTP requests into the
validation server. The two main functions are to hand out compiled json
objects of revisions and comparisons and to start and monitor the
creation of comparison plots.

Definition at line 651 of file validationserver.py.

Constructor & Destructor Documentation

__init__()

def __init__	(		self,
			working_folder,
			gitlab_object,
			gitlab_config,
			gitlab_map
	)

class initializer, which takes the path to the folders containing the
validation run results and plots (aka comparison), gitlab object and
config

Definition at line 661 of file validationserver.py.

    def __init__(self, working_folder, gitlab_object, gitlab_config, gitlab_map):
        """
        class initializer, which takes the path to the folders containing the
        validation run results and plots (aka comparison), gitlab object and
        config
        """
 
        
        self.working_folder = working_folder
 
        
        self.last_restart = datetime.datetime.now()
 
        
        self.version = validationfunctions.get_compact_git_hash(
            os.environ["BELLE2_LOCAL_DIR"]
        )
 
        
        self.gitlab_object = gitlab_object
 
        
        self.gitlab_config = gitlab_config
 
        
        self.gitlab_map = gitlab_map
 
        
        self.file_path = None
        
        self.revision_label = None
        
        self.contact = None
 

Member Function Documentation

check_comparison_status()

def check_comparison_status

(

self

)

Checks on the status of a comparison creation

Definition at line 744 of file validationserver.py.

    def check_comparison_status(self):
        """
        Checks on the status of a comparison creation
        """
        progress_key = cherrypy.request.json["input"]
        logging.debug("Checking status for plot creation: " + str(progress_key))
        status = check_plotting_status(progress_key)
        return status
 

comparisons()

def comparisons	(		self,
			comparison_label = None
	)

return the json file of the comparison results of one specific
comparison

Definition at line 859 of file validationserver.py.

    def comparisons(self, comparison_label=None):
        """
        return the json file of the comparison results of one specific
        comparison
        """
 
        warn_wrong_directory()
 
        # todo: Make this independent of our working directory!
        path = os.path.join(
            os.path.relpath(
                validationpath.get_html_plots_tag_comparison_folder(
                    self.working_folder, comparison_label.split(",")
                ),
                validationpath.get_html_folder(self.working_folder),
            ),
            "comparison.json",
        )
 
        # check if this comparison actually exists
        if not os.path.isfile(path):
            raise cherrypy.HTTPError(
                404, f"Json Comparison file {path} does not exist"
            )
 
        return deliver_json(path)
 

create_comparison()

def create_comparison

(

self

)

Triggers the start of a now comparison between the revisions supplied
in revision_list

Definition at line 698 of file validationserver.py.

    def create_comparison(self):
        """
        Triggers the start of a now comparison between the revisions supplied
        in revision_list
        """
        rev_list = cherrypy.request.json["revision_list"]
        logging.debug("Creating plots for revisions: " + str(rev_list))
        progress_key = start_plotting_request(
            rev_list,
            validationpath.get_results_folder(self.working_folder),
        )
        return {"progress_key": progress_key}
 

create_issue()

def create_issue	(		self,
			title,
			description
	)

Call the functions to create the issue and redirect
to the created Gitlab issue page.

Definition at line 920 of file validationserver.py.

    def create_issue(self, title, description):
        """
        Call the functions to create the issue and redirect
        to the created Gitlab issue page.
        """
 
        # check if this is a plot/script based on file format
        issue_type = 'plot'
        if 'log' == self.file_path.split("/")[-1].split(".")[-1]:
            issue_type = 'script'
        default_section = self.gitlab_config['global']['default']
        project_id = self.gitlab_config[default_section]['project_id']
        # Create issue in the Gitlab project and save it
        project = get_project_object(self.gitlab_object, project_id)
        uploaded_file = upload_file_gitlab(self.file_path, project)
        assignees = {
            'gitlab_ids': [],
            'usernames': [],
        }
        file_name = self.file_path.split("/")[-1].split(".log")[0]
        file_package = self.file_path.split("/")[-2]
        if self.gitlab_map:
            assignees = parse_contact(
                self.contact, self.gitlab_map, file_package, self.gitlab_object
            )
        description += "\n\n---\n\n:robot: Automated code, please do not delete\n\n" + \
            f"Relevant {issue_type}: {file_name}\n\n" + \
            f"Revision label: {self.revision_label}\n\n---"
        issue_id = create_gitlab_issue(
            title, description, uploaded_file, assignees, file_package, project
        )
        project.save()
 
        # Update JSON with created issue id - script and plot info reside
        # in different locations and also have different structures.
        # todo - maybe this can be combined?
        if issue_type == 'script':
            revision_json_path = os.path.join(
                validationpath.get_results_folder(self.working_folder),
                self.revision_label,
                validationpath.file_name_results_json,
            )
            update_scriptfile_issues_json(
                revision_json_path, file_name, file_package, issue_id)
        else:
            comparison_json_path = os.path.join(
                validationpath.get_html_plots_folder(self.working_folder),
                self.file_path.split("/")[-3],
                "comparison.json",
            )
            update_plot_issues_json(
                comparison_json_path, file_name, file_package, issue_id)
 
        issue_url = self.gitlab_config[default_section]['project_url'] \
            + "/-/issues/" \
            + str(issue_id)
        raise cherrypy.HTTPRedirect(
            issue_url
        )
 

index()

def index

(

self

)

forward to the static landing page if
the default url is used (like http://localhost:8080/)

Definition at line 712 of file validationserver.py.

    def index(self):
        """
        forward to the static landing page if
        the default url is used (like http://localhost:8080/)
        """
        raise cherrypy.HTTPRedirect("/static/validation.html")
 

issue()

def issue	(		self,
			file_path,
			rev_label,
			contact
	)

Return a template issue creation interface
for the user to add title and description.

Definition at line 981 of file validationserver.py.

    def issue(self, file_path, rev_label, contact):
        """
        Return a template issue creation interface
        for the user to add title and description.
        """
        self.file_path = os.path.join(
            validationpath.get_html_folder(self.working_folder), file_path
        )
 
        self.revision_label = rev_label
        self.contact = contact
 
        if not self.gitlab_object:
            return "ERROR: Gitlab integration not set up, verify config file."
 
        raise cherrypy.HTTPRedirect("/static/validation_issue.html")
 

issue_redirect()

def issue_redirect	(		self,
			iid
	)

Redirect to the Gitlab issue page.

Definition at line 999 of file validationserver.py.

    def issue_redirect(self, iid):
        """
        Redirect to the Gitlab issue page.
        """
        default_section = self.gitlab_config['global']['default']
        if not self.gitlab_config[default_section]['project_url']:
            return "ERROR: Gitlab integration not set up, verify config file."
 
        issue_url = self.gitlab_config[default_section]['project_url'] \
            + "/-/issues/" \
            + str(iid)
        raise cherrypy.HTTPRedirect(
                issue_url
            )
 

plots()

def plots	(		self,
		*	args
	)

Serve file from the html/plot directory.
:param args: For the request /plots/a/b/c, these will be the strings
    "a", "b", "c"

Definition at line 720 of file validationserver.py.

    def plots(self, *args):
        """
        Serve file from the html/plot directory.
        :param args: For the request /plots/a/b/c, these will be the strings
            "a", "b", "c"
        """
 
        warn_wrong_directory()
 
        if len(args) < 3:
            raise cherrypy.HTTPError(404)
 
        tag_folder = os.path.relpath(
            validationpath.get_html_plots_tag_comparison_folder(
                self.working_folder, args[:-2]
            ),
            validationpath.get_html_folder(self.working_folder),
        )
        path = os.path.join(tag_folder, *args[-2:])
        return cherrypy.lib.static.serve_file(path)
 

retrieve_file_metadata()

def retrieve_file_metadata	(		self,
			filename
	)

Returns:
    Metadata(str) of the file

Definition at line 908 of file validationserver.py.

    def retrieve_file_metadata(self, filename):
        """
        Returns:
            Metadata(str) of the file
        """
        cherrypy.response.headers['Content-Type'] = 'text/plain'
        metadata = validationfunctions.get_file_metadata(filename)
        return metadata
 

revisions()

def revisions	(		self,
			revision_label = None
	)

Return a combined json object with all revisions and
mark the newest one with the field most_recent=true

Definition at line 755 of file validationserver.py.

    def revisions(self, revision_label=None):
        """
        Return a combined json object with all revisions and
        mark the newest one with the field most_recent=true
        """
 
        # get list of available revision
        rev_list = get_json_object_list(
            validationpath.get_results_folder(self.working_folder),
            validationpath.file_name_results_json,
        )
 
        # always add the reference revision
        combined_list = []
        reference_revision = json.loads(
            json_objects.dumps(json_objects.Revision(label="reference"))
        )
 
        # load and combine
        for r in rev_list:
            full_path = os.path.join(
                validationpath.get_results_folder(self.working_folder),
                r,
                validationpath.file_name_results_json,
            )
 
            # update label, if dir has been moved
            lbl_folder = get_revision_label_from_json_filename(full_path)
            j = deliver_json(full_path)
            j["label"] = lbl_folder
            combined_list.append(j)
 
        # Sorting
 
        # Order by categories (nightly, release, etc.) first, then by date
        # A pure chronological order doesn't make sense, because we do not
        # have a linear history ((pre)releases branch off) and for the builds
        # the date corresponds to the build date, not to the date of the
        # actual commit.
        def sort_key(label: str):
            if "-" not in label:
                logging.warning(
                    f"Misformatted label encountered: '{label}' "
                    f"(doesn't seem to include date?)"
                )
                return label
            category, datetag = label.split("-", maxsplit=1)
            print(category, datetag)
            # Will later reverse order to bring items in the same category
            # in reverse chronological order, so the following list will have
            # the items in reverse order as well:
            order = ["release", "prerelease", "nightly"]
            try:
                index = order.index(category)
            except ValueError:
                index = 9
                logging.warning(
                    f"Misformatted label encountered: '{label}' (doesn't seem "
                    f"to belong to any known category?)"
                )
            return f"{index}-{datetag}"
 
        combined_list.sort(key=lambda rev: sort_key(rev["label"]), reverse=True)
 
        # reference always on top
        combined_list = [reference_revision] + combined_list
 
        # Set the most recent one ...
        newest_date = None
        newest_rev = None
        for r in combined_list:
            rdate_str = r["creation_date"]
            if isinstance(rdate_str, str):
                if len(rdate_str) > 0:
                    try:
                        rdate = time.strptime(rdate_str, "%Y-%m-%d %H:%M")
                    except ValueError:
                        # some old validation results might still contain
                        # seconds and therefore cannot properly be converted
                        rdate = None
 
                    if rdate is None:
                        continue
 
                    if newest_date is None:
                        newest_date = rdate
                        newest_rev = r
                    if rdate > newest_date:
                        newest_date = rdate
                        newest_rev = r
 
        for c in combined_list:
            if c["most_recent"] is not None:
                c["most_recent"] = False
 
        # if there are no revisions at all, this might also be just None
        if newest_rev:
            newest_rev["most_recent"] = True
 
        # topmost item must be dictionary for the ractive.os template to match
        return {"revisions": combined_list}
 

system_info()

def system_info

(

self

)

Returns:
    JSON file containing git versions and time of last restart

Definition at line 888 of file validationserver.py.

    def system_info(self):
        """
        Returns:
            JSON file containing git versions and time of last restart
        """
 
        warn_wrong_directory()
 
        # note: for some reason %Z doesn't work like this, so we use
        # time.tzname for the time zone.
        return {
            "last_restart": self.last_restart.strftime("%-d %b %H:%M ")
            + time.tzname[1],
            "version_restart": self.version,
            "version_current": validationfunctions.get_compact_git_hash(
                os.environ["BELLE2_LOCAL_DIR"]
            ),
        }
 

update_issue()

def update_issue	(		self,
			id,
			file_path,
			rev_label
	)

Update existing issue in Gitlab with current result plot
and redirect to the updated Gitlab issue page.

Definition at line 1015 of file validationserver.py.

    def update_issue(self, id, file_path, rev_label):
        """
        Update existing issue in Gitlab with current result plot
        and redirect to the updated Gitlab issue page.
        """
 
        if not self.gitlab_object:
            return "ERROR: Gitlab integration not set up, verify config file."
 
        plot_path = os.path.join(
            validationpath.get_html_folder(self.working_folder), file_path
        )
 
        default_section = self.gitlab_config['global']['default']
        project_id = self.gitlab_config[default_section]['project_id']
        project = get_project_object(self.gitlab_object, project_id)
        uploaded_file = upload_file_gitlab(plot_path, project)
        update_gitlab_issue(
            id, uploaded_file, project, plot_path, rev_label
        )
        project.save()
 
        issue_url = self.gitlab_config[default_section]['project_url'] \
            + "/-/issues/" \
            + str(id)
 
        raise cherrypy.HTTPRedirect(
            issue_url
        )
 
 

Member Data Documentation

contact

contact

placeholder variable for contact

Definition at line 693 of file validationserver.py.

file_path

file_path

placeholder variable for path

Definition at line 689 of file validationserver.py.

gitlab_config

gitlab_config

Gitlab config.

Definition at line 683 of file validationserver.py.

gitlab_map

gitlab_map

Gitlab usermap.

Definition at line 686 of file validationserver.py.

gitlab_object

gitlab_object

Gitlab object.

Definition at line 680 of file validationserver.py.

last_restart

last_restart

Date when this object was instantiated.

Definition at line 672 of file validationserver.py.

revision_label

revision_label

placeholder variable for revision label

Definition at line 691 of file validationserver.py.

version

version

Git version.

Definition at line 675 of file validationserver.py.

working_folder

working_folder

html folder that contains plots etc.

Definition at line 669 of file validationserver.py.

---

The documentation for this class was generated from the following file:

• validation/scripts/validationserver.py