Belle II Software development
Public Member Functions | Public Attributes | Protected Member Functions | Static Protected Member Functions | Protected Attributes | Static Protected Attributes | List of all members
Chi2Test Class Reference
Inheritance diagram for Chi2Test:
PvalueTest ComparisonBase

Public Member Functions

 __init__ (self, *args, **kwargs)
 
 ensure_compute (self)
 
 comparison_result (self)
 
 comparison_result_long (self)
 
 can_compare (self)
 

Public Attributes

 object_a = object_a
 store the first object to compare
 
 object_b = object_b
 store the second object to compare
 
 mop = mop
 MetaOptionParser.
 
 debug = debug
 enable debug?
 
bool computed = False
 used to store, whether the quantities have already been compared
 

Protected Member Functions

 _ensure_zero_error_has_no_content (self, a, b)
 
None _compute (self)
 
str _get_comparison_result_long (self)
 
str _get_comparison_result (self)
 
bool _has_correct_types (self)
 
None _raise_has_correct_types (self)
 
bool _has_compatible_bins (self)
 
None _raise_has_compatible_bins (self)
 

Static Protected Member Functions

 _convert_teff_to_hist (teff_a)
 

Protected Attributes

 _chi2 = None
 chi2
 
 _chi2ndf = None
 chi2 / number of degrees of freedom
 
tuple _ndf = None
 number of degrees of freedom
 
 _pvalue = None
 pvalue
 
float _pvalue_warn = self.mop.pvalue_warn()
 pvalue below which a warning is issued
 
float _pvalue_error = self.mop.pvalue_error()
 pvalue below which an error is issued
 
str _comparison_result = "not_compared"
 Comparison result, i.e.
 
str _comparison_result_long = ""
 Longer description of the comparison result (e.g.
 

Static Protected Attributes

float _default_pvalue_warn = 0.99
 Default pvalue below which a warning is issued (unless supplied in metaoptions)
 
float _default_pvalue_error = 0.01
 Default pvalue below which an error is issued (unless supplied in metaoptions)
 

Detailed Description

Perform a Chi2Test for ROOT objects. The chi2 test method is e.g. described
in the documentation of TH1::Chi2Test. Basically this class wraps around
this Chi2Test function, and takes care that we can call perform these
tests for a wider selection of ROOT objects.

Definition at line 399 of file validationcomparison.py.

Constructor & Destructor Documentation

◆ __init__()

__init__ ( self,
* args,
** kwargs ) 
Initialize Chi2Test.
:param args: See arguments of :class:`ComparisonBase`
:param kwargs:  See arguments of :class:`ComparisonBase`

Definition at line 408 of file validationcomparison.py.

408 def __init__(self, *args, **kwargs):
409 """
410 Initialize Chi2Test.
411 :param args: See arguments of :class:`ComparisonBase`
412 :param kwargs: See arguments of :class:`ComparisonBase`
413 """
414 super().__init__(*args, **kwargs)
415
416 # The following attributes will be set in :meth:`_compute`
417
418
419 self._chi2 = None
420
421 self._chi2ndf = None
422
423 self._ndf = None
424

Member Function Documentation

◆ _compute()

None _compute ( self)
protected 
Performs the actual Chi^2 test
@return: None

Reimplemented from PvalueTest.

Definition at line 443 of file validationcomparison.py.

443 def _compute(self) -> None:
444 """
445 Performs the actual Chi^2 test
446 @return: None
447 """
448 self._raise_has_correct_types()
449 self._raise_has_compatible_bins()
450
451 local_object_a = self.object_a
452 local_object_b = self.object_b
453
454 # very special handling for TEfficiencies
455 if self.object_a.ClassName() == "TEfficiency":
456 local_object_a = self._convert_teff_to_hist(self.object_a)
457 local_object_b = self._convert_teff_to_hist(self.object_b)
458 if self.debug:
459 print("Converting TEfficiency objects to histograms.")
460
461 nbins = local_object_a.GetNbinsX()
462
463 if nbins < 2:
464 raise TooFewBins(
465 f"{nbins} bin(s) is too few to perform the Chi2 test."
466 )
467
468 weighted_types = ["TProfile", "TH1D", "TH1F"]
469 comp_weight_a = local_object_a.ClassName() in weighted_types
470 comp_weight_b = local_object_b.ClassName() in weighted_types
471
472 # clone, because possibly some content of profiles will
473 # be set to zero
474 first_obj = local_object_a.Clone()
475 second_obj = local_object_b.Clone()
476
477 if comp_weight_a and not comp_weight_b:
478 # switch histograms, because ROOT can only have the first one
479 # to be unweighted
480 first_obj, second_obj = second_obj, first_obj
481 if self.debug:
482 print(
483 "Debug: Warning: Switching the two objects, because "
484 "ROOT can only have the first one to be unweighted"
485 )
486
487 # Construct the option string for the Chi2Test call
488 comp_options = "P " # for debugging output
489 if comp_weight_a and comp_weight_b:
490 comp_options += "WW"
491 elif comp_weight_a or comp_weight_b:
492 comp_options += "UW"
493 else:
494 comp_options += "UU"
495
496 if comp_weight_a and comp_weight_b:
497 self._ensure_zero_error_has_no_content(first_obj, second_obj)
498
499 # use numpy arrays to support ROOT's pass-by-reference interface here
500 res_chi2 = numpy.array([1], numpy.float64)
501 res_igood = numpy.array([1], numpy.int32)
502 res_ndf = numpy.array([1], numpy.int32)
503
504 res_pvalue = first_obj.Chi2TestX(
505 second_obj, res_chi2, res_ndf, res_igood, comp_options
506 )
507
508 if self.debug:
509 print("Performing our own chi2 test, with bin-by-bin results: ")
510 print()
511 print_contents_and_errors(first_obj, second_obj)
512 print()
513 print(
514 f"Here's what ROOT's Chi2Test gave us (comp_options: '{comp_options}'): "
515 )
516
517 tp = TablePrinter(3, width=(10, 10, 40))
518 print()
519 tp.print_divider()
520 tp.print(["Key", "Value", "Comment"])
521 tp.print_divider()
522 tp.print(
523 [
524 "chi2",
525 numpy.asscalar(res_chi2),
526 "Should roughly match above 'Total chi2'",
527 ]
528 )
529 tp.print(["ndf", numpy.asscalar(res_ndf), "#Non-empty bins - 1"])
530 tp.print(["chi2/ndf", numpy.asscalar(res_chi2 / res_ndf), ""])
531 tp.print(
532 [
533 "igood",
534 numpy.asscalar(res_igood),
535 "a debug indicator, 0 if all good",
536 ]
537 )
538 tp.print(["pvalue", res_pvalue, ""])
539 tp.print_divider()
540 print()
541 print(
542 "See https://root.cern.ch/doc/master/classTH1.html for more "
543 "information."
544 )
545 print()
546
547 if res_ndf < 1:
548 raise TooFewBins("res_ndf (<1) is too few to perform the Chi2 test.")
549
550 res_chi2ndf = res_chi2 / res_ndf
551
552 self._pvalue, self._chi2, self._chi2ndf, self._ndf = (
553 res_pvalue,
554 res_chi2[0],
555 res_chi2ndf[0],
556 res_ndf[0],
557 )
558

◆ _convert_teff_to_hist()

_convert_teff_to_hist ( teff_a)
staticprotectedinherited 
Convert the content of a TEfficiency plot to a histogram and set
the bin content and errors

Definition at line 313 of file validationcomparison.py.

313 def _convert_teff_to_hist(teff_a):
314 """
315 Convert the content of a TEfficiency plot to a histogram and set
316 the bin content and errors
317 """
318 conv_hist = teff_a.GetTotalHistogram()
319 xbin_count = conv_hist.GetNbinsX()
320 xbin_low = conv_hist.GetXaxis().GetXmin()
321 xbin_max = conv_hist.GetXaxis().GetXmax()
322
323 th1 = ROOT.TH1D(
324 teff_a.GetName() + "root_conversion",
325 teff_a.GetName(),
326 xbin_count,
327 xbin_low,
328 xbin_max,
329 )
330 # starting from the first to the last bin, ignoring the under/overflow
331 # bins
332 for i in range(1, xbin_count):
333 th1.SetBinContent(i, teff_a.GetEfficiency(i))
334 th1.SetBinError(i, teff_a.GetEfficiencyErrorLow(i))
335
336 return th1
337
338

◆ _ensure_zero_error_has_no_content()

_ensure_zero_error_has_no_content ( self,
a,
b )
protected 
Ensure there are no bins which have a content set, but 0 error
This bin content will be set to 0 to disable this bin completely during
the comparison

Definition at line 425 of file validationcomparison.py.

425 def _ensure_zero_error_has_no_content(self, a, b):
426 """
427 Ensure there are no bins which have a content set, but 0 error
428 This bin content will be set to 0 to disable this bin completely during
429 the comparison
430 """
431 nbins = a.GetNbinsX()
432 for ibin in range(1, nbins + 1):
433 if a.GetBinError(ibin) <= 0.0 and b.GetBinError(ibin) <= 0.0:
434 # set the bin content of the profile plots to zero so ROOT
435 # will ignore this bin in its comparison
436 a.SetBinContent(ibin, 0.0)
437 b.SetBinContent(ibin, 0.0)
438 if self.debug:
439 print(
440 f"DEBUG: Warning: Setting bin content of bin {ibin} to zero for both histograms, because both " +
441 "histograms have vanishing errors there.")
442

◆ _get_comparison_result()

str _get_comparison_result ( self)
protectedinherited 
 Used to format the value of :attr:`_comparison_result`.

Reimplemented from ComparisonBase.

Definition at line 370 of file validationcomparison.py.

370 def _get_comparison_result(self) -> str:
371 if self._pvalue is None:
372 return "error"
373
374 if self._pvalue < self._pvalue_error:
375 return "error"
376 elif self._pvalue < self._pvalue_warn:
377 return "warning"
378 else:
379 return "equal"
380

◆ _get_comparison_result_long()

str _get_comparison_result_long ( self)
protected 
 Used to format the value of :attr:`_comparison_result_long`.

Reimplemented from PvalueTest.

Definition at line 559 of file validationcomparison.py.

559 def _get_comparison_result_long(self) -> str:
560 if self._pvalue is None or self._chi2ndf is None or self._chi2 is None:
561 return (
562 r"Could not perform $\chi^2$-Test between {{revision1}} "
563 r"and {{revision2}} due to an unknown error. Please "
564 r"submit a bug report."
565 )
566
567 return (
568 r"Performed $\chi^2$-Test between {{revision1}} "
569 r"and {{revision2}} "
570 r"($\chi^2$ = {chi2:.4f}; NDF = {ndf}; "
571 r"$\chi^2/\text{{{{NDF}}}}$ = {chi2ndf:.4f})."
572 r" <b>p-value: {pvalue:.6f}</b> (p-value warn: {pvalue_warn}, "
573 r"p-value error: {pvalue_error})".format(
574 chi2=self._chi2,
575 ndf=self._ndf,
576 chi2ndf=self._chi2ndf,
577 pvalue=self._pvalue,
578 pvalue_warn=self._pvalue_warn,
579 pvalue_error=self._pvalue_error,
580 )
581 )
582
583
584# ------------------------------------------------------------------------------
585# Kolmogorov Test
586# ------------------------------------------------------------------------------
587
588

◆ _has_compatible_bins()

bool _has_compatible_bins ( self)
protectedinherited 
Check if both ROOT objects have the same amount of bins
@return: True if the bins are equal, otherwise False

Definition at line 275 of file validationcomparison.py.

275 def _has_compatible_bins(self) -> bool:
276 """
277 Check if both ROOT objects have the same amount of bins
278 @return: True if the bins are equal, otherwise False
279 """
280 if (
281 self.object_a.ClassName()
282 == "TEfficiency"
283 == self.object_b.ClassName()
284 ):
285 nbins_a = self.object_a.GetTotalHistogram().GetNbinsX()
286 nbins_b = self.object_b.GetTotalHistogram().GetNbinsX()
287 else:
288 nbins_a = self.object_a.GetNbinsX()
289 nbins_b = self.object_b.GetNbinsX()
290
291 return nbins_a == nbins_b
292

◆ _has_correct_types()

bool _has_correct_types ( self)
protectedinherited 
@return: True if the two objects have a) a type supported for
    comparison and b) can be compared with each other

Definition at line 231 of file validationcomparison.py.

231 def _has_correct_types(self) -> bool:
232 """
233 @return: True if the two objects have a) a type supported for
234 comparison and b) can be compared with each other
235 """
236 if self.object_a is None or self.object_b is None:
237 return False
238
239 # check if the supplied object inherit from one of the supported types
240 # and if they are of the same type
241 supported_types = ["TProfile", "TH1D", "TH1F", "TEfficiency"]
242 if self.object_a.ClassName() != self.object_b.ClassName():
243 return False
244 if self.object_a.ClassName() not in supported_types:
245 return False
246
247 if self.object_a.ClassName() == "TEfficiency":
248 # can only handle TEfficiencies with dimension one atm
249 if self.object_a.GetDimension() > 1:
250 return False
251
252 return True
253

◆ _raise_has_compatible_bins()

None _raise_has_compatible_bins ( self)
protectedinherited 
Raise Exception if not both ROOT objects have the same amount of bins
@return: None

Definition at line 293 of file validationcomparison.py.

293 def _raise_has_compatible_bins(self) -> None:
294 """
295 Raise Exception if not both ROOT objects have the same amount of bins
296 @return: None
297 """
298 if not self._has_compatible_bins():
299 msg = (
300 "The objects have differing x bin count: {} has {} vs. {} "
301 "has {}."
302 )
303 raise DifferingBinCount(
304 msg.format(
305 self.object_a.GetName(),
306 self.object_a.GetNbinsX(),
307 self.object_b.GetName(),
308 self.object_b.GetNbinsX(),
309 )
310 )
311

◆ _raise_has_correct_types()

None _raise_has_correct_types ( self)
protectedinherited 
Raise Exception if not the two objects have a) a type supported for
comparison and b) can be compared with each other
@return: None

Definition at line 254 of file validationcomparison.py.

254 def _raise_has_correct_types(self) -> None:
255 """
256 Raise Exception if not the two objects have a) a type supported for
257 comparison and b) can be compared with each other
258 @return: None
259 """
260 if not self._has_correct_types():
261 msg = (
262 "Comparison of {} (Type {}) with {} (Type {}) not "
263 "supported.\nPlease open a GitLab issue (validation "
264 "label) if you need this supported. "
265 )
266 raise ObjectsNotSupported(
267 msg.format(
268 self.object_a.GetName(),
269 self.object_a.ClassName(),
270 self.object_b.GetName(),
271 self.object_b.ClassName(),
272 )
273 )
274

◆ can_compare()

can_compare ( self)
inherited 
@return: True if the two objects can be compared, False otherwise

Definition at line 225 of file validationcomparison.py.

225 def can_compare(self):
226 """
227 @return: True if the two objects can be compared, False otherwise
228 """
229 return self._has_correct_types() and self._has_compatible_bins()
230

◆ comparison_result()

comparison_result ( self)
inherited 
 Comparison result, i.e. pass/warning/error

Definition at line 210 of file validationcomparison.py.

210 def comparison_result(self):
211 """ Comparison result, i.e. pass/warning/error """
212 self.ensure_compute()
213 return self._comparison_result
214

◆ comparison_result_long()

comparison_result_long ( self)
inherited 
 Longer description of the comparison result

Definition at line 216 of file validationcomparison.py.

216 def comparison_result_long(self):
217 """ Longer description of the comparison result """
218 self.ensure_compute()
219 return self._comparison_result_long
220

◆ ensure_compute()

ensure_compute ( self)
inherited 
Ensure all required quantities get computed and are cached inside the
class

Definition at line 159 of file validationcomparison.py.

159 def ensure_compute(self):
160 """
161 Ensure all required quantities get computed and are cached inside the
162 class
163 """
164 if self.computed:
165 return
166
167 if self.mop.has_option("nocompare"):
168 # is comparison disabled for this plot ?
169 self._comparison_result_long = "Testing is disabled for this plot"
170 return
171
172 fail_message = "Comparison failed: "
173
174 # Note: default for comparison_result is "not_compared"
175 try:
176 self._compute()
177 except ObjectsNotSupported as e:
178 self._comparison_result_long = fail_message + str(e)
179 except DifferingBinCount as e:
180 self._comparison_result = "error"
181 self._comparison_result_long = fail_message + str(e)
182 except TooFewBins as e:
183 self._comparison_result_long = fail_message + str(e)
184 except ComparisonFailed as e:
185 self._comparison_result = "error"
186 self._comparison_result_long = fail_message + str(e)
187 except Exception as e:
188 self._comparison_result = "error"
189 self._comparison_result_long = (
190 "Unknown error occurred. Please "
191 "submit a bug report. " + str(e)
192 )
193 else:
194 # Will be already set in case of errors above and we don't want
195 # to overwrite this.
196 self._comparison_result_long = self._get_comparison_result_long()
197 self._comparison_result = self._get_comparison_result()
198
199 self.computed = True
200

Member Data Documentation

◆ _chi2

_chi2 = None
protected

chi2

Definition at line 419 of file validationcomparison.py.

◆ _chi2ndf

_chi2ndf = None
protected

chi2 / number of degrees of freedom

Definition at line 421 of file validationcomparison.py.

◆ _comparison_result

str _comparison_result = "not_compared"
protectedinherited

Comparison result, i.e.

equal/warning/error

Definition at line 154 of file validationcomparison.py.

◆ _comparison_result_long

str _comparison_result_long = ""
protectedinherited

Longer description of the comparison result (e.g.

'performed Chi2 Test ... with chi2 = ...').

Definition at line 157 of file validationcomparison.py.

◆ _default_pvalue_error

float _default_pvalue_error = 0.01
staticprotectedinherited

Default pvalue below which an error is issued (unless supplied in metaoptions)

Definition at line 348 of file validationcomparison.py.

◆ _default_pvalue_warn

float _default_pvalue_warn = 0.99
staticprotectedinherited

Default pvalue below which a warning is issued (unless supplied in metaoptions)

Definition at line 344 of file validationcomparison.py.

◆ _ndf

tuple _ndf = None
protected

number of degrees of freedom

Definition at line 423 of file validationcomparison.py.

◆ _pvalue

_pvalue = None
protectedinherited

pvalue

Definition at line 359 of file validationcomparison.py.

◆ _pvalue_error

float _pvalue_error = self.mop.pvalue_error()
protectedinherited

pvalue below which an error is issued

Definition at line 363 of file validationcomparison.py.

◆ _pvalue_warn

float _pvalue_warn = self.mop.pvalue_warn()
protectedinherited

pvalue below which a warning is issued

Definition at line 361 of file validationcomparison.py.

◆ computed

bool computed = False
inherited

used to store, whether the quantities have already been compared

Definition at line 151 of file validationcomparison.py.

◆ debug

debug = debug
inherited

enable debug?

Definition at line 148 of file validationcomparison.py.

◆ mop

mop = mop
inherited

MetaOptionParser.

Definition at line 145 of file validationcomparison.py.

◆ object_a

object_a = object_a
inherited

store the first object to compare

Definition at line 137 of file validationcomparison.py.

◆ object_b

object_b = object_b
inherited

store the second object to compare

Definition at line 140 of file validationcomparison.py.

The documentation for this class was generated from the following file: