Pull Request Comparisons

After running the pull request tests, it is run a comparison of DQM root files generated by runTheMatrix.py of IB + PR compared to IB. For example, after you run the tests for CMSSW_7_5_X_2015-05-05-1100 + 8978, the comparisons will be triggered for the files generated for this tests against the baseline files generated by CMSSW_7_5_X_2015-05-05-1100. This is done by compare-root-files-short-matrix which runs run-pr-comparisons.


The comparisons need the baseline files to run correctly

For example, to run the comparisons of CMSSW_7_5_X_2015-05-05-1100 + 8978 the baseline files for CMSSW_7_5_X_2015-05-05-1100 must have been generated correctly. If not, the comparisons will fail, this the main cause for the failure of the comparisons. These files are generated by baseline-ib-results which runs run-ib-comparison-baseline. In order to ensure that the comparisons run always in the same architecture where the node that did the tests was deployed, the Jenkins slaves are labelled with their hardware architecture using the vendor id, “GenuineIntel” or “AuthenticAMD”. The baseline files are generated for the each architecture. The pull requests tests can run in any of the 2 labels. But when they trigger the comparison, they pass a parameter informing in which architecture they ran. The comparison will download the corresponding baseline.

Not all release queues generate the baseline files

config.map controls for which release queues the baseline files are generated with the parameter ADITIONAL_TESTS. When the IB is promoted after it is installed, it triggers the generation of the baseline files depending on the configuration in config.map. For more information, see IB Additional Tests.

If there are errors in the IB, it will still try to upload the files for the successfully run workflows

baseline-ib-results will fail, but it will still upload the files from the workflows that ran successfully. The comparisons that are done against that IB will indicate this when writing the result in the pull request thread.

If the pull request had errors in the relvals the comparisons will no be triggered.

The file used to trigger the comparisons is only created if it is determined that the relvals ran sucessfully.

Scripts used by the comparisons

You can see the scripts used for the comparisons here.

Quick start