After running the pull request tests, it is run a comparison of DQM root files generated by runTheMatrix.py of IB + PR compared to IB. For example, after you run the tests for CMSSW_7_5_X_2015-05-05-1100 + 8978, the comparisons will be triggered for the files generated for this tests against the baseline files generated by CMSSW_7_5_X_2015-05-05-1100. This is done by compare-root-files-short-matrix which runs run-pr-comparisons.
For example, to run the comparisons of CMSSW_7_5_X_2015-05-05-1100 + 8978 the baseline files for CMSSW_7_5_X_2015-05-05-1100 must have been generated correctly. If not, the comparisons will fail, this the main cause for the failure of the comparisons. These files are generated by baseline-ib-results which runs run-ib-comparison-baseline. In order to ensure that the comparisons run always in the same architecture where the node that did the tests was deployed, the Jenkins slaves are labelled with their hardware architecture using the vendor id, “GenuineIntel” or “AuthenticAMD”. The baseline files are generated for the each architecture. The pull requests tests can run in any of the 2 labels. But when they trigger the comparison, they pass a parameter informing in which architecture they ran. The comparison will download the corresponding baseline.
config.map controls for which release queues the baseline files are generated
with the parameter ADITIONAL_TESTS. When the IB is promoted after it is installed, it triggers the generation of the baseline files
depending on the configuration in config.map.
For more information, see IB Additional Tests.
baseline-ib-results will fail, but it will still upload the files from the workflows that ran successfully. The comparisons that are done against that IB will indicate this when writing the result in the pull request thread.
The file used to trigger the comparisons is only created if it is determined that the relvals ran sucessfully.
You can see the scripts used for the comparisons here.