Automated Build of Releases

The github issues can be used to build CMSSW Releases. The main idea is that an authorized user creates an issue with the name “Build <release_name>”. Then, cms-bot guides you through the process.

Only authorized people can request of a release through a github issue. To request the build of a release your username must be in the list REQUEST_BUILD_RELEASE in the categories.py file. Likewise, the build must be approved from one of the users in the list APPROVE_BUILD_RELEASE.

if you are not authorized to request a build, cms-bot will inform it and ignore the issue. In this diagram you can see a detailed description of the process.

Click here to access the diagram in Google Drive.

Notice that there is a label for each state of the process, this is the way in which cms-bot can tell the status of a build issue.

General Workflow

Here you can see an example of the process. This process is inteded to be very similar to the manual workflow and at any time the release manager should be able to manually continue it or solve a failure.

• Step 1: Create a new issue with the name “Build <release_name>”. Only if you are in REQUEST_BUILD_RELEASE your issue will be processed.
• Step 2: someone from APPROVE_BUILD_RELEASE needs to write “+1” in a comment to start the build. When that happens, the release is created in github, the cmssw tag created, and the build is queued in jenkins.
• Step 3: When the builds for different architectures finish sucessfully, cms-bot informs it in the conversation, it also runs some tests on the release. You can check the logs for the builds and the tests to decide if it is ok to be uploaded and installed. If everything is ok, you can write “upload all” in the conversation and cms-bot will start to upload the builds for each architecture as soon as they finish successfully. The instalation in AFS is triggered automatically after the upload finishes.
• Step 4: You can generate the release notes by writing “release-notes since <previous_release>”. The workspaces used for the build in the jenkins nodes will be cleaned up 2 days after the release notes are generated.

Important: The state of the build is marked using the labels of the Github Issue. There is a main label that represents the state of the process, and a label per each architecture that indicates the particular state of that architecture. You can modify the labels of the issue to change the state of the build. For example, if the build for the architecture slc6_amd64_gcc481 had an error, the issue will get the label slc6_amd64_gcc481-build-error. If you can manually fix the issue in the jenkins node, you can remove that label replace it with slc6_amd64_gcc481-build-ok. cms-bot will interpret it as if the build for that architecture went ok and continue with the workflow.

Scripts and Jenkins Jobs

• process-build-release-request: script that handles the github issue used to request the build. It orchestates the triggering of the next scripts according to the status of the build. Run by query-build-release-issues.
• build-release: script used to build a release which has been requested through a Github issue. It can also build only cmssw-tool-conf if requested. Run by build-release (jenkins).
• upload-release: script used to upload a release to the repository. It connvects via SSH to the build machine which has the release and executes this script. Run by upload-release (jenkins). It is triggered after the comment “upload all” is found in the issue.
• report-build-release-status script used to report the status of the build.
• release-deploy-afs: takes care of installing the release in afs, it is run by release-deploy-afs (jenkins). It is triggered automatically after the upload finished correctly.
• release-notes: generates the release notes for the release, is run by release-produce-changelog.
• cleanup-auto-build: cleans the workspace of a build in the node where it was built. It is triggered 2 days after the release has been announced (2 days after the command to generate the release notes was written). Run by cleanup-auto-build (jenkins).
• kill-build-release: kills the process that was building the release, when the build is aborted. kill-build-release (jenkins). It logs into the machine that is building the release, and kills the process. build-release writes the pid in the file $WORKSPACE/BUILD_PID, this is used by kill-build-release to get the pid to kill.

Commands

During the process, cms-bot will indicate to the user which commands they can write in the comments depending on the status of the build. You need to be in the list APPROVE_BUILD_RELEASE in the file categories.py, otherwise your commands will be ignored. Currently the commands are:

• “build cmssw-tool-conf”: Before the actual build starts you can request to build cmssw-tool-conf, this package will be built and uploaded for each architecture of the release queue.
• “+1”: Start the build of the release per each architecture.
• “Abort”: Aborts the build, deletes the release in github, the cmssw and cmsdist tags, and kills the builds in jenkins. The issue goes to the state build-aborted and a new issue must be created to start over the build.
• “upload all”: starts the upload of the builds per each architecture as soon as the build is finished successfully.
• “release-notes since <previous-release>”: triggers the generation of the release notes since the release indicated. A template for the announcement is generated. The issue goes to the state release-announced, it is responsibility of the user to send the announcement email.
• “release-notes”: triggers the generation of the release-notes, but guesses the previous release. In any case, you can trigger the generation of the release notes again if necessary by writing again the command.

Quick start

• Main page
• Automated Builds
• Pull Request Testing
• Pull Request Comparisons
• IB Pages
• IB Additional Tests