The idea is to automate the build and deployment of various Jenkins jobs to a specific environment based on Github tags.
What is required from the development team?
Development team will create the tag for their respective github repositories using a Jenkins job.
How to create tag?
Go to the Jenkins job and choose the repository (multi-select enabled)
Provide the release branch from where the tag needs to be created
Click on Build
In the cosole log, the tag will be displayed
What happens next?
Jenkins jobs in staging environment will poll the repositories every 10 minutes to check if any new tags are created in the repo.
If new tag is found, the code will check if the tag is created in the deployment window. Lets say 16:00:00 to 18:00:00.
The start and end times are configurable using 2 variables in Jenkins - START_TIME, END_TIME
The tag will also be checked if it has a _RC string and whether it matchs the current release pattern. This is to prevent non RC tags to not trigger deployments and hotfix tags to not trigger staging deployements.
Example -
release-2.10.0_RC1 is valid
release-2.10.0, release-2.10.0_ABC, release-2.8.7_RC1 are invalid
Once the above check is passed, a new jenkins job is automatically created with the tag name.
Based on the job names, the respective build, upload and deploy jobs will be triggered.
A slack notification will be sent for successful build and successful deployements.
For intermediate jobs like uploading the docker image, artifacts etc, success notifications will not be sent and only failure notifications will be sent
Few images depicting the above details
Start and End time variables
New jobs get created based on tags
Downstream jobs triggered automatically based on the status of upstream job
Example - Portal build (few images)
Stages:
Player Build
Player Image docker image upload
Player Tenant Build
Player Tenant docker image upload
Player CDN Build
Player Deploy
Slack Notifications
Current challenges to discuss
Few repos have more than one build job, so when tag gets created, all the jobs of this repo will get triggered as we cannot distinguish which job should trigger
Current design: All these jobs will be triggered and deployed
Solution desired: Spiltting the service per repo
Portal and Editors have a strong dependency. So during portal and editor tag creation, portal needs to wait till editors deployment is completed and then start as it downloads the editors from blob
Current design: Dev team should co-ordinate and create editor tags first and ensure its deployed. After this, Portal should create the tag
Solution desired: Decouple these in someway
Certain builds when triggered parallely, causes overwriting of maven artifacts with a different code
Current design: Do nothing. If things break, it will be a reason to push teams to distinguish their maven artifacts
Solution desired: Version the maven artifatcs properly in pom.xml and also use different naming conventions per service. Or separate submodules. Or even better, remove the submodules.
Few Yarn jobs have an option to select only a specific samza job to deploy
Current design: Deploy all samza jobs always as we dont have away to distinguish which samza job to deploy based on tag
Solution desired: Unknown
Planned Change: In Kubernetes, we will implement deployment of specific jobs
Depoyment tracker updates still need to be done by dev team for a couple more days for artifact promotion tracking and variables tracking. Also dev teams need to ensure variables are commited to private repo before creating tags
Current design: Variable updates are manual and cannot be automated
Planned change: A new summary job which will hold details on all the jobs that were run / triggered in that release. The summary file can be used for next environment promotion along with job details, tags etc.
Certain deploy jobs which dont have a corresponding build job needs to be triggered manually - Example: Admin dashboard jobs
Certain deploy jobs which have non standard parameters needs to be triggered manually - Example: Neo4jSyncTool
Different repos have different release version
Current design: Trigger those manually as long as they are in same version
Solution desired: Follow release cycle to start using the automated jobs
Submodule changes will not trigger a new build
Current design: Ask teams to create dummy tag in main repo
Soultion desired: Tagging of submodules and then updating submodule repository to point to these tags instread of branch / Removing the submodules all together
Tags for staging cannot be created outside the deployment window. Hotfix tag for preprod and above can be created any anytime of the day.
Updates - 28/04/2020
Deployment tracker will still be created for each release. This is to track only manual steps like:
New jenkins job from dev jenkins
Jenkins job configuration changes
Infra changes and provision jobs
Anything that needs to be run manually
Jobs which are not covered under the automated build and deploy process
For all ansible code repos like sunbird-devops, sunbird-data-pipeline, sunbird-learning-platform, branch will be used to deploy and no need to create tag for these repos in automated deployments
Final tag for ansible code repos like sunbird-devops, sunbird-data-pipeline, sunbird-learning-platform will be created before promotion to next environment.
Any new tags post promotion for ansible code will be done by respective dev teams
If there is a change in build step of Jenkinsfile, the dev teams should also ensure to make this change in a file called auto_build_deploy which is present in every repoFlag notifications