The idea is to automate the build and deployment of various Jenkins jobs to a specific environment based on Github tags.
What is required from the development team?
Development team will create the tag for their respective github repositories using a Jenkins job.
How to create tag?
Go to the Jenkins job and choose the repository (multi-select enabled)
Provide the release branch from where the tag needs to be created
Click on Build
In the cosole log, the tag will be displayed
What happens next?
Jenkins jobs in staging environment will poll the repositories every 10 minutes to check if any new tags are created in the repo.
If new tag is found, the code will check if the tag is created in the deployment window. Lets say 16:00:00 to 18:00:00.
The start and end times are configurable using 2 variables in Jenkins - START_TIME, END_TIME
The tag will also be checked if it has a _RC string and whether it matchs the current release pattern. This is to prevent non RC tags to not trigger deployments and hotfix tags to not trigger staging deployements.
Example -
release-2.10.0_RC1 is valid
release-2.10.0, release-2.10.0_ABC are invalid
Once the above check is passed, a new jenkins job is automatically created with the tag name.
Based on the job names, the respective build, upload and deploy jobs will be triggered.
A slack notification will be sent for successful build and successful deployements.
For intermediate jobs like uploading the docker image, artifacts etc, success notifications will not be sent and only failure notifications will be sent
Few images depicting the above details
Start and End time variables
New jobs get created based on tags
Downstream jobs triggered automatically based on the status of upstream job
Example - Portal build (few images)
Stages:
Player Build
Player Image docker image upload
Player Tenant Build
Player Tenant docker image upload
Player CDN Build
Player Deploy
Slack Notifications
Current challenges to discuss
Few repos have more than one build job, so when tag gets created, all the jobs of this repo will get triggered as we cannot distinguish which job should trigger
Current design: All these jobs will be triggered and deployed
Solution desired: Spiltting the service per repo
Portal and Editors have a strong dependency. So during portal and editor tag creation, portal needs to wait till editors deployment is completed and then start as it downloads the editors from blob
Current design: Manual intervention to stop portal builds so that editor builds complete
Solution desired: Decouple these in someway
Planned change: Automated portal build in waiting state in case any editors builds are currently being built. If editor team creates tags, auto build portal again to include new editors
Certain builds when triggered parallely, causes overwriting of maven artifacts with a different code
Current design: Do nothing. If things break, it will be a reason to push teams to ditinguish their maven artifacts
Solution desired: Version the maven artifatcs properly in pom.xml and also use different naming conventions per service
Few Yarn jobs have an option to select only a specific samza job to deploy
Current design: Deploy all samza jobs always as we dont have away to distinguish which samza job to deploy based on tag
Solution desired: Unknown
Depoyment tracker updates still need to be done by dev team for a couple more days for artifact promotion tracking and variables tracking. Also dev teams need to ensure variables are commited to private repo before creating tags
Current design: Variable updates are manual and cannot be automated
Planned change: A new summary job which will hold details on all the jobs that were run / triggered in that release. The summary file can be used for next environment promotion along with job details, tags etc
Certain deploy jobs which dont have a corresponding build job needs to be triggered manually
Certain deploy jobs which have non standard parameters needs to be triggered manually
Different repos have different release version
Current design: Trigger those manually
Planned change: Have a list of relase versions that we can match against to solve different repos having different release versions. Example - release-2.10.0, release-1.2.0, secor-0.25