This document details out the design to setup Sunbird Ed production ready environment
Approach1:
Leverage Easy installer (cli mode) to setup all containerised and non-containerised workloads in kuberentes cluster where all the non-containerised workloads like databases to be run in in HA mode using the respective kubernetes operators. Provide Rancher as part of setup to manage the k8s clsuter
Advantages:
Installation and upgrade process will be easy
Cost of infra will be less as all the stateful workloads also run in k8s cluster
Step towards k8s native
Open source package unavailability issues will not occur as all the libraries are packaged as container image for non-containerised workloads
Challenges:
Need additional effort to setup operators for each stateful app and test
Database management/operations on k8s need to be taken care by SRE team post the installation
We need to do loadtest and benchmark the throughput
Approach2:
Leverage the Jenkins setup where containerised services are deployed to the kuberentes cluster and non-containerised services are deployed on to Virtual machines. Create wrapper jenkins jobs to provision & configure the services and deploy the services
Advantages:
Leverage existing ansible scripts to setup databases on Virtual Machines (Compute instances)
Jenkins based installation which user can visualise the builds and deployments
Challenges:
Manage self managed servers
Always need to depend on package managers. If a old lib/package is removed, installation scripts will fail
Need to create Jenkins master jobs which in turn trigger other jenkins jobs of service provision, deploy and configure jobs in automated way
Design discussion points:
We can go ahead with implementation of setting up containerised and non-containerised workloads on k8s cluster. Post the installation SRE team should be able to manage the infra including databases.
Add a disclaimer saying, if the SRE team is not capable enough to manage the database workloads on k8s cluster, it is recommended to setup the databases using cloud managed services or on self managed compute instances
Do the load testing and benchmark the setup for 2000 tps.
If the adopter has requirement of more than 2000 tps, it is recommended to setup the databases using cloud managed services
Enable configurability to connect to the managed database services from microservices and test with one CSP
Document the best practices to setup and manage the environment including security, scale, Monitoring and logging