Deployment Document
Table of Content
...
Objective
Hardware Requirements
IAM user and Role creation for S3 connectivity
...
cQube Deployment Process
...
Appendix
AWS - Network Architecture
Microservice Details
cQube Deployment Procedure
Objective
...
This document mainly focuses on the hardware requirements/ AWS network architecture for installing cQube and the deployment process step-by-step.
AWS - Network Architecture:
The following steps define the cQube setup and workflow completion processes in AWS. cQube mainly comprises the areas mentioned below:
EC2 Server
IAM user and Role creation for S3 connectivity.
The cQube network setup process is described in the block diagram below:
Ec2 Instance
Hardware Requirements
Create an EC2 instance with below configurations to install all the cQube micro services.
...
Create a domain name
Configure cname of ec2 instance to the domain name
Create a SSL certificate for the domain name.
Following are the details of the microservices which get installed in the cqube server.
Ingestion-ms: The ingestion-ms is used to upload the data of the events, datasets, dimensions, transformers and pipeline. All these apis will be to ingesting the data into the cQube.
Spec-ms: The spec-ms is used to import schema of the events, datasets, dimensions, transformers and pipeline. All these specs will be defined by the cQube platform prior to ingesting the data into the cQube. These specifications are derived by considering the KPIs as the Indicator.
Generator-ms: The generator-ms is used to create the specs & transformers for the derived datasets. Performed aggregation logics, updating data to datasets based on transformation. Status update of file processing
Nifi-ms: Apache NiFi is used as a real-time integrated data logistics and simple event processing platform
Postgres-ms: Postgres microservice contains the schema and tables
Nginx-ms: It is commonly used as a reverse proxy and load balancer to manage incoming traffic and distribute it to slower upstream servers
Kong-ms: It is a lightweight API Gateway that secures, manages, and extends APIs and microservices.
IAM user and Role creation for S3 connectivity
...
Create a IAM policy from AWS IAM
Provide access to list, read and write the objects to s3 buckets
cQube Deployment Process
...
Step - 1: Connect to the cqube AWS ec2 instance
For linux and macOS:
Download the .pem file which is generated while creating the EC2 instance
Open the terminal and navigate to the folder where .pem file has been downloaded
Then give the read permission to the .pem file using following command
sudo chmod 400 <aws.pem>
Use the following command to connect to the instance
ssh -i <path_to_the_pem_file> <user_name>@<public_ip_of_the_instance>
For windows:
Download the .pem file which is generated while creating the EC2 instance
Use puttygen to connect to the instance.
Refer following link to use puttyGen for connecting.
Step - 2: Clone the cqube-devops repository using following command
...
Step - 5: Install cqube with non root user with sudo privileges
sudo ./install.sh
...
Install
...
Install.sh file contains a shell script where it will run shell scripts and ansible-playbook to setup the cqube
Step - 6: User Input Variables - These are the variables which need to be entered by the user by following the Hint provided
state_name ( Enter the required state code by referring to the state list provided )
api_end_point ( Enter the url in which cqube to be configured )
s3_access_key
s3_secret_key
s3 archived bucket name
s3 error bucket name
...
Step - 7: Optional_variables- Database credentials contain default values. If the user wishes to enter their own credentials then the user should opt for yes to enter their credentials otherwise can opt for no when the question pops up
db_user_name ( Enter the postgres database username )
db_name ( Enter the postgres database name )
db_password ( Enter the postgres password )
...
Step - 8: Once the config file is generated, A preview of the config file is displayed followed by a question where the user gets an option to re enter the configuration values on choosing yes. If option no is selected then the install.sh moves to the next section.
...
Step - 9: Once the installation is completed, You will be prompted with the following messages and required reference urls.
cQube Installed Successfully
cQube ingestion api can be accessible using <domain_name>
...
Appendix
AWS - Network Architecture
The following steps define the cQube setup and workflow completion processes in AWS. cQube mainly comprises the areas mentioned below:
EC2 Server
IAM user and Role creation for S3 connectivity.
The cQube network setup process is described in the block diagram below:
Micro services Details
Following are the details of the micro-services which get installed in the cqube server.
Ingestion-ms: The ingestion-ms is used to upload the data of the events, datasets, dimensions, transformers and pipeline. All these apis will be to ingesting the data into the cQube.
Spec-ms: The spec-ms is used to import schema of the events, datasets, dimensions, transformers and pipeline. All these specs will be defined by the cQube platform prior to ingesting the data into the cQube. These specifications are derived by considering the KPIs as the Indicator.
Generator-ms: The generator-ms is used to create the specs & transformers for the derived datasets. Performed aggregation logics, updating data to datasets based on transformation. Status update of file processing
Nifi-ms: Apache NiFi is used as a real-time integrated data logistics and simple event processing platform
Postgres-ms: Postgres microservice contains the schema and tables
Nginx-ms: It is commonly used as a reverse proxy and load balancer to manage incoming traffic and distribute it to slower upstream servers
Kong-ms: It is a lightweight API Gateway that secures, manages, and extends APIs and microservices.
cQube Deployment Procedure
Install.sh sh file contains a shell script where it will run following shell scripts and ansible-playbook to setup the cqube
Basic_requirements.sh:
This This script basically updates and upgrades the software packages in the server and installs the basic softwares such as
...
Config_file_generator.sh:
This script is used to generate a configuration file which contains some constant values and few required variables should be entered by the user. Following are the variables which get added in the config file.
Note: Users should follow the Hints provided in the description and should enter the variables accordingly. If the entered value is wrong then an error message gets displayed and the user should modify the variable value accordingly.
Constant Constant Variables: These variables are auto generated
System_user_name
base_dir
Private_ip
aws_default_region
Step - 6: User Input Variables-These are variables which need to be entered by the user by following the Hint provided
state_name ( Enter the required state code by referring to the state list provided )
api_end_point ( Enter the url in which cqube to be configured )
s3_access_key
s3_secret_key
s3 archived bucket name
s3 error bucket name
...
name
Optional_variables - Database credentials contain default values. If the user wishes to enter their own credentials then the user should opt for yes to enter their credentials otherwise can opt for no when the question pops up
db_user_name ( Enter the postgres database username )
db_name ( Enter the postgres database name )
db_password ( Enter the postgres password )
...
Step - 8: Once the config file is generated, A preview of the config file is displayed followed by a question where the user gets an option to re enter the configuration values on choosing yes. If option no is selected then the install.sh moves to the next section.
Repository_clone.sh:
This script clones the following repositories in the microservices directory and checkout to the required release branch
...
cQube Installed Successfully
...
We can check the containers running status by using following command
sudo docker ps
...