Table of Content
Objective
Hardware Requirements
IAM user and Role creation for S3 connectivity
cQube Deployment Process
Appendix
AWS - Network Architecture
Microservice Details
cQube Deployment Procedure 8
Objective
This document mainly focuses on the hardware requirements/ AWS network architecture for installing cQube and the deployment process step-by-step.
...
An AWS Identity and Access Management (IAM) user is an entity that is created in AWS to represent the person or application that uses it to interact with AWS. A user in AWS contains a name and credentials. An IAM user with administrator permissions is different from the AWS account root user. One has to create an IAM user with a supported role to provide the connectivity between EC2 and S3. The role should have list, read and write permissions
S3 Buckets:
Create following s3 buckets
...
Archiving
...
.
IAM User:
Create a IAM user
Assign IAM policy to the user
Download the access key and secret key
...
state_name ( Enter the required state code by referring to the state list provided )
api_end_point ( Enter the url in which cqube to be configured )
Storage_type (Enter the storage_type as aws or local or azure. If User opting for aws You will be prompted with the following AWS s3 credentials to enter and it will create the s3 bucket. If s3_bucket already exists it will prompt you to enter a unique s3 bucket name. And it will be generated in the config.yml.)
s3_access_key
s3_secret_key
s3 archived bucket name
s3 error bucket name
...
Storage_type (Enter the storage_type as local, minio will install and configure the minio username and minio password and create the minio bucket And it will be generated in the config.yml.)
...
Storage_type (Enter the storage_type as Azure, If User opting for Azure you will be prompted with the following Azure credentials to enter and it will create an azure container , if azure container exists it will prompt to enter a unique azure container name. And it will be generated in the config.yml.)
azure_connection_string
azure_account_name
azure_account_key
...
Step - 7: Optional_variables- Database credentials contain default values. If the user wishes to enter their own credentials then the user should opt for yes to enter their credentials otherwise can opt for no when the question pops up
db_user_name ( Enter the postgres database username )
db_name ( Enter the postgres database name )
db_password ( Enter the postgres password )
...
Step - 8: Optional_variables- Read Only Database credentials contain default values. If the user wishes to enter their own credentials then the user should opt for yes to enter their credentials otherwise can opt for no when the question pops up
read_only_db_user ( Enter the read only postgres database username )
read_only_db_password ( Enter the read onlypostgres password )
Step - 9: Once the config file is generated, A preview of the config file is displayed followed by a question where the user gets an option to re enter the configuration values on choosing yes. If option no is selected then the install.sh moves to the next section.
...
Step - 910: Once the installation is completed, You will be prompted with the following messages and required reference urls.
...
cQube ingestion api can be accessible using <domain_name>
...
Appendix
AWS - Network Architecture
...
The cQube network setup process is described in the block diagram below:
Micro services Microservices Details
Following are the details of the micro-services microservices which get installed in the cqube server.
Ingestion-ms: The ingestion-ms is used to upload the data of the events, datasets, dimensions, transformers and pipeline. All these apis will be to ingesting the data into the cQube.
Spec-ms: The spec-ms is used to import schema of the events, datasets, dimensions, transformers and pipeline. All these specs will be defined by the cQube platform prior to ingesting the data into the cQube. These specifications are derived by considering the KPIs as the Indicator.
Generator-ms: The generator-ms is used to create the specs & transformers for the derived datasets. Performed aggregation logics, updating data to datasets based on transformation. Status update of file processing
Nifi-ms: Apache NiFi is used as a real-time integrated data logistics and simple event processing platform
Postgres-ms: Postgres microservice contains the schema and tables
Nginx-ms: It is commonly used as a reverse proxy and load balancer to manage incoming traffic and distribute it to slower upstream servers
Kong-ms: It is a lightweight API Gateway that secures, manages, and extends APIs and microservices.
Dashboard-ms: It consists of an angular app, it is used to visualize the datasets present in postgres-ms in the form of charts. On run time it requests spec-ms to fetch data from postgres-ms and load it into the client side(Browser)
cQube Deployment Procedure
...
state_name ( Enter the required state code by referring to the state list provided )
api_end_point ( Enter the url in which cqube to be configured )
storage_type
s3_access_key
s3_secret_key
s3 archived bucket names3 error bucket name bucket name
Optional_variables - Database credentials contain default values. If the user wishes to enter their own credentials then the user should opt for yes to enter their credentials otherwise can opt for no when the question pops up
db_user_name ( Enter the postgres database username )
db_name ( Enter the postgres database name )
db_password ( Enter the postgres password )
Optional_variables- Read Only Database credentials contain default values. If the user wishes to enter their own credentials then the user should opt for yes to enter their credentials otherwise can opt for no when the question pops up
read_only_db_user ( Enter the read only postgres database username )
read_only_db_password ( Enter the read only postgres password )
Once the config file is generated, A preview of the config file is displayed followed by a question where the user gets an option to re enter the configuration values on choosing yes. If option no is selected then the install.sh moves to the next section.
...
...
Note: If the repository is already cloned then the script will pull the updated code.
...
Ingestion-ms
Spec-ms
Generator-ms
Postgres-ms
Nifi-ms
Dashboard-ms
Query_builder-ms
Kong-ms
Nginx-ms
...
compose.yml:
A docker compose ansible script gets triggered where it will up all the containers to running state.
...
cQube Installed Successfully
...
We can check the containers running status by using following command
sudo docker ps
...