Infra Provisioning Based on Historical Data and Field Inputs

Problem:

It is expected that, by end of June, 2019, about 28 States will be on DIKSHA, and the platform needs to support all that load.

Based on historical usage data (telemetry and API logs), and the field data known to Implementation, can we get an estimate of the Infra needed?

Input Data:

Field Inputs: What is known about the profile of the State that is going to come – such as

1. Number of students enrolled in Public Schools and Demographics (distribution by Class)

2. Subject, Medium, Class

3. Textbook descriptors such as a number of QR codes, size of the Textbook etc..

Historical Data:

1. Telemetry Events

2. API Logs

Load Testing Data

1. tps VS infra configuration needed

Output Data:

1. For every Service, provide an estimate of the Infra configuration needed

Stage-1:  Filter

Task

Filter the events and logs between a certain time period, based on the Field Inputs to create a segment. The same filter needs to be applied to telemetry events and API logs.

Expected Output

Same as original event structure, except that they are filtered.

Challenges

Every Filed input might not directly translate to a filter dimension. Have to use approximations/proxies in such cases.

Stage-2a:  Group By

Task

1. For every Telemetry event type, get the "COUNT" of those events for every "HOUR" by "DAY". (All words capital case are variables). 

2. For every API log event type, get the "MAXIMUM" number of those class for every "HOUR" by "DAY". (All words capital case are variables). 

3. For each day, get NEW number of devices (or Apps installs). This will be the adoption curve for that particular state.

Expected Output

Event Time Series

API Log Time Series

Adoption Curve

Challenges

Some heuristics might have to be developed to populate the columns. Some events (SCAN) may not be directly available, but it may have to derived from low level telemetry events like INTERACT  etc... May be, we have to use work flow summarizer to get these data.

Stage-2b: Cluster

Task

Cluster the event and API logs to create several clusters. Each cluster represents a type of a load profile. For clustering, consider the "HOUR" as the dimension and "DAY" as a new instance. In this example, for each Event type cluster will have 24 data points, one at each hour. It is effectively clustering time-series data. 

Expected Output

Event Time Series Cluster

API Log Cluster

code for clustering (using python sklearn and pyspark mllib

Challenges

Plot the clusters, and see if they are making sense. Iterative on the "number" of clusters

Stage-3:  Map Number of Events to API Calls

Task

Map Events  to API Calls. Run the following Seemingly Unrelated Regression

y₁(k)  = β₀+  β₁₁(k) x₁(k)  + β₁₁(k) x₁(k)  +... + β_1m(k) x_m(k) 

y₂(k)  = β₀+  β₂₁(k) x₁(k)  + β₂₁(k) x₁(k)  +... + β_2m(k) x_m(k) 

.....

y_n(k)  = β₀+  β_n1(k) x₁(k)  + β_n1(k) x₂(k)  +... + β_nm(k) x_m(k) 

Here 

y_i(k) is the maximum number calls between kth to (k+1) hour of the i-th API. Example, y₁(1) is the maximum no of CompositeSearch calls at between 1am to 2am. 

_xi(k) is the total count of i-th event between kth to (k+1) hour. Example, x₁(2) is the total number of SCAN events  between 1am to 2am. 

code for the SUR

Challenges

Try to interpret the coefficients. Here we are assuming that:

1) Event to API call mapping does not depend on the when they are called
2) A single event can trigger multiple API calls
3) And that mapping is linear

Do those assumptions make sense, in the light of the data?

Stage-4: Load to Infra

Task

Map "concurrent calls" to actual Infra requirements. Example:

Expected Output

code for mapping, given load testing data

With the above four pieces in places, one can play with a profile of a state (inputs) and look at the intermediate data at every stage and look at the eventual the infra requirements. he mappings can be improved over time. A little UI can be built to run different scenarios, adjust the mappings until  you get confidence and  go for the kill.