Page Comparison

Introduction

This wiki details the architecture of enabling reporting framework to operate at scale. It discusses the high level design problems to be solved and introduces the proposed architecture for the same. 

Key Design Problems

TBA

...

Reporting Architecture

Image Added

...

Druid Architecture

...

Image Added

...

Druid Data

...

Model

Raw Telemetry

...

Telemetry Event Fields

Summary Events

Dimensions

...

Dimension in Druid

...

Field in Summary event

...

Description

...

Data Type

...

eid

...

eid

...

Event Id

...

String

...

ver

...

ver

...

Version

...

String

...

syncts

...

syncts

...

Sync timestamp

...

Long

...

uid

...

uid

...

User Id

...

String

...

context_date_range_from

...

context.date_range.from

...

Start Date for the summary

...

String

...

context_date_range_to

...

context.date_range.to

...

End Date for the summary

...

String

...

context_rollup_l1

...

context.rollup.l1

...

Context level1 rollup

...

String

...

context_rollup_l2

...

context.rollup.l2

...

Context level2 rollup

...

String

...

context_rollup_l3

...

context.rollup.l3

...

Context level3 rollup

...

String

...

context_rollup_l4

...

context.rollup.l4

...

Context level4 rollup

...

String

...

channel_id

...

dimensions.channel

...

Channel Id as dimension from raw telemetry

...

String

...

device_id

...

dimensions.did

...

Device Id as dimension from raw telemetry

...

String

...

producer_id

...

dimensions.pdata.id

...

Producer Id as dimension from raw telemetry

...

String

...

producer_pid

...

dimensions.pdata.pid

...

Producer Process Id as dimension from raw telemetry

...

String

...

session_id

...

dimensions.sid

...

Session Id as dimension

...

String

...

session_type

...

dimension.type

...

Type of summary

...

String

...

session_mode

...

dimension.mode

...

Mode of action in the session

...

String

...

object_id

...

object.id

...

Content Id

...

String

...

object_type

...

object.type

...

Content Type

...

String

...

object_type

...

object.type

...

Content Type

...

String

...

object_version

...

object.ver

...

Content version

...

String

...

object_rollup_l1

...

object.rollup.l1

...

Object level1 rollup

...

String

...

object_rollup_l2

...

object.rollup.l2

...

Object level2 rollup

...

String

...

object_rollup_l3

...

object.rollup.l3

...

Object level3 rollup

...

String

...

object_rollup_l4

...

object.rollup.l4

...

Object level4 rollup

...

String

...

time_spent

...

edata.eks.time_spent

...

Time spent in the session excluding idle time

...

String

...

time_difference

...

edata.eks.time_diff

...

Total time in a session including idle time

...

String

...

interaction_count

...

edata.eks.interact_events_count

...

Total count of interact events in a session

...

Long

...

summary_env

...

edata.eks.env_summary.env

...

High level env within the app (content, domain, resources, community)

...

String

...

summary_env_count

...

edata.eks.env_summary.count

...

Count of times the environment has been visited

...

Integer

...

summary_env_time_spent

...

edata.eks.env_summary.time_spent

...

Time spent per env

...

Double

...

summary_page_id

...

edata.eks.page_summary.id

...

Page id

...

String

...

summary_page_type

...

edata.eks.page_summary.type

...

Type of page e.g. view/edit

...

String

...

summary_page_visit_count

...

edata.eks.page_summary.visit_count

...

Number of times each page was visited

...

String

...

summary_page_time_spent

...

edata.eks.page_summary.time_spent

...

Time taken per page

...

Double

...

item_responses_item_id

...

edata.eks.item_responses.itemId

...

Question Id passed in the ASSESS event

...

String

...

item_responses_time_spent

...

edata.eks.item_responses.timeSpent

...

Time spent in seconds from ASSESS event

...

String

...

item_responses_pass

...

edata.eks.item_responses.pass

...

Pass response for a question from ASSESS event

...

String

...

item_responses_score

...

edata.eks.item_responses.score

...

Score from ASSESS event

...

Array[Integer]

...

item_responses_max_score

...

edata.eks.item_responses.maxScore

...

Max Score from ASSESS event

...

Array[Integer]

...

item_responses_timestamp

...

edata.eks.item_responses.time_stamp

...

Timestamp for each response from ASSESS event

...

String

...

tags

...

tags

...

Tags attached to a summary event

...

Array[String]

Aggregates

Summary Events

Aggregates

Granularity → DAY

...

Report JSON Spec

JSON Schema

[{
    id: String, // Required. Report ID.
    label: String, // Required. Report Label (will be shown up as menu)
    title: String, // Optional. Report title. Defaults to report label
    description: String, // Optional. Report description. HTML text can be included as description
    dataSource: String, // Required. Location of the data source to show the report. Can be an expression. For ex: /<report_id>/{{channel}}/report.json
    charts: [{ // Optional
 		datasets: [{
			data: Array[Number], // Required if `dataExpr` is not provided. Array of Number. Data points to show in the chart
			dataExpr: String, // Required if `data` is not provided. Expression pointing to the data in dataSource. For ex: {{data.noOfDownloads}}
			label: String // Required. Label to display on the chart
		}],
		labels: Array[String], // Required if `labelsExpr` is not provided. Labels to show on the x-axis
		labelsExpr: String, // Required if `labels` is not provided. Expression pointing to the data in dataSource. For ex: {{data.Date}}
		chartType: String, // Optional. Defaults to line. Available types - line, bar, radar, pie, polarArea & doughnut
		colors: [""], // Optional. Color to show for each dataset. Defaults to ["#024F9D"].
		options: { // Optional. options for display. Full set of options look at https://valor-software.com/ng2-charts/
			responsive: Boolean, // Defaults to true
			...
		}, 
		legend: Boolean // Optional. Whether to show the legend below/above the chart. Defaults to true and position to top.
    }],
    table: { // Optional
        "columns": Array[String], // Required if `columnsExpr` is not provided. Columns to show.
        "values": Array[Array[String]], // Required if `valuesExpr` is not provided. Column data.
        "columnsExpr": String, // Required if `columns` is not provided. Expression pointing to the data in dataSource. For ex: {{keys}}
        "valuesExpr": String // Required if `values` is not provided. Expression pointing to the data in dataSource. For ex: {{tableData}}
    },
    downloadUrl: String // Location to download the data as CSV
}]

Following is a example schema to show the general usage report 

{
    id: "usage",
    label: "Diksha Usage Report",
    title: "Diksha Usage Report",
    description: "The report provides a quick summary of the data analysed by the analytics team to track progess of Diksha across states. This report will be used to consolidate insights using various metrics on which Diksha is currently being mapped and will be shared on a weekly basis. The first section of the report will provide a snapshot of the overall health of the Diksha App. This will be followed by individual state sections that provide state-wise status of Diksha",
    dataSource: "/usage/$state/report.json",
    charts: [
        {
        	datasets: [{
        		dataExpr: "data.Number_of_downloads",
        		label: "# of downloads"
        	}],
        	labelsExpr: "data.Date",
            chartType: "line"
        },
        {
        	datasets: [{
        		dataExpr: "data.Number_of_succesful_scans",
        		label: "# of successful scans"
        	}],
        	labelsExpr: "data.Date",
            chartType: "bar"
        }
    ],
    table: {
        "columnsExpr": "key",
        "valuesExpr": "tableData"
    },
    downloadUrl: "<report_id>/$state/report.csv"
}