Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 20 Next »

Introduction

This wiki details the architecture of enabling reporting framework to operate at scale. It discusses the high level design problems to be solved and introduces the proposed architecture for the same. 

Key Design Problems

TBA


Reporting Architecture


Druid Architecture


Druid Data Model

Raw Telemetry


Dimension in Druid

Field in Telemetry

Description

Data Type

1etsetsEvent timestampLong
2

eid

eid

Event Id

String

3

syncts

syncts

Sync Timestamp

Long

4

actor_id

actor.id

Actor Id of the event

String

5

actor_type

actor.type

Type of the actor

String

6

context_channel

context.channel

Channel Id

String

7

context_pdata_id

context.pdata.id

Producer Id

String

8

context_pdata_pid

context.pdata.pid

Producer Process Id

String

9context_pdata_vercontext.pdata.verProducer version numberString
10

context_env

context.env

Context Environment

String

11

context_sid

context.sid

Session Id

String

12

context_did

context.did

Device Id

String

13

context_cdata_type

context.cdata.type

Correlation Data Type

Array[String]

14

context_cdata_id

context.cdata.id

Correlation Data Id

Array[String]

15

object_id

object.id

Content Id

String

16

object_type

object.type

Content Type

String

17

object_version

object.ver

Content Version

String

18

tags

tags

Tags

Array[String]

19

edata_type

edata.type

Event type

String

20

edata_subtype

edata.subtype

Event subtype

String

21

edata_mode

edata.mode

START event Mode of start

String

22

edata_pageid

edata.pageid

Unique pageid

String

23

edata_uri

edata.uri

IMPRESSION event Relative URI of the content

String

24

edata_id

edata.id

Event data Id

String

25

edata_duration

edata.duration

Duration of the event

Double

26

edata_index

edata.index

ASSESS event Index of the question within a content

String

27

edata_pass

edata.pass

ASSESS event Field to identify pass or fail for assessments

String

28

edata_score

edata.score

ASSESS event Assessment score

Double

29

edata_resvalues

edata.resvalues

ASSESS event Assessment results

Array[Object]

30

edata_item_id

edata.item.id

ASSESS event Assessment item id

String

31

edata_item_title

edata.item.title

ASSESS event Assessment item title

String

32

edata_item_maxscore

edata.item.maxscore

ASSESS event Assessment item max score

Double

33

edata_target_id

edata.target.id

ASSESS event Assessment item target id

String

34

edata_target_type

edata.target.type

ASSESS event Assessment item target type

String

35

edata_rating

edata.rating

FEEDBACK event Ratings

Integer

36

edata_comments

edata.comments

FEEDBACK event Comments

String

37

edata_dir

edata.dir

SHARE event direction

String

38

edata_items_id

edata.items.id

SHARE event shared item ids

Array[String]

39

edata_items_type

edata.items.type

SHARE item types

Array[String]

40

edata_items_origin_id

edata.items.origin.id

SHARE event source id

Array[String]

41

edata_items_origin_type

edata.items.origin.type

SHARE event source type

Array[String]

42

edata_items_to_id

edata.items.to.id

SHARE event destination id

Array[String]

43

edata_items_to_type

edata.items.to.type

SHARE event destination type

Array[String]

44

edata_state

edata.state

AUDIT event current state

String

45

edata_prevstate

edata.prevstate

AUDIT event previous state

String

46

edata_size

edata.size

SEARCH event result size

Integer

47

edata_filters_dialcodes

edata.filters.dialcodes

SEARCH event List of dialcodes

Array[String]

48edata_topn_identifieredata.topn.identifierSEARCH event topn resultsArray[String]
49edata_visits_objidedata.visits.objidIMPRESSION event unique id for object visitedArray[String]
50edata_visits_objtypeedata.visits.objtypeIMPRESSION event type of object visitedArray[String]
51edata_visits_objveredata.visits.objverIMPRESSION event version of object visitedArray[String]
52edata_visits_indexedata.visits.indexIMPRESSION event index of object within listArray[Integer]
53

device_loc_state

ldata.state

State location information for the device

String

54

device_loc_state_code

ldata.state_code

State ISO code information for the device

String

55

device_loc_city

ldata.city

City location information for the device

String

56

device_loc_country_code

ldata.country_code

Country ISO code information for the device

String

57

device_loc_country

ldata.country

Country location information for the device

String

58device_osdevice.osDevice OS nameString
59device_makedevice.makeDevice make and modelString
60device_iddevice.idPhysical device id if available from OSString
61device_memdevice.memTotal memory in mbInteger
62device_idiskdevice.idiskTotal interanl diskInteger
63device_ediskdevice.ediskTotal external diskInteger
64device_scrndevice.scrnScreen size in inchesInteger
65device_cameradevice.cameraPrimary & secondary camera specString
66device_cpudevice.cpuProcessor nameString
67device_simsdevice.simsNumber of sim cardsInteger
68device_ua_specua_specUser agent spec of the browserMap
69device_first_accessfirst_accessFirst access of the deviceDate
70content_namenameName of the contentString
71content_object_typeobjectTypeType of the contentString
72content_typecontentTypeType of the resourceString
73content_media_typemediaTypeType of the media of the resourceString
74content_languagelanguageList of languages in the contentArray[String]
75content_mediummeidumLanguage medium of the boardString
76content_mimetypemimeTypeMimetype of the resource in the contentString
77content_frameworkframework
String
78content_boardboardBoard of affiliationString
79content_statusstatusStatus of the content - Draft, Published etc.String
80content_versionversionKeyVersion of the contentDouble
81content_last_submitted_onlastSubmittedOnLast submitted date of the contentDate
82content_last_published_onlastPublishedOnLast submitted date of the contentDate
83user_first_namefirstnameFirst name of the userString
84user_last_namelastnameLastname of the userString
85user_grade_listgrade_listList of grades taughtArray[String]
86user_language_listlanguage_listList of languages knownArray[String]
87user_subject_listsubject_listList of subjects taughtArray[String]
88user_nameusernameUsername of the userString
89user_typeuserTypeType of userString
90user_loc_stateuser_stateState info of the userString
91user_loc_districtuser_districtDistrict info of the userString
92user_loc_blockuser_blockBlock info of the userString

Summary Events


Dimension in Druid

Field in Summary event

Description

Data Type

1etsetsEvent timestampLong
2

eid

eid

Event Id

String

3

ver

ver

Version

String

4

syncts

syncts

Sync timestamp

Long

5

uid

uid

User Id

String

6

context_date_range_from

context.date_range.from

Start Date for the summary

Date

7

context_date_range_to

context.date_range.to

End Date for the summary

Date

8

context_rollup_l1

context.rollup.l1

Context level1 rollup

String

9

context_rollup_l2

context.rollup.l2

Context level2 rollup

String

10

context_rollup_l3

context.rollup.l3

Context level3 rollup

String

11

context_rollup_l4

context.rollup.l4

Context level4 rollup

String

12

dimension_channel

dimensions.channel

Channel Id as dimension from raw telemetry

String

13

dimensions_did

dimensions.did

Device Id as dimension from raw telemetry

String

14

dimensions_pdata_id

dimensions.pdata.id

Producer Id as dimension from raw telemetry

String

15

dimensions_pdata_pid

dimensions.pdata.pid

Producer Process Id as dimension from raw telemetry

String

16dimensions_pdata_verdimensions.pdata.verProducer Process Ver as dimension from raw telemetryString
17

dimensions_sid

dimensions.sid

Session Id as dimension

String

18

dimensions_type

dimensions.type

Type of summary

String

19

dimensions_mode

dimensions.mode

Mode of action in the session

String

20

object_id

object.id

Content Id

String

21

object_type

object.type

Content Type

String

22

object_type

object.type

Content Type

String

23

object_version

object.ver

Content version

String

24

object_rollup_l1

object.rollup.l1

Object level1 rollup

String

25

object_rollup_l2

object.rollup.l2

Object level2 rollup

String

26

object_rollup_l3

object.rollup.l3

Object level3 rollup

String

27

object_rollup_l4

object.rollup.l4

Object level4 rollup

String

28

tags

tags

Tags attached to a summary event

Array[String]

29

edata_time_spent

edata.eks.time_spent

Time spent in the session excluding idle time

Double

30

edata_time_difference

edata.eks.time_diff

Total time in a session including idle time

Double

31

edata_interaction_count

edata.eks.interact_events_count

Total count of interact events in a session

Long

32

edata_env_summary_env

edata.eks.env_summary.env

High level env within the app

(content, domain, resources, community)

Array[String]

33

edata_env_summary_count

edata.eks.env_summary.count

Count of times the environment has been visited

Array[Integer]

34

edata_env_summary_time_spent

edata.eks.env_summary.time_spent

Time spent per env

Array[Double]

35

edata_page_summary_id

edata.eks.page_summary.id

Page id

Array[String]

36

edata_page_summary_type

edata.eks.page_summary.type

Type of page e.g. view/edit

Array[String]

37edata_page_summary_envedata.eks.page_summary.envEnv of pageArray[String]
38

edata_page_summary_visit_count

edata.eks.page_summary.visit_count

Number of times each page was visited

Array[Integer]

39

edata_page_summary_time_spent

edata.eks.page_summary.time_spent

Time taken per page

Array[Double]

40

edata_item_responses_item_id

edata.eks.item_responses.itemId

Question Id passed in the ASSESS event

Array[String]

41

edata_item_responses_time_spent

edata.eks.item_responses.timeSpent

Time spent in seconds from ASSESS event

Array[Double]

42

edata_item_responses_pass

edata.eks.item_responses.pass

Pass response for a question from ASSESS event

Array[String]

43

edata_item_responses_score

edata.eks.item_responses.score

Score from ASSESS event

Array[Integer]

44

edata_item_responses_max_score

edata.eks.item_responses.maxScore

Max Score from ASSESS event

Array[Integer]

45

edata_item_responses_timestamp

edata.eks.item_responses.time_stamp

Timestamp for each response from ASSESS event

Array[Date]

46

device_loc_state

ldata.state

State location information for the device

String

47

device_loc_state_code

ldata.state_code

State ISO code information for the device

String

48

device_loc_city

ldata.city

City location information for the device

String

49

device_loc_country_code

ldata.country_code

Country ISO code information for the device

String

50

device_loc_country

ldata.country

Country location information for the device

String

51device_osdevice.osDevice OS nameString
52device_makedevice.makeDevice make and modelString
53device_iddevice.idPhysical device id if available from OSString
54device_memdevice.memTotal memory in mbInteger
55device_idiskdevice.idiskTotal interanl diskInteger
56device_ediskdevice.ediskTotal external diskInteger
57device_scrndevice.scrnScreen size in inchesInteger
58device_cameradevice.cameraPrimary & secondary camera specString
59device_cpudevice.cpuProcessor nameString
60device_simsdevice.simsNumber of sim cardsInteger
61device_ua_specua_specUser agent spec of the browserMap
62device_first_accessfirst_accessFirst access of the deviceDate
63content_namenameName of the contentString
64content_object_typeobjectTypeType of the contentString
65content_typecontentTypeType of the resourceString
66content_media_typemediaTypeType of media of the contentString
67content_languagelanguageList of languagesArray[String]
68content_mediummediumLanguage medium of the boardString
69content_mimetypemimeTypeMimetype of the contentString
70content_frameworkframework
String
71content_boardboardBoard of affiliationString
72content_statusstatusStatus of the content - Draft, Published etc.String
73content_versionversionKeyVersion of the contentDouble
74content_last_submitted_onlastSubmittedOnLast submitted date of the contentDate
75content_last_published_onlastPublishedOnLast submitted date of the contentDate
76user_first_namefirstnameFirst name of the userString
77user_last_namelastnameLast name of the userString
78user_grade_listgrade_listList of grades taughtArray[String]
79user_language_listlanguage_listList of languages knownArray[String]
80user_subject_listsubject_listList of subjects taughtArray[String]
81user_nameusernameUser name of the userString
82user_typeusertypeType of the userString
83user_loc_stateuser_stateState info of the userString
84user_loc_districtuser_districtDistrict info of the userString
85user_loc_blockuser_blockBlock info of the userString
Aggregates

Granularity → DAY

Druid field name

Druid source field

Aggregate Type

total_interactions

interaction_count

SUM

total_time_spent

time_spent

SUM

total_sessionsmidCOUNT






Report JSON Spec

JSON Schema

[{
    id: String, // Required. Report ID.
    label: String, // Required. Report Label (will be shown up as menu)
    title: String, // Optional. Report title. Defaults to report label
    description: String, // Optional. Report description. HTML text can be included as description
    dataSource: String, // Required. Location of the data source to show the report. Can be an expression. For ex: /<report_id>/{{channel}}/report.json
    charts: [{ // Optional
 		datasets: [{
			data: Array[Number], // Required if `dataExpr` is not provided. Array of Number. Data points to show in the chart
			dataExpr: String, // Required if `data` is not provided. Expression pointing to the data in dataSource. For ex: {{data.noOfDownloads}}
			label: String // Required. Label to display on the chart
		}],
		labels: Array[String], // Required if `labelsExpr` is not provided. Labels to show on the x-axis
		labelsExpr: String, // Required if `labels` is not provided. Expression pointing to the data in dataSource. For ex: {{data.Date}}
		chartType: String, // Optional. Defaults to line. Available types - line, bar, radar, pie, polarArea & doughnut
		colors: [""], // Optional. Color to show for each dataset. Defaults to ["#024F9D"].
		options: { // Optional. options for display. Full set of options look at https://valor-software.com/ng2-charts/
			responsive: Boolean, // Defaults to true
			...
		}, 
		legend: Boolean // Optional. Whether to show the legend below/above the chart. Defaults to true and position to top.
    }],
    table: { // Optional
        "columns": Array[String], // Required if `columnsExpr` is not provided. Columns to show.
        "values": Array[Array[String]], // Required if `valuesExpr` is not provided. Column data.
        "columnsExpr": String, // Required if `columns` is not provided. Expression pointing to the data in dataSource. For ex: {{keys}}
        "valuesExpr": String // Required if `values` is not provided. Expression pointing to the data in dataSource. For ex: {{tableData}}
    },
    downloadUrl: String // Location to download the data as CSV
}]

Following is a example schema to show the general usage report 

{
    id: "usage",
    label: "Diksha Usage Report",
    title: "Diksha Usage Report",
    description: "The report provides a quick summary of the data analysed by the analytics team to track progess of Diksha across states. This report will be used to consolidate insights using various metrics on which Diksha is currently being mapped and will be shared on a weekly basis. The first section of the report will provide a snapshot of the overall health of the Diksha App. This will be followed by individual state sections that provide state-wise status of Diksha",
    dataSource: "/usage/$state/report.json",
    charts: [
        {
        	datasets: [{
        		dataExpr: "data.Number_of_downloads",
        		label: "# of downloads"
        	}],
        	labelsExpr: "data.Date",
            chartType: "line"
        },
        {
        	datasets: [{
        		dataExpr: "data.Number_of_succesful_scans",
        		label: "# of successful scans"
        	}],
        	labelsExpr: "data.Date",
            chartType: "bar"
        }
    ],
    table: {
        "columnsExpr": "key",
        "valuesExpr": "tableData"
    },
    downloadUrl: "<report_id>/$state/report.csv"
}



  • No labels