Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Introduction

This wiki details the architecture of enabling reporting framework to operate at scale. It discusses the high level design problems to be solved and introduces the proposed architecture for the same. 

Key Design Problems

TBA

...

Reporting Architecture

...

Druid Architecture

...

Druid Data Model

Raw Telemetry

314objectidobjectidContent Id15objecttypeobjecttypeContent Type16objectversionobjectverContent VersionBlock info of the user


Dimension in Druid

Field in Telemetry

Description

Data Type

1etsetsEvent timestampLong
2

eid

eid

Event Id

String

23

syncts

syncts

Sync Timestamp

Long

4@timstamp@timstampSync Timestamp in StringString
5

actor_id

actor.id

Actor Id of the event

String

46

actor_type

actor.type

Type of the actor

String

57

context_channel

context.channel

Channel Id

String

68

context_pdata_id

context.pdata.id

Producer Id

String

79

context_pdata_pid

context.pdata.pid

Producer Process Id

String

810context_pdata_vercontext.pdata.verProducer version numberString
911

context_env

context.env

Context Environment

String

1012

context_sid

context.sid

Session Id

String

1113

context_did

context.did

Device Id

String

1214

context_cdata_type

context.cdata.type

Correlation Data Type

Array[String]

1315

context_cdata_id

context.cdata.id

Correlation Data Id

Array[String]

16context_rollup_l1context.rollup.l1Context level 1 rollupString
17context_rollup_l2context.rollup.l2Context level 2 rollupString
18context_rollup_l3context.rollup.l3Context level 3 rollupString
17

tags

tags

Tags

Array[String]

18

edata_type

edata.type

Event type

String

19

edata_subtype

edata.subtype

Event subtype

String

20

edata_mode

edata.mode

START event Mode of start

String

21

edata_pageid

edata.pageid

Unique pageid

String

22

edata_uri

edata.uri

IMPRESSION event Relative URI of the content

String

23

edata_id

edata.id

Event data Id

String

24

edata_duration

edata.duration

Duration of the event

String

25

edata_index

edata.index

ASSESS event Index of the question within a content

String

26

edata_pass

edata.pass

ASSESS event Field to identify pass or fail for assessments

String

27

edata_score

edata.score

ASSESS event Assessment score

Double

28

edata_resvalues

edata.resvalues

ASSESS event Assessment results

Array[Object]

29

edata_item_id

edata.item.id

ASSESS event Assessment item id

String

30

edata_item_title

edata.item.title

ASSESS event Assessment item title

String

31

edata_item_maxscore

edata.item.maxscore

ASSESS event Assessment item max score

Double

32

edata_target_id

edata.target.id

ASSESS event Assessment item target id

String

33

edata_target_type

edata.target.type

ASSESS event Assessment item target type

String

34

edata_rating

edata.rating

FEEDBACK event Ratings

String

35

edata_comments

edata.comments

FEEDBACK event Comments

String

36

edata_dir

edata.dir

SHARE event direction

String

37

edata_items_id

edata.items.id

SHARE event shared item ids

String

38

edata_items_type

edata.items.type

SHARE item types

String

39

edata_items_origin_id

edata.items.origin.id

SHARE event source id

String

40

edata_items_origin_type

edata.items.origin.type

SHARE event source type

String

41

edata_items_to_id

edata.items.to.id

SHARE event destination id

String

42

edata_items_to_type

edata.items.to.type

SHARE event destination type

String

43

edata_state

edata.state

AUDIT event current state

String

44

edata_prevstate

edata.prevstate

AUDIT event previous state

String

45

edata_size

edata.size

SEARCH event result size

Integer

46

edata_filters_dialcodes

edata.filters.dialcodes

SEARCH event List of dialcodes

Array[String]

47edata_topnedata.topnSEARCH event topn resultsArray[Object]
48edata_visitsedata.visitsIMPRESSION event object visit detailsArray[Object]
49

device_loc_state

ldata.state

State location information for the device

String

50

device_loc_state_code

ldata.state_code

State ISO code information for the device

String

51

device_loc_city

ldata.city

City location information for the device

String

52

device_loc_country_code

ldata.country_code

Country ISO code information for the device

String

53

device_loc_country

ldata.country

Country location information for the device

String

54device_osdevice.osDevice OS nameString
55device_makedevice.makeDevice make and modelString
56device_iddevice.idPhysical device id if available from OSString
57device_memdevice.memTotal memory in mbInt
58device_idiskdevice.idiskTotal interanl diskInt
59device_ediskdevice.ediskTotal external diskInt
60device_scrndevice.scrnScreen size in inchesInt
61device_cameradevice.cameraPrimary & secondary camera specString
62device_cpudevice.cpuProcessor nameString
63device_simsdevice.simsNumber of sim cardsInt
64device_ua_specua_specUser agent spec of the browserMap
65device_first_accessfirst_accessFirst access of the deviceString
66content_namenameName of the contentString
67content_object_typeobjectTypeType of the contentString
68content_typecontentTypeType of the resourceString
69content_media_typemediaTypeType of the media of the resourceString
70content_languagelanguageList of languages in the contentArray[String]
71content_mediummeidumLanguage medium of the boardString
72content_mimetypemimeTypeMimetype of the resource in the contentString
73content_frameworkframeworkString
74content_boardboardBoard of affiliationString
75content_statusstatusStatus of the content - Draft, Published etc.String
76content_mimetypes_countmimeTypesCountCount grouped by mimetypesJSON Object
77content_types_countcontentTypesCountCounts grouped by each content typeJSON Object
78content_versionversionKeyVersion of the contentDouble
79content_last_submitted_onlastSubmittedOnLast submitted date of the contentString
80content_last_published_onlastPublishedOnLast submitted date of the contentString
81user_first_namefirstnameFirst name of the userString
82user_last_namelastnameLastname of the userString
83user_grade_listgrade_listList of grades taughtArray[String]
84user_language_listlanguage_listList of languages knownArray[String]
85user_subject_listsubject_listList of subjects taughtArray[String]
86user_nameusernameUsername of the userString
87user_typeuserTypeType of userString
88user_loc_stateuser_stateState info of the userString
89user_loc_districtuser_districtDistrict info of the userString
90user_loc_blockuser_block19context_rollup_l4context.rollup.l4Context level 4 rollupString
20

object_id

object.id

Content Id

String

21

object_type

object.type

Content Type

String

22

object_version

object.ver

Content Version

String

23object_rollup_l1object.rollup.l1Content level 1 rollupString
24object_rollup_l2object.rollup.l2Content level 2 rollupString
25object_rollup_l3object.rollup.l3Content level 3 rollupString
26object_rollup_l4object.rollup.l4Content level 4 rollupString
27

tags

tags

Tags

Array[String]

28

edata_type

edata.type

Event type

String

29

edata_subtype

edata.subtype

Event subtype

String

30

edata_mode

edata.mode

START event Mode of start

String

31

edata_pageid

edata.pageid

Unique pageid

String

32

edata_uri

edata.uri

IMPRESSION event Relative URI of the content

String

33

edata_id

edata.id

Event data Id

String

34

edata_duration

edata.duration

Duration of the event

Double

35

edata_index

edata.index

ASSESS event Index of the question within a content

Integer

36

edata_pass

edata.pass

ASSESS event Field to identify pass or fail for assessments

String

37

edata_score

edata.score

ASSESS event Assessment score

Double

38

edata_resvalues

edata.resvalues

ASSESS event Assessment results

Array[Object]

39

edata_item_id

edata.item.id

ASSESS event Assessment item id

String

40

edata_item_title

edata.item.title

ASSESS event Assessment item title

String

41

edata_item_maxscore

edata.item.maxscore

ASSESS event Assessment item max score

Double

42

edata_target_id

edata.target.id

ASSESS event Assessment item target id

String

43

edata_target_type

edata.target.type

ASSESS event Assessment item target type

String

44

edata_rating

edata.rating

FEEDBACK event Ratings

Integer

45

edata_comments

edata.comments

FEEDBACK event Comments

String

46

edata_dir

edata.dir

SHARE event direction

String

47

edata_items_id

edata.items.id

SHARE event shared item ids

Array[String]

48

edata_items_type

edata.items.type

SHARE item types

Array[String]

49

edata_items_origin_id

edata.items.origin.id

SHARE event source id

Array[String]

50

edata_items_origin_type

edata.items.origin.type

SHARE event source type

Array[String]

51

edata_items_to_id

edata.items.to.id

SHARE event destination id

Array[String]

52

edata_items_to_type

edata.items.to.type

SHARE event destination type

Array[String]

53

edata_state

edata.state

AUDIT event current state

String

54

edata_prevstate

edata.prevstate

AUDIT event previous state

String

55

edata_size

edata.size

SEARCH event result size

Integer

56

edata_filters_dialcodes

edata.filters.dialcodes

SEARCH event List of dialcodes

Array[String]

57edata_topn_identifieredata.topn.identifierSEARCH event topn resultsArray[String]
58edata_visits_objidedata.visits.objidIMPRESSION event unique id for object visitedArray[String]
59edata_visits_objtypeedata.visits.objtypeIMPRESSION event type of object visitedArray[String]
60edata_visits_objveredata.visits.objverIMPRESSION event version of object visitedArray[String]
61edata_visits_indexedata.visits.indexIMPRESSION event index of object within listArray[Integer]
62

device_loc_state

devicedata.state

State location information for the device

String

63

device_loc_state_code

devicedata.statecode

State ISO code information for the device

String

64

device_loc_iso_state_code

devicedata.iso3166statecodeState ISO-3166 code information for the deviceString
65

device_loc_city

devicedata.city

City location information for the device

String

66

device_loc_country_code

devicedata.countrycode

Country ISO code information for the device

String

67

device_loc_country

devicedata.country

Country location information for the device

String

68device_osdevicedata.osDevice OS nameString
69device_makedevicedata.makeDevice make and modelString
70device_iddevicedata.idPhysical device id if available from OSString
71device_memdevicedata.memTotal memory in mbInteger
72device_idiskdevicedata.idiskTotal interanl diskInteger
73device_ediskdevicedata.ediskTotal external diskInteger
74device_scrndevicedata.scrnScreen size in inchesInteger
75device_cameradevicedata.cameraPrimary & secondary camera specString
76device_cpudevicedata.cpuProcessor nameString
77device_simsdevicedata.simsNumber of sim cardsInteger
78device_uaspec_agentdevicedata.uaspec.agentUser agent of the browserString
79device_uaspec_verdevicedata.uaspec.verUser agent version of the browserString
80device_uaspec_systemdevicedata.uaspec.systemUser agent system identification of the browserString
81device_uaspec_platformdevicedata.uaspec.platformUser agent client platform of the browserString
82device_uaspec_rawdevicedata.uaspec.rawRaw user agent of the browserString
83device_first_accessdevicedata.firstaccessFirst access of the deviceLong (Epoch)
84content_namecontentdata.nameName of the contentString
85content_object_typecontentdata.objecttypeType of the contentString
86content_typecontentdata.contenttypeType of the resourceString
87content_media_typecontentdata.mediatypeType of the media of the resourceString
88content_languagecontentdata.languageList of languages in the contentArray[String]
89content_mediumcontentdata.mediumLanguage medium of the boardArray[String]
90content_mimetypecontentdata.mimetypeMimetype of the resource in the contentString
91content_frameworkcontentdata.framework
String
92content_boardcontentdata.boardBoard of affiliationString
93content_statuscontentdata.statusStatus of the content - Draft, Published etc.String
94content_versioncontentdata.pkgversionVersion of the contentDouble
95content_last_submitted_oncontentdata.lastsubmittedonLast submitted date of the contentLong (Epoch)
96content_last_published_oncontentdata.lastpublishedonLast submitted date of the contentLong (Epoch)
97content_last_updated_oncontentdata.lastupdatedonLast updated date of the contentLong (Epoch)
98user_grade_listuserdata.gradelistList of grades taughtArray[String]
99user_language_listuserdata.languagelistList of languages knownArray[String]
100user_subject_listuserdata.subjectlistList of subjects taughtArray[String]
101user_typeuserdata.typeType of userString
102user_loc_stateuserdata.stateState info of the userString
103user_loc_districtuserdata.districtDistrict info of the userString
104user_loc_blockuserdata.blockBlock info of the userString
105dialcode_channeldialcodedata.channelChannel for which dialcode is generatedString
106dialcode_batchcodedialcodedata.batchcodeBatch for which dialcode belongs toString
107dialcode_publisherdialcodedata.publisherPublisher of the dialcodeString
108dialcode_generated_ondialcodedata.generatedonDialcode generated onLong (Epoch)
109dialcode_published_ondialcodedata.publishedonDialcode published onLong (Epoch)
110dialcode_statusdialcodedata.statusStatus of the dialcodeString
111dialcode_object_typedialcodedata.objecttypeObject type - DialCode as a valueString

Summary Events

6715edata_page_summary_id34353637394043727477


Dimension in Druid

Field in Summary event

Description

Data Type

1etsetsEvent timestampLong
2

eid

eid

Event Id

String

23

ver

ver

Version

String

34

syncts

syncts

Sync timestamp

Long

45

uid

uid

User Id

String

56

context_date_range_from

context.date_range.from

Start Date for the summary

String

Long (Epoch)

7

context_date_range_to

context.date_range.to

End Date for the summary

String

Long (Epoch)

8

context_rollup_l1

context.rollup.l1

Context level1 rollup

String

89

context_rollup_l2

context.rollup.l2

Context level2 rollup

String

910

context_rollup_l3

context.rollup.l3

Context level3 rollup

String

1011

context_rollup_l4

context.rollup.l4

Context level4 rollup

String

1112

dimension_channel

dimensions.channel

Channel Id as dimension from raw telemetry

String

1213

dimensions_did

dimensions.did

Device Id as dimension from raw telemetry

String

1314

dimensions_pdata_id

dimensions.pdata.id

Producer Id as dimension from raw telemetry

String

1415

dimensions_pdata_pid

dimensions.pdata.pid

Producer Process Id as dimension from raw telemetry

String

16dimensions_pdata_verdimensions.pdata.verProducer Process Ver as dimension from raw telemetryString
17

dimensions_sid

dimensions.sid

Session Id as dimension

String

1618

dimensions_type

dimensions.type

Type of summary

String

1719

dimensions_mode

dimensions.mode

Mode of action in the session

String

18

object_id

object.id

Content Idsession

String

1920

object_typeid

object.typeid

Content TypeId

String

2021

object_type

object.type

Content Type

String

2122

object_version

object.ver

Content version

String

2223

object_rollup_l1

object.rollup.l1

Object level1 rollup

String

2324

object_rollup_l2

object.rollup.l2

Object level2 rollup

String

24

object_rollup_l3

object.rollup.l3

Object level3 rollup

String

25

object_rollup_l4

object.rollup.l4

Object level4 rollup

String

26

tags

tags

Tags attached to a summary event

Array[String]

27

edata_time_spent

edata.eks.time_spent

Time spent in the session excluding idle time

String

28

edata_time_difference

edata.eks.time_diff

Total time in a session including idle time

String

29

edata_interaction_count

edata.eks.interact_events_count

Total count of interact events in a session

Long

30

edata_summary_env

edata.eks.env_summary.env

High level env within the app

(content, domain, resources, community)

String

31

edata_env_summary_count

edata.eks.env_summary.count

Count of times the environment has been visited

Integer

32

edata_env_summary_time_spent

edata.eks.env_summary.time_spent

Time spent per env

Double

33

Object level2 rollup

String

25

object_rollup_l3

object.rollup.l3

Object level3 rollup

String

26

object_rollup_l4

object.rollup.l4

Object level4 rollup

String

27

tags

tags

Tags attached to a summary event

Array[String]

28

edata_time_spent

edata.eks.time_spent

Time spent in the session excluding idle time

Double

29

edata_time_difference

edata.eks.time_diff

Total time in a session including idle time

Double

30

edata_interaction_count

edata.eks.interact_events_count

Total count of interact events in a session

Long

31

edata_env_summary_env

edata.eks.env_summary.env

High level env within the app

(content, domain, resources, community)

Array[String]

32

edata_env_summary_count

edata.eks.env_summary.count

Count of times the environment has been visited

Array[Integer]

33

edata_env_summary_time_spent

edata.eks.env_summary.time_spent

Time spent per env

Array[Double]

34

edata_page_summary_id

edata.eks.page_summary.id

Page id

Array[String]

35

edata_page_summary_type

edata.eks.page_summary.type

Type of page e.g. view/edit

Array[String]

36edata_page_summary_envedata.eks.page_summary.envEnv of pageArray[String]
37

edata_page_summary_visit_count

edata.eks.page_summary.id

Page id

String

visit_count

Number of times each page was visited

Array[Integer]

38

edata_page_summary_time_typespent

edata.eks.page_summary.type

Type of page e.g. view/edit

String

time_spent

Time taken per page

Array[Double]

39

edata_pageitem_summaryresponses_visititem_countid

edata.eks.pageitem_summaryresponses.visit_count

Number of times each page was visited

String

itemId

Question Id passed in the ASSESS event

Array[String]

40

edata_pageitem_summaryresponses_time_spent

edata.eks.pageitem_summaryresponses.time_spenttimeSpent

Time taken per page

Double

spent in seconds from ASSESS event

Array[Double]

41

edata_item_responses_item_idpass

edata.eks.item_responses.itemIdQuestion Id passed in the pass

Pass response for a question from ASSESS event

Array[String]

3842

edata_item_responses_time_spentscore

edata.eks.item_responses.timeSpentTime spent in seconds score

Score from ASSESS event

String

Array[Integer]

43

edata_item_responses_max_passscore

edata.eks.item_responses.passPass response for a question maxScore

Max Score from ASSESS event

String

Array[Integer]

44

edata_item_responses_scoretimestamp

edata.eks.item_responses.scoreScore time_stamp

Timestamp for each response from ASSESS event

Array[Integer]

41

edata_item_responses_max_score

edata.eks.item_responses.maxScore

Max Score from ASSESS event

Array[Integer]

42

edata_item_responses_timestamp

edata.eks.item_responses.time_stamp

Timestamp for each response from ASSESS event

String

Long (Epoch)]

45

device_loc_state

devicedata.state

State location information for the device

String

46

device_loc_state_code

devicedata.statecode

State ISO code information for the device

String

47

device_loc_iso_state_code

devicedata.iso3166statecodeState ISO-3166 code information for the deviceString
48

device_loc_statecity

ldatadevicedata.statecity

State City location information for the device

String

4449

device_loc_statecountry_codeldata

devicedata.state_codecountrycode

State Country ISO code information for the device

String

4550

device_loc_citycountry

ldatadevicedata.citycountry

City Country location information for the device

String

4651device_loc_country_code

ldata.country_code

Country ISO code information for the device

String

47

device_loc_country

ldata.country

Country location information for the device

String

48device_osdevice.osDevice OS nameString
49device_makedevice.makeDevice make and modelString
50device_iddevice.idPhysical device id if available from OSString
51device_memdevice.memTotal memory in mbInt
52device_idiskdevice.idiskTotal interanl diskInt
53device_ediskdevice.ediskTotal external diskInt
54device_scrndevice.scrnScreen size in inchesInt
55device_cameradevice.cameraPrimary & secondary camera specString
56device_cpudevice.cpuProcessor nameString
57device_simsdevice.simsNumber of sim cardsInt
58device_ua_specua_specUser agent spec of the browserMap
59device_first_accessfirst_accessFirst access of the deviceString
60content_nameosdevicedata.osDevice OS nameString
52device_makedevicedata.makeDevice make and modelString
53device_iddevicedata.idPhysical device id if available from OSString
54device_memdevicedata.memTotal memory in mbInteger
55device_idiskdevicedata.idiskTotal interanl diskInteger
56device_ediskdevicedata.ediskTotal external diskInteger
57device_scrndevicedata.scrnScreen size in inchesInteger
58device_cameradevicedata.cameraPrimary & secondary camera specString
59device_cpudevicedata.cpuProcessor nameString
60device_simsdevicedata.simsNumber of sim cardsInteger
61device_uaspec_agentdevicedata.uaspec.agentUser agent of the browserString
62device_uaspec_verdevicedata.uaspec.verUser agent version of the browserString
63device_uaspec_systemdevicedata.uaspec.systemUser agent system identification of the browserString
64device_uaspec_platformdevicedata.uaspec.platformUser agent client platform of the browserString
65device_uaspec_rawdevicedata.uaspec.rawRaw user agent of the browserString
66device_first_accessdevicedata.firstaccessFirst access of the deviceLong (Epoch)
67content_namecontentdata.nameName of the contentString
6168content_object_typeobjectTypecontentdata.objecttypeType of the contentString
6269content_typecontentTypecontentdata.contenttypeType of the resourceString
6370content_media_typecontentdata.mediatypemediaTypeType of media of the contentString
6471content_languagecontentdata.languageList of languagesArray[String]
6572content_mediumcontentdata.mediumLanguage medium of the boardString
6673content_mimetypemimeTypecontentdata.mimetypeMimetype of the contentString
6774content_frameworkcontentdata.framework
String
6875content_boardcontentdata.boardBoard of affiliationString
6976content_statuscontentdata.statusStatus of the content - Draft, Published etc.String
70content_mimetypes_countmimeTypesCountCount grouped by mimetypesJSON Object
71content_content_types_countcontentTypesCountCounts grouped by each content typeJSON Object
the content - Draft, Published etc.String
77content_versionversionKeycontentdata.pkgversionVersion of the contentDouble
7378content_last_submitted_onlastSubmittedOncontentdata.lastsubmittedonLast submitted date of the contentStringLong (Epoch)
79content_last_published_onlastPublishedOncontentdata.lastpublishedonLast submitted published date of the contentString
75user_first_namefirstnameFirst name of the userString
76user_last_namelastnameLast name of the userString
Long (Epoch)
80content_last_updated_oncontentdata.lastupdatedonLast updated date of the contentLong (Epoch)
81user_grade_listgrade_listuserdata.gradelistList of grades taughtArray[String]
7882user_language_listlanguage_listuserdata.languagelistList of languages knownArray[String]
7983user_subject_listsubject_listuserdata.subjectlistList of subjects taughtArray[String]
8084user_nameusernameUser name of the userString81user_typeusertypetypeuserdata.typeType of the userString
8285user_loc_stateuser_userdata.stateState info of the userString
8386user_loc_districtuser_userdata.districtDistrict info of the userString
8487user_loc_blockuser_userdata.blockBlock info of the userString
Aggregates

Granularity → DAY

Druid field name

Druid source field

Aggregate Type

total_interactions

interaction_count

SUM

total_time_spent

time_spent

SUM

total_sessionsmidCOUNT





...

Report JSON Spec

JSON Schema

Code Block
languagejs
[{
    id: String, // Required. Report ID.
    label: String, // Required. Report Label (will be shown up as menu)
    title: String, // Optional. Report title. Defaults to report label
    description: String, // Optional. Report description. HTML text can be included as description
    dataSource: String, // Required. Location of the data source to show the report. Can be an expression. For ex: /<report_id>/{{channel}}/report.json
    charts: [{ // Optional
 		datasets: [{
			data: Array[Number], // Required if `dataExpr` is not provided. Array of Number. Data points to show in the chart
			dataExpr: String, // Required if `data` is not provided. Expression pointing to the data in dataSource. For ex: {{data.noOfDownloads}}
			label: String // Required. Label to display on the chart
		}],
		labels: Array[String], // Required if `labelsExpr` is not provided. Labels to show on the x-axis
		labelsExpr: String, // Required if `labels` is not provided. Expression pointing to the data in dataSource. For ex: {{data.Date}}
		chartType: String, // Optional. Defaults to line. Available types - line, bar, radar, pie, polarArea & doughnut
		colors: [""], // Optional. Color to show for each dataset. Defaults to ["#024F9D"].
		options: { // Optional. options for display. Full set of options look at https://valor-software.com/ng2-charts/
			responsive: Boolean, // Defaults to true
			...
		}, 
		legend: Boolean // Optional. Whether to show the legend below/above the chart. Defaults to true and position to top.
    }],
    table: { // Optional
        "columns": Array[String], // Required if `columnsExpr` is not provided. Columns to show.
        "values": Array[Array[String]], // Required if `valuesExpr` is not provided. Column data.
        "columnsExpr": String, // Required if `columns` is not provided. Expression pointing to the data in dataSource. For ex: {{keys}}
        "valuesExpr": String // Required if `values` is not provided. Expression pointing to the data in dataSource. For ex: {{tableData}}
    },
    downloadUrl: String // Location to download the data as CSV
}]

...