Reports@Scale

Introduction

This wiki details the architecture of enabling reporting framework to operate at scale. It discusses the high level design problems to be solved and introduces the proposed architecture for the same.

Key Design Problems

TBA

Reporting Architecture

Druid Architecture

Druid Data Model

Raw Telemetry

	Dimension in Druid	Field in Telemetry	Description	Data Type

	Dimension in Druid	Field in Telemetry	Description	Data Type
1	ets	ets	Event timestamp	Long
2	eid	eid	Event Id	String
3	syncts	syncts	Sync Timestamp	Long
4	@timstamp	@timstamp	Sync Timestamp in String	String
5	actor_id	actor.id	Actor Id of the event	String
6	actor_type	actor.type	Type of the actor	String
7	context_channel	context.channel	Channel Id	String
8	context_pdata_id	context.pdata.id	Producer Id	String
9	context_pdata_pid	context.pdata.pid	Producer Process Id	String
10	context_pdata_ver	context.pdata.ver	Producer version number	String
11	context_env	context.env	Context Environment	String
12	context_sid	context.sid	Session Id	String
13	context_did	context.did	Device Id	String
14	context_cdata_type	context.cdata.type	Correlation Data Type	Array[String]
15	context_cdata_id	context.cdata.id	Correlation Data Id	Array[String]
16	context_rollup_l1	context.rollup.l1	Context level 1 rollup	String
17	context_rollup_l2	context.rollup.l2	Context level 2 rollup	String
18	context_rollup_l3	context.rollup.l3	Context level 3 rollup	String
19	context_rollup_l4	context.rollup.l4	Context level 4 rollup	String
20	object_id	object.id	Content Id	String
21	object_type	object.type	Content Type	String
22	object_version	object.ver	Content Version	String
23	object_rollup_l1	object.rollup.l1	Content level 1 rollup	String
24	object_rollup_l2	object.rollup.l2	Content level 2 rollup	String
25	object_rollup_l3	object.rollup.l3	Content level 3 rollup	String
26	object_rollup_l4	object.rollup.l4	Content level 4 rollup	String
27	tags	tags	Tags	Array[String]
28	edata_type	edata.type	Event type	String
29	edata_subtype	edata.subtype	Event subtype	String
30	edata_mode	edata.mode	START event Mode of start	String
31	edata_pageid	edata.pageid	Unique pageid	String
32	edata_uri	edata.uri	IMPRESSION event Relative URI of the content	String
33	edata_id	edata.id	Event data Id	String
34	edata_duration	edata.duration	Duration of the event	Double
35	edata_index	edata.index	ASSESS event Index of the question within a content	Integer
36	edata_pass	edata.pass	ASSESS event Field to identify pass or fail for assessments	String
37	edata_score	edata.score	ASSESS event Assessment score	Double
38	edata_resvalues	edata.resvalues	ASSESS event Assessment results	Array[Object]
39	edata_item_id	edata.item.id	ASSESS event Assessment item id	String
40	edata_item_title	edata.item.title	ASSESS event Assessment item title	String
41	edata_item_maxscore	edata.item.maxscore	ASSESS event Assessment item max score	Double
42	edata_target_id	edata.target.id	ASSESS event Assessment item target id	String
43	edata_target_type	edata.target.type	ASSESS event Assessment item target type	String
44	edata_rating	edata.rating	FEEDBACK event Ratings	Integer
45	edata_comments	edata.comments	FEEDBACK event Comments	String
46	edata_dir	edata.dir	SHARE event direction	String
47	edata_items_id	edata.items.id	SHARE event shared item ids	Array[String]
48	edata_items_type	edata.items.type	SHARE item types	Array[String]
49	edata_items_origin_id	edata.items.origin.id	SHARE event source id	Array[String]
50	edata_items_origin_type	edata.items.origin.type	SHARE event source type	Array[String]
51	edata_items_to_id	edata.items.to.id	SHARE event destination id	Array[String]
52	edata_items_to_type	edata.items.to.type	SHARE event destination type	Array[String]
53	edata_state	edata.state	AUDIT event current state	String
54	edata_prevstate	edata.prevstate	AUDIT event previous state	String
55	edata_size	edata.size	SEARCH event result size	Integer
56	edata_filters_dialcodes	edata.filters.dialcodes	SEARCH event List of dialcodes	Array[String]
57	edata_topn_identifier	edata.topn.identifier	SEARCH event topn results	Array[String]
58	edata_visits_objid	edata.visits.objid	IMPRESSION event unique id for object visited	Array[String]
59	edata_visits_objtype	edata.visits.objtype	IMPRESSION event type of object visited	Array[String]
60	edata_visits_objver	edata.visits.objver	IMPRESSION event version of object visited	Array[String]
61	edata_visits_index	edata.visits.index	IMPRESSION event index of object within list	Array[Integer]
62	device_loc_state	devicedata.state	State location information for the device	String
63	device_loc_state_code	devicedata.statecode	State ISO code information for the device	String
64	device_loc_iso_state_code	devicedata.iso3166statecode	State ISO-3166 code information for the device	String
65	device_loc_city	devicedata.city	City location information for the device	String
66	device_loc_country_code	devicedata.countrycode	Country ISO code information for the device	String
67	device_loc_country	devicedata.country	Country location information for the device	String
68	device_os	devicedata.os	Device OS name	String
69	device_make	devicedata.make	Device make and model	String
70	device_id	devicedata.id	Physical device id if available from OS	String
71	device_mem	devicedata.mem	Total memory in mb	Integer
72	device_idisk	devicedata.idisk	Total interanl disk	Integer
73	device_edisk	devicedata.edisk	Total external disk	Integer
74	device_scrn	devicedata.scrn	Screen size in inches	Integer