Versions Compared

Old Version 20

changes.mady.by.user Sowmya Dixit

Saved on

compared with

New Version 21

changes.mady.by.user Anand Parthasarathy

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  1. Configure Report API - This API will be used to submit a request for configuration of a new report.
  2. Job Scheduler Engine - This Scheduler will submit the reports for execution based on execution frequency.
  3. Disable Report API - This API will mark an existing report as disabled and will be excluded from the list of reports to be executed.
  4. Report Data Generator - The report data generator will be a spark job which will generate report data file by executing the query configured in the report configuration against the druid data store. The report data file will then be exported to cloud storage to complete the report execution.


Configure Report API:

Input

...

- ReportName

- QueryEngine - druid/cassandra/ES

- ExecutionFrequency - DAILY/WEEKLY/MONTHLY

- ChannelId

- ReportInterval

- Query

- OutputFormat - json/csv

...

parameters

Parameter

Mandatory

Description

Comments

report_name

Yes

Name of the report


query_engine

Yes

Data Source

DRUID, CASSANDRA, ELASTICSEARCH

execution_frequency

Yes

Report generation frequency

DAILY, WEEKLY, MONTHLY

channel_id

No

ChannelId for filtering


report_interval

Yes

Date range for queries

  1. YESTERDAY
  2. LAST_7_DAYS,
  3. LAST_WEEK,
  4. LAST_30_DAYS,
  5. LAST_MONTH,
  6. LAST_QUARTER,
  7. LAST_3_MONTHS,
  8. LAST_6_MONTHS
  9. LAST_YEAR

query

Yes

Query to be executed


output_format

Yes

Output format of the report

json, csv

output_file_pattern

No

Report output filename pattern

Supported Placeholders are:

  1. report_name
  2. date
  3. timestamp


  • Request Object

    Code Block
    language
    themeRDark
    borderStylesolid
    linenumberstrue
    collapsefalse
     {
  "id":"sunbird.analytics.report.submit",
  "ver":"1.0",
  "ts":"2019-03-07T12:40:40+05:30",
  "params":{
     "msgid":"4406df37-cd54-4d8a-ab8d-3939e0223580",
     "client_key":"analytics-team"
  },
  "request":{
     "channel_id":"in.ekstep",
     "report_name":"avg_collection_downloads",
     "query_engine": "druid",
     "execution_frequency": "DAILY",
     "report_interval":"LAST_7_DAYS",
     "output_format": "json"
     "query_json":{
        "queryType":"groupBy",
        "dataSource":"telemetry-events",
        "granularity":"day",
        "dimensions":[
           "eid"
        ],
        "aggregations":[
           { "type":"count", "name":"context_did", fieldName":"context_did" }
        ],
        "filter":{
           "type":"and",
           "fields":[
              { "type":"selector", "name":"eid", fieldName":"IMPRESSION" },
              { "type":"selector", "name":"edata_type", fieldName":"detail" },
              { "type":"selector", "name":"edata_pageid", fieldName":"collection-detail" },
              { "type":"selector", "name":"context_pdata_id", fieldName":"prod.diksha.app" }
           ]
        },
        "postAggregations":[
           {
              "type":"arithmetic",
              "name":"avg__edata_value",
              "fn":"/",
              "fields":[
                 { "type":"fieldAccess", "name":"total_edata_value", "fieldName":"total_edata_value" },
                 { "type":"fieldAccess", "name":"rows", "fieldName":"rows" }
              ]
           }
        ],
        "intervals":[
           "2019-02-20T00:00:00.000/2019-01-27T23:59:59.000"
        ]
     }
  }
 }


  • Output:

...

  • Location and file format of query in azure:

    Code Block
    themeRDark
    borderStylesolid
    /druid-reports/query/druid/report-id.json
/druid-reports/query/cassandra/report-id.cql

    Report Intervals:

    Code Block
    themeRDark
    borderStylesolid
       YESTERDAY
   LAST_7_DAYS
   LAST_30_DAYS
   LAST_MONTH
   LAST_6_MONTHS
   LAST_YEAR
  • Available Placeholders for output file pattern:

...

themeRDark
  •


Job Scheduler Engine:



  • Input:

...