Problem Statement 1:
...
Use Elastic search scroll api . 'Scroll API ' can be used to retrieve large numbers of results (or even all results) from a single search request, it will work in same way as cursor on a traditional database.
Pros | Cons |
|---|---|
We can retrieve large data set | We can not use scroll api for real time user request |
We can slice the data based upon shards | Performance issues while using it for real time request |
...
| Code Block |
|---|
Path: /{{IndexName}}/{{type}}/_search?scroll=1m
Request Data{
"query": {//Contains the query required to fetch the data
},
"size" : 1000,
}
Returns → {"scrollId":"SCROLL ID"hits:["data"]}After receiving the scroll Id IdWeWe need to send this request till we get all the resultPath:
/_search/scroll{
"scroll": "1m",
"scroll_id":"Scroll id" // received in the previous request
}
Returns {
"_scroll_id": "Scroll Id",
"hits": {
"total": 263,
"max_score": 0.11207403,
"hits": [
{data}//result data from scroll api
]
}
}
|
Approach 1:
We can't start the service instantly or we can generate the batch metrics by running this service once in a day, it should be Async process , and process id need to be track. This process will generate file and upload to some storage and link will be share to user on email. second time we might use same file for particular date range : Ex , if user request for stats for a course batch and for that course batch report is already generated and report validity time not expire then we can re-use it , instead of re-generating.
...
Using relation database management system. RDBMS provide both pagination and sorting available.
Pros | Cons |
|---|---|
We It can handle large data set | We have to manage a new DB |
Syncing data will be an issue |
...
Using filter inside the current implementation as reading more than 10 thousand records is not feasible by user.
Filters that can be added
Filtering based on userName
Filtering based up on enrolled date (eg:- between start and end date)
Filtering based upon progress status ( eg:- user progress between 20-40% )
Pros | Cons |
|---|---|
We just need to handle the extra query parameters | If after adding applying search filters, the result data is more than 10000 results, then user will not be able to read all the data |
Solution 3:
Use Elastic search scroll api . 'Scroll API ' can be used to retrieve large numbers of results (or even all results) from a single search request, it will work in same way as cursor on a traditional database.
Pros | Cons |
|---|---|
Can fetch data more than 10000 | Performance issues while using it for real time request |