Problem Statement:
Extend this design to support the download for batches stats upto 1L participants. By default elasticsearch elastic search support max 10k records , in one search query.
...
Use Elastic search scroll api . 'Scroll API ' can be used to retrieve large numbers of results (or even all results) from a single search request, it will work in same way as cursor on a traditional database.
| Pros | Cons |
|---|---|
We can retrieve large data set | We can not use scroll api for real time user request |
We can slice the data based upon shards | Performance issues while using it for real time request |
Example:
| Code Block |
|---|
Path: /{{IndexName}}/{{type}}/_search?scroll=1m
Request Data{
"query": {//Contains the query required to fetch the data
},
"size" : 1000,
}Returns → {"scrollId":"SCROLL ID"hits:["data"]}After receiving the scroll IdWe need to send this request till we get all the resultPath: /_search/scroll{
"scroll": "1m",
"scroll_id":"Scroll id" // received in the previous request}Returns {
"_scroll_id": "Scroll Id",
"hits": {
"total": 263,
"max_score": 0.11207403,
"hits": [ ]
}
}
|
...