Problem Statement 1:
Extend this design to support the download for batches stats upto 1L participants. By default elastic search support max 10k records , in one search query.
Use Elastic search scroll api . 'Scroll API ' can be used to retrieve large numbers of results (or even all results) from a single search request, it will work in same way as cursor on a traditional database.
Pros | Cons |
|---|---|
We can retrieve large data set | We can not use scroll api for real time user request |
We can slice the data based upon shards | Performance issues while using it for real time request |
...