Problem Statement:
Extend this design to support the download for batches stats upto 1L participants.
Solutions
By default elasticsearch support max 10k records , in one search query.
Solution 1 :
Use elasticsearch scroll api . 'Scroll API ' can be used to retrieve large numbers of results (or even all results) from a single search request, it will work in much the same way as you would use a cursor on a traditional database.
| Pros |
|---|
...
| Cons | |
|---|---|
We can retrieve large data set | We can |
...
Cons:
...
not use scroll api for real time user request | |
We can slice the data based upon shards | Performance issues |
Approach 1:
Start the service instantly to download the batches stats
...
- No extra efforts require to handle the request
Cons:
- Duplicate request will entertain multiple times
...
We can queue all the request coming for batch stats. Our scheduler will check the queue and start processing the request one by one
Pros:
- We can avoid duplicate requests
Cons:
- We have to maintain a queue to handle all the request.
- We have to schedule a scheduler.
...