Elasticsearch upgrade in Sunbird Platform
Problem Statement
Sunbird is using ES 5.4.3 whereas other platforms (LP/DP) are running on ES 6.3. Hence the Sunbird ES 5.4.3 needs to be upgraded to 6.3.
SB-11085 - Getting issue details... STATUS
Solution Approach
To accomplish the upgrade we can take below approach
Full cluster restart
- Disable shard allocation
- Stop indexing and sync flush
- shutdown all nodes
- upgrade each nodes and provide stored data path
- Start each upgraded node
- reenable allocation which was disabled in first step
backup and restore process.
Steps -
- Register a repository in the existing ES 5.4.3
- Create a snapshot of the data from ES
- Register the same repo in the new ES 6.3
- Call the restore API in ES 6.3 for the snapshot created in step 2
Pros and cons
| Approach | pros | cons |
|---|---|---|
| Full cluster restart | downtime | |
| Backup and restore | minimal downtime | needs extra running nodes |
Note
Snapshots are incremental which means that once sunbird in pointing to the new ES 6.3, we can repeat the snapshot and backup process to push any mew data created in between.
Problem Statement
How to register a repository?
Solution
For registering a repository, we need to define/add path.repo in elastisearch.yml as below
# # Path to directory where to store the data (separate multiple locations by comma): # #path.data: /path/to/data # # Path to log files: # #path.logs: /path/to/logs # path.repo: ["/home/elastic/repo"]
Once we restart the elasticsearch with this config, It needs an API call as below
Request
PUT /_snapshot/{repoName}
{
"type": "fs",
"settings": {
"location": "{location}", //the location where the repo should be, the value should be present in path.repo
"compress": true
}
}
Response
{
"acknowledged": true
}
example
curl -X PUT \
http://localhost:9200/_snapshot/repo \
-H 'Content-Type: application/json' \
-H 'Postman-Token: 1b3d36e5-0609-47e4-8e19-d2dc945acddd' \
-H 'cache-control: no-cache' \
-d '{
"type": "fs",
"settings": {
"location": "/home/elastic/repo",
"compress": true
}
}'
Please note that created repo can be verified by below API call
Request
GET /_snapshot/{repoName}
Response
{
"type": "fs",
"settings": {
"location": "home/elastic/repo",
"compress": true
}
}
Problem Statement
How to create a snapshot in repo?
Solution
Once the repository is created, we can take snapshot by below API
Request
PUT /_snapshot/{repoName}/{snapshotName}
Response
{
"accepted": true
}
example
curl -X PUT \
http://localhost:9200/_snapshot/repo/snapshot_1 \
-H 'cache-control: no-cache'
It starts the snapshot process and the status of this process can be verified as below
Request
GET /_snapshot/{repoName}/{snapshotName}
Response
{
"snapshots": [
{
"snapshot": "snapshot",
"uuid": "Ht4gT_joQKKEoBF-Qcj5Vg",
"version_id": 5010199,
"version": "5.1.1",
"indices": [
"searchindex",
"sunbirdplugin",
"sunbirddataaudit",
".kibana"
],
"state": "SUCCESS",
"start_time": "2019-03-08T07:07:22.416Z",
"start_time_in_millis": 1552028842416,
"end_time": "2019-03-08T07:07:25.128Z",
"end_time_in_millis": 1552028845128,
"duration_in_millis": 2712,
"failures": [],
"shards": {
"total": 16,
"failed": 0,
"successful": 16
}
}
]
}
Further details and considerations
Problem Statement
How to restore a snapshot?
Solution
A snapshot created can be restored by calling the below API, however it should be ensured that the same repository is registered where we want to restore
Request
POST /_snapshot/{repoName}/{snapshotName}/_restore
Response
{
"accepted": true
}
example
curl -X POST \
http://localhost:9200/_snapshot/repo/snapshot/_restore \
-H 'cache-control: no-cache'
Problem Statement
How to get the status of snapshot or recovery process
Solution
Below API is used to get the status
Problem Statement
Cluster consideration while backup and restore
Solution
For cluster implementation, the path.repo should be configured on all nodes with the same value which will be shared file storage.
We can also use s3 for elasticsearch backup and restore
References
- https://www.elastic.co/guide/en/elasticsearch/reference/5.5/modules-snapshots.html
- https://www.elastic.co/guide/en/elasticsearch/plugins/5.5/repository-s3.html
- https://www.elastic.co/guide/en/elasticsearch/reference/6.3/restart-upgrade.html