Overview :
Type mapping inside elasticsearch can be dynamic or static. Both mapping has it's own pros and cons. For initial usage dynamic mapping is fine, because we won't have complete knowledge of data model and it's usability. But for production, dynamic mapping upper limit can get consumed very quickly.
Problem statement :
Sunbird currently is using dynamic mapping for elasticsearch indexes., Inside Elastic search default index limit is 1000. So if your index reached 1000 limit , then it won't allow any other key to be indexed.
Sunbird platform has reached this limit, and introduction of new key would throw error.
Solution 1:
Use of static mapping, where static mapping can be of two type.
- Flexible static mapping: In this case first time we can define all attribute and it's type, and elasticseaerch will do indexing for only those attribute ,which is already defined in mapping, and it will allow user to store other attribute as well without indexing
- Strict mapping: In this case first time we can define all attribute and it's type, and elasticseaerch will do indexing for only those attribute ,which is already defined in mapping, and it will throw error if any new attribute is introduced.
Syntax for enabling above strategies:
Flexible static mapping: { "dynamic": false, "properties": { } }
Strict static mapping: { "dynamic": "strict", "properties": { } }
Solution 2:
Use of dynamic mapping and adjust the keep increasing the upper limit as target is reached.
We can increase limit of index form 1000 to 1500 as follow:
curl -X PUT http://localhost:9200/searchindex/settings -H 'cache-control: no-cache' -H 'content-type: application/json' -H 'postman-token: e2b31e11-6678-ae40-288b-32f51705f7f7' -d '{ "index.mapping.totalfields.limit": 1500 }'
Curl command to do mapping:
Method : Put
http://localhost:9200/searchindex/batch/_mapping
{
"dynamic": false,
"properties": {
"all_fields": {
"type": "text",
"fields": {
"raw": {
"type": "text",
"analyzer": "keylower"
}
},
"analyzer": "cs_index_analyzer",
"search_analyzer": "cs_search_analyzer"
},
"countDecrementDate": {
"type": "text",
"fields": {
"raw": {
"type": "text",
"analyzer": "keylower",
"fielddata": true
}
},
"copy_to": [
"all_fields"
],
"analyzer": "cs_index_analyzer",
"search_analyzer": "cs_search_analyzer",
"fielddata": true
},
"countDecrementStatus": {
"type": "boolean",
"fields": {
"raw": {
"type": "boolean"
}
}
},
"countIncrementDate": {
"type": "text",
"fields": {
"raw": {
"type": "text",
"analyzer": "keylower",
"fielddata": true
}
},
"copy_to": [
"all_fields"
],
"analyzer": "cs_index_analyzer",
"search_analyzer": "cs_search_analyzer",
"fielddata": true
},
"countIncrementStatus": {
"type": "boolean",
"fields": {
"raw": {
"type": "boolean"
}
}
},
"courseCreator": {
"type": "text",
"fields": {
"raw": {
"type": "text",
"analyzer": "keylower",
"fielddata": true
}
},
"copy_to": [
"all_fields"
],
"analyzer": "cs_index_analyzer",
"search_analyzer": "cs_search_analyzer",
"fielddata": true
},
"courseId": {
"type": "text",
"fields": {
"raw": {
"type": "text",
"analyzer": "keylower",
"fielddata": true
}
},
"copy_to": [
"all_fields"
],
"analyzer": "cs_index_analyzer",
"search_analyzer": "cs_search_analyzer",
"fielddata": true
},
"batchId": {
"type": "text",
"fields": {
"raw": {
"type": "text",
"analyzer": "keylower",
"fielddata": true
}
},
"copy_to": [
"all_fields"
],
"analyzer": "cs_index_analyzer",
"search_analyzer": "cs_search_analyzer",
"fielddata": true
},
"createdBy": {
"type": "text",
"fields": {
"raw": {
"type": "text",
"analyzer": "keylower",
"fielddata": true
}
},
"copy_to": [
"all_fields"
],
"analyzer": "cs_index_analyzer",
"search_analyzer": "cs_search_analyzer",
"fielddata": true
},
"createdDate": {
"type": "text",
"fields": {
"raw": {
"type": "text",
"analyzer": "keylower",
"fielddata": true
}
},
"copy_to": [
"all_fields"
],
"analyzer": "cs_index_analyzer",
"search_analyzer": "cs_search_analyzer",
"fielddata": true
},
"createdFor": {
"type": "text",
"fields": {
"raw": {
"type": "text",
"analyzer": "keylower",
"fielddata": true
}
},
"copy_to": [
"all_fields"
],
"analyzer": "cs_index_analyzer",
"search_analyzer": "cs_search_analyzer",
"fielddata": true
},
"description": {
"type": "text",
"fields": {
"raw": {
"type": "text",
"analyzer": "keylower",
"fielddata": true
}
},
"copy_to": [
"all_fields"
],
"analyzer": "cs_index_analyzer",
"search_analyzer": "cs_search_analyzer",
"fielddata": true
},
"endDate": {
"type": "date",
"fields": {
"raw": {
"type": "date"
}
}
},
"enrollmentType": {
"type": "text",
"fields": {
"raw": {
"type": "text",
"analyzer": "keylower",
"fielddata": true
}
},
"copy_to": [
"all_fields"
],
"analyzer": "cs_index_analyzer",
"search_analyzer": "cs_search_analyzer",
"fielddata": true
},
"hashTagId": {
"type": "text",
"fields": {
"raw": {
"type": "text",
"analyzer": "keylower",
"fielddata": true
}
},
"copy_to": [
"all_fields"
],
"analyzer": "cs_index_analyzer",
"search_analyzer": "cs_search_analyzer",
"fielddata": true
},
"id": {
"type": "text",
"fields": {
"raw": {
"type": "text",
"analyzer": "keylower",
"fielddata": true
}
},
"copy_to": [
"all_fields"
],
"analyzer": "cs_index_analyzer",
"search_analyzer": "cs_search_analyzer",
"fielddata": true
},
"identifier": {
"type": "text",
"fields": {
"raw": {
"type": "text",
"analyzer": "keylower",
"fielddata": true
}
},
"copy_to": [
"all_fields"
],
"analyzer": "cs_index_analyzer",
"search_analyzer": "cs_search_analyzer",
"fielddata": true
},
"mentors": {
"type": "text",
"fields": {
"raw": {
"type": "text",
"analyzer": "keylower",
"fielddata": true
}
},
"copy_to": [
"all_fields"
],
"analyzer": "cs_index_analyzer",
"search_analyzer": "cs_search_analyzer",
"fielddata": true
},
"name": {
"type": "text",
"fields": {
"raw": {
"type": "text",
"analyzer": "keylower",
"fielddata": true
}
},
"copy_to": [
"all_fields"
],
"analyzer": "cs_index_analyzer",
"search_analyzer": "cs_search_analyzer",
"fielddata": true
},
"startDate": {
"type": "date",
"fields": {
"raw": {
"type": "date"
}
}
},
"status": {
"type": "long",
"fields": {
"raw": {
"type": "long"
}
}
},
"updatedDate": {
"type": "text",
"fields": {
"raw": {
"type": "text",
"analyzer": "keylower",
"fielddata": true
}
},
"copy_to": [
"all_fields"
],
"analyzer": "cs_index_analyzer",
"search_analyzer": "cs_search_analyzer",
"fielddata": true
}
}
}
-- New Mapping files
-- Old Mapping files: