Live Reindexing in Elasticsearch
Since you can’t change field mappings on an existing index, you’ll need to create a new index, copy the data over and point an alias at it. Using the reindex API and aliases, you can achieve this without downtime for your searches.
Let’s assume you have a books index and you want to apply a new analyzer to the description field, or you want to change its type. Regardless of what you’re doing, you’ll be reindexing your data. You only need to stop your pipelines once to alias your index. From here on, you can migrate without stopping them.
Aliasing
Since Elasticsearch doesn’t allow an alias with the same name as an index, you can’t create an alias called books. By cloning your index to books_v1, you’ll be able to proceed. You only need to do this once.
You need to stop your pipelines for the next few steps, as the index must be set to read-only for cloning. Any writes that hit this will return a 403 cluster_block_exception. If you have any pipelines running, they should retry these writes.
PUT /books/_settings
{
"index.blocks.write": true
}
POST /books/_clone/books_v1
Because the clone hard-links the segment files (if supported by your file system), this should be quick. The index will also inherit the write block, which you’ll remove after aliasing.
When the response returns, the new index may not have its shards allocated yet, so you’ll need to wait until it’s “green”.
GET /_cluster/health/books_v1?wait_for_status=green&timeout=30m
You want to wait until the index is green so the replicas are allocated, as you’ll be deleting books in the next step. You don’t want to have just your primary holding the data.
This is where most of the pipeline pause will be spent. The clone is fast, but it’ll take time to replicate the data to its replicas. Depending on how large your index is, you’ll want to increase the timeout (it defaults to 30 seconds).
At this point no data can be written to either index, so you can compare the counts.
GET /books/_count
GET /books_v1/_count
You can now swap your index for the alias. This is done in one _aliases request.
POST /_aliases
{
"actions": [
{
"remove_index": {
"index": "books"
}
},
{
"add": {
"index": "books_v1",
"alias": "books"
}
}
]
}
Both actions will happen atomically, so your alias will always point to an index. Your searches will continue to work and now point to books_v1.
Remove the write block and start your pipelines.
PUT /books_v1/_settings
{
"index.blocks.write": false
}
Any writes to books will now go to books_v1.
New mapping
Create your new index with the updated mapping.
PUT /books_v2
{
"settings": {
"analysis": {
"analyzer": {
"custom_analyzer": {
"tokenizer": "standard",
"filter": ["lowercase", "stop"]
}
}
}
},
"mappings": {
"properties": {
"title": {
"type": "text",
"analyzer": "standard"
},
"author": {
"type": "keyword"
},
"description": {
"type": "text",
"analyzer": "custom_analyzer"
},
"publication_year": {
"type": "integer"
}
}
}
}
Dual writes
Your pipelines should now write all documents to both books_v1 and books_v2, using the actual index names. If you wrote to the books alias, it’d break when you move the alias. Both writes would go to books_v2, leaving books_v1 stagnant.
Once your dual writes are going, you can start the reindex. Because you’re reindexing from books_v1, any new writes will be covered by the dual writes. Your searches will continue to read from books_v1, which is being written to, so no data will be lost.
You now have 2 writes per document. One may fail and the other may not. Make sure your pipelines can retry each write separately.
Reindexing
The reindex is reading from a point-in-time snapshot of books_v1. Any writes your pipelines have made to books_v2 are newer than the snapshot. By default, the reindex would overwrite these documents with the old version. By setting the op_type to create, you’ll only copy over documents that don’t already exist, and with conflicts: proceed it’ll ignore documents that do already exist.
This assumes your pipelines are writing whole documents. If you update or delete a document that hasn’t been reindexed, it’ll return a 404 for books_v2, and deleted documents will be reindexed. Make sure your pipelines log these failed writes so you can replay them once the reindex finishes.
Start the reindex in the background.
POST /_reindex?wait_for_completion=false
{
"conflicts": "proceed",
"source": {
"index": "books_v1"
},
"dest": {
"index": "books_v2",
"op_type": "create"
}
}
If you have a large index, you may want to throttle the reindex with requests_per_second.
POST /_reindex?wait_for_completion=false
{
"conflicts": "proceed",
"source": {
"index": "books_v1",
"size": 5000
},
"dest": {
"index": "books_v2",
"op_type": "create"
},
"requests_per_second": 1000
}
You’ll be returned a TASK_ID you can use to check the progress.
Monitoring
Check on the status of your reindex.
GET /_tasks/TASK_ID
Or see it live (nice dopamine hits on a large reindex).
# Watch progress every 5 seconds
watch -n 5 'curl -s -X GET "http://your-cluster:9200/_tasks/TASK_ID" | jq ".task.status"'
You can also see all running reindex tasks.
GET /_tasks?actions=*reindex&detailed=true
Verification
Once complete, you’ll get the following back from GET /_tasks/TASK_ID.
{
"completed": true,
"response": {
"total": 1250000,
"created": 1198431,
"version_conflicts": 51569,
"failures": [],
...
},
...
}
The total will be the number of documents that were read from books_v1. They’ll either have been created in books_v2 or failed with a version_conflicts because your pipeline wrote to it. These numbers should equal your total, and failures should be empty.
If they don’t equal your total, you can just run the same reindex again. It’ll skip any documents that already exist in books_v2.
Both indexes should now have the same number of documents. Depending on when you run these requests, there may be a small difference from writes that land between them.
GET /books_v1/_count
GET /books_v2/_count
If you have more documents in books_v2, it’s probably a deleted document that the reindex recreated. Replaying your failed writes will correct it.
Cutover
Time to move the alias. Again, you can do this in one _aliases request.
POST /_aliases
{
"actions": [
{
"remove": {
"index": "books_v1",
"alias": "books"
}
},
{
"add": {
"index": "books_v2",
"alias": "books"
}
}
]
}
All clients that are using books won’t know the difference.
You can keep your pipelines writing to both indexes. You know books_v1 is up to date, so if something looks wrong, you can just move the alias back.
Cleanup
When you’re confident nothing has been lost, stop your pipelines from writing to books_v1.
Because index creation is enabled by default, if a write comes in after you delete books_v1, it’ll recreate the index with a dynamic mapping. Your pipeline will think it succeeded and no one will know the index exists. So to catch any writers, you can put the block back on.
PUT /books_v1/_settings
{
"index.blocks.write": true
}
Any writes that come in now will return a 403. Once you’re confident there are no more writes, delete the index in its own request.
DELETE /books_v1
You can now always create an index with a number after it and use the alias to point to it. When you need to update a field mapping again, just create books_v3 and skip cloning and stopping your pipelines.