./ahmedhashim

Painless Enrichment

At some point your search needs will become larger than what your index schema can support. Maybe you’re working with products and want some derived fields that you can filter on and/or report against. You can add these to your current products index.

Elasticsearch has a scripting language called Painless. Assuming you already have the original fields, you can use /_update_by_query to add these derived fields to your current documents without needing to reindex everything.

Map the new fields

You’ll first need to add these derived fields to your products index mapping. If you don’t, they’ll be dynamically mapped when you add them to your documents, which means your price_band field would be mapped as text with a .keyword subfield.

PUT /products/_mapping
{
  "properties": {
    "discount_pct": { "type": "integer" },
    "margin_pct":   { "type": "integer" },
    "price_band":   { "type": "keyword" },
    "stock_state":  { "type": "keyword" }
  }
}

Dry run

Use the /_scripts/painless/_execute API to test your math with some values. This dry run will help to catch any problems with your formula on a single product instead of your entire catalog.

POST /_scripts/painless/_execute
{
  "script": {
    "source": """
      double discount = (params.list_price - params.price) * 100.0 / params.list_price;
      double margin = (params.price - params.cost) * 100.0 / params.price;

      return [
        "discount_pct": (int) Math.round(discount),
        "margin_pct": (int) Math.round(margin)
      ];
    """,
    "params": {
      "price": 79.99,
      "list_price": 99.99,
      "cost": 42.00
    }
  }
}

Store the script

Next, save a Painless script to your cluster so update jobs can call it by ID on all the documents that need these fields. It skips any field whose inputs are missing and adds price_band and stock_state.

PUT /_scripts/products-enrichment-v1
{
  "script": {
    "lang": "painless",
    "source": """
      if (ctx._source.list_price != null &&
          ctx._source.list_price > 0 &&
          ctx._source.price != null) {
        double discount =
          (ctx._source.list_price - ctx._source.price) * 100.0 / ctx._source.list_price;
        ctx._source.discount_pct = (int) Math.round(discount);
      }

      if (ctx._source.price != null &&
          ctx._source.price > 0 &&
          ctx._source.cost != null) {
        double margin =
          (ctx._source.price - ctx._source.cost) * 100.0 / ctx._source.price;
        ctx._source.margin_pct = (int) Math.round(margin);
      }

      if (ctx._source.price != null) {
        if (ctx._source.price < 25) {
          ctx._source.price_band = "budget";
        } else if (ctx._source.price < 100) {
          ctx._source.price_band = "mid";
        } else {
          ctx._source.price_band = "premium";
        }
      }

      if (ctx._source.inventory_count != null) {
        if (ctx._source.inventory_count > 0) {
          ctx._source.stock_state = "in_stock";
        } else {
          ctx._source.stock_state = "out_of_stock";
        }
      }
    """
  }
}

Keep in mind you’re multiplying by 100.0 before you divide. The _source will give you what you have in your index, so if that’s "price": 80 and "list_price": 100, the script will be working with 2 integers. With integer division, discount_pct would round down to 0. The dry run won’t show this since it uses prices that are decimals.

Backfill

You can run this query async, so the response will return immediately. The query only matches documents that are missing at least one of the new fields, so if you run it again, it’ll skip any documents that already have them.

POST /products/_update_by_query?conflicts=proceed&wait_for_completion=false&slices=auto
{
  "query": {
    "bool": {
      "should": [
        { "bool": { "must_not": { "exists": { "field": "discount_pct" } } } },
        { "bool": { "must_not": { "exists": { "field": "margin_pct" } } } },
        { "bool": { "must_not": { "exists": { "field": "price_band" } } } },
        { "bool": { "must_not": { "exists": { "field": "stock_state" } } } }
      ],
      "minimum_should_match": 1
    }
  },
  "script": {
    "id": "products-enrichment-v1"
  }
}

You’ll receive a task ID:

{
  "task": "r1A2WoRbTwKZ516z6NEs5A:36619"
}

Check the status of your task every 5 seconds. Once it reports "completed": true, your documents have been updated.

watch -n 5 'curl -s "http://your-cluster:9200/_tasks/r1A2WoRbTwKZ516z6NEs5A:36619" | jq "{completed: .completed, updated: .task.status.updated, version_conflicts: .task.status.version_conflicts, failures: .response.failures}"'

Verify

Now you can query some of your documents to see the new fields.

GET /products/_search
{
  "size": 3,
  "_source": [
    "name",
    "price",
    "discount_pct",
    "margin_pct",
    "price_band",
    "stock_state"
  ],
  "query": {
    "match_all": {}
  }
}

This can only be done when adding fields. If you need to change the field type or analyzer of an existing field, you’ll need to reindex your data.