Update a transform Generally available; Added in 7.2.0

POST /_transform/{transform_id}/_update

Updates certain properties of a transform.

All updated properties except description do not take effect until after the transform starts the next checkpoint, thus there is data consistency in each checkpoint. To use this API, you must have read and view_index_metadata privileges for the source indices. You must also have index and read privileges for the destination index. When Elasticsearch security features are enabled, the transform remembers which roles the user who updated it had at the time of update and runs with those privileges.

Required authorization

  • Index privileges: read,index,view_index_metadata
  • Cluster privileges: manage_transform

Path parameters

  • transform_id string Required

    Identifier for the transform.

Query parameters

  • defer_validation boolean

    When true, deferrable validations are not run. This behavior may be desired if the source index does not exist until after the transform is created.

  • timeout string

    Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.

    Values are -1 or 0.

application/json

Body Required

  • dest object
    Hide dest attributes Show dest attributes object
    • index string
    • pipeline string

      The unique identifier for an ingest pipeline.

  • description string

    Free text description of the transform.

  • frequency string

    A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

  • _meta object
    Hide _meta attribute Show _meta attribute object
    • * object Additional properties
  • source object
    Hide source attributes Show source attributes object
    • index string | array[string] Required
    • runtime_mappings object
      Hide runtime_mappings attribute Show runtime_mappings attribute object
      • * object Additional properties
        Hide * attributes Show * attributes object
        • fields object

          For type composite

          Hide fields attribute Show fields attribute object
          • * object Additional properties
            Hide * attribute Show * attribute object
            • type string Required

              Values are boolean, composite, date, double, geo_point, geo_shape, ip, keyword, long, or lookup.

        • fetch_fields array[object]

          For type lookup

          Hide fetch_fields attributes Show fetch_fields attributes object
          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • format string
        • format string

          A custom format for date type runtime fields.

        • input_field string

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • target_field string

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • target_index string
        • script object
          Hide script attributes Show script attributes object
          • source string

            The script source.

          • id string
          • params object

            Specifies any named parameters that are passed into the script as variables. Use parameters instead of hard-coded values to decrease compile time.

            Hide params attribute Show params attribute object
            • * object Additional properties
          • lang string

            Any of:

            Values are painless, expression, mustache, or java.

          • options object
            Hide options attribute Show options attribute object
            • * string Additional properties
        • type string Required

          Values are boolean, composite, date, double, geo_point, geo_shape, ip, keyword, long, or lookup.

    • query object

      A query clause that retrieves a subset of data from the source index.

      Query DSL
  • settings object

    The source of the data for the transform.

    Hide settings attributes Show settings attributes object
    • align_checkpoints boolean

      Specifies whether the transform checkpoint ranges should be optimized for performance. Such optimization can align checkpoint ranges with the date histogram interval when date histogram is specified as a group source in the transform config. As a result, less document updates in the destination index will be performed thus improving overall performance.

    • dates_as_epoch_millis boolean

      Defines if dates in the ouput should be written as ISO formatted string or as millis since epoch. epoch_millis was the default for transforms created before version 7.11. For compatible output set this value to true.

    • deduce_mappings boolean

      Specifies whether the transform should deduce the destination index mappings from the transform configuration.

    • docs_per_second number

      Specifies a limit on the number of input documents per second. This setting throttles the transform by adding a wait time between search requests. The default value is null, which disables throttling.

    • max_page_search_size number

      Defines the initial page size to use for the composite aggregation for each checkpoint. If circuit breaker exceptions occur, the page size is dynamically adjusted to a lower value. The minimum value is 10 and the maximum is 65,536.

    • unattended boolean Generally available; Added in 8.5.0

      If true, the transform runs in unattended mode. In unattended mode, the transform retries indefinitely in case of an error which means the transform never fails. Setting the number of retries other than infinite fails in validation.

  • sync object
    Hide sync attribute Show sync attribute object
    • time object
      Hide time attributes Show time attributes object
      • delay string

        A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

      • field string Required

        Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

  • retention_policy object | string | null

    Defines a retention policy for the transform. Data that meets the defined criteria is deleted from the destination index.

    One of:

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
    • authorization object
      Hide authorization attributes Show authorization attributes object
      • api_key object
        Hide api_key attributes Show api_key attributes object
        • id string Required

          The identifier for the API key.

        • name string Required

          The name of the API key.

      • roles array[string]

        If a user ID was used for the most recent update to the transform, its roles at the time of the update are listed in the response.

      • service_account string

        If a service account was used for the most recent update to the transform, the account name is listed in the response.

    • create_time number Required
    • description string Required
    • dest object Required
      Hide dest attributes Show dest attributes object
      • index string Required
      • op_type string

        Values are index or create.

      • pipeline string

        The name of the pipeline to use.

      • routing string
      • version_type string

        Values are internal, external, external_gte, or force.

    • frequency string

      A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

    • id string Required
    • latest object
      Hide latest attributes Show latest attributes object
      • sort string Required

        Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • unique_key array[string] Required

        Specifies an array of one or more fields that are used to group the data.

    • pivot object
      Hide pivot attributes Show pivot attributes object
      • aggregations object

        Defines how to aggregate the grouped data. The following aggregations are currently supported: average, bucket script, bucket selector, cardinality, filter, geo bounds, geo centroid, geo line, max, median absolute deviation, min, missing, percentiles, rare terms, scripted metric, stats, sum, terms, top metrics, value count, weighted average.

      • group_by object

        Defines how to group the data. More than one grouping can be defined per pivot. The following groupings are currently supported: date histogram, geotile grid, histogram, terms.

        Hide group_by attribute Show group_by attribute object
        • * object Additional properties
          Hide * attributes Show * attributes object
          • date_histogram object
            Hide date_histogram attributes Show date_histogram attributes object
            • calendar_interval string

              Values are second, 1s, minute, 1m, hour, 1h, day, 1d, week, 1w, month, 1M, quarter, 1q, year, or 1y.

            • extended_bounds object
            • hard_bounds object
            • field string

              Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

            • fixed_interval string

              A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

            • format string

              The date format used to format key_as_string in the response. If no format is specified, the first date format specified in the field mapping is used.

            • interval string

              A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

            • min_doc_count number

              Only returns buckets that have min_doc_count number of documents. By default, all buckets between the first bucket that matches documents and the last one are returned.

            • missing
            • offset string

              A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

            • order
            • params object
            • script object
            • time_zone string
            • keyed boolean

              Set to true to associate a unique string key with each bucket and return the ranges as a hash rather than an array.

          • geotile_grid object
            Hide geotile_grid attributes Show geotile_grid attributes object
            • field string

              Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

            • precision number
            • shard_size number

              Allows for more accurate counting of the top cells returned in the final result the aggregation. Defaults to returning max(10,(size x number-of-shards)) buckets from each shard.

            • size number

              The maximum number of buckets to return.

            • bounds
          • histogram object
            Hide histogram attributes Show histogram attributes object
            • extended_bounds object
            • hard_bounds object
            • field string

              Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

            • interval number

              The interval for the buckets. Must be a positive decimal.

            • min_doc_count number

              Only returns buckets that have min_doc_count number of documents. By default, the response will fill gaps in the histogram with empty buckets.

            • missing number

              The value to apply to documents that do not have a value. By default, documents without a value are ignored.

            • offset number

              By default, the bucket keys start with 0 and then continue in even spaced steps of interval. The bucket boundaries can be shifted by using the offset option.

            • order
            • script object
            • format string
            • keyed boolean

              If true, returns buckets as a hash instead of an array, keyed by the bucket keys.

          • terms
    • retention_policy object
      Hide retention_policy attribute Show retention_policy attribute object
      • time object
        Hide time attributes Show time attributes object
        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • max_age string Required

          A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

    • settings object Required

      The source of the data for the transform.

      Hide settings attributes Show settings attributes object
      • align_checkpoints boolean

        Specifies whether the transform checkpoint ranges should be optimized for performance. Such optimization can align checkpoint ranges with the date histogram interval when date histogram is specified as a group source in the transform config. As a result, less document updates in the destination index will be performed thus improving overall performance.

      • dates_as_epoch_millis boolean

        Defines if dates in the ouput should be written as ISO formatted string or as millis since epoch. epoch_millis was the default for transforms created before version 7.11. For compatible output set this value to true.

      • deduce_mappings boolean

        Specifies whether the transform should deduce the destination index mappings from the transform configuration.

      • docs_per_second number

        Specifies a limit on the number of input documents per second. This setting throttles the transform by adding a wait time between search requests. The default value is null, which disables throttling.

      • max_page_search_size number

        Defines the initial page size to use for the composite aggregation for each checkpoint. If circuit breaker exceptions occur, the page size is dynamically adjusted to a lower value. The minimum value is 10 and the maximum is 65,536.

      • unattended boolean Generally available; Added in 8.5.0

        If true, the transform runs in unattended mode. In unattended mode, the transform retries indefinitely in case of an error which means the transform never fails. Setting the number of retries other than infinite fails in validation.

    • source object Required
      Hide source attributes Show source attributes object
      • index string | array[string] Required
      • query object

        An Elasticsearch Query DSL (Domain Specific Language) object that defines a query.

        External documentation
      • remote object
        Hide remote attributes Show remote attributes object
        • connect_timeout string

          A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

        • headers object

          An object containing the headers of the request.

          Hide headers attribute Show headers attribute object
          • * string Additional properties
        • host string Required
        • username string
        • password string
        • socket_timeout string

          A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

      • size number

        The number of documents to index per batch. Use it when you are indexing from remote to ensure that the batches fit within the on-heap buffer, which defaults to a maximum size of 100 MB.

      • slice object
        Hide slice attributes Show slice attributes object
        • field string

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • id string Required
        • max number Required
      • sort string | object | array[string | object]

        One of:

        Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        One of:

        Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • _source string | array[string]
      • runtime_mappings object
        Hide runtime_mappings attribute Show runtime_mappings attribute object
        • * object Additional properties
          Hide * attributes Show * attributes object
          • fields object

            For type composite

            Hide fields attribute Show fields attribute object
            • * object Additional properties
              Hide * attribute Show * attribute object
              • type string Required

                Values are boolean, composite, date, double, geo_point, geo_shape, ip, keyword, long, or lookup.

          • fetch_fields array[object]

            For type lookup

            Hide fetch_fields attributes Show fetch_fields attributes object
            • field string Required

              Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

            • format string
          • format string

            A custom format for date type runtime fields.

          • input_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • target_field string

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • target_index string
          • script object
            Hide script attributes Show script attributes object
            • source string

              The script source.

            • id string
            • params object

              Specifies any named parameters that are passed into the script as variables. Use parameters instead of hard-coded values to decrease compile time.

              Hide params attribute Show params attribute object
              • * object Additional properties
            • lang string

              Any of:

              Values are painless, expression, mustache, or java.

            • options object
              Hide options attribute Show options attribute object
              • * string Additional properties
          • type string Required

            Values are boolean, composite, date, double, geo_point, geo_shape, ip, keyword, long, or lookup.

    • sync object
      Hide sync attribute Show sync attribute object
      • time object
        Hide time attributes Show time attributes object
        • delay string

          A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

    • version string Required
    • _meta object
      Hide _meta attribute Show _meta attribute object
      • * object Additional properties
POST /_transform/{transform_id}/_update
curl \
 --request POST 'http://api.example.com/_transform/{transform_id}/_update' \
 --header "Authorization: $API_KEY" \
 --header "Content-Type: application/json" \
 --data '"{\n  \"source\": {\n    \"index\": \"kibana_sample_data_ecommerce\",\n    \"query\": {\n      \"term\": {\n        \"geoip.continent_name\": {\n          \"value\": \"Asia\"\n        }\n      }\n    }\n  },\n  \"pivot\": {\n    \"group_by\": {\n      \"customer_id\": {\n        \"terms\": {\n          \"field\": \"customer_id\",\n          \"missing_bucket\": true\n        }\n      }\n    },\n    \"aggregations\": {\n      \"max_price\": {\n        \"max\": {\n          \"field\": \"taxful_total_price\"\n        }\n      }\n    }\n  },\n  \"description\": \"Maximum priced ecommerce data by customer_id in Asia\",\n  \"dest\": {\n    \"index\": \"kibana_sample_data_ecommerce_transform1\",\n    \"pipeline\": \"add_timestamp_pipeline\"\n  },\n  \"frequency\": \"5m\",\n  \"sync\": {\n    \"time\": {\n      \"field\": \"order_date\",\n      \"delay\": \"60s\"\n    }\n  },\n  \"retention_policy\": {\n    \"time\": {\n      \"field\": \"order_date\",\n      \"max_age\": \"30d\"\n    }\n  }\n}"'
Request example
Run `POST _transform/simple-kibana-ecomm-pivot/_update` to update a transform that uses the pivot method.
{
  "source": {
    "index": "kibana_sample_data_ecommerce",
    "query": {
      "term": {
        "geoip.continent_name": {
          "value": "Asia"
        }
      }
    }
  },
  "pivot": {
    "group_by": {
      "customer_id": {
        "terms": {
          "field": "customer_id",
          "missing_bucket": true
        }
      }
    },
    "aggregations": {
      "max_price": {
        "max": {
          "field": "taxful_total_price"
        }
      }
    }
  },
  "description": "Maximum priced ecommerce data by customer_id in Asia",
  "dest": {
    "index": "kibana_sample_data_ecommerce_transform1",
    "pipeline": "add_timestamp_pipeline"
  },
  "frequency": "5m",
  "sync": {
    "time": {
      "field": "order_date",
      "delay": "60s"
    }
  },
  "retention_policy": {
    "time": {
      "field": "order_date",
      "max_age": "30d"
    }
  }
}
Response examples (200)
A successful response when creating a transform.
{
  "id": "simple-kibana-ecomm-pivot",
  "authorization": {
    "roles": [
      "superuser"
    ]
  },
  "version": "10.0.0",
  "create_time": 1712951576767,
  "source": {
    "index": [
      "kibana_sample_data_ecommerce"
    ],
    "query": {
      "term": {
        "geoip.continent_name": {
          "value": "Asia"
        }
      }
    }
  },
  "dest": {
    "index": "kibana_sample_data_ecommerce_transform_v2",
    "pipeline": "add_timestamp_pipeline"
  },
  "frequency": "15m",
  "sync": {
    "time": {
      "field": "order_date",
      "delay": "120s"
    }
  },
  "pivot": {
    "group_by": {
      "customer_id": {
        "terms": {
          "field": "customer_id",
          "missing_bucket": true
        }
      }
    },
    "aggregations": {
      "max_price": {
        "max": {
          "field": "taxful_total_price"
        }
      }
    }
  },
  "description": "Maximum priced ecommerce data by customer_id in Asia",
  "settings": {},
  "retention_policy": {
    "time": {
      "field": "order_date",
      "max_age": "30d"
    }
  }
}