Upgrading Elasticsearch

Important

Before upgrading Elasticsearch:

  • Consult the breaking changes docs.
  • Use the Elasticsearch Migration Plugin to detect potential issues before upgrading.
  • Test upgrades in a dev environment before upgrading your production cluster.
  • Always back up your data before upgrading. You cannot roll back to an earlier version unless you have a backup of your data.
  • If you are using custom plugins, check that a compatible version is available.

Elasticsearch can usually be upgraded using a rolling upgrade process, resulting in no interruption of service. This section details how to perform both rolling upgrades and upgrades with full cluster restarts.

To determine whether a rolling upgrade is supported for your release, please consult this table:

Upgrade From Upgrade To Supported Upgrade Type

1.x

5.x

Reindex to upgrade

2.x

2.y

Rolling upgrade (where y > x)

2.x

5.x

Full cluster restart

5.0.0 pre GA

5.x

Full cluster restart

5.x

5.y

Rolling upgrade (where y > x)

Important

Indices created in Elasticsearch 1.x or before

Elasticsearch is able to read indices created in the previous major version only. For instance, Elasticsearch 5.x can use indices created in Elasticsearch 2.x, but not those created in Elasticsearch 1.x or before.

This condition also applies to indices backed up with snapshot and restore. If an index was originally created in 1.x, it cannot be restored into a 5.x cluster even if the snapshot was made by a 2.x cluster.

Elasticsearch 5.x nodes will fail to start in the presence of too old indices.

See Reindex to upgrade for more information about how to upgrade old indices.

Rolling upgrades

A rolling upgrade allows the Elasticsearch cluster to be upgraded one node at a time, with no downtime for end users. Running multiple versions of Elasticsearch in the same cluster for any length of time beyond that required for an upgrade is not supported, as shards will not be replicated from the more recent version to the older version.

Consult this table to verify that rolling upgrades are supported for your version of Elasticsearch.

To perform a rolling upgrade:

  1. Disable shard allocation

    When you shut down a node, the allocation process will wait for one minute before starting to replicate the shards that were on that node to other nodes in the cluster, causing a lot of wasted I/O. This can be avoided by disabling allocation before shutting down a node:

    PUT _cluster/settings
    {
      "transient": {
        "cluster.routing.allocation.enable": "none"
      }
    }
  2. Stop non-essential indexing and perform a synced flush (Optional)

    You may happily continue indexing during the upgrade. However, shard recovery will be much faster if you temporarily stop non-essential indexing and issue a synced-flush request:

    POST _flush/synced

    A synced flush request is a “best effort” operation. It will fail if there are any pending indexing operations, but it is safe to reissue the request multiple times if necessary.

  3. Stop and upgrade a single node

    Shut down one of the nodes in the cluster before starting the upgrade.

    Tip

    When using the zip or tarball packages, the config, data, logs and plugins directories are placed within the Elasticsearch home directory by default.

    It is a good idea to place these directories in a different location so that there is no chance of deleting them when upgrading Elasticsearch. These custom paths can be configured with the path.conf, path.logs, and path.data settings.

    The Debian and RPM packages place these directories in the appropriate place for each operating system.

    To upgrade using a Debian or RPM package:

    • Use rpm or dpkg to install the new package. All files should be placed in their proper locations, and config files should not be overwritten.

    To upgrade using a zip or compressed tarball:

    • Extract the zip or tarball to a new directory, to be sure that you don’t overwrite the config or data directories.
    • Either copy the files in the config directory from your old installation to your new installation, or use the -E path.conf= option on the command line to point to an external config directory.
    • Either copy the files in the data directory from your old installation to your new installation, or configure the location of the data directory in the config/elasticsearch.yml file, with the path.data setting.
  4. Upgrade any plugins

    Elasticsearch plugins must be upgraded when upgrading a node. Use the elasticsearch-plugin script to install the correct version of any plugins that you need.

  5. Start the upgraded node

    Start the now upgraded node and confirm that it joins the cluster by checking the log file or by checking the output of this request:

    GET _cat/nodes
  6. Reenable shard allocation

    Once the node has joined the cluster, reenable shard allocation to start using the node:

    PUT _cluster/settings
    {
      "transient": {
        "cluster.routing.allocation.enable": "all"
      }
    }
  7. Wait for the node to recover

    You should wait for the cluster to finish shard allocation before upgrading the next node. You can check on progress with the _cat/health request:

    GET _cat/health

    Wait for the status column to move from yellow to green. Status green means that all primary and replica shards have been allocated.

    Important

    During a rolling upgrade, primary shards assigned to a node with the higher version will never have their replicas assigned to a node with the lower version, because the newer version may have a different data format which is not understood by the older version.

    If it is not possible to assign the replica shards to another node with the higher version — e.g. if there is only one node with the higher version in the cluster — then the replica shards will remain unassigned and the cluster health will remain status yellow.

    In this case, check that there are no initializing or relocating shards (the init and relo columns) before proceding.

    As soon as another node is upgraded, the replicas should be assigned and the cluster health will reach status green.

    Shards that have not been sync-flushed may take some time to recover. The recovery status of individual shards can be monitored with the _cat/recovery request:

    GET _cat/recovery

    If you stopped indexing, then it is safe to resume indexing as soon as recovery has completed.

  8. Repeat

    When the cluster is stable and the node has recovered, repeat the above steps for all remaining nodes.

Full cluster restart upgrade

Elasticsearch requires a full cluster restart when upgrading across major versions. Rolling upgrades are not supported across major versions. Consult this table to verify that a full cluster restart is required.

The process to perform an upgrade with a full cluster restart is as follows:

  1. Disable shard allocation

    When you shut down a node, the allocation process will immediately try to replicate the shards that were on that node to other nodes in the cluster, causing a lot of wasted I/O. This can be avoided by disabling allocation before shutting down a node:

    PUT _cluster/settings
    {
      "persistent": {
        "cluster.routing.allocation.enable": "none"
      }
    }
  2. Perform a synced flush

    Shard recovery will be much faster if you stop indexing and issue a synced-flush request:

    POST _flush/synced

    A synced flush request is a “best effort” operation. It will fail if there are any pending indexing operations, but it is safe to reissue the request multiple times if necessary.

  3. Shutdown and upgrade all nodes

    Stop all Elasticsearch services on all nodes in the cluster. Each node can be upgraded following the same procedure described in [upgrade-node].

  4. Upgrade any plugins

    Elasticsearch plugins must be upgraded when upgrading a node. Use the elasticsearch-plugin script to install the correct version of any plugins that you need.

  5. Start the cluster

    If you have dedicated master nodes — nodes with node.master set to true(the default) and node.data set to false —  then it is a good idea to start them first. Wait for them to form a cluster and to elect a master before proceeding with the data nodes. You can check progress by looking at the logs.

    As soon as the minimum number of master-eligible nodes have discovered each other, they will form a cluster and elect a master. From that point on, the _cat/health and _cat/nodes APIs can be used to monitor nodes joining the cluster:

    GET _cat/health
    
    GET _cat/nodes

    Use these APIs to check that all nodes have successfully joined the cluster.

  6. Wait for yellow

    As soon as each node has joined the cluster, it will start to recover any primary shards that are stored locally. Initially, the _cat/health request will report a status of red, meaning that not all primary shards have been allocated.

    Once each node has recovered its local shards, the status will become yellow, meaning all primary shards have been recovered, but not all replica shards are allocated. This is to be expected because allocation is still disabled.

  7. Reenable allocation

    Delaying the allocation of replicas until all nodes have joined the cluster allows the master to allocate replicas to nodes which already have local shard copies. At this point, with all the nodes in the cluster, it is safe to reenable shard allocation:

    PUT _cluster/settings
    {
      "persistent": {
        "cluster.routing.allocation.enable": "all"
      }
    }

    The cluster will now start allocating replica shards to all data nodes. At this point it is safe to resume indexing and searching, but your cluster will recover more quickly if you can delay indexing and searching until all shards have recovered.

    You can monitor progress with the _cat/health and _cat/recovery APIs:

    GET _cat/health
    
    GET _cat/recovery

    Once the status column in the _cat/health output has reached green, all primary and replica shards have been successfully allocated.

Reindex to upgrade

Elasticsearch is able to use indices created in the previous major version only. For instance, Elasticsearch 5.x can use indices created in Elasticsearch 2.x, but not those created in Elasticsearch 1.x or before.

Note

Elasticsearch 5.x nodes will fail to start in the presence of too old indices.

If you are running an Elasticsearch 2.x cluster which contains indices that were created before 2.x, you will either need to delete those old indices or to reindex them before upgrading to 5.x. See Reindex in place.

If you are running an Elasticsearch 1.x cluster, you have two options:

Reindex in place

The easiest way to reindex old (1.x) indices in place is to use the Elasticsearch Migration Plugin. You will need to upgrade to Elasticsearch 2.3.x or 2.4.x first.

The reindex utility provided in the migration plugin does the following:

  • Creates a new index with the Elasticsearch version appended to the old index name (e.g. my_index-2.4.1), copying the mappings and settings from the old index. Refresh is disabled on the new index and the number of replicas is set to 0 for efficient reindexing.
  • Sets the old index to read only to ensure that no data is written to the old index.
  • Reindexes all documents from the old index to the new index.
  • Resets the refresh_interval and number_of_replicas to the values used in the old index, and waits for the index to become green.
  • Adds any aliases that existed on the old index to the new index.
  • Deletes the old index.
  • Adds an alias to the new index with the old index name, e.g. alias my_index points to index my_index-2.4.1.

At the end of this process, you will have a new 2.x index which can be used by an Elasticsearch 5.x cluster.

Upgrading with reindex-from-remote

If you are running a 1.x cluster and would like to migrate directly to 5.x without first migrating to 2.x, you can do so using reindex-from-remote.

Warning

Elasticsearch includes backwards compatibility code that allows indices from the previous major version to be upgraded to the current major version. By moving directly from Elasticsearch 1.x to 5.x, you will have to solve any backwards compatibility issues yourself.

You will need to set up a 5.x cluster alongside your existing 1.x cluster. The 5.x cluster needs to have access to the REST API of the 1.x cluster.

For each 1.x index that you want to transfer to the 5.x cluster, you will need to:

  • Create a new index in 5.x with the appropriate mappings and settings. Set the refresh_interval to -1 and set number_of_replicas to 0 for faster reindexing.
  • Use reindex-from-remote to pull documents from the 1.x index into the new 5.x index.
  • If you run the reindex job in the background (with wait_for_completion set to false), the reindex request will return a task_id which can be used to monitor progress of the reindex job in the task API: GET _tasks/TASK_ID.
  • Once reindex has completed, set the refresh_interval and number_of_replicas to the desired values (the defaults are 30s and 1 respectively).
  • Once the new index has finished replication, you can delete the old index.

The 5.x cluster can start out small, and you can gradually move nodes from the 1.x cluster to the 5.x cluster as you migrate indices across.