Important System Configuration

Ideally, Elasticsearch should run alone on a server and use all of the resources available to it. In order to do so, you need to configure your operating system to allow the user running Elasticsearch to access more resources than allowed by default.

The following settings must be addressed before going to production:

Development mode vs production mode

By default, Elasticsearch assumes that you are working in development mode. If any of the above settings are not configured correctly, a warning will be written to the log file, but you will be able to start and run your Elasticsearch node.

As soon as you configure a network setting like network.host, Elasticsearch assumes that you are moving to production and will upgrade the above warnings to exceptions. These exceptions will prevent your Elasticsearch node from starting. This is an important safety measure to ensure that you will not lose data because of a malconfigured server.

Configuring system settings

Where to configure systems settings depends on which package you have used to install Elasticsearch, and which operating system you are using.

When using the .zip or .tar.gz packages, system settings can be configured:

When using the RPM or Debian packages, most system settings are set in the system configuration file. However, systems which use systemd require that system limits are specified in a systemd configuration file.

ulimit

On Linux systems, ulimit can be used to change resource limits on a temporary basis. Limits usually need to be set as root before switching to the user that will run Elasticsearch. For example, to set the number of open file handles (ulimit -n) to 65,536, you can do the following:

sudo su  
ulimit -n 65536 
su elasticsearch 

Become root.

Change the max number of open files.

Become the elasticsearch user in order to start Elasticsearch.

The new limit is only applied during the current session.

You can consult all currently applied limits with ulimit -a.

/etc/security/limits.conf

On Linux systems, persistent limits can be set for a particular user by editing the /etc/security/limits.conf file. To set the maximum number of open files for the elasticsearch user to 65,536, add the following line to the limits.conf file:

elasticsearch  -  nofile  65536

This change will only take effect the next time the elasticsearch user opens a new session.

Note

Ubuntu and limits.conf

Ubuntu ignores the limits.conf file for processes started by init.d. To enable the limits.conf file, edit /etc/pam.d/su and uncomment the following line:

# session    required   pam_limits.so

Sysconfig file

When using the RPM or Debian packages, system settings and environment variables can be specified in the system configuration file, which is located in:

RPM

/etc/sysconfig/elasticsearch

Debian

/etc/default/elasticsearch

However, for systems which uses systemd, system limits need to be specified via systemd.

Systemd configuration

When using the RPM or Debian packages on systems that use systemd, system limits must be specified via systemd.

The systemd service file (/usr/lib/systemd/system/elasticsearch.service) contains the limits that are applied by default.

To override these, add a file called /etc/systemd/system/elasticsearch.service.d/elasticsearch.conf and specify any changes in that file, such as:

[Service]
LimitMEMLOCK=infinity

Setting JVM options

The preferred method of setting Java Virtual Machine options (including system properties and JVM flags) is via the jvm.options configuration file. The default location of this file is config/jvm.options (when installing from the tar or zip distributions) and /etc/elasticsearch/jvm.options (when installing from the Debian or RPM packages). This file contains a line-delimited list of JVM arguments, which must begin with -. You can add custom JVM flags to this file and check this configuration into your version control system.

An alternative mechanism for setting Java Virtual Machine options is via the ES_JAVA_OPTS environment variable. For instance:

export ES_JAVA_OPTS="$ES_JAVA_OPTS -Djava.io.tmpdir=/path/to/temp/dir"
./bin/elasticsearch

When using the RPM or Debian packages, ES_JAVA_OPTS can be specified in the system configuration file.

Set JVM heap size via jvm.options

By default, Elasticsearch tells the JVM to use a heap with a minimum and maximum size of 2 GB. When moving to production, it is important to configure heap size to ensure that Elasticsearch has enough heap available.

Elasticsearch will assign the entire heap specified in jvm.options via the Xms (minimum heap size) and Xmx (maximum heap size) settings.

The value for these setting depends on the amount of RAM available on your server. Good rules of thumb are:

  • Set the minimum heap size (Xms) and maximum heap size (Xmx) to be equal to each other.
  • The more heap available to Elasticsearch, the more memory it can use for caching. But note that too much heap can subject you to long garbage collection pauses.
  • Set Xmx to no more than 50% of your physical RAM, to ensure that there is enough physical RAM left for kernel file system caches.
  • Don’t set Xmx to above the cutoff that the JVM uses for compressed object pointers (compressed oops); the exact cutoff varies but is near 32 GB. You can verify that you are under the limit by looking for a line in the logs like the following:

    heap size [1.9gb], compressed ordinary object pointers [true]
  • Even better, try to stay below the threshold for zero-based compressed oops; the exact cutoff varies but 26 GB is safe on most systems, but can be as large as 30 GB on some systems. You can verify that you are under the limit by starting Elasticsearch with the JVM options -XX:+UnlockDiagnosticVMOptions -XX:+PrintCompressedOopsMode and looking for a line like the following:

    heap address: 0x000000011be00000, size: 27648 MB, zero based Compressed Oops

    showing that zero-based compressed oops are enabled instead of

    heap address: 0x0000000118400000, size: 28672 MB, Compressed Oops with base: 0x00000001183ff000

Here are examples of how to set the heap size via the jvm.options file:

-Xms2g 
-Xmx2g 

Set the minimum heap size to 2g.

Set the maximum heap size to 2g.

It is also possible to set the heap size via an environment variable. This can be done by commenting out the Xms and Xmx settings in the jvm.options file and setting these values via ES_JAVA_OPTS:

ES_JAVA_OPTS="-Xms2g -Xmx2g" ./bin/elasticsearch 
ES_JAVA_OPTS="-Xms4000m -Xmx4000m" ./bin/elasticsearch 

Set the minimum and maximum heap size to 2 GB.

Set the minimum and maximum heap size to 4000 MB.

Note

Configuring the heap for the Windows service is different than the above. The values initially populated for the Windows service can be configured as above but are different after the service has been installed. Consult the Windows service documentation for additional details.

Disable swapping

Most operating systems try to use as much memory as possible for file system caches and eagerly swap out unused application memory. This can result in parts of the JVM heap being swapped out to disk.

Swapping is very bad for performance and for node stability and should be avoided at all costs. It can cause garbage collections to last for minutes instead of milliseconds and can cause nodes to respond slowly or even to disconnect from the cluster.

There are three approaches to disabling swapping:

Enable bootstrap.memory_lock

The first option is to use mlockall on Linux/Unix systems, or VirtualLock on Windows, to try to lock the process address space into RAM, preventing any Elasticsearch memory from being swapped out. This can be done, by adding this line to the config/elasticsearch.yml file:

bootstrap.memory_lock: true
Warning

mlockall might cause the JVM or shell session to exit if it tries to allocate more memory than is available!

After starting Elasticsearch, you can see whether this setting was applied successfully by checking the value of mlockall in the output from this request:

GET _nodes?filter_path=**.mlockall

If you see that mlockall is false, then it means that the mlockall request has failed. You will also see a line with more information in the logs with the words Unable to lock JVM Memory.

The most probable reason, on Linux/Unix systems, is that the user running Elasticsearch doesn’t have permission to lock memory. This can be granted as follows:

.zip and .tar.gz
Set ulimit -l unlimited as root before starting Elasticsearch, or set memlock to unlimited in /etc/security/limits.conf.
RPM and Debian
Set MAX_LOCKED_MEMORY to unlimited in the system configuration file (or see below for systems using systemd).
Systems using systemd
Set LimitMEMLOCK to infinity in the systemd configuration.

Another possible reason why mlockall can fail is that the temporary directory (usually /tmp) is mounted with the noexec option. This can be solved by specifying a new temp directory using the ES_JAVA_OPTS environment variable:

export ES_JAVA_OPTS="$ES_JAVA_OPTS -Djava.io.tmpdir=/path/to/temp/dir"
./bin/elasticsearch

or setting this JVM flag in the jvm.options configuration file.

Disable all swap files

The second option is to completely disable swap. Usually Elasticsearch is the only service running on a box, and its memory usage is controlled by the JVM options. There should be no need to have swap enabled.

On Linux systems, you can disable swap temporarily by running: sudo swapoff -a. To disable it permanently, you will need to edit the /etc/fstab file and comment out any lines that contain the word swap.

On Windows, the equivalent can be achieved by disabling the paging file entirely via System Properties → Advanced → Performance → Advanced → Virtual memory.

Configure swappiness

The second option available on Linux systems is to ensure that the sysctl value vm.swappiness is set to 1. This reduces the kernel’s tendency to swap and should not lead to swapping under normal circumstances, while still allowing the whole system to swap in emergency conditions.

File Descriptors

Note

This is only relevant for Linux and macOS and can be safely ignored if running Elasticsearch on Windows. On Windows that JVM uses an API limited only by available resources.

Elasticsearch uses a lot of file descriptors or file handles. Running out of file descriptors can be disastrous and will most probably lead to data loss. Make sure to increase the limit on the number of open files descriptors for the user running Elasticsearch to 65,536 or higher.

For the .zip and .tar.gz packages, set ulimit -n 65536 as root before starting Elasticsearch, or set nofile to 65536 in /etc/security/limits.conf.

RPM and Debian packages already default the maximum number of file descriptors to 65536 and do not require further configuration.

You can check the max_file_descriptors configured for each node using the Nodes Stats API, with:

GET _nodes/stats/process?filter_path=**.max_file_descriptors

Virtual memory

Elasticsearch uses a hybrid mmapfs / niofs directory by default to store its indices. The default operating system limits on mmap counts is likely to be too low, which may result in out of memory exceptions.

On Linux, you can increase the limits by running the following command as root:

sysctl -w vm.max_map_count=262144

To set this value permanently, update the vm.max_map_count setting in /etc/sysctl.conf. To verify after rebooting, run sysctl vm.max_map_count.

The RPM and Debian packages will configure this setting automatically. No further configuration is required.

Number of threads

Elasticsearch uses a number of thread pools for different types of operations. It is important that it is able to create new threads whenever needed. Make sure that the number of threads that the Elasticsearch user can create is at least 2048.

This can be done by setting ulimit -u 2048 as root before starting Elasticsearch, or by setting nproc to 2048 in /etc/security/limits.conf.