Bootstrap Checks

Collectively, we have a lot of experience with users suffering unexpected issues because they have not configured important settings. In previous versions of Elasticsearch, misconfiguration of some of these settings were logged as warnings. Understandably, users sometimes miss these log messages. To ensure that these settings receive the attention that they deserve, Elasticsearch has bootstrap checks upon startup.

These bootstrap checks inspect a variety of Elasticsearch and system settings and compare them to values that are safe for the operation of Elasticsearch. If Elasticsearch is in development mode, any bootstrap checks that fail appear as warnings in the Elasticsearch log. If Elasticsearch is in production mode, any bootstrap checks that fail will cause Elasticsearch to refuse to start.

There are some bootstrap checks that are always enforced to prevent Elasticsearch from running with incompatible settings. These checks are documented individually.

Development vs. production mode

By default, Elasticsearch binds to localhost for HTTP and transport (internal) communication. This is fine for downloading and playing with Elasticsearch, and everyday development but it’s useless for production systems. To form a cluster, Elasticsearch instances must be reachable via transport communication so they must bind transport to an external interface. Thus, we consider an Elasticsearch instance to be in development mode if it does not bind transport to an external interface (the default), and is otherwise in production mode if it does bind transport to an external interface. Note that HTTP can be configured independently of transport via http.host and transport.host; this can be useful for configuring a single instance to be reachable via HTTP for testing purposes without triggering production mode.

Heap size check

If a JVM is started with unequal initial and max heap size, it can be prone to pauses as the JVM heap is resized during system usage. To avoid these resize pauses, it’s best to start the JVM with the initial heap size equal to the maximum heap size. Additionally, if bootstrap.memory_lock is enabled, the JVM will lock the initial size of the heap on startup. If the initial heap size is not equal to the maximum heap size, after a resize it will not be the case that all of the JVM heap is locked in memory. To pass the heap size check, you must configure the heap size.

File descriptor check

File descriptors are a Unix construct for tracking open "files". In Unix though, everything is a file. For example, "files" could be a physical file, a virtual file (e.g., /proc/loadavg), or network sockets. Elasticsearch requires lots of file descriptors (e.g., every shard is composed of multiple segments and other files, plus connections to other nodes, etc.). This bootstrap check is enforced on OS X and Linux. To pass the file descriptor check, you might have to configure file descriptors.

Memory lock check

When the JVM does a major garbage collection it touches every page of the heap. If any of those pages are swapped out to disk they will have to be swapped back in to memory. That causes lots of disk thrashing that Elasticsearch would much rather use to service requests. There are several ways to configure a system to disallow swapping. One way is by requesting the JVM to lock the heap in memory through mlockall (Unix) or virtual lock (Windows). This is done via the Elasticsearch setting bootstrap.memory_lock. However, there are cases where this setting can be passed to Elasticsearch but Elasticsearch is not able to lock the heap (e.g., if the elasticsearch user does not have memlock unlimited). The memory lock check verifies that if the bootstrap.memory_lock setting is enabled, that the JVM was successfully able to lock the heap. To pass the memory lock check, you might have to configure mlockall.

Maximum number of threads check

Elasticsearch executes requests by breaking the request down into stages and handing those stages off to different thread pool executors. There are different thread pool executors for a variety of tasks within Elasticsearch. Thus, Elasticsearch needs the ability to create a lot of threads. The maximum number of threads check ensures that the Elasticsearch process has the rights to create enough threads under normal use. This check is enforced only on Linux. If you are on Linux, to pass the maximum number of threads check, you must configure your system to allow the Elasticsearch process the ability to create at least 2048 threads. This can be done via /etc/security/limits.conf using the nproc setting (note that you might have to increase the limits for the root user too).

Maximum size virtual memory check

Elasticsearch and Lucene use mmap to great effect to map portions of an index into the Elasticsearch address space. This keeps certain index data off the JVM heap but in memory for blazing fast access. For this to be effective, the Elasticsearch should have unlimited address space. The maximum size virtual memory check enforces that the Elasticsearch process has unlimited address space and is enforced only on Linux. To pass the maximum size virtual memory check, you must configure your system to allow the Elasticsearch process the ability to have unlimited address space. This can be done via /etc/security/limits.conf using the as setting to unlimited (note that you might have to increase the limits for the root user too).

Maximum map count check

Continuing from the previous point, to use mmap effectively, Elasticsearch also requires the ability to create many memory-mapped areas. The maximum map count check checks that the kernel allows a process to have at least 262,144 memory-mapped areas and is enforced on Linux only. To pass the maximum map count check, you must configure vm.max_map_count via sysctl to be at least 262144.

Client JVM check

There are two different JVMs provided by OpenJDK-derived JVMs: the client JVM and the server JVM. These JVMs use different compilers for producing executable machine code from Java bytecode. The client JVM is tuned for startup time and memory footprint while the server JVM is tuned for maximizing performance. The difference in performance between the two VMs can be substantial. The client JVM check ensures that Elasticsearch is not running inside the client JVM. To pass the client JVM check, you must start Elasticsearch with the server VM. On modern systems and operating systems, the server VM is the default. Additionally, Elasticsearch is configured by default to force the server VM.

Use serial collector check

There are various garbage collectors for the OpenJDK-derived JVMs targeting different workloads. The serial collector in particular is best suited for single logical CPU machines or extremely small heaps, neither of which are suitable for running Elasticsearch. Using the serial collector with Elasticsearch can be devastating for performance. The serial collector check ensures that Elasticsearch is not configured to run with the serial collector. To pass the serial collector check, you must not start Elasticsearch with the serial collector (whether it’s from the defaults for the JVM that you’re using, or you’ve explicitly specified it with -XX:+UseSerialGC). Note that the default JVM configuration that ship with Elasticsearch configures Elasticsearch to use the CMS collector.

OnError and OnOutOfMemoryError checks

The JVM options OnError and OnOutOfMemoryError enable executing arbitrary commands if the JVM encounters a fatal error (OnError) or an OutOfMemoryError (OnOutOfMemoryError). However, by default, Elasticsearch system call filters (seccomp) are enabled and these filters prevent forking. Thus, using OnError or OnOutOfMemoryError and system call filters are incompatible. The OnError and OnOutOfMemoryError checks prevent Elasticsearch from starting if either of these JVM options are used and system call filters are enabled. This check is always enforced. To pass this check do not enable OnError nor OnOutOfMemoryError; instead, upgrade to Java 8u92 and use the JVM flag ExitOnOutOfMemoryError. While this does not have the full capabilities of OnError nor OnOutOfMemoryError, arbitrary forking will not be supported with seccomp enabled.

G1GC check

Early versions of the HotSpot JVM that shipped with JDK 8 are known to have issues that can lead to index corruption when the G1GC collector is enabled. The versions impacted are those earlier than the version of HotSpot that shipped with JDK 8u40. The G1GC check detects these early versions of the HotSpot JVM.