Fine Tuning Payara Server 5 in Production

One of the biggest challenges when developing applications for the web is to understand how they need to be fine-tuned when releasing them into a production environment. This is no exception for Java Enterprise applications deployed on a Payara Server installation. 

Running a Payara Server setup is simple: download the current distribution suited for your needs (full, web); head to the /bin folder and start the default domain (domain1)! However, keep in mind that this default domain is tailored for development purposes (a trait inherited from GlassFish Server Open Source). When developing a web application, it’s better to quickly code features, deploy them quickly, test them, un-deploy (or redeploy) them, and continue with the next set of features until a stable state is reached. 

(last updated 06/04/2021)

In a production environment, however, a proper configuration to make the system commit to its expected quality attributes (performance, availability, reliability, etc.) is a must-have. The intention of this blog post is to provide a set of commonly recommended optimizations to apply when preparing your production environment and show how to fine-tune the critical aspects related to them.

The production Template

Usually, when fine-tuning an application server, some of the usual parameterizations done to a server configuration deal with the following aspects:

• Java Virtual Machine (JVM) Initialization Options
• General System Properties
• Networking and HTTP Listeners 
• Class loading and File Handling
• JDBC Settings
• EJB Container Settings

Payara Server Enterprise 5 includes a  domain “template” called production. This domain is pre-configured with some common settings that deal with these aspects described earlier and helps improve the performance in almost all scenarios. This production template is no longer available on the Community edition as Community is not suited for running applications in a critical environment. Payara Community has no upgrade guarantees and contains experimental implementations and changes that can affect stability.

How do you make use of this template? Instead of running the default domain or a custom domain, run this domain instead and use it to deploy your applications:

./asadmin start-domain production 

This domain template uses the same default ports that are configured for the domain1 domain (8080 for HTTP, 8181 for HTTPS, 4848 for the DAS, etc.), so you do not have to worry about reconfiguring these ports if you rely on the default configuration.

Are you wondering what optimizations are included in this domain template? We’ll highlight them in the following sections as we detail the configuration settings to be applied in a production environment.

As a simpler convention to illustrate these optimizations, we will use the corresponding element in the domain.xml configuration file that needs to be updated. This is to keep consistency across the article. Also, assume relevant configurations will be done on the server-config configuration group. 

JVM Options

The default JVM options are not suitable for a Java application that should be constantly running in production, so the following changes are always recommended:

Fine Tune Heap Size (Included with production):

./asadmin delete-jvm-options '-Xmx512m'
./asadmin create-jvm-options '-Xmx2048m:-Xms2048m'

Adjust the Xmx and Xms options to increase the amount of memory allocated for the Heap space. Also, set both options with the same value so the server allocates all memory available at startup time instead of executing multiple re-allocations of memory at runtime.

As a personal recommendation, allocate an amount of memory between 2 and 16 Gigabytes. More than that size would mean an unmanageable number of objects that need to be managed by the garbage collector, so this can affect the performance of the JVM. If your applications live in a Heap space that needs to be that big, consider implementing a clustered arrangement so the workload is reduced overall.

On the production template, the heap space is set to 2G by default, so remember to adjust according to your memory specifications.

Fine Tune the MetaSpace size (Included with Production):

./asadmin create-jvm-options '-XX\:MetaspaceSize=256m:-XX\:MaxMetaspaceSize=2g'

Increase both the initial and maximum size of memory to be allocated to the MetaSpace, which is the memory space that replaced the PermGen space in Java 8. The MetaSpace handles memory allocation for all static content in an automatic manner, meaning that this space is subject to garbage collection as well. Configure the size of this space with sensible defaults that consider the amount of memory to contain references to the static content of all classes loaded by the JVM. 

By default, on the production template the initial size is set to 256M and the maximum size is set to 2G, which is more than good enough for most production environments.

Disable Secure Client-Initiated Renegotiation:

./asadmin create-jvm-options -Djdk.tls.rejectClientInitiatedRenegotiation=true

This is necessary in order to avoid DDoS attacks occurring on the server due to the TLS protocol feature of secure client-initiated renegotiation. This option is enabled for all domain templates, so you should never disable it.

Enable Isolated Class loading

./asadmin create-jvm-options -Dfish.payara.classloading.delegate=false

Consider enabling isolated class loading (by disabling class loader delegation) of all classes that are included in third-party libraries and modules that are already part of Payara Server. This setting will be applied to all libraries resting under the lib folder of the domain in question and those included inside the applications as well. It’s a good idea to enable this feature if your applications heavily depend on common third-party libraries like Apache Commons and Google Guava, for example.

Fine-Tune Garbage Collection

A Garbage Collector reclaims “unused” heap space allocated on the JVM by identifying all objects that are no longer needed. Since this process of locating and removing these objects can slow down the performance of an application, an efficient garbage collector algorithm should be provided to maximize the performance of the server under heavy load.

Fine-tuning the Garbage Collector is a huge topic that can fill books by itself, so it is outside the scope of this article. Our recommendation is to choose wisely the Garbage Collector algorithm that the server’s JVM will use based on the following metrics:

• Percentage of CPU usage
• Frequency of CPU pauses
• Shorter vs. Longer CPU pauses

For most production environments, the use of the G1 garbage collector is recommended. The G1 collector is server-oriented and targeted for multi-processor machines with large memory pools and it is usually a good match considering that it can:

• Work concurrently with application threads
• Compact free space without lengthy GC-induced pause times
• Rely on predictable GC pause durations
• Prevent the sacrifice of significant throughput performance
• Require moderate heap space

To configure the use of this collector, just set the following JVM setting:

./asadmin create-jvm-options '-XX\:+UseG1GC'

An additional optimization to consider along the G1 garbage collector is to enable String Deduplication, which allows the G1 collector to remove duplicate string objects from all memory regions. This feature can be enabled with the following JVM setting: 

./asadmin create-jvm-options '-XX\:+UseStringDeduplication'

Both of these settings are configured by default on the production domain template. 

We also recommend disabling explicit garbage collections triggered by the System.gc method with the following property:

./asadmin create-jvm-options '-XX\:+DisableExplicitGC'

In rare cases, developers will make use of such feature of the JDK but disabling this method will prevent high CPU usage peaks on the server due to unwanted garbage collection iterations.

Disable Development Features

You should always disable both the auto deployment and dynamic reloading of applications in your production environment. These features help speed up the development of an application but are not needed at all on a normal operation, since they affect the general performance (by using server resources). 

To disable these features, adjust the attributes of the das-config tag in the domain.xml file:

./asadmin set configs.config.server-config.admin-service.das-config.dynamic-reload-enabled=false configs.config.server-config.admin-service.das-config.autodeploy-enabled=false

It’s also a good practice to disable the JspServlet dynamic reloading so it isn’t constantly checking changes on JSP files. Configure the management of all String values as static character arrays, since this will lower the memory used by the server.

To implement both changes, you will need to edit the servlet configuration under the domain/config/default-web.xml configuration file:

    <servlet>
        <servlet-name>jsp</servlet-name>
        <servlet-class>org.apache.jasper.servlet.JspServlet</servlet-class>
        <init-param>
            <param-name>development</param-name>
            <param-value>false</param-value>
        </init-param>
        <init-param>
            <param-name>genStrAsCharArray</param-name>
            <param-value>true</param-value>
        </init-param>
        …
    </servlet>

The features are disabled by default on the production domain template.

EJB Pool Settings

When an application uses a set of EJBs to model business components, keep in mind that the application server's EJB container caches, pools and reuses EJB component instances to improve performance and promote efficient resource access. If your applications make use of Stateless EJBs you need to fine-tune the pool settings to maintain performance:

./asadmin set configs.config.server-config.ejb-container.pool-resize-quantity=2 configs.config.server-config.ejb-container.max-pool-size=120 configs.config.server-config.ejb-container.steady-pool-size=10

Assign appropriate values to the minimum size of the pool (steady-pool-size), maximum pool size, and the resize quantity used by the container when resizing the EJB pool. I recommend assigning values based on the number of concurrent clients your applications will manage: set the maximum pool size to handle the anticipated high load of the system, the minimum/initial pool size to handle a moderate load and the resize quantity to a large number if the system is expecting frequent peaks of high load on its lifetime. The resize quantity should be small as performance will degrade as many EJB beans doing heavy initialization will slow down request handling when resize is happening.

The production template sets the maximum value of the pool to 128 by default, which is a sensible default for applications that make use of intensive EJB calls.

If your application makes use of remote EJB interfaces, you must configure the ORB settings accordingly as well. The ORB (CORBA Object Request Broker) works using the RMI/IIOP protocols for remote communications with client applications (either Java EE application clients or classical stand-alone applications). To service requests from these remote clients, a thread pool is used to handle the availability of threads to handle remote call executions on the server. 

Adjust the maximum and minimum sizes of the thread pool assigned to the ORB service (by default assigned to the name thread-pool-1). Pay special attention to the maximum value since you can use it to improve performance: if your applications will be handling a lot of calls from remote clients, an appropriate number of threads should be used to service them without slowing down the server.

./asadmin set configs.config.server-config.thread-pools.thread-pool.thread-pool-1.min-thread-pool-size=150 configs.config.server-config.thread-pools.thread-pool.thread-pool-1.max-thread-pool-size=500

The production template sets the maximum value of this thread pool to 250 by default, which accounts for around 250 concurrent users. It is recommended to adjust this value for the correct number of concurrent requests.

HTTP and Network Configuration

When releasing a web application on a production environment, it is extremely important to configure the HTTP network listeners appropriately, since the server needs to handle HTTP client requests without compromising its availability and reliability. 

By default, each domain comes pre-configured with 3 network listeners: 

• admin-listener, which handles HTTP connections for the admin console web interface. This listener rarely needs fine-tuning.
• http-listener-1, which handles HTTP connections for all web applications.
• http-listener-2, which handles HTTPS exclusively connections for all web applications. 

HTTP or HTTPS?

If you are planning to use HTTPS-only connections for your production environment (this is a common occurrence in cloud environments), it is recommended that you disable http-listener-1:

./asadmin set configs.config.server-config.network-config.network-listeners.network-listener.http-listener-1.enabled=false

In some cases, if no HTTPS access is required by your security infrastructure (intranet environments or applications that handle public or non-securable data), you can disable http-listener-2 instead. If you are considering of using HTTP version 2 instead, you will have to configure secure access regardless (since this is mandated by browsers that implement the protocol), so keep this in mind.

If you need to redirect all HTTP traffic to HTTPS, we recommend you use a Load Balancer setup or a simple web server that fronts your Payara Server to handle this task. 

HTTP Settings

Set the maximum number of requests per connection available to your network listener depending on the number of concurrent users your web application is expecting. Keep in mind that the number of request x connections specified is divided equally among the keep-alive threads to serve HTTP requests.

Set the timeout for the maximum time in seconds that the server holds HTTP connections open. Setting this timeout is a tricky optimization since a client can keep a connection to the server open so that multiple requests to the server can be serviced by a single network connection. Since the server can only handle a limited number of open connections, you need to make sure that connections are open if possible, in case the system handles a heavy load of requests. 

Finally, enable the file cache of the listener. This cache contains information about static files like HTML, CSS, images or plain text files that are hosted by the server. Enabling this cache can improve the performance of the server by reducing the number of search operations the server will do on the file system. Also, configure an appropriate time for the max age (in seconds) these files will live on the cache before expiring.

Configure these parameters in the protocol element of the corresponding network listener:

By default, file caching is enabled on both the default domain (domain1) and the production domain template, so there’s no reason for you to disable it.

If the machine the server lives on only possesses one network interface, set the Network Address of the network listener to the specific IP address of this network interface. This will improve performance since the server won’t have to open socket listeners for each available address to the operating system:

./asadmin set configs.config.server-config.network-config.protocols.protocol.http-listener-1.http.max-connections=500 configs.config.server-config.network-config.protocols.protocol.http-listener-1.http.timeout-seconds=60
./asadmin set configs.config.server-config.network-config.protocols.protocol.http-listener-1.http.file-cache.max-age-seconds=3600 configs.config.server-config.network-config.protocols.protocol.http-listener-1.http.file-cache.enabled=true

If the server machine has multiple network interface cards, it is a good practice to create multiple network listeners and assign them the IP addresses of each network interface available. Use a load balancer or web server to distribute requests among these network listeners for your client applications. 

Acceptor and Request Threads 

Configure the number of acceptor threads for the transport service you want to use. An acceptor thread works by listening to new requests on a socket and passing them to request threads. It is recommended to set this number of threads to be equal to the number of CPUs the server machine has. For example, if your server has 4 Quad-Core CPUs, the total number of acceptor threads should be 16 (4x4). Set this setting in the TCP transport element:

./asadmin set configs.config.server-config.network-config.transports.transport.tcp.acceptor-threads=16

On the other hand, request threads execute HTTP requests. These are handled by a thread-pool in the same vein that ORB requests are handled as well. The thread pool for both the http-listener-1 and http-listener-2 listeners is called http-thread-pool. Assign the minimum and maximum values to this pool with values ranging from 50 to 500 depending on the expected load and user base your applications have. It is always a good idea to adjust these values after carefully monitoring how many threads are used by the pool when the system is on a heavy load and correct the values incrementally:

./asadmin set configs.config.server-config.thread-pools.thread-pool.http-thread-pool.min-thread-pool-size=25 configs.config.server-config.thread-pools.thread-pool.http-thread-pool.max-thread-pool-size=350

On the production domain template, this thread pool has a configured maximum value of 50 threads, which is a good starting number in most scenarios as stated previously.

JDBC Pool Configuration

If you have applications that make intensive use of database connections, it’s best to tune the JDBC Connection Pool settings accordingly. Since the server should open, reuse and close these connections based on the pool’s configuration, fine-tuning these settings will guarantee a stable performance under heavy load.

Assign the minimum and maximum pool size depending on how many open connections your application needs: Consider setting a higher number that is proportional to the number of requests you will receive.

Also, if the gap between the minimum and maximum sizes is too big, consider setting a higher number for the resize quantity so more connections are created faster by the server.

Also, set the maximum wait time of the connection pool to zero (0) if possible. By doing this, every caller thread will become blocked until a connection is available. This way, you can make the server avoid tracking the elapsed wait time for each connection request and can improve performance. This is a double-edged sword: if your applications tend to hog JDBC connections, it’s possible the pool can become a bottleneck by making a lot of connection requests wait indefinitely.

./asadmin set resources.jdbc-connection-pool.<pool-name>.max-pool-size=64 \ 
   resources.jdbc-connection-pool.<pool-name>.max-wait-time-in-millis=0 \
   resources.jdbc-connection-pool.<pool-name>.steady-pool-size=10

Domain Data Grid

Payara Server 5 introduced the domain data grid, an in-memory data structure that allows the creation of distributed deployments and clustering across a set of Payara Server and or Payara Micro instances. The domain data grid is always turned on, which introduces the following benefits for production environments.

• Easy creation and configuration of clusters composed of multiple Payara Server (and Payara Micro) instances.
• The use of the JCache API, introduces distributed caching mechanisms with automatic failover and data replication across the cluster.
• Light-weight messaging via remote CDI events (this is a proprietary feature of the Payara Platform)
• Functions as a backing store for web sessions, persistent EJB timers, performance metrics, clustered singletons and other specific data that requires replication across the cluster.

Under the covers, the domain data grid is implemented using a customized Hazelcast in-memory data grid. If you are familiar with the Hazelcast features that were implemented in Payara Server 4, you will recognize that the domain data grid is an amplification of that set of features (many of the internal configurations and available commands remain the same for example).

For production environments, the configuration of the data grid will depend on the specific topology and architecture of the applications being deployed, so I won’t be covering much information on this topic. The only recommendation that can be given to configure the data grid in a production environment is to change both the name and password of the cluster from the default values. This can be done by running the following asadmin commands:

./asadmin set-hazelcast-configuration --clusterName=<cluster-name> --clusterPassword=<cluster-password>

And as of 5.2021.1 which uses Hazelcast 4.x that doesn't use the cluster password anymore.

./asadmin set-hazelcast-configuration --clusterName=<cluster-name> 

This is a recommended configuration in order to prevent the domain data grid from being “breached” by allowing other Hazelcast nodes to join it by using the default credentials (which could be an embarrassing situation to have since sensitive data could be contained in the grid)  and also to prevent unwanted cross-cluster talk.

Dockerizing a Production Environment

Many production environments nowadays are provisioned using Docker containers due to the ease of managing a whole environment with simple files (Infrastructure as Code) and the excellent synergy that the JVM has with most containers. For production environments, my recommendation is to use the official payara/server-full image, which is already fine-tuned with the following settings:

• Runs the Payara Server domain in the foreground by using a custom script that prepares the start domain instructions in the proper way.
• Configures the underlying JVM process memory management by:
• Setting the -XX:+UseCGroupMemoryLimitForHeap JVM option which sets the maximum amount of memory (and heap space) available to the JVM to the current value limited for the container as defined by the cgroup memory limits. This option was introduced in JDK 9 but was backported to JDK1.8u131. 
• Setting the -XX:MaxRAMFraction=1 JVM option which instructs the container to use all available memory to allocate it between heap space and other memory regions as well. Since the container’s only responsibility is hosting the server’s domain all memory should be used by the JVM.
• Remove the Xmx and Xms options since memory management is dictated by the 2 previous options.
• Allow the deployment of all application binaries (JAR, WAR, EAR and RAR files) located in the default deployment directory of the container, identified by the $DEPLOY_DIR environment variable, whose default value is /opt/payara/deployments
• Allow the configuration of the domain by specifying asadmin domains that are detailed in either the following:
• A pre-boot commands file that is referenced by the $PREBOOT_COMMANDS environment variable, whose default value is /opt/payara/config/pre-boot-commands.asadmin
• A post-boot commands file that is referenced by the $POSTBOOT_COMMANDS environment variable, whose default value is /opt/payara/config/post-boot-commands.asadmin

As you can see, the default image already covers a lot of ground with the customization options available to it. In the case of fine-tuning, since memory management is deferred to the container’s available memory and the default domain being used is the production domain there is not much to be done. In order to run a production-level container you can do it like this:

docker run -p 8080:8080 -v ~/my-applications-dir:/opt/payara/deployments payara/server-full

You can notice that the run command is mounting the local directory my-applications to the deployment directory in the container’s filesystem, which will allow all application binaries (suffixed with either .ear, .war, .rar or .jar) to be deployed when the server starts. 

How about the data grid configuration we mentioned previously? In that case, we must run the corresponding administration commands by creating a post-boot command script like this:

#File datagrid-conf.asadmin

set-hazelcast-configuration --clusterName=production-cluster --clusterPassword=mypassword

Then mount the directory that contains the script to the default configuration directory of the container like this:

mv datagrid-conf.asadmin ~/command-scripts/post-boot-commands.asadmin
docker run -p 8080:8080 -v ~/my-applications-dir:/opt/payara/deployments -v ~/command-scripts/:/opt/payara/config payara/server-full

However, a much simpler solution in most cases is to create a custom Docker image that is based on the official image like this:

FROM payara/server-full

COPY my-application.war ${DEPLOY_DIR}

COPY datagrid-conf.asadmin ${CONFIG_DIR}/post-boot-commands.asadmin

And that’s it! If you want to fine-tune additional configuration settings, just add the corresponding asadmin commands in the post-boot command script and that should be enough. Do you want to apply these optimizations but don’t know the specific commands needed to apply them? Worry not, the Asadmin recorder feature will help you to set the correct commands! 

Fine Tuning Prevents Performance Degradation and Measures System Performance

Fine-tuning Payara Server for a production environment is a task that should ALWAYS be done to prevent performance degradation and measure how the system behaves under different weights of workload. Installing a server and letting it stick to the default configuration is something that should NEVER be done, so plan carefully what optimizations are needed for your software architecture and apply them when applicable. 

The production template is a huge help in this regard since it is configured with tried-and-tested sensible defaults for most common use cases! Just don’t blindly depend on the optimizations already included within it, be sure to test your system under the appropriate conditions and add any additional optimizations as they are needed.

Note: This blog is about Payara Server 5. If you're still running Payara Server 4, you'll want to look at Fine Tuning Payara Server 4 in Production. Also note, Payara Server 4 is maintained only for customers at this time with no new features or enhancements - so you should be looking toward upgrading to Payara Server 5 or consider paid Payara support services.

Learn more about the benefits of Payara Enterprise