One of the biggest challenges when developing applications for the web is understanding how they need to be fine-tuned when releasing them into a production environment. This is no exception for Jakarta EE applications deployed on a Payara Server installation.  

Running a Payara Server setup is straightforward. First, download the distribution that best fits your needs (either full or web). Next, navigate to the /bin folder and start the default domain (domain1). It's important to note that the default domain included with the Community edition is designed for development purposes, a feature inherited from GlassFish Server Open Source. Payara Server Enterprise, however, comes with an optimized production domain, ideal for production deployments. When developing a web application, it’s beneficial to iterate quickly until you achieve a stable state.  

In a production environment, proper configuration is essential to ensure the system meets its expected quality attributes, such as performance, availability, and reliability. This blog post offers a set of commonly recommended optimizations that can be applied when preparing your production environment. It also presentshow to fine-tune the critical aspects related to these attributes. 

The production Template 

When fine-tuning an application server, common parameterizations for a server configuration address the following aspects: 

• Java Virtual Machine (JVM) Initialization Options 

• General System Properties 

• Networking and HTTP Listeners  

• Class loading and File Handling 

• JDBC Settings 

• EJB Container Settings 

 
Starting with version 5, Payara Server Enterprise includes a domain “template” called production. This domain is pre-configured with some common settings for these aspects and helps improve performance in most scenarios. This production template is no longer available in the Community edition, as the platform is not suited for applications in a critical environment. Payara Community lacks upgrade guarantees and contains experimental implementations as well as changes that can affect stability. 

How do you make use of this template? Instead of running the default domain or a custom domain, run this domain instead and use it to deploy your applications: 

./asadmin start-domain production  

This domain template uses the same default ports that are configured for the domain1 domain (8080 for HTTP, 8181 for HTTPS, 4848 for the DAS, etc.), so you do not have to worry about reconfiguring these ports if you rely on the default configuration. 

Are you wondering what optimizations are included in this domain template? We’ll highlight these in the following sections as we detail the configuration settings to be applied in a production environment. 

As a simpler convention to illustrate these optimizations, we will use the corresponding element in the domain.xml configuration file that needs to be updated. This is to keep consistency across the post. Also, assume relevant configurations will be done on the server-config configuration group.  

JVM Options 

The default JVM options are not always suitable for a Java application running in a production environment. The following changes are recommended for optimizing performance: 

Fine Tune Heap Size (included with production): 

./asadmin delete-jvm-options '-Xmx512m' 
./asadmin create-jvm-options '-Xmx2048m:-Xms2048m' 

Adjust the Xmx and Xms options to increase the amount of memory allocated for the Heap space. Also, set both options to the same value so the server allocates all memory available at startup, avoiding runtime reallocations. This practice ensures that the JVM does not have to negotiate with the operating system to allocate memory during runtime. When different sizes are set for the initial (-Xms) and maximum (-Xmx) heap size, the JVM must dynamically allocate and deallocate memory as needed. This constant allocation and deallocation can add overhead, impacting application performance. Additionally, setting the initial heap size to be the same as the maximum heap size improves the application’s startup time by avoiding the overhead of growing the heap during startup. 

As a personal recommendation, allocate an amount of memory between 2 and 16 Gigabytes. Allocating more than that size might lead to an unmanageable number of objects that need to be managed by the garbage collector, potentially impacting JVM performance. If your applications live in a Heap space that needs to be that big, consider implementing a clustered arrangement to distribute the workload. 

In the production template, the heap space is set to 2G by default, so remember to adjust according to your memory specifications. 

Fine Tune the MetaSpace size (included with Production): 

./asadmin create-jvm-options '-XX\:MetaspaceSize=256m:-XX\:MaxMetaspaceSize=2g' 

Increase both the initial and maximum size of memory to be allocated to the MetaSpace, which is the memory space that replaces the PermGen space in Java 11. The MetaSpace handles memory allocation for all static content, such as class definitions and method definitions, in an automatic manner. It is subject to garbage collection. Configure the size of this space with sensible defaults to ensure that there is adequate memory for the JVM to manage these definitions efficiently.  

By default, in the production template the initial size is set to 256M, and the maximum size is set to 2G, which is more than good enough for most production environments. 

Disable Secure Client-Initiated Renegotiation: 

./asadmin create-jvm-options -Djdk.tls.rejectClientInitiatedRenegotiation=true 

This is necessary in order to avoid distributed denial-of-service (DDoS) attacks occurring on the server due to the TLS protocol feature of secure client-initiated renegotiation. This option is enabled for all domain templates, so you should never disable it, as doing so will make your server vulnerable to a DoS attack. 

Enable Isolated Class loading 

./asadmin create-jvm-options -Dfish.payara.classloading.delegate=false 

Payara Server extends the default JVM classloading and provides additional classes, which are loaded by the class loaders included by default on this distribution. 

Consider enabling isolated class loading (by disabling class loader delegation) of all classes that are included in third-party libraries and modules that are already part of the Payara Server. This setting will be applied to all libraries resting under the lib folder of the domain in question and those included inside the applications as well. It’s a good idea to enable this feature if your applications heavily depend on common third-party libraries like Apache Commons, for example. 

Thread Stack Size 

The default value for the -Xss JVM parameter, which sets the thread stack size, varies depending on the Java version and the operating system. Setting the -Xss parameter is important for controlling the memory usage of your application, especially if it uses many threads. A smaller stack size can help save memory, but if it is set too low and the threads require more stack space than allocated, it can cause StackOverflowError. 

./asadmin create-jvm-options '-Xss512k' 

If your application is big and has a considerable number of classes and calls across multiple methods, increasing the stack size is recommended as a preventive measure. 

Fine-Tune Garbage Collection

A Garbage Collector (GC) reclaims “unused” heap space allocated on the JVM by identifying all objects that are no longer needed. Since this process of locating and removing these objects can slow down the performance of an application, an efficient garbage collector algorithm should be provided to maximize the performance of the server under heavy load. 

Fine-tuning the GC is a vast topic that can fill books on its own, so it is outside the scope of this post. Our recommendation is to choose wisely the GC algorithm that the server’s JVM will use based on the following metrics: 

• Percentage of CPU usage 

• Frequency of CPU pauses 

• Shorter vs. Longer CPU pauses 

For most production environments, the use of the G1 GC is recommended. For Java 11, G1 GC is the default GC algorithm. G1 collector is server-oriented and targeted for multi-processor machines with large memory pools and it is usually a good match considering that it can: 

• Work concurrently with application threads 

• Compact free space without lengthy GC-induced pause times 

• Rely on predictable GC pause durations 

• Prevent the sacrifice of significant throughput performance 

• Require moderate heap space 

Versions of Java after 11 have also introduced new and experimental GCs. You should test them and find the most appropriate for your workload.  

To configure the use of this collector, just set the following JVM setting: 

./asadmin create-jvm-options '-XX\:+UseG1GC' 

An additional optimization to consider along the G1 GC is to enable String Deduplication, which allows the G1 collector to remove long-lived duplicate string objects from all memory regions. This feature can be enabled with the following JVM setting:  

./asadmin create-jvm-options '-XX\:+UseStringDeduplication' 

Both of these settings are configured by default on the production domain template.  

We also recommend disabling explicit garbage collections triggered by the System.gc method with the following property: 

./asadmin create-jvm-options '-XX\:+DisableExplicitGC' 

In rare cases, developers will make use of such feature of the JDK, but disabling this method will prevent high CPU usage peaks on the server. This is due to unwanted stop-the-world full garbage collection iterations, which further freezes the JVM. 

In most cases, G1 GC is sufficient. However, if your big application has an extremely large heap size, you may consider configuring the Z Garbage Collector (ZGC) with specific settings. 

Disable Development Features 

You should always disable both the auto deployment and dynamic reloading of applications in your production environment. These features help speed up the development of an application but are not needed at all on a normal operation, since they affect the general performance (by using server resources).  

To disable these features, adjust the attributes of the das-config tag in the domain.xml file: 

./asadmin set configs.config.server-config.admin-service.das-config.dynamic-reload-enabled=false configs.config.server-config.admin-service.das-config.autodeploy-enabled=false 

It’s also a good practice to disable the JspServlet dynamic reloading, so it isn’t constantly checking changes on JSP files. Configure the management of all String values as static character arrays, since this will lower the memory used by the server. 

To implement both changes, you will need to edit the servlet configuration under the domain/config/default-web.xml configuration file: 

    <servlet> 

       <servlet-name>jsp</servlet-name> 

       <servlet-class>org.glassfish.wasp.servlet.JspServlet</servlet-class>   

         <init-param> 

            <param-name>development</param-name> 

            <param-value>false</param-value> 

        </init-param> 

        <init-param> 

            <param-name>genStrAsCharArray</param-name> 

            <param-value>true</param-value> 

        </init-param> 

        … 

    </servlet> 

The features are disabled by default on the production domain template. 

EJB Pool Settings 

When an application uses a set of EJBs to model business components, keep in mind that the application server's EJB container caches, pools and reuses EJB component instances to improve performance and promote efficient resource access. If your applications make use of Stateless EJBs you need to fine-tune the pool settings to maintain performance: 

./asadmin set configs.config.server-config.ejb-container.pool-resize-quantity=2 configs.config.server-config.ejb-container.max-pool-size=120 configs.config.server-config.ejb-container.steady-pool-size=10 

Assign appropriate values to the minimum size of the pool (steady-pool-size), maximum pool size, and the resize quantity used by the container when resizing the EJB pool. I recommend assigning values based on the number of concurrent clients your applications will manage. Set the maximum pool size to handle the anticipated high load of the system, the minimum/initial pool size to handle a moderate load and the resize quantity to a large number if the system is expecting frequent peaks of high load on its lifetime. The resize quantity should be small, as performance will degrade. In effect, many EJB beans doing heavy initialization will slow down request handling when resize is happening. 

The production template sets the maximum value of the pool to 128 by default, which is a sensible value for applications that make use of intensive EJB calls. 

If your application makes use of remote EJB interfaces, you must configure the ORB settings accordingly as well. The ORB (CORBA Object Request Broker) works using the RMI/IIOP protocols for remote communications with client applications (either Java EE application clients or classical stand-alone applications). To service requests from these remote clients, a thread pool is used to handle the availability of threads to handle remote call executions on the server.  

Adjust the maximum and minimum sizes of the thread pool assigned to the ORB service. By default, this is assigned to the name thread-pool-1. Pay special attention to the maximum value, since you can use it to improve performance. If your applications will be handling a lot of calls from remote clients, an appropriate number of threads should be used to service them without slowing down the server. 

./asadmin set configs.config.server-config.thread-pools.thread-pool.thread-pool-1.min-thread-pool-size=150 configs.config.server-config.thread-pools.thread-pool.thread-pool-1.max-thread-pool-size=500 

The production template sets the maximum value of this thread pool to 250 by default, which accounts for around 250 concurrent users. It is recommended to adjust this value for the correct number of concurrent requests. 

HTTP and Network Configuration 

When releasing a web application on a production environment, it is extremely important to configure the HTTP network listeners appropriately, since the server needs to handle HTTP client requests without compromising its availability and reliability.  

 
By default, each domain comes pre-configured with 3 network listeners:  

• admin-listener, which handles HTTP connections for the admin console web interface. This listener rarely needs fine-tuning. 

• http-listener-1, which handles HTTP connections for all web applications. 

• http-listener-2, which exclusively handles HTTPS connections for all web applications.  

HTTP or HTTPS? 

If you are planning to use HTTPS-only connections for your production environment, which is a common occurrence in cloud environments, it is recommended that you disable http-listener-1: 

./asadmin set configs.config.server-config.network-config.network-listeners.network-listener.http-listener-1.enabled=false 

In some cases, if no HTTPS access is required by your security infrastructure (intranet environments or applications that handle public or non-securable data), you can disable http-listener-2 instead. If you are considering using HTTP version 2 instead, you will have to configure secure access regardless, since this is mandated by browsers that implement the protocol. So keep this in mind. 

If you need to redirect all HTTP traffic to HTTPS, we recommend you use a Load Balancer setup or a simple web server that fronts your Payara Server to handle this task.  

HTTP Settings 

Set the maximum number of requests per connection available to your network listener depending on the number of concurrent users your web application is expecting. Keep in mind that the number of request x connections specified is divided equally among the keep-alive threads to serve HTTP requests. 

Set the timeout for the maximum time in seconds that the server holds HTTP connections open. Setting this timeout is a tricky optimization, since a client can keep a connection to the server open so that multiple requests to the server can be serviced by a single network connection. Since the server can only handle a limited number of open connections, you need to make sure that connections are open, if possible, in case the system handles a heavy load of requests.  

Finally, enable the file cache of the listener. This cache contains information about static files like HTML, CSS, images or plain text files that are hosted by the server. Enabling this cache can improve the performance of the server by reducing the number of search operations the server will do on the file system. Also, configure an appropriate time for the max age (in seconds) these files will live on the cache before expiring. 

By default, file caching is enabled on both the default domain (domain1) and the production domain template, so there’s no reason for you to disable it. 

If the machine where the server lives on only possesses one network interface, set the Network Address of the network listener to the specific IP address of this network interface. This will improve performance, since the server won’t have to open socket listeners for each available address to the operating system: 

./asadmin set configs.config.server-config.network-config.protocols.protocol.http-listener-1.http.max-connections=500 configs.config.server-config.network-config.protocols.protocol.http-listener-1.http.timeout-seconds=60 

 
./asadmin set configs.config.server-config.network-config.protocols.protocol.http-listener-1.http.file-cache.max-age-seconds=3600 configs.config.server-config.network-config.protocols.protocol.http-listener-1.http.file-cache.enabled=true 

If the server machine has multiple network interface cards, it is a good practice to create multiple network listeners and assign them the IP addresses of each network interface available. Use a load balancer or web server to distribute requests among these network listeners for your client applications.  

Acceptor and Request Threads  

Configure the number of acceptor threads for the transport service you want to use. An acceptor thread works by listening to new requests on a socket and passing them to request threads. It is recommended to set this number of threads to be equal to the number of CPUs the server machine has. For example, if your server has 4 Quad-Core CPUs, the total number of acceptor threads should be 16 (4x4). Set this setting in the TCP transport element: 

./asadmin set configs.config.server-config.network-config.transports.transport.tcp.acceptor-threads=16 

On the other hand, request threads execute HTTP requests. These are handled by a thread-pool in the same vein that ORB requests are handled as well. The thread pool for both the http-listener-1 and http-listener-2 listeners is called http-thread-pool. Assign the minimum and maximum values to this pool with values ranging from 50 to 500, depending on the expected load and user base your applications have. It is always a good idea to adjust these values after carefully monitoring how many threads are used by the pool when the system is on a heavy load and correct the values incrementally: 

./asadmin set configs.config.server-config.thread-pools.thread-pool.http-thread-pool.min-thread-pool-size=25 configs.config.server-config.thread-pools.thread-pool.http-thread-pool.max-thread-pool-size=350 

In the production domain template, this thread pool has a configured maximum value of 50 threads, which is a good starting number in most scenarios as stated previously. 

JDBC Pool Configuration 

If you have applications that make intensive use of database connections, it’s best to tune the JDBC Connection Pool settings accordingly. Since the server should open, reuse and close these connections based on the pool’s configuration, fine-tuning these settings will guarantee a stable performance under heavy load. 

Assign the minimum and maximum pool size depending on how many open connections your application needs: Consider setting a higher number that is proportional to the number of requests you will receive. 

Also, if the gap between the minimum and maximum sizes is too big, consider setting a higher number for the resize quantity so more connections are created faster by the server. 

Also, set the maximum wait time of the connection pool to zero (0), if possible. By doing this, every caller thread will become blocked until a connection is available. This way, you can make the server avoid tracking the elapsed wait time for each connection request and can improve performance. This is a double-edged sword: if your applications tend to hog JDBC connections, it’s possible the pool can become a bottleneck by making a lot of connection requests wait indefinitely. 

./asadmin set resources.jdbc-connection-pool.<pool-name>.max-pool-size=64 \  
   resources.jdbc-connection-pool.<pool-name>.max-wait-time-in-millis=0 \ 
   resources.jdbc-connection-pool.<pool-name>.steady-pool-size=10 

Ensure that your application properly closes connections to avoid connection leaks, which can lead to exhausted connection pools and application downtime.  

Domain Data Grid 

Payara Server 6 continues to enhance the Domain Data Grid functionality that was introduced in Payara Server 5. This in-memory data structure supports distributed deployments and clustering across Payara Server and Payara Micro instances. The domain data grid is always turned on, which introduces the following benefits for production environments. 

• Easy creation and configuration of clusters composed of multiple Payara Server (and Payara Micro) instances. 

• The use of the JCache API continues to provide robust distributed caching mechanisms with automatic failover and data replication across the cluster. Payara Server 6 improves on these capabilities by offering better integration with Hazelcast 5.x, enhancing cache performance and reliability. 

• Light-weight messaging via remote CDI events (this is a proprietary feature of the Payara Platform) 

• Functions as a backing store for web sessions, persistent EJB timers, performance metrics, clustered singletons and other specific data that requires replication across the cluster. 

Under the covers, the domain data grid is implemented using a customized Hazelcast in-memory data grid. Payara Server 6 integrates Hazelcast 5.x, which brings performance improvements, new features, and better compatibility with modern cloud environments. Hazelcast 5.x offers improved memory management, enhanced scalability, and better support for distributed computing patterns. 

For production environments, the configuration of the data grid will depend on the specific topology and architecture of the applications being deployed, so I won’t be covering much information on this topic. The only recommendation that can be given to configure the data grid in a production environment is to change the domain’s Hazelcast cluster name. This can be done by running the following asadmin commands: 

./asadmin set-hazelcast-configuration --clusterName=<cluster-name> 

This is a recommended configuration to prevent the domain data grid from being “breached” by allowing other Hazelcast nodes to join it by using the default credentials (which could be an embarrassing situation to have since sensitive data could be contained in the grid) and also to prevent unwanted cross-cluster talk. 

It is recommended to disable the Hazelcast PhoneHome service on the production environment. This can be done  via either the HZ_PHONE_HOME_ENABLED environment variable (set it to false) or by using the hazelcast.phone.home.enabled system property. 

./asadmin create-jvm-options --target server-config "-Dhazelcast.phone.home.enabled=false" 

Dockerizing a Production Environment

Many production environments are provisioned using Docker containers due to the ease of managing a whole environment with simple files (Infrastructure as Code) and the excellent synergy that the JVM has with most containers. For production environments, my recommendation is to use the official payara/server-full image, which is already fine-tuned with the following settings: 

• Runs the Payara Server domain in the foreground by using a custom script that prepares the start domain instructions in the proper way. 

• Configures the underlying JVM process memory management by: 

• • Setting the -XX:+UseCGroupMemoryLimitForHeap JVM option which sets the maximum amount of memory (and heap space) available to the JVM to the current value limited for the container as defined by the cgroup memory limits. This option was introduced in JDK 9 but was backported to JDK1.8u131.  

• • Setting the -XX:MaxRAMFraction=1 JVM option which instructs the container to use all available memory to allocate it between heap space and other memory regions as well. Since the container’s only responsibility is hosting the server’s domain all memory should be used by the JVM. 

• • Remove the Xmx and Xms options since memory management is dictated by the 2 previous options. 

• Allow the deployment of all application binaries (JAR, WAR, EAR and RAR files) located in the default deployment directory of the container, identified by the $DEPLOY_DIR environment variable, whose default value is /opt/payara/deployments. 

• Allow the configuration of the domain by specifying asadmin domains that are detailed in either the following: 

• • A pre-boot commands file that is referenced by the $PREBOOT_COMMANDS environment variable, whose default value is /opt/payara/config/pre-boot-commands.asadmin 

• • A post-boot commands file that is referenced by the $POSTBOOT_COMMANDS environment variable, whose default value is /opt/payara/config/post-boot-commands.asadmin 

 
As you can see, the default image already covers a lot of ground with the customization options available to it. In the case of fine-tuning, since memory management is deferred to the container’s available memory and the default domain being used is the production domain there is not much to be done. In order to run a production-level container you can do it like this: 

docker run -p 8080:8080 -v ~/my-applications-dir:/opt/payara/deployments payara/server-full 

You can notice that the run command is mounting the local directory my-applications to the deployment directory in the container’s filesystem, which will allow all application binaries (suffixed with either .ear, .war, .rar or .jar) to be deployed when the server starts.  

How about the data grid configuration we mentioned previously? In that case, we must run the corresponding administration commands by creating a post-boot command script like this: 

#File datagrid-conf.asadmin 

set-hazelcast-configuration --clusterName=production-cluster 

Then mount the directory that contains the script to the default configuration directory of the container like this: 

mv datagrid-conf.asadmin ~/command-scripts/post-boot-commands.asadmin 
docker run -p 8080:8080 -v ~/my-applications-dir:/opt/payara/deployments -v ~/command-scripts/:/opt/payara/config payara/server-full 

However, a much simpler solution in most cases is to create a custom Docker image that is based on the official image like this: 

FROM payara/server-full 
 
COPY my-application.war ${DEPLOY_DIR} 
 
COPY datagrid-conf.asadmin ${CONFIG_DIR}/post-boot-commands.asadmin 

And that’s it! If you want to fine-tune additional configuration settings, just add the corresponding asadmin commands in the post-boot command script and that should be enough. Do you want to apply these optimizations but don’t know the specific commands needed to apply them? Worry not, the Asadmin recorder feature will help you to set the correct commands!  

Fine Tuning Prevents Performance Degradation and Measures System Performance 

Fine-tuning Payara Server for a production environment is a task that should ALWAYS be done to prevent performance degradation and measure how the system behaves under different weights of workload. Installing a server and letting it stick to the default configuration is something that should NEVER be done, so plan carefully what optimizations are needed for your software architecture and apply them when applicable.  

The production template is a huge help in this regard since it is configured with tried-and-tested sensible defaults for most common use cases! Just don’t blindly depend on the optimizations already included within it, be sure to test your system under the appropriate conditions and add any additional optimizations as they are needed.