Fine Tuning Payara Server 4 in Production

One of the biggest challenges when developing applications for the web is understanding how applications need to be fine-tuned when releasing them into a production environment. This is no exception for Java Enterprise applications deployed on Payara Server.

Running a Payara Server installation is simple: download the current distribution suited for your needs (full, web, micro), etc.; head to the /bin folder and start the default domain!

However, keep in mind that Payara Server is tailored for development purposes (a trait shared with GlassFish). When developing a web application, it’s better to quickly code features, execute a fast deploy, test them, un-deploy the application (or redeploy it) and continue with the next set of features.

However, in a production environment, a proper configuration to make the system fulfill its expected quality attributes (performance, availability, reliability, etc.) is a must. The intention of this blog post is to provide a set of optimizations to apply when preparing your production environment using Payara Server; and show how to fine tune the critical aspects related to these optimizations.

The payaradomain Template in Payara Server 4

NOTE: In Payara Server 5, this domain template is called "production" instead of "payaradomain" to reflect its purpose. Payara Server 4 is maintained only for customers so you should be looking toward upgrading to Payara Server 5 or consider commercial Payara support services. If you're already running Payara Server 5, read Fine Tuning Payara Server 5 in Production.

Usually, when fine-tuning an application server, some of the usual parameterizations done to a server configuration deal with the following aspects:

• Java Virtual Machine (JVM) Initialization Options
• General System Properties
• Networking and HTTP Listeners
• Class loading and File Handling
• JDBC Settings

Starting with Payara Server v4.1.1.152.1 a new domain “template” called payaradomain is included by default. This domain is configured with some common settings that deal with these aspects described earlier and help improve performance in almost all scenarios.

How do you make use of this template? It’s simple! Instead of running the default domain or a custom domain, run this domain instead and use it to deploy your applications:

./asadmin start-domain payaradomain

Are you wondering what optimizations are included in this domain template? We’ll mark them in the following sections when we detail each one of the configuration settings we consider should be applied in a production environment.

As a simpler convention to illustrate these optimizations, we will use the corresponding element in the domain/config/domain.xml file needed to be changed/updated. This is to keep consistency across this article. Also, assume relevant configurations will be done on the server-config configuration group as well.

JVM Options

The default JVM options are not suitable for a Java application that should be constantly running in production, so the following changes are always recommended:

Set ‘server’ mode (Included with payaradomain):

 <jvm-options>-client</jvm-options> à <jvm-options>-server</jvm-options>

The server option enables the JVM to run on server mode, which is intended to be used for a long running operation that works at peak speed.

Fine tune Heap size (Included with payaradomain):

<jvm-options>-Xmx2048m</jvm-options>

<jvm-options>-Xms2048m</jvm-options>

Adjust the Xmx and Xms options to increase the amount of memory allocated for the Heap space. Also, set both options with the same value so the server allocates all memory available at startup time instead of executing multiple re-allocations of memory at runtime.

As a personal recommendation, allocate an amount of memory between 2 and 16 Gigabytes. More than that size would mean an unmanageable number of objects that need to be managed by the garbage collector, so this can affect the performance of the JVM. If your applications live in a Heap space that needs to be that big, consider implementing a clustered arrangement so the workload is reduced overall.

On the payaradomain template, the heap space is set to 1G, so remember to adjust it to a more appropriate value.

Increase PermGen max size (Included with payaradomain):

<jvm-options>-XX:MaxPermSize=512m</jvm-options>

Increase the maximum size of memory to be allocated to the PermGen space per the number of dependencies and/or classes to be used by them. Remember that the PermGen space will be filled with the metadata of each class loaded by your applications, so an appropriate value should be configured with this number in mind. By default, on the payaradomain template, this value is set to 512 Megabytes.

Also, keep in mind that this option is only available to all JDKs before version 8, since in this version the PermGen space was removed and was replaced by the Metaspace instead.

Disable Secure Client-Initiated Renegotiation (Included with payaradomain):

<jvm-options>-Djdk.tls.rejectClientInitiatedRenegotiation=true</jvm-options>

This is necessary in order to avoid DDoS attacks occurring on the server due to the TLS protocol feature of secure client-initiated renegotiation. Keep in mind this option is available for JDK 8 only.

Enable Isolated Class Loading

<jvm-options>-Dfish.payara.classloading.delegate=false</jvm-options>

Consider enabling isolated class loading of classes that are included in third party libraries that can conflict with previous versions of these classes already included within the Payara Server modules. This setting will be applied to all libraries resting under the domain/lib folder and those included inside the applications as well. It’s a good idea to enable this feature if your applications heavily depend on common third party libraries (Apache Commons, Google Guava, Spring Framework, etc.)

Fine-Tune Garbage Collection

Garbage collection reclaims “unused” heap space allocated on the JVM by identifying all objects that are no longer needed. Since this process of locating and removing these objects can slow down the performance of an application, an efficient garbage collection configuration is needed to maximize the performance of the server under heavy load.

Fine-tuning the Garbage Collector is a huge topic that can fill books by itself, so it is outside the scope of this article. Our recommendation is to choose wisely the GC algorithm that the server’s JDK will use based on these metrics:

• Percentage of CPU usage
• Frequency of CPU pauses
• Shorter vs. Longer CPU pauses

We also recommend to disable explicit garbage collections triggered by the System.gc method with the following property:

<jvm-options>-XX:+DisableExplicitGC</jvm-options>

In rare cases, developers will make use of such feature of the JDK, but disabling this method will prevent high CPU usage peaks on the server due to unwanted garbage collection iterations.

Disable Development Features

You should always disable both the auto deployment and dynamic reloading of applications in your production environment. These features help speed up the development of an application but are not needed on a daily basis at all, since they can affect performance (by using server resources to monitor application state or changes on file directories for example).

To disable these features, adjust the attributes of the das-config element in the domain.xml file:

<das-config dynamic-reload-enable=“false” autodeploy-enabled=”false”></das-config>

It’s also a good practice to disable the JspServlet dynamic reloading, so it isn’t constantly checking changes on JSP files. Configure the management of all String values as static character arrays, since this will lower the memory used by the server.

To implement both changes, you will need to edit the servlet configuration under the domain/config/default-web.xml file:

<servlet>
    <servlet-name>jsp</servlet-name>
    <servlet-class>org.apache.jasper.servlet.JspServlet</servlet-class>
    <init-param>
        <param-name>development</param-name>
        <param-value>false</param-value>
    </init-param>
    <init-param>
        <param-name>genStrAsCharArray</param-name>
        <param-value>true</param-value>
    </init-param>
     …
</servlet>

EJB Pool Settings

When an application uses a set of EJBs to model business components, keep in mind that the application server's EJB container caches, pools and reuses EJB component instances to improve performance and promote efficient resource access. If your applications make use of Stateless EJBs, you need to fine tune the pool settings to maintain performance:

<ejb-container pool-resize-quantity="16" max-pool-size="120" steady-pool-size="10">

    <ejb-timer-service></ejb-timer-service>

</ejb-container>

Assign appropriate values to the minimum size of the pool (steady-pool-size), maximum pool size and the resize quantity used by the container when resizing the EJB pool. I recommend assigning values based on the number of concurrent clients your applications will manage: set the maximum pool size to handle the anticipated high load of the system, the minimum/initial pool size to handle a moderate load and the resize quantity to a large number if the system is expecting frequent peaks of high load on its lifetime.

If your application makes use of remote EJB interfaces, you must configure the ORB settings accordingly as well. The ORB (CORBA Object Request Broker) works using the RMI/IIOP protocols for remote communications with client applications (either Java EE application clients or classical stand-alone applications). To service requests from these remote clients, a thread-pool is used to handle the availability of threads to handle remote call executions on the server.

Adjust the maximum and minimum sizes of the thread pool assigned to the ORB service (on the payaradomain domain it’s called thread-pool-1). Pay special attention to the maximum value since you can use it to improve performance: if your applications will be handling a lot of calls from remote clients, an appropriate number of threads should be used to service them without slowing down the server.

<thread-pool name="thread-pool-1" min-thread-pool-size="50"

      max-thread-pool-size="200"></thread-pool>

HTTP and Network Configuration

When releasing a web application on a production environment, it is extremely important to configure the HTTP network listeners appropriately, since the server needs to handle client requests without compromising availability and reliability.

By default, each domain comes configured with 3 network listeners:

• admin-listener, which handles HTTP connections for the admin console web interface.
• http-listener-1, which handles HTTP connections for all web applications.
• http-listener-2, which handles HTTPS exclusively connections for all web applications.

HTTP or HTTPS?

If you are planning to use HTTPS only connections for your production environment (this is a common occurrence on cloud environments), it is recommended that you disable http-listener-1:

<network-listeners>

     <network-listener protocol="http-listener-1" port="8080" name="http-listener-1"

      thread-pool="http-thread-pool" transport="tcp"

      enabled="false"></network-listener>

   …

</network-listeners>

In some cases, if no HTTPS access is required by your security infrastructure (intranet environments or applications that handle public or non-securable data), you can disable http-listener-2 instead.

If you need to redirect all HTTP traffic to HTTPS, we recommend you to use a Load Balancer setup or a simple web server that fronts your Payara Server to handle this task.

HTTP Settings

Set the maximum number of requests per connection available to your network listener depending on the number of concurrent users your web application is expecting. Keep in mind that the number of request x connections specified is divided equally among the keep alive threads to serve HTTP requests.

Set the timeout for the maximum time in seconds that the server holds HTTP connections open. Setting this timeout is a tricky optimization, since a client can keep a connection to the server open so that multiple requests to the server can be serviced by a single network connection. Since the server can only handle a limited number of open connections, you need to make sure that connections are open if possible in the case the system handles a heavy load of requests.

Finally, enable the file cache of the listener. This cache contains information about static files like HTML, CSS, images or plain text files that are hosted by the server. Enabling this cache can improve the performance of the server by reducing the number of search operation the server will do on the file system. Also, configure an appropriate time for the max age (in seconds) these files will live on the cache before expiring.

Configure these parameters in the protocol element of the corresponding network listener:

<protocol name="http-listener-1">

   <http max-connections="500" timeout-seconds="60" default-virtual-server="server">

        <file-cache enabled="true" max-age-seconds="3600"></file-cache>

   </http>

   …

</protocol>

On the payaradomain template, the number of max connections is set to 500 by default, which is a good average value for cloud environments.

If the machine the server lives on only possess one network interface card, set the network address of the network listener to the specific IP address of this network interface. This will improve performance since the server won’t have to query the operating system for this IP address:

<network-listener protocol="http-listener-1" address="192.168.1.103" port="8080"                             name="http-listener-1" thread-pool="http-thread-pool" transport="tcp">

</network-listener>

If the server machine has multiple network interface card, it is a good practice to create multiple network listeners and assign them the IP addresses of each network interface available. Use a load balancer or web server to distribute requests among these network listeners for your client applications.

Acceptor and Request Threads

Configure the number of acceptor threads the transport service you want to use. An acceptor thread works listening to new requests on a socket always and passing them to request threads. It is recommended to set this number of threads to be equal to the number of CPUs the server machine has. For example, if your server has 4 Quad-Core CPUs, the total number of acceptor threads should be 16 (4x4). Set this setting in the tcp transport element:

<transports>

    <transport name="tcp" acceptor-threads="16" byte-buffer-type="HEAP"></transport>

</transports>

On the other hand, request threads execute HTTP requests. These are handled by a thread-pool in the same vein that ORB requests are handled as well. The thread pool for both the http-listener-1 and http-listener-2 listeners is called http-thread-pool. Assign the minimum and maximum values to this pool with values ranging from 50 to 500 depending on the expected load and user base your applications have. It is always a good idea to adjust these values after carefully monitoring how many threads are used by the pool when the system is on a heavy load and correct the values incrementally:

<thread-pool name="http-thread-pool" min-thread-pool-size="25"

             max-thread-pool-size="350"></thread-pool>

On the payaradomain template, the thread pool used for requests threads for the Web admin console, admin-thread-pool, has configured a range of 1-15. This is a consistent average value for most production environments.

JDBC Pool Configuration

If you have applications that make an intensive use of database connections, it’s best to tune the JDBC Connection Pool settings accordingly. Since the server should open, reuse and closes these connections based on the pool’s configuration, fine tuning these settings is necessary so a good performance is kept under heavy load.

Assign the minimum and maximum pool size depending on how many open connections your application needs: Consider setting a higher number that is proportional to the number of requests you will receive. Also, if the gap between the minimum and maximum sizes is too big, consider setting a higher number for the resize quantity so more connections are created faster by the server.

Also, set the maximum wait time of the connection pool to zero (0) if possible. By doing this, every caller thread will become blocked until a connection is available. This way, you can make the server avoid tracking the elapsed wait time for each connection request and can improve performance. This is a double-edged sword though, since if your applications tend to hog JDBC connections, it’s possible the pool can become a bottleneck by making too many connection requests wait indefinitely.

<jdbc-connection-pool steady-pool-size="10" max-pool-size="64" pool-resize-quantity="12"
is-isolation-level-guaranteed="false" max-wait-time-in-millis="0" datasource-classname="org.apache.derby.jdbc.EmbeddedDataSource" name="DerbyPool" res-type="javax.sql.DataSource">
<property name="databaseName"
value="${com.sun.aas.instanceRoot}/lib/databases/embedded_default">
      </property>
      <property name="connectionAttributes" value=";create=true"></property>
</jdbc-connection-pool>

Hazelcast and JCache

Enabling Hazelcast is required when you want to enable the use of the JCache API in your applications or when taking advantage of the auto-clustering capabilities of the server instead of using the Shoal-based clustering.

Keep in mind that you should implement the following configuration changes when enabling Hazelcast for your server:

• Change the name of your cluster group! The default group name is development, so assign a proper name for your cluster group. Since the default discovery of the cluster group is done using multicast, your server may accidentally join a default cluster on a test environment and cause unwanted behavior if left unconfigured.
• Change the password the server should use to join the cluster group as well, to prevent unwanted members to join these clusters.
• Consider configuring the server as a lite member of the cluster group when it’s applications are stateless in nature and do not store information locally. For more information about the benefits of lite members, check out this blog post.

<hazelcast-runtime-configuration cluster-group-name="production-cluster"

           cluster-group-password="securePassword"

    lite="true" enabled="true">

</hazelcast-runtime-configuration>

Dockerizing a Production Environment

Many production environments nowadays are provisioned using Docker containers due to the ease of managing a whole environment with simple files (Infrastructure as Code) and the excellent synergy the JVM has with most containers. If your application is deployed using a custom Docker image and it’s based on our official payara/server-full image, then you can apply all our recommended optimizations using asadmin commands.

Here I provide a sample Dockerfile with some of the discussed optimizations to illustrate:

FROM payara/server-full

ENV AS_ADMIN $PAYARA_PATH/bin/asadmin 
ENV DOMAIN payaradomain

#In order to edit the default-web-xml file to disable development features, it's easier to have a copy of the file with theses optimization applied and then replace the file in the server
ADD custom-default-web-xml ~./default-web-xml

RUN $AS_ADMIN start-domain $DOMAIN && \
    $AS_ADMIN set configs.config.server-config.admin-service.das-config.dynamic-reload-enabled=false
                  configs.config.server-config.admin-service.das-config.autodeploy-enabled=false && \
    $AS_ADMIN create-jvm-options -client:
                             -Xmx1024m:-Xms512m && \
    $AS_ADMIN create-jvm-options -server: 
                                 -Xmx2048m:-Xms2048m:                                 
                                 -Dfish.payara.classloading.delegate=false && \
    $AS_ADMIN set-hazelcast-configuration --clusterName=my-production-cluster 
  --clusterPassword=***** --target=server-config && \
    $AS_ADMIN stop-domain $DOMAIN && \
    mv ~./default-web-xml $PAYARA_PATH/glassfish/domains/payaradomain/config/

ENTRYPOINT $AS_ADMIN start-domain $DOMAIN 

You can even use Docker environment variables and parametrize your provisioning configuration! This is extremely useful since you can build an optimizable domain container template for different applications. Do you want to apply your own optimizations but don’t know the specific asadmin commands needed to apply them? Worry not, the asadmin recorder feature included in the 163 release will have you covered!

Optimize Payara Server in a Production Environment

Fine tuning Payara Server 4 for a production environment is a task that should ALWAYS be done to prevent performance degradation, and measure out how the system behaves under different weights of workload. Installing a server and letting it stick to the default configuration is something that should NEVER be done, so plan carefully what optimizations are needed for your software architecture and apply them when applicable.

Don’t forget about the payaradomain template as well! This template will simplify your work by including many of our recommended optimizations. We are planning to include more optimizations in the future, depending on new features shipped with Payara Server or updated GlassFish features. Be sure to check them out in the official documentation as well. You might benefit from these future inclusions when upgrading your server installation, so that’s a nice bonus!

If you're using or planning to use Payara Server 5, read about how to Fine Tune Payara Server 5 for production environments here: https://blog.payara.fish/fine-tuning-payara-server-5-in-production