The Health Check Service provides automatic self-monitoring in order to detect future problems as soon as possible. The Health Check Service was introduced in Payara Server and Payara Micro version 161 and some new metrics have been added in version 162.
All the functionality of the Health Check Service that is available in Payara Server is also included in Payara Micro. However, as Payara Micro differs in some concepts from Payara Server, the usage and configuration of the Health Check Service is slightly different. In this post, we will focus on how to use it in Payara Micro.
The Health Check Service periodically checks several metrics, such as CPU and memory usage. If any of these metrics exceed a configurable threshold, then a message is logged to the standard output. This helps to rapidly detect problems or work out what happened after problems have occurred. For most of the checks, threshold configurations can be specified in 3 different levels: good, warning and critical. When the threshold is not met, the Health Check Service will in turn log a message with the level of Warning, Error or Critical respectively and will continue to do so after some time until the threshold is met.
We need to capture the warning messages to make them useful. The easiest way to capture them is to redirect the standard output of Payara Micro into a file. The messages could then be captured by a monitoring tool, or displayed in a log viewer:
How to configure the monitoring services
The Health Check Service in Payara Micro is not enabled by default, therefore we need to enable and configure it in order to get warnings in the log. Although Payara Micro provides completely different means to execute asadmin commands compared to Payara Server, we can use all the standard asadmin commands related to the Health Check Service.
Unlike Payara Server, it is not possible to manage the configuration in Payara Micro remotely using the asadmin tool. On the other hand, Payara Micro provides an internal API to execute the same asadmin commands from within a deployed web application. In order to access the API, we need to include payara-micro as a compile time dependency. For maven projects, this is as easy as adding the following into the
This Maven Artifact is available from the maven central repository and should be used with
provided scope so that it is not bundled into the final WAR package.
PayaraMicroRuntime object is available to our application. This object provides methods to execute any asadmin command. For example, in order to enable the Health Check Service, we could execute the
healthcheck-configure asadmin command in the following way from within our application:
PayaraMicro.getInstance().getRuntime().run("healthcheck-configure", "--enabled=true", "--dynamic=true");
@Startupannotation). The following code will enable the Health Check Service, and configure the CPU consumption and available machine memory checkers:
The above code creates the following configuration:
healthcheck-cpumetric service to check each 5 seconds. This metric provides information about the CPU load caused by all JVM threads together.
configures thresholds for the metric:
status CRITICAL when CPU load is over 90%
status WARNING when CPU load is in range 50% - 90%
status GOOD when CPU load is below 50%
healthcheck-machinememmetric service to check each 5 seconds. This metric provides information about how much RAM memory is used compared to its full capacity.
- configures the same thresholds for this metric as for
- configures the same thresholds for this metric as for
As a result, each 5 seconds we get something like this in the standard output:
The above 2 lines in the log mean that:
- the CPU metric is over the WARNING threshold, with CPU load at 56.5%
- the machine memory metric is over CRITICAL threshold, with 97.35% memory used out of 7 GB
How to fine-tune alert logging
All the health check messages are logged using a logger named
fish.payara.nucleus.healthcheck.HealthCheckService. We can further configure this logger using
set-log-levels asadmin command to allow logging only for some severity levels. For example, if we want to log health check messages only for level SEVERE (status CRITICAL), we would use the following code in our application:
NOTE: Payara Micro does not automatically refresh JUL logging configuration after an asadmin command, therefore it is necessary to execute
As Payara Micro uses JUL logging to log messages, we can also provide an alternative configuration file via
java.util.logging.config.file system property:
java -Djava.util.logging.config.file=logging.properties -jar payara-micro.jar --deploy app.war
We need to be careful however, as this will completely overwrite the default logging configuration. We should always use the default logging.properties file used by payara-micro.jar as a template - we can either extract it from the payara-micro.jar (from path
config/logging.properties), or dump the domain directory using
--rootDir command line option and copy it from that domain directory.
How to separate the configuration from the application
It is good practice to separate the configuration from the application package. We can achieve that by moving the configuration singleton into a separate WAR package. If we do that, we can deploy both WAR packages on the same Payara Micro instance. It is simple as repeating the
--deploy argument multiple times for each package.
For example, let's name our application package
app.war. Then we'll create another package called
app-config.war , which will contain only the configuration singleton that will execute on startup. Then we can deploy our application on Payara Micro like this:
java -jar payara-micro.jar --deploy app-config.war --deploy app.war
We should always place the configuration package first on the command line. The configuration package will then be deployed first, effectively updating the configuration before the main application is deployed.
This way we can easily update the configuration without rebuilding whole application.
Building an executable JAR
When we are satisfied with our configuration, we can optionally build an executable JAR file (uberJar). Such a JAR file could then be executed without any external dependencies.
This feature is completely new and was introduced with Payara Micro 162. If we previously separated the configuration, we can specify to include both our packages in the executable JAR file, just by using standard
--deploy argument. It is even possible to specify any other Payara Micro command-line arguments that should be used by default when the uberJar is executed later.
The following command will build an executable JAR from our two WAR packages:
java -jar payara-micro.jar --deploy app-config.war --deploy app.war --outputUberJar executableApp.jar --autoBindHttp
--autoBindHttp argument will make the application automatically bind HTTP listener to an available port if the configured port is not available:
We can afterwards simply execute the JAR file to start our application. We can also provide additional Payara Micro arguments which will be merged with the ones provided with
--outputUberJar argument previously.
For example, the following command will execute an application that will listen on port 10080:
java -jar executableApp.jar --port 10080
When we execute the above command again, the second application will start listening on port 10081 automatically, as the port 10080 is already occupied. This is due to
--autoBindHttp argument provided before and now included in the uber JAR.
The Health Check Service is a really nice addition to Payara Micro monitoring capabilities. Together with other new useful features, such as ability to deploy multiple packages or to build an executable JAR, Payara Micro strives to be a flexible and productive runtime both for development and production scenarios.
For more information, you can always refer to the Health Check Service documentation page, which provides up-to-date information and will keep being updated when new metrics are added in future versions of Payara Server & Payara Micro. And if you want to find out more about the Health Check Service in Payara Server specifically, check out my previous blog here.