The HealthCheck Service In-Depth - Payara Micro

Originally published on 14 Jun 2016
Last updated on 14 Jun 2016

The HealthCheck Service provides automatic self-monitoring in order to detect future problems as soon as possible. The HealthCheck Service was introduced in Payara Server and Payara Micro version 161 and some new metrics have been added in version 162.

All the functionality of the HealthCheck Service that is available in Payara Server is also included in Payara Micro. However, as Payara Micro differs in some concepts from Payara Server, the usage and configuration of the HealthCheck Service is slightly different. In this post, we will focus on how to use it in Payara Micro.

The HealthCheck Service periodically checks several metrics, such as CPU and memory usage. If any of these metrics exceed a configurable threshold, then a message is logged to the standard output. This helps to rapidly detect problems or work out what happened after problems have occurred. For most of the checks, threshold configurations can be specified in 3 different levels: good, warning and critical. When the threshold is not met, the HealthCheck Service will in turn log a message with the level of Warning, Error or Critical respectively and will continue to do so after some time until the threshold is met.

We need to capture the warning messages to make them useful. The easiest way to capture them is to redirect the standard output of Payara Micro into a file. The messages could then be captured by a monitoring tool, or displayed in a log viewer:

How to configure the monitoring services

The Health CheckService in Payara Micro is not enabled by default, therefore we need to enable and configure it in order to get warnings in the log. Although Payara Micro provides completely different means to execute asadmin commands compared to Payara Server, we can use all the standard asadmin commands related to the HealthCheck Service.

Unlike Payara Server, it is not possible to manage the configuration in Payara Micro remotely using the asadmin tool. On the other hand, Payara Micro provides an internal API to execute the same asadmin commands from within a deployed web application. In order to access the API, we need to include payara-micro as a compile time dependency. For maven projects, this is as easy as adding the following into the pom.xml:

<dependency>
    <groupId>fish.payara.extras</groupId>
    <artifactId>payara-micro</artifactId>
    <version>4.1.1.162</version>
    <scope>provided</scope>
</dependency>

This Maven Artifact is available from the maven central repository and should be used with provided scope so that it is not bundled into the final WAR package.

Afterwards, the PayaraMicroRuntime object is available to our application. This object provides methods to execute any asadmin command. For example, in order to enable the HealthCheck Service, we could execute the healthcheck-configure asadmin command in the following way from within our application:

PayaraMicro.getInstance().getRuntime().run("healthcheck-configure", "--enabled=true", "--dynamic=true");

If we want to set up the HealthCheck Service before our application is started and ready to accept connections, we may put our initialization code into a singleton EJB and start it during application start-up phase (using @Startup annotation). The following code will enable the HealthCheck Service, and configure the CPU consumption and available machine memory checkers:

@Singleton
@Startup
public class ApplicationStartup {
     
    @PostConstruct
    public void start() {
        final PayaraMicroRuntime pmRuntime = PayaraMicro.getInstance().getRuntime();
        pmRuntime.run("healthcheck-configure", "--enabled=true", "--dynamic=true");
        pmRuntime.run("healthcheck-configure-service", "--serviceName=healthcheck-cpu", "--enabled=true",
            "--time=5", "--unit=SECONDS", "--dynamic=true");
        pmRuntime.run("healthcheck-configure-service-threshold", "--serviceName=healthcheck-cpu",
            "--thresholdCritical=90", "--thresholdWarning=50", "--thresholdGood=0", "--dynamic=true");
        pmRuntime.run("healthcheck-configure-service", "--serviceName=healthcheck-machinemem",
            "--enabled=true", "--dynamic=true", "--time=5","--unit=SECONDS");
        pmRuntime.run("healthcheck-configure-service-threshold", "--serviceName=healthcheck-machinemem",
            "--thresholdCritical=90", "--thresholdWarning=50", "--thresholdGood=0", "--dynamic=true");
    }
 
}

The above code creates the following configuration:

enables healthcheck-cpu metric service to check each 5 seconds. This metric provides information about the CPU load caused by all JVM threads together.
- configures thresholds for the metric:
  - status CRITICAL when CPU load is over 90%
  - status WARNING when CPU load is in range 50% - 90%
  - status GOOD when CPU load is below 50%
enables healthcheck-machinemem metric service to check each 5 seconds. This metric provides information about how much RAM memory is used compared to its full capacity.
- configures the same thresholds for this metric as for healthcheck-cpu metric

As a result, each 5 seconds we get something like this in the standard output:

[2016-05-20T11:19:56.904+0200] [Payara Micro 4.1] [WARNING] [] [fish.payara.nucleus.healthcheck.HealthCheckService] [tid: _ThreadID=114 _ThreadName=healthcheck-service-19] [timeMillis: 1463735996904] [levelValue: 800] CPU:Health Check Result:[[status=WARNING, message='CPU%: 56.50, Time CPU used: 12 seconds 251 milliseconds'']']
 
[2016-05-20T11:19:56.908+0200] [Payara Micro 4.1] [SEVERE] [] [fish.payara.nucleus.healthcheck.HealthCheckService] [tid: _ThreadID=112 _ThreadName=healthcheck-service-18] [timeMillis: 1463735996908] [levelValue: 1000] MMEM:Health Check Result:[[status=CRITICAL, message='Physical Memory Used: 7 Gb - Total Physical Memory: 7 Gb - Memory Used%: 97.35%'']']

The above 2 lines in the log mean that:

the CPU metric is over the WARNING threshold, with CPU load at 56.5%
the machine memory metric is over CRITICAL threshold, with 97.35% memory used out of 7 GB

How to fine-tune alert logging

All the health check messages are logged using a logger named fish.payara.nucleus.healthcheck.HealthCheckService. We can further configure this logger using set-log-levels asadmin command to allow logging only for some severity levels. For example, if we want to log health check messages only for level SEVERE (status CRITICAL), we would use the following code in our application:

PayaraMicro.getInstance().getRuntime().run("set-log-levels", "fish.payara.nucleus.healthcheck.HealthCheckService=SEVERE");
LogManager.getLogManager().readConfiguration(); 
[2016-05-20T11:19:56.908+0200] [Payara Micro 4.1] [SEVERE] [] [fish.payara.nucleus.healthcheck.HealthCheckService] [tid: _ThreadID=112 _ThreadName=healthcheck-service-18] [timeMillis: 1463735996908] [levelValue: 1000] MMEM:Health Check Result:[[status=CRITICAL, message='Physical Memory Used: 7 Gb - Total Physical Memory: 7 Gb - Memory Used%: 97.35%'']']

_{NOTE: Payara Micro does not automatically refresh JUL logging configuration after an asadmin command, therefore it is necessary to execute readConfiguration() on java.util.logging.LogManager.}

As Payara Micro uses JUL logging to log messages, we can also provide an alternative configuration file via java.util.logging.config.file system property:

java -Djava.util.logging.config.file=logging.properties -jar payara-micro.jar --deploy app.war

We need to be careful however, as this will completely overwrite the default logging configuration. We should always use the default logging.properties file used by payara-micro.jar as a template - we can either extract it from the payara-micro.jar (from path config/logging.properties), or dump the domain directory using --rootDir command line option and copy it from that domain directory.

How to separate the configuration from the application

It is good practice to separate the configuration from the application package. We can achieve that by moving the configuration singleton into a separate WAR package. If we do that, we can deploy both WAR packages on the same Payara Micro instance. It is simple as repeating the --deploy argument multiple times for each package.

For example, let's name our application package app.war. Then we'll create another package called app-config.war , which will contain only the configuration singleton that will execute on startup. Then we can deploy our application on Payara Micro like this:

java -jar payara-micro.jar --deploy app-config.war --deploy app.war

Building an executable JAR

When we are satisfied with our configuration, we can optionally build an executable JAR file (uberJar). Such a JAR file could then be executed without any external dependencies.

This feature is completely new and was introduced with Payara Micro 162. If we previously separated the configuration, we can specify to include both our packages in the executable JAR file, just by using standard --deploy argument. It is even possible to specify any other Payara Micro command-line arguments that should be used by default when the uberJar is executed later.

The following command will build an executable JAR from our two WAR packages:

java -jar payara-micro.jar --deploy app-config.war --deploy app.war --outputUberJar executableApp.jar --autoBindHttp

The additional --autoBindHttp argument will make the application automatically bind HTTP listener to an available port if the configured port is not available:

We can afterwards simply execute the JAR file to start our application. We can also provide additional Payara Micro arguments which will be merged with the ones provided with --outputUberJar argument previously.

For example, the following command will execute an application that will listen on port 10080:

java -jar executableApp.jar --port 10080

When we execute the above command again, the second application will start listening on port 10081 automatically, as the port 10080 is already occupied. This is due to --autoBindHttp argument provided before and now included in the uber JAR.

More information

The HealthCheck Service is a really nice addition to Payara Micro monitoring capabilities. Together with other new useful features, such as ability to deploy multiple packages or to build an executable JAR, Payara Micro strives to be a flexible and productive runtime both for development and production scenarios.

For more information, you can always refer to the HealthCheck Service documentation page, which provides up-to-date information and will keep being updated when new metrics are added in future versions of Payara Server & Payara Micro. And if you want to find out more about the HealthCheck Service in Payara Server specifically, check out my previous blog here.