The HealthCheck Service In-Depth - Payara Micro

The HealthCheck Service provides automatic self-monitoring in order to detect future problems as soon as possible. The HealthCheck Service was introduced in Payara Server and Payara Micro version 161 and some new metrics have been added in version 162.

All the functionality of the HealthCheck Service that is available in Payara Server is also included in Payara Micro. However, as Payara Micro differs in some concepts from Payara Server, the usage and configuration of the HealthCheck Service is slightly different. In this post, we will focus on how to use it in Payara Micro.

The HealthCheck Service periodically checks several metrics, such as CPU and memory usage. If any of these metrics exceed a configurable threshold, then a message is logged to the standard output. This helps to rapidly detect problems or work out what happened after problems have occurred. For most of the checks, threshold configurations can be specified in 3 different levels: good, warning and critical. When the threshold is not met, the HealthCheck Service will in turn log a message with the level of Warning, Error or Critical respectively and will continue to do so after some time until the threshold is met.

We need to capture the warning messages to make them useful. The easiest way to capture them is to redirect the standard output of Payara Micro into a file. The messages could then be captured by a monitoring tool, or displayed in a log viewer:

How to configure the monitoring services

The Health CheckService in Payara Micro is not enabled by default, therefore we need to enable and configure it in order to get warnings in the log. Although Payara Micro provides completely different means to execute asadmin commands compared to Payara Server, we can use all the standard asadmin commands related to the HealthCheck Service.

Unlike Payara Server, it is not possible to manage the configuration in Payara Micro remotely using the asadmin tool. On the other hand, Payara Micro provides an internal API to execute the same asadmin commands from within a deployed web application. In order to access the API, we need to include payara-micro as a compile time dependency. For maven projects, this is as easy as adding the following into the pom.xml:

<dependency>
    <groupId>fish.payara.extras</groupId>
    <artifactId>payara-micro</artifactId>
    <version>4.1.1.162</version>
    <scope>provided</scope>
</dependency>

This Maven Artifact is available from the maven central repository and should be used with provided scope so that it is not bundled into the final WAR package.

Afterwards, the PayaraMicroRuntime object is available to our application. This object provides methods to execute any asadmin command. For example, in order to enable the HealthCheck Service, we could execute the healthcheck-configure asadmin command in the following way from within our application:

PayaraMicro.getInstance().getRuntime().run("healthcheck-configure", "--enabled=true", "--dynamic=true");

@Singleton
@Startup
public class ApplicationStartup {
     
    @PostConstruct
    public void start() {
        final PayaraMicroRuntime pmRuntime = PayaraMicro.getInstance().getRuntime();
        pmRuntime.run("healthcheck-configure", "--enabled=true", "--dynamic=true");
        pmRuntime.run("healthcheck-configure-service", "--serviceName=healthcheck-cpu", "--enabled=true",
            "--time=5", "--unit=SECONDS", "--dynamic=true");
        pmRuntime.run("healthcheck-configure-service-threshold", "--serviceName=healthcheck-cpu",
            "--thresholdCritical=90", "--thresholdWarning=50", "--thresholdGood=0", "--dynamic=true");
        pmRuntime.run("healthcheck-configure-service", "--serviceName=healthcheck-machinemem",
            "--enabled=true", "--dynamic=true", "--time=5","--unit=SECONDS");
        pmRuntime.run("healthcheck-configure-service-threshold", "--serviceName=healthcheck-machinemem",
            "--thresholdCritical=90", "--thresholdWarning=50", "--thresholdGood=0", "--dynamic=true");
    }
 
}

The above code creates the following configuration:

• enables healthcheck-cpu metric service to check each 5 seconds. This metric provides information about the CPU load caused by all JVM threads together.

  • configures thresholds for the metric:

    • status CRITICAL when CPU load is over 90%

    • status WARNING when CPU load is in range 50% - 90%

    • status GOOD when CPU load is below 50%

• enables healthcheck-machinemem metric service to check each 5 seconds. This metric provides information about how much RAM memory is used compared to its full capacity.
  • configures the same thresholds for this metric as for healthcheck-cpu metric

As a result, each 5 seconds we get something like this in the standard output:

[2016-05-20T11:19:56.904+0200] [Payara Micro 4.1] [WARNING] [] [fish.payara.nucleus.healthcheck.HealthCheckService] [tid: _ThreadID=114 _ThreadName=healthcheck-service-19] [timeMillis: 1463735996904] [levelValue: 800] CPU:Health Check Result:[[status=WARNING, message='CPU%: 56.50, Time CPU used: 12 seconds 251 milliseconds'']']
 
[2016-05-20T11:19:56.908+0200] [Payara Micro 4.1] [SEVERE] [] [fish.payara.nucleus.healthcheck.HealthCheckService] [tid: _ThreadID=112 _ThreadName=healthcheck-service-18] [timeMillis: 1463735996908] [levelValue: 1000] MMEM:Health Check Result:[[status=CRITICAL, message='Physical Memory Used: 7 Gb - Total Physical Memory: 7 Gb - Memory Used%: 97.35%'']']

The above 2 lines in the log mean that:

• the CPU metric is over the WARNING threshold, with CPU load at 56.5%
• the machine memory metric is over CRITICAL threshold, with 97.35% memory used out of 7 GB

How to fine-tune alert logging

All the health check messages are logged using a logger named fish.payara.nucleus.healthcheck.HealthCheckService. We can further configure this logger using set-log-levels asadmin command to allow logging only for some severity levels. For example, if we want to log health check messages only for level SEVERE (status CRITICAL), we would use the following code in our application:

PayaraMicro.getInstance().getRuntime().run("set-log-levels", "fish.payara.nucleus.healthcheck.HealthCheckService=SEVERE");
LogManager.getLogManager().readConfiguration(); 
[2016-05-20T11:19:56.908+0200] [Payara Micro 4.1] [SEVERE] [] [fish.payara.nucleus.healthcheck.HealthCheckService] [tid: _ThreadID=112 _ThreadName=healthcheck-service-18] [timeMillis: 1463735996908] [levelValue: 1000] MMEM:Health Check Result:[[status=CRITICAL, message='Physical Memory Used: 7 Gb - Total Physical Memory: 7 Gb - Memory Used%: 97.35%'']']

_{NOTE: Payara Micro does not automatically refresh JUL logging configuration after an asadmin command, therefore it is necessary to execute readConfiguration() on java.util.logging.LogManager.}

As Payara Micro uses JUL logging to log messages, we can also provide an alternative configuration file via java.util.logging.config.file system property:

java -Djava.util.logging.config.file=logging.properties -jar payara-micro.jar --deploy app.war

We need to be careful however, as this will completely overwrite the default logging configuration. We should always use the default logging.properties file used by payara-micro.jar as a template - we can either extract it from the payara-micro.jar (from path config/logging.properties), or dump the domain directory using --rootDir command line option and copy it from that domain directory.

How to separate the configuration from the application

It is good practice to separate the configuration from the application package. We can achieve that by moving the configuration singleton into a separate WAR package. If we do that, we can deploy both WAR packages on the same Payara Micro instance. It is simple as repeating the --deploy argument multiple times for each package.

For example, let's name our application package app.war. Then we'll create another package called app-config.war , which will contain only the configuration singleton that will execute on startup. Then we can deploy our application on Payara Micro like this:

java -jar payara-micro.jar --deploy app-config.war --deploy app.war

Building an executable JAR

When we are satisfied with our configuration, we can optionally build an executable JAR file (uberJar). Such a JAR file could then be executed without any external dependencies.

This feature is completely new and was introduced with Payara Micro 162. If we previously separated the configuration, we can specify to include both our packages in the executable JAR file, just by using standard --deploy argument. It is even possible to specify any other Payara Micro command-line arguments that should be used by default when the uberJar is executed later.

The following command will build an executable JAR from our two WAR packages:

java -jar payara-micro.jar --deploy app-config.war --deploy app.war --outputUberJar executableApp.jar --autoBindHttp

java -jar executableApp.jar --port 10080