MicroProfile Metrics in Payara Micro

One of the new features in Payara Server and Payara Micro 4.1.2.181 and 5.181 is full compatibility with MicroProfile 1.2. In this blog, I will introduce Metrics 1.0, a new API of MicroProfile 1.2 to use with Payara Micro (and Payara Server).

Needless to say, monitoring the underlying runtime and system, as well as your own application, is important to keep them reliable. There is already JMX as a Java standard to monitor and expose metrics, but remote JMX is not easy to deal with. In addition, JMX is specific to Java so it can be even more difficult to use in the case that both Java and non-Java environments are mixed. MicroProfile Metrics is simple, easy to use and suitable for polyglot environment API to expose system and application metrics.

Key Features

MicroProfile Metrics is fairly simple to use, and it is even enabled by default in both Payara Server and Payara Micro. To understand the inner workings of the specification, there are three key features you need to know:

• REST endpoints
  Prometheus is configured to collect metrics from HTTP endpoints on /metrics by default, so there is no complicated configuration needed.
  
  
• Registry
  Metric values are stored in a registry with three scopes:
  • Base (JVM metrics, common to every implementation)
  • Vendor (Vendor specific metrics)
  • Application (User metrics)
    
    
• Format
  Both JSON and Prometheus text format are available. Prometheus text format is the default, but requesting application/json in the request header will return JSON formatted metrics.

First Steps with Metrics on Payara Micro

In this section, I'll show you the basics to get started with Metrics and view Base and Vendor scoped metrics with any existing application. There is no need to deploy any application to Payara Micro at this stage; both base and vendor metrics will be available by default.

For example, using curl command is as follows;

curl http://localhost:8080/metrics

You may see such an output as follows. This is the default output, which is Prometheus text format.

# TYPE vendor:system_cpu_load gauge
# HELP vendor:system_cpu_load Display the "recent cpu usage" for the whole system. This value is a double in the [0.0,1.0] interval. A value of 0.0 means that all CPUs were idle during the recent period of time observed, while a value of 1.0 means that all CPUs were actively running 100% of the time during the recent period being observed. All values betweens 0.0 and 1.0 are possible depending of the activities going on in the system. If the system recent cpu usage is not available, the method returns a negative value.
vendor:system_cpu_load 0.1491192437579042
# TYPE base:classloader_total_loaded_class_count counter
# HELP base:classloader_total_loaded_class_count Displays the total number of classes that have been loaded since the JVM has started execution.
base:classloader_total_loaded_class_count 11250
# TYPE base:cpu_system_load_average gauge
# HELP base:cpu_system_load_average Displays the system load average for the last minute. The system load average is the sum of the number of runnable entities queued to the available processors and the number of runnable entities running on the available processors averaged over a period of time. The way in which the load average is calculated is operating system specific but is typically a damped time-dependent average. If the load average is not available, a negative value is displayed. This attribute is designed to provide a hint about the system load and may be queried frequently. The load average may be unavailable on some platform where it is expensive to implement this method.
base:cpu_system_load_average -1.0
# TYPE base:thread_count counter
# HELP base:thread_count Displays the current number of live threads including both daemon and non-daemon threads.
base:thread_count 73
# TYPE base:classloader_current_loaded_class_count counter
# HELP base:classloader_current_loaded_class_count Displays the number of classes that are currently loaded in the JVM.
base:classloader_current_loaded_class_count 10992
 
...etc

If you are monitoring your system by Prometheus, or with another product which can collect Prometheus-format metrics, this would be very useful.

JSON Format

Metrics can also be retrieved with another output format - JSON. Simply access the URL with an "Accept" HTTP header. Metrics formatted as JSON can be retrieved using cURL as follows;

curl -H "Accept: application/json" http://localhost:8080/metrics

You may see such a JSON format output as follows. (This is formatted for ease to see.)

{
  "base": {
    "memory.maxHeap": 487587840,
    "classloader.currentLoadedClass.count": 11150,
    "thread.count": 90,
    "memory.committedNonHeap": 111009792,
    "gc.PS Scavenge.time": 296,
    "classloader.totalLoadedClass.count": 11193,
    "thread.max.count": 137,
    "jvm.uptime": 256681,
    "memory.committedHeap": 487587840,
    "memory.maxNonHeap": -1,
    "memory.usedNonHeap": 97943648,
    "gc.PS MarkSweep.time": 2433,
    "cpu.availableProcessors": 4,
    "memory.usedHeap": 134214048,
    "classloader.totalUnloadedClass.count": 44,
    "cpu.systemLoadAverage": -1,
    "thread.daemon.count": 85,
    "gc.PS MarkSweep.count": 7,
    "gc.PS Scavenge.count": 15
  },
  "vendor": {"system.cpu.load": 0}
}

Here, we have seen metrics in the scopes of Base and Vendor. Base metrics are required by the specification, thus all MicroProfile vendors have to implement these. These are also mainly exposing JVM statistics. Vendor metrics are additional metirics provided by each vendor, so will vary between implementations of the Metrics specification. Both of these scopes are used for reporting on the status of the container.

So far we see the way to obtain all the metrics. You can focus the scope of metrics by specifying the endpoint as follows:

• All metrics: /metrics
• Base metrics: /metrics/base
• Vendor metrics: /metrics/vendor
• Application metrics: /metrics/application

We'll take a look at the application scope in the next section, and look at an example of exposing metrics from your own application.

Application metrics

There are several types of Metrics. Some of these can be seen in the example output above, and all can be used to provide data from your own applications in the application scope.

• Gauge - simple metric type for continuous value, e.g. memory.usedHeap, memory.committedHeap, memory.maxHeap
• Counter - simple metric type for discrete value, e.g. thread.count, thread.daemon.count 
• Meter - complex metric type comprised of multiple key/values; values are count, meatRate, oneMinRate, fiveMinRate, fifteenMinRate
• Histogram - complex metric type comprised of multiple key/values; values are count, min, max, mean, stddev, p50, p75, p95, p98, p99, p999
• Timer - complex metric type comprised of multiple key/values; values are count, meanRate, oneMinRate, fiveMinRate, fifteenMinRate, min, max, mean, stddev, p50, p75, p95, p98, p99, p999

In the previous section, we see Base and Vendor metrics. Metrics are also implemented on your applications. You may provide the way of health check for each applications. It's called Application metrics.

First, add the following dependency to the pom.xml of your application.

 pom.xml

<dependency>
  <groupId>org.eclipse.microprofile</groupId>
  <artifactId>microprofile</artifactId>
  <type>pom</type>
  <version>1.2</version>
</dependency>

Next, we can create our simple example using @Counter and @Meter. It exposes "employeeCount" as Counter, and both "createEmployee", "deleteEmployee" as Meter.

import org.eclipse.microprofile.metrics.Counter;
import org.eclipse.microprofile.metrics.annotation.Metered;
import org.eclipse.microprofile.metrics.annotation.Metric;
 
import javax.annotation.PostConstruct;
import javax.enterprise.context.ApplicationScoped;
import javax.inject.Inject;
import javax.ws.rs.*;
import java.util.ArrayList;
 
@Path("employees")
@ApplicationScoped
public class EmployeeResource {
 
  private ArrayList<String> employees = new ArrayList<String>() ;
 
  @Inject
  @Metric
  Counter employeeCount;
 
  @PostConstruct
  private void init() {
    // initialise counter with beginning number
    employeeCount.inc(4);
  }
 
  @GET
  @Path("{id}")
  public String getEmployeeById(@PathParam("id") int id) {
    // return specified employee by id via path parameter
    return employees.get(id);
  }
 
  @GET
  public String getAllEmployees() throws InterruptedException {
    // return all employees without parameters
    return employees.toString();
  }
 
  @POST
  @Metered
  public void createEmployee(@FormParam("name") String emp) {
    // add an employee from form parameter and increment "employeeCount"
    if (employees.add(emp)) employeeCount.inc();
  }
 
  @DELETE
  @Path("{id}")
  @Metered
  public void deleteEmployee(@PathParam("id") int id) {
    // delete an employee by id via path parameter, and decrement "employeeCount"
    if (id >= employees.size()) {
      return;
    } else {
      employees.remove(id);
      employeeCount.dec();
    }
  }
}

The counter "employeeCount" is an example of a Field level annotation which will always show the current count of employees who registered. At first, it's four ("John", "Paul", "George" and "Ringo"). Since a counter always initialises at 0, we need to increment it by 4 in our example to keep the number at the correct level. Every time the HTTP DELETE or POST is used, the counter is manually incremented.

The meter "createEmployee" and "deleteEmployee" are examples of Method level annotations which are changed by every method call. We do not need to manually set any values here, since the meters are showing the number of times that each endpoint has been hit. This kind of monitoring can be very useful in collecting data about how users make use of an application.

Conclusion

Before the API, we had to struggle with remote JMX to monitor the system properly, and would often have to create complicated configurations to get good data. There is the option of a JMX-HTTP bridge such as Jolokia but that still required additional set up. Metrics is REST based, simple and flexible monitoring API that's fully configured out-of-the box. Because it's Prometheus compatible, it's also well placed to use in a Kubenetes environment. See also https://github.com/payara/Payara-Examples/tree/master/microprofile to check more example of Metrics.

You can download the spec from https://github.com/eclipse/microprofile-metrics/releases/tag/1.0 for a more comprehensive overview.