Troubleshooting your Java EE Applications - Part 2

Originally published on 29 Dec 2018
Last updated on 09 Oct 2019

In the previous blog of this series, we learned different ways to troubleshoot Java EE application. This blog will continue to focus on different ways and techniques to catch potential issues in the early stages and how to find the root cause of application performance issues.

MicroProfile Metrics

Monitoring is an essential function in production environments that helps to ensure optimal performance and service uptime. In production, Operations teams wants to gather information about container to proactively check CPU throughput, Thread counts, Memory and Disk IO usage and view that data in a realtime dashboard (e.g Prometheus). To get a complete visibility into the applications and different components, Operations teams also need information about application e.g number of requests served, request duration, or the number active requests, fault tolerance responses, and several other metrics to find any potential problems in the early stage. So If the monitoring system raises an alarm because some resource is almost depleted, action can be taken to mitigate that problem.

The goal of Microprofile Metrics is to provide the monitoring support for applications as well as the underlying runtime and the metrics data are made available at the HTTP endpoints ‘/metrics’ with JSON and Prometheus format.Using a web browser to visit http://localhost:8080/metrics will result in an output of Prometheus text format as shown below:

# TYPE base:classloader_current_loaded_class_count counter 
# HELP base:classloader_current_loaded_class_count Displays the number of classes that are currently loaded in the JVM. 
base:classloader_current_loaded_class_count 65932

# TYPE base:classloader_total_loaded_class_count counter 
# HELP base:classloader_total_loaded_class_count Displays the total number of classes that have been loaded since the JVM has started execution. 
base:classloader_total_loaded_class_count 11384

# TYPE base:thread_count counter 
# HELP base:thread_count Displays the current number of live threads including both daemon and non-daemon threads. 
base:thread_count 83

# TYPE base:memory_committed_non_heap_bytes gauge
# HELP base:memory_committed_non_heap_bytes Displays the amount of memory in bytes that is committed for the JVM to use.
base:memory_committed_non_heap_bytes 1.54742108E7

# TYPE base:memory_committed_heap_bytes gauge
# HELP base:memory_committed_heap_bytes Displays the amount of memory in bytes that is committed for the JVM to use.
base:memory_committed_heap_bytes 4.78452712E8

...
...

The Prometheus server (an open-source leading monitoring solution) can scrapes the metrics data at an interval from ‘/metrics’ endpoint and store them in a time-series database, so you may extract and analyze metrics information in the Prometheus UI. Because it's Prometheus compatible, it's also well placed to use in a Kubernetes environment.

Prometheus server

The MicroProfile Metrics configuration page can be found under Configurations → <config-name> → MicroProfile → Metrics tab.

Payara Server

MicroProfile Metrics is also configurable through asadmin commands. The set-metrics-configuration asadmin command is used to configure MicroProfile Metrics.

asadmin set-metrics-configuration --enabled=true

Find out more information about set-metrics-configuration asadmin command here.

MicroProfile Metrics specification defines 3 types of scopes for metrics :

Base: JVM statistics specific metrics, that every compliant vendor has to support.
Endpoint: /metrics/base
Vendor: optional vendor-specific metrics that are not portable between different implementations.
Endpoint: /metrics/vendor
Application: deployed applications specific metrics.
Endpoint: /metrics/application

MicroProfile Metrics provides a way to register application-specific metrics either using Java Annotation or using programmatic api :

@Counted - simple incrementing and decrementing counter.
@Timed - tracks the duration of an event.
@Metered - measures the rate at which a set of events occur.
@Gauge - instantaneous reading of a particular value.

In most cases, it is enough to just provide one of the annotation @Counted, @Gauge etc, on the method or field you want to expose and the implementation will do the rest for you.

For explicit registration of metrics, MetricRegistry api can be used which maintains a collection of metrics along with their metadata. The metric registry can be obtained via CDI injection.

@Inject
@RegistryType(type = MetricRegistry.Type.APPLICATION)
MetricRegistry metricRegistry;

The MetricRegistry class has methods that can retrieve or register metrics using programmatic api :

Metadata metadata = Metadata.builder()
    .withName("myGauge")
    .withDescription("Example Gauge")
    .withType(MetricType.GAUGE)
    .build();

Gauge gauge = new Gauge<Double>() {

      @Override
      public Double getValue() {
          return hits / total;
      }
    };
}

metricRegistry.register(metadata, gauge);

Using annotation or programmatic api is few lines of code that provide a huge amount of insight into an application running in production.

MicroProfile OpenTracing

The Microprofile Opentracing specification defines behaviors that allows you to trace the flow of a request across service boundaries. This is essential in a microservices environment where a request typically flows through the multiple services. By default tracing is enabled for all JAX-RS resources. To enable/disable tracing explicitly, the JAX-RS resource or non-JAX-RS class/method can be annotated with @Traced annotation. The annotation provides just two options: whether the method should be traced (the default value option of the annotation), and what the created Span should be called (operationName):

 @Traced(operationName = "hello-trace")
 @GET
 @Path("/hello")
 public String hello() {
     return "hello World";
 }

 @Traced(value = false)
 @GET
 @Path("/bye")
 public String notTraced() {
     return "no trace";
 }

Now if the “/hello” endpoint is called, the following similar JSON representation should be available in log (if service-log notifier enabled) :

{
  "traceSpans": [
    {
      "operationName": "processContainerRequest",
      "spanContext": {
        "spanId": "0652a697-ba9f-4dc6-ae09-ca4de5640ece",
        "traceId": "47df4c93-bf79-495f-9933-443333ea23ac"
      },
      "startTime": "2018-11-21T10:46:01.654+05:30[Asia/Calcutta]",
      "endTime": "2018-11-21T10:46:01.673+05:30[Asia/Calcutta]",
      "traceDuration": "19000000",
      "spanTags": [
        {
          "Server": "server"
        },
        {
          "Domain": "domain1"
        }
      ]
    },
    {
      "operationName": "processWebserviceRequest",
      "spanContext": {
        "spanId": "efff8d70-cfa4-48f8-97c3-5ff43750c9aa",
        "traceId": "47df4c93-bf79-495f-9933-443333ea23ac"
      },
      "startTime": "2018-11-21T10:46:01.655+05:30[Asia/Calcutta]",
      "endTime": "2018-11-21T10:46:01.671+05:30[Asia/Calcutta]",
      "traceDuration": "16000000",
      "spanTags": [
        {
          "accept-language": "[en-US,en;q=0.9]"
        },
        {
          "cookie": "[JSESSIONID=47ce20d819f72d2af49de936ace6; treeForm_tree-hi=treeForm:tree:configurations:server-config:monitor]"
        },
        {
          "ResponseStatus": "200"
        },
        {
          "host": "[localhost:8080]"
        },
        {
          "upgrade-insecure-requests": "[1]"
        },
        {
          "connection": "[keep-alive]"
        },
        {
          "Method": "GET"
        },
        {
          "URL": "http://localhost:8080/rest-client/resources/hello/hello2"
        },
        {
          "accept-encoding": "[gzip, deflate, br]"
        },
        {
          "user-agent": "[Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36]"
        },
        {
          "accept": "[text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8]"
        }
      ],
      "references": [
        {
          "spanContext": {
            "spanId": "0652a697-ba9f-4dc6-ae09-ca4de5640ece",
            "traceId": "47df4c93-bf79-495f-9933-443333ea23ac"
          },
          "relationshipType": "ChildOf"
        }
      ]
    },
    {
      "operationName": "hello-trace",
      "spanContext": {
        "spanId": "cd4fb874-f486-4df7-acf5-f8346c433161",
        "traceId": "47df4c93-bf79-495f-9933-443333ea23ac"
      },
      "startTime": "2018-11-21T10:46:01.667+05:30[Asia/Calcutta]",
      "endTime": "2018-11-21T10:46:01.671+05:30[Asia/Calcutta]",
      "traceDuration": "4000000",
      "spanTags": [
        {
          "http.status_code": "200"
        },
        {
          "component": "jaxrs"
        },
        {
          "span.kind": "server"
        },
        {
          "http.url": "http://localhost:8080/trace-sample/resources/hello"
        },
        {
          "http.method": "GET"
        }
      ],
      "references": [
        {
          "spanContext": {
            "spanId": "0652a697-ba9f-4dc6-ae09-ca4de5640ece",
            "traceId": "47df4c93-bf79-495f-9933-443333ea23ac"
          },
          "relationshipType": "ChildOf"
        },
        {
          "spanContext": {
            "spanId": "0652a697-ba9f-4dc6-ae09-ca4de5640ece",
            "traceId": "47df4c93-bf79-495f-9933-443333ea23ac"
          },
          "relationshipType": "ChildOf"
        }
      ]
    }
  ]
}

Payara Platform is using the existing Request Tracing Service feature for MicroProfile OpenTracing implementation. The Request Tracing Service configuration page can be found under Configurations → <config-name> → Request Tracing. To enable Request Tracing Service select Enabled :

Payara Server

Aside from this configuration settings, you can also define which notifiers will be used to relay the request tracing events by moving them to the Selected Notifiers box.

Selected Notifiers box

Request Tracing Service is also configurable through asadmin commands. By default, Request Tracing Service is disabled. The set-requesttracing-configuration asadmin command is used to configure Request Tracing Service.

asadmin set-requesttracing-configuration --enabled=true

Find out more information about set-requesttracing-configuration asadmin command here.

REST Monitoring

REST Monitoring Service exposes JMX MBeans over standard HTTP connection with a REST API in a JSON format, to stop JMX monitoring requiring RMI. The REST monitoring is hosted on the admin-listener (port 4848).

The REST Monitoring configuration page can be found under Configurations → <config-name> → Monitoring → REST. By default, REST monitoring is disabled. To enable REST Monitoring select Enabled :

Payara Server REST Monitoring

REST monitoring is also configurable through asadmin commands. By default, REST monitoring is disabled. The set-rest-monitoring-configuration asadmin command is used to configure REST monitoring.

asadmin set-rest-monitoring-configuration --enabled=true --contextroot=${CONTEXT_ROOT}

Find out more information about set-rest-monitoring-configuration asadmin command here.

The REST API is hosted on the following URL: http://<HOSTNAME>:<ADMIN_PORT>/<CONTEXT_ROOT>/rest

Which by default is http://localhost:4848/rest-monitoring/rest.

To read the particular MBean "java.lang:type=Threading", make a GET request to

http://localhost:4848/fang/rest/read/java.lang:type=Threading endpoint,

which will result in a JSON representation of MBean similar to the following output:

{
  "request": {
    "mbean": "java.lang:type=Threading",
    "type": "read"
  },
  "value": {
    "ThreadAllocatedMemoryEnabled": true,
    "ThreadAllocatedMemorySupported": true,
    "CurrentThreadCpuTime": 2140625000,
    "CurrentThreadUserTime": 1718750000,
    "ThreadCount": 130,
    "TotalStartedThreadCount": 476,
    "ThreadCpuTimeSupported": true,
    "ThreadContentionMonitoringEnabled": false,
    "ThreadCpuTimeEnabled": true,
    "DaemonThreadCount": 119,
    "PeakThreadCount": 160,
    "CurrentThreadCpuTimeSupported": true,
    "ObjectMonitorUsageSupported": true,
    "SynchronizerUsageSupported": true,
    "ThreadContentionMonitoringSupported": true,
    "AllThreadIds": [
      488,
      486,

….,

      20,
      14
    ],
    "ObjectName": "java.lang:type=Threading"
  },
  "timestamp": 1542774156309,
  "status": 200
}

You can also get the individual attributes returned by a composite MBean. So for the output above you can get a breakdown of each attribute in the array "value". For example, going to

http://localhost:4848/fang/rest/read/java.lang:type=Threading/DaemonThreadCount

returns the following response :

{
  "request": {
    "mbean": "java.lang:type=Threading",
    "attribute": "DaemonThreadCount",
    "type": "read"
  },
  "value": 119,
  "timestamp": 1542774346941,
  "status": 200
}

REST Monitoring can be used to keep track of things like memory pool usage or thread count when looking for bottlenecks or other potential issues. This helps you to make better decisions about how to maintain & optimize server resources and performance-tune code.

Additional Rest Monitoring Resources:

Consuming a REST Service

Create a RESTful Web Service with Payara Server and NetBeans (Video)

Securing a REST Service

I hope you have enjoyed this tutorial and that it’s given you a decent amount of confidence to troubleshoot your application in production!