Demystifying Request Tracing in Payara Server

Introduction

In today's complex, distributed applications, pinpointing performance issues and understanding the flow of requests can be a hard task. Payara Server's Request Tracing Service provides a reliable toolset for tracing requests across various protocols and components. This blog post looks at request tracing in Payara Server, offering a quick guide to its features, configuration and best practices.

Key Concepts

Before diving in, let’s take a look at a breakdown of the essential terminology:

• Trace: A record of the entire request lifecycle within the Payara Platform, including all associated units of work.
• Span: A single, timed operation representing a segment of work within a trace, e.g. JAX-RS method execution. Spans can have parent-child relationships.
• Tag: Key-value metadata attached to a span for additional context.
• Log: A time stamped tag, usually representing an event during a span.

Request Tracing Service

At its core, the Request Tracing Service does the following:

• Tracks Requests: Monitors various request types across the Payara Server.
• Traces Long-Running Requests: Logs requests that exceed configured thresholds, helping identify slowdowns.
• Generates Insights: Extracts valuable trace data for troubleshooting and performance analysis.

Supported Request Types

Payara Server's Request Tracing Service covers a wide range of requests:

• JAX-RS endpoints and client calls
• MicroProfile REST Client calls
• Servlets (HTTP requests)
• SOAP Web Services
• WebSockets
• EJB timer executions
• Inbound JMS messages
• JBatch jobs
• Injected Managed Executors

OpenTelemetry vs. OpenTracing: The Evolution of Distributed Tracing

• OpenTracing: OpenTracing emerged as a vendor-neutral project designed to provide a standard API and guidelines for instrumenting applications with distributed tracing. This approach helped developers avoid vendor lock-in and simplified the process of integrating tracing across various parts of an application.
  
  
• OpenTelemetry: OpenTelemetry builds upon the foundations of OpenTracing and another project called OpenCensus. It aims to be a comprehensive observability solution, not only covering distributed tracing but also encompassing metrics and logging. Key advantages of OpenTelemetry include:
  
  
  • Unified Standard: OpenTelemetry provides a single, standardized way to collect all types of telemetry data.
  • Vendor-neutrality: You can choose your preferred backend tools for analyzing and visualizing traces without being being restricted to a single vendor.
  • Implementation Provided: OpenTelemetry offers official SDKs and libraries, reducing the implementation effort for developers compared to OpenTracing's API-focused approach.

OpenTelemetry and OpenTracing Support

Payara Server recognizes the significance of these standards and prioritises OpenTelemetry while still providing support for legacy systems:

• OpenTelemetry: The Primary Focus
  
  
  • Payara Server now natively supports OpenTelemetry APIs.
  • This means you can directly use OpenTelemetry SDKs to instrument your applications.
  • You can export traces using the OTLP (OpenTelemetry Protocol) for compatibility with a wide range of tracing backends.
    
    
• OpenTracing: Backward Compatibility
  
  
  • To simplify migration for applications already using OpenTracing, Payara Server offers an OpenTracing Compatibility Layer.
  • This layer translates OpenTracing API calls into the OpenTelemetry format.

If you are starting a new project or considering a switch, OpenTelemetry is the recommended choice for its expanded capabilities, broader scope and active development community. Payara's focus on OpenTelemetry reflects the industry shift towards this comprehensive observability standard.

Configuring Request Tracing

Payara Server provides multiple ways to manage its Request Tracing Service:

• Web Administration Console: A user-friendly interface for configuration.
• Administration Commands (asadmin): Fine-grained control through the command line.
• domain.xml: Direct editing for advanced scenarios.

Key Configuration Options

• Sampling Rates: Determines what percentage of requests get traced (flat probability or adaptive).
• Thresholds: Traces requests that surpass specified time limits.
• Trace Stores: Stores historical trace data for review.
• Notifiers: Sends trace information to various destinations, e.g. log files, Slack.

Using Request Tracing

Automatic Tracing

For ease of use, Payara automatically traces several common request types:

• JAX-RS endpoints: Your JAX-RS resource methods will be traced without any additional code.
• MicroProfile REST Client calls: Outbound calls made using the MicroProfile REST client are automatically tracked.
• Other Request Types: Servlets, SOAP Web Services, WebSockets and more are also covered by default.

Manual Tracing

When you need finer-grained control over your tracing, you have two primary options:

• @Traced Annotation:

  • Simply add the @Traced annotation (from the MicroProfile OpenTracing specification) to any method you want to include in tracing.
  • Example:

@RequestScoped

public class MyJaxRsResource {




    @Traced // Traces the entire method

    @GET

    @Path("/items")

    public List<Item> getItems() {

        // ... your item retrieval logic ...

    }
    @Traced(operationName = "processItem") // Custom operation name

    private Item processItem(Item rawItem) {

        // ...  your item processing logic ...

    }

}

In this example:

• • getItems() will be automatically traced, providing insights into its overall execution time.
  • processItem() is also traced; the span name is explicitly set to "processItem".

• Tracer Class:

Obtain an instance of the io.opentracing.Tracer class through @Inject, then use the Tracer to create and manage tracing spans for more customized tracking.

Example:

@RequestScoped

public class MyServiceBean {

    

    @Inject

    Tracer tracer;




    public void doSomeWork() {

        try (Span span = tracer.buildSpan("myCustomSpan").startActive(true)) {

            // ... code to be traced within the span ...




            span.setTag("dataProcessed", "importantData");  // Add a tag

            span.log("Processing step completed");          // Add a log

        }

    }

}

In this example:

• • A new span named "myCustomSpan" is created manually.
  • Tags and logs provide additional context within the span.

Choosing the Right Approach

For basic tracing of most common components, automatic tracing often suffices.

If you need specific tracing behavior, customizing which operations are traced or adding additional data to traces, the @Traced annotation or the Tracer class will be your tools of choice.

Methods for Disabling Automatic Tracing

@Traced(false) Annotation

• Method-level control: Add the @Traced(false) annotation directly to specific methods or classes you wish to exclude from tracing.
  
  
• Example:

@Traced(false) 

@GET

public Response getHealthStatus() {

    // ... health check logic

    return Response.ok("Healthy!").build();

}




@Traced(false) // Disables tracing for all methods in the class

public class InternalService { 

    // ... 

}

MicroProfile Config Properties

Pattern-based control: Define skip patterns using regular expressions within MicroProfile Config properties. This allows you to disable tracing for sets of resources that match the pattern.

Example (in your MicroProfile config file):

mp.opentracing.server.skip-pattern=/health.*

This pattern would disable tracing for any JAX-RS resource paths starting with "/health".

Specific Use Cases

• Health Checks and Internal Endpoints: You might want to exclude routine health checks or internal endpoints from tracing to reduce noise and overhead.

• Performance-Sensitive Operations: For highly performance-critical code sections, you might choose to disable tracing to avoid any potential overhead.

• Troubleshooting: Selectively disabling automatic tracing can help isolate performance issues or unexpected behavior.

Note that:

• The @Traced(false) annotation is available for JAX-RS methods and MicroProfile REST Client interfaces.

• Payara Server prioritizes the MicroProfile Config pattern over the @Traced(false) annotation, meaning if both are present, the config pattern will take effect.

Conclusions

Unlock the power of Payara Server's Request Tracing Service to debug issues, optimize performance and understand how your distributed applications work. Combine it with metrics and logging for a complete picture of your system's health. This approach will streamline troubleshooting, enhance user experience and help you build more reliable applications. Download Payara Community today and explore the Request Tracing documentation to get started. Happy Coding!