New Feature: Remote EJB Tracing in Payara Server Community 5.2020.4

New in the Payara Server 5.2020.4 release (August 2020), RMI calls to an EJB hosted on Payara Server from a Java SE client will now have the active Span Context automatically propagated to the server, with a counterpart server-side span being created as a child of this client call. This fills in a gap in our existing Request Tracing service, and comes with the added benefit of allowing a user to propagate key:value pairs from a Java SE client to Payara Server (or simply from server to server) in a request as OpenTracing baggage items - of particular interest to those who utilise Context Propagation or similar features in other application servers.

Setup

No additional setup is required on the client side over what is already required for making regular remote EJB calls - a dependency on `payara-embedded-all` or the appclient of a Payara Server instance, and the use of the `SerialInitContextFactory` Initial Context factory.

import javax.naming.Context;
import javax.naming.InitialContext;
import javax.naming.NamingException;
import java.util.Properties;

...

Properties contextProperties = new Properties();
contextProperties.setProperty(Context.INITIAL_CONTEXT_FACTORY, "com.sun.enterprise.naming.SerialInitContextFactory");

try {
    Context context = new InitialContext(contextProperties);
    EjbRemote zhuli = (EjbRemote) context.lookup(java:global/myRemoteEjb/Ejb);
} catch (NamingException ne) {
    logger.warning("Failed performing lookup:\n" + ne.getMessage());
}

Getting a Tracer Instance

Injection of an OpenTracing tracer is not supported on Java SE clients, so you must create an instance yourself and register it to the OpenTracing.io `GlobalTracer`. Manual creation of a Tracer instance and registration of it is only required if using a third-party tracer such as Zipkin or Jaeger - if using the built-in Payara Request Tracing service, the initialisation of a Tracer instance and registration will automatically happen during creation of the initial context.

While the built-in Payara Request Tracing service unfortunately does not yet support the tracing of applications running outside of the Payara Server runtime, it will still propagate the Span Context, baggage items, and create a Child Of Span on the server side.

import io.opentracing.Tracer;
import io.opentracing.util.GlobalTracer;

import javax.naming.Context;
import javax.naming.InitialContext;
import javax.naming.NamingException;
import java.util.Properties;

...

Properties contextProperties = new Properties();
contextProperties.setProperty(Context.INITIAL_CONTEXT_FACTORY, "com.sun.enterprise.naming.SerialInitContextFactory");

try {
    Context context = new InitialContext(contextProperties);
    EjbRemote zhuli = (EjbRemote) context.lookup(java:global/myRemoteEjb/Ejb);

    Tracer tracer = GlobalTracer.get();
} catch (NamingException ne) {
    logger.warning("Failed performing lookup:\n" + ne.getMessage());
}

Starting a Trace

The MicroProfile OpenTracing @Traced annotation is not supported on Java SE clients, so spans must be started and finished manually. Note that the Span Context of the active span will be propagated to the server, so it is recommended that you use a try-with-resources to help ensure your Span Scope is the one you expect.

import io.opentracing.Scope;
import io.opentracing.Span;
import io.opentracing.Tracer;
import io.opentracing.util.GlobalTracer;

import javax.naming.Context;
import javax.naming.InitialContext;
import javax.naming.NamingException;
import java.util.Properties;

...

Properties contextProperties = new Properties();
contextProperties.setProperty(Context.INITIAL_CONTEXT_FACTORY, "com.sun.enterprise.naming.SerialInitContextFactory");

try {
    Context context = new InitialContext(contextProperties);
    EjbRemote zhuli = (EjbRemote) context.lookup(java:global/myRemoteEjb/Ejb);

    Tracer tracer = GlobalTracer.get();

    try (Scope scope = tracer.buildSpan("theThing").startActive(true)) {
        zhuli.doTheThing();
    }
} catch (NamingException ne) {
    logger.warning("Failed performing lookup:\n" + ne.getMessage());
}

Once your span has been started, you can attach any desired baggage items and these will be propagated to the server along with the Span Context allowing you to retrieve them within the EJB
implementation hosted on Payara Server:

Client:

import io.opentracing.Scope;
import io.opentracing.Span;
import io.opentracing.Tracer;
import io.opentracing.util.GlobalTracer;

import javax.naming.Context;
import javax.naming.InitialContext;
import javax.naming.NamingException;
import java.util.Properties;

...

Properties contextProperties = new Properties();
contextProperties.setProperty(Context.INITIAL_CONTEXT_FACTORY, "com.sun.enterprise.naming.SerialInitContextFactory");

try {
    Context context = new InitialContext(contextProperties);
    EjbRemote zhuli= (EjbRemote) context.lookup(java:global/myRemoteEjb/Ejb);

    Tracer tracer = GlobalTracer.get();

    try (Scope scope = tracer.buildSpan("theThing").startActive(true)) {
        scope.span().setBaggageItem("The Thing", "Dress up as Platybus Bear");
        zhuli.doTheThing();
    }
} catch (NamingException ne) {
    logger.warning("Failed performing lookup:\n" + ne.getMessage());
}

Server:

import io.opentracing.Span;
import io.opentracing.Tracer;

import javax.ejb.Stateless;
import javax.inject.Inject;

@Stateless
public class Ejb implements EjbRemote {

    @Inject
    Tracer tracer;

...

    Span activeSpan = tracer.activeSpan();
    if (activeSpan != null) {
        String toDo = activeSpan.getBaggageItem("The Thing");
    }

With request tracing enabled and configured in such a way that the execution of the EJB method takes longer than the trace threshold, you should see a trace akin to this in the server logs, denoting the trace of the EJB implementation method and showcasing that the call came from a client with its Child Of relationship (formatted and highlighted for readability):

{
  "operationName":"rmi",
  "spanContext":{
    "spanId":"350c7c41-79af-4641-9a8c-6c82f10b76e5",
    "traceId":"7f6cf010-f33c-4c6a-aae1-a5fa970c8e05"
  },
  "startTime":"2020-08-11T15:14:27.459+01:00[Europe/London]",
  "endTime":"2020-08-11T15:14:28.346+01:00[Europe/London]",
  "traceDuration":"887000000",
  "spanTags":[
    {
      "component":"ejb"
    },
    {
      "Server":"server"
    },
    {
      "Domain":"domain1"
    }
  ],
  "spanLogs":[
    {
      "logDetails":[
        {
          "logEvent":"jtaContextBeginEvent"
        },
        {
          "Transaction ID":"0000000000000002_00"
        },
        {
          "Remaining Timeout":"0"
        }
      ],
      "logTime":"1597155267461"
    },
    {
      "logDetails":[
        {
          "logEvent":"enterEjbMethodEvent"
        },
        {
          "ApplicationName":"null"
        },
        {
          "ComponentName":"Ejb"
        },
        {
          "ComponentType":"STATELESS_SESSION_BEAN"
        },
        {
          "ModuleName":"lok-reference"
        },
        {
          "EJBClass":"fish.payara.samples.remote.ejb.tracing.Ejb"
        },
        {
          "EJBMethod":"doTheThing"
        },
        {
          "CallerPrincipal":"ANONYMOUS"
        },
        {
          "TX-ID":"JavaEETransactionImpl: txId=2 nonXAResource=null jtsTx=null localTxStatus=0 syncs=[com.sun.ejb.containers.ContainerSynchronization@149c717b]"
        }
      ],
      "logTime":"1597155267461"
    }
  ],
  "references":[
    {
      "spanContext":{
        "spanId":"1876af92-bed3-40ff-9f2d-413fcf3cd780",
        "traceId":"7f6cf010-f33c-4c6a-aae1-a5fa970c8e05"
      },
      "relationshipType":"ChildOf"
    }
  ]
}

You should also notice that other information such as the transaction ID and the EJB method name are automatically tagged - no extra actions are required by the user to get this information tagged.

Additional information can be found in our documentation(such as usage of the @Traced annotation and help with setting up alternative OpenTracing providers), but if you have any feedback please feel empowered to reach out to us via the usual channels such as onGitHub, ourforum, orTwitter. 

The Remote EJB Tracing feature has been added to Payara Server as a result of a Payara Enterprise customer request.