Building AI Agents with Jakarta EE

Artificial Intelligence (AI) and Large Language Models (LLMs) are revolutionizing enterprise Java applications, unlocking new capabilities, and driving unprecedented innovation and automation. But how can you bring the power of AI and LLMs like OpenAI or Ollama into your Jakarta EE/MicroProfile projects, without wrestling with complex, low-level integrations?

For developers, especially those leveraging the Payara Platform, the open-source SmallRye LLM project offers an elegant solution. It simplifies the creation of AI-powered features, often referred to as 'AI agents,' by integrating LLM capabilities through familiar Jakarta EE paradigms like CDI (Contexts and Dependency Injection).

In this blog, we’ll delve into how SmallRye LLM works and explore its integration within the Payara Platform, demonstrating how it supports the creation of AI Jakarta EE applications. We'll then guide you through a practical 'payara-car-booking' demo, showcasing AI-driven conversations and seamless backend integration."

Why AI Agents in Jakarta EE Applications?

Software applications are increasingly expected to do more with less, from summarizing vast content, automating customer support, generating complex documents, or answering complex user queries, often with fewer resources. AI agents can help address these challenges and become a gamechanger for Jakarta EE applications, as they can enrich enterprise system with new opportunities and capabilities.

In particular, AI agents can help applications become smarter and more interactive, as these agents can:

• Process natural language and handle unstructured data.
• Learn from their interactions (in line with retrieval-augmented generation, RAG).
• Deliver personalized and context-aware  responses. 

As a result, it is possible to make the user experience more engaging and useful. For instance, you might add an AI-powered chatbot to your application to handle customer queries, provide tailored recommendations or automate routine support tasks.

What is SmallRye LLM?

SmallRye LLM is an experimental, lightweight Java library designed to seamlessly integrate LLM-powered capabilities into Jakarta EE, MicroProfile and Quarkus applications. It empowers developers to easily inject LangChain4j AI Services into their code using CDI. 

At its core, SmallRye LLM leverages LangChain4j, a Java-native framework inspired by the popular Python-based LangChain (which also explains the project's alias, "Langchain4J Microprofile"). LangChain4j provides the foundation for creating sophisticated AI agents capable of processing information, responding to users and interacting with external tools.

By bridging LangChain4j's capabilities with these Java ecosystems, SmallRye LLM offers several key advantages:

• CDI Integration: Easily inject LangChain4j AI Services (interfaces you define, backed by LLMs) as CDI beans, often using annotations like @RegisterAiService.
• Flexible Model Configuration: Utilize MicroProfile Config to seamlessly select and configure various LLM models (e.g., OpenAI, Ollama, Hugging Face) and their parameters.
• Tool Integration: Enable AI Services to interact with your existing backend services or other external data sources through LangChain4j's declarative @Tool mechanism, managed within the CDI environment.
• Visibility and Monitoring: Benefit from built-in support for crucial enterprise features like observability (metrics, tracing), fault tolerance, and context propagation.

SmallRye LLM Key Capabilities and Ecosystem Integration

The SmallRye LLM project, available on GitHub atsmallrye/smallrye-llm, is designed to integrate seamlessly with the broader Jakarta EE and MicroProfile landscape. In this repository, you can find several modules to support Jakarta EE developers, each focusing on different aspects of LLM integration:

• Core Module: Provides the foundational CDI mechanisms. This allows developers to inject and manage LangChain4j AI Services as standard CDI beans within their applications
• Config Module: All configurations, such as LLM provider details (e.g., API keys, endpoints for OpenAI, Ollama), model parameters and timeouts, are managed through MicroProfile Config
• Fault Tolerance Module: Developers can apply MicroProfile Fault Tolerance policies (like @Retry, @Fallback, @CircuitBreaker) to CDI beans, making interactions with LLMs more robust and reliable.
• Telemetry Module: By integrating with MicroProfile Telemetry, developers can gain valuable insights into the performance, usage, and behavior of their LLM-powered features.
• Portable and Build-Compatible Extensions: A key design principle is to ensure that AI Service integrations are portable across various Jakarta EE and MicroProfile compatible platforms

How Does SmallRye LLM Work: The Payara Example

To understand the practical benefits of SmallRye LLM, let's dive into a working example deployed on Payara Server: the payara-car-booking application. This demonstration is inspired by the insightful "Java meets AI" talk from Lize Raes at Devoxx Belgium 2023 (with further contributions from Jean-François James and Dmytro Liubarskyi). It showcases how you can build a conversational AI for a car booking system. The demo provides a hands-on look at integrating sophisticated LLM capabilities into a standard Jakarta EE application.

Don't have Payara Platform Community yet? Download it from here!

Before we explore the code's logic, we will do a quick look at the dependencies in pom.xml file:

• Jakarta EE Core: The application is built upon the standard jakarta.platform:jakarta.jakartaee-api, ensuring it leverages the robust and familiar Jakarta EE 10 environment provided by Payara Server
• SmallRye LLM Integration: The key enabler is the io.smallrye.llm:smallrye-llm-langchain4j-portable-extension dependency. This artifact is the bridge, bringing LangChain4j's AI service capabilities into the Jakarta EE fold through CDI. The portable-extension nature underscores its design for compatibility across compliant application servers.
• LangChain4j Foundation: Underneath SmallRye LLM, we have:
  • dev.langchain4j:langchain4j: The core LangChain4j library providing the tools and abstractions to interact with LLMs
  • dev.langchain4j:langchain4j-open-ai and dev.langchain4j:langchain4j-hugging-face: These specific LangChain4j modules allow the application to connect to different LLM providers. In this case, it demonstrates flexibility with OpenAI and Hugging Face models

Another relevant file to explore is llm-config.properties. This file leverages MicroProfile Config to define and customize the various AI services and components used by the application. Let's break down the key configurations:

1. Chat Model Configuration (The LLM Core):

   • smallrye.llm.plugin.chat-model.class=dev.langchain4j.model.openai.OpenAiChatModel: specifies the use of OpenAI chat model
   • smallrye.llm.plugin.chat-model.config.base-url=http://localhost:11434/v1: points the application to a local Ollama instance (running on localhost:11434) rather than OpenAI's public API
   • smallrye.llm.plugin.chat-model.config.model-name=llama3.1: specifies the model to be used via Ollama (in this case, llama3.1)
   • smallrye.llm.plugin.chat-model.config.api-key=not-needed: Confirms that no API key is needed, typical for local Ollama setups
     
     

2. Document RAG Retriever Configuration (Knowledge Augmentation):

   • smallrye.llm.plugin.docRagRetriever.class=dev.langchain4j.rag.content.retriever.EmbeddingStoreContentRetriever: sets up a content retriever for RAG, allowing the LLM to use information from your documents
   • smallrye.llm.plugin.docRagRetriever.config.embeddingStore=lookup:default and smallrye.llm.plugin.docRagRetriever.config.embeddingModel=lookup:default: these lookup:default values instruct SmallRye LLM to find the default CDI beans for the embedding store and embedding model
   • maxResults and minScore: control how many documents are retrieved and their threshold
     
     

3. Chat Memory Configuration (Maintaining Conversation Context):

   • Blocks like smallrye.llm.plugin.chat-ai-service-memory.*  define how conversation history is managed
   • class=dev.langchain4j.memory.chat.MessageWindowChatMemory: specifies a sliding window approach to keep recent messages
   • scope=jakarta.enterprise.context.ApplicationScoped: defines the CDI scope for the chat memory bean
   • config.maxMessages=10: sets the number of messages to retain in the conversation window
     
     

4. Embedding Store Persistence:

   • smallrye.llm.embedding.store.in-memory.file=embedding.json: this configures an in-memory embedding store to persist its data to a local embedding.json file
     
     

5. Application-Specific Parameters:

   • Properties like chat.memory.max.messages, fraud.memory.max.messages and app.docs-for-rag.dir are custom values used by other parts of the application to further configure AI components or locate resources like RAG documents.

In summary, the llm-config.properties file provides a declarative way to define and tune all aspects of LLM integration, from the core model interaction and RAG capabilities to chat memory management.

Inside docs-for-rag: The AI's Reference Material

The smallrye-llm/examples/payara-car-booking/docs-for-rag/ directory is where the raw text files reside that provide the payara-car-booking AI agent with its specialized knowledge. These documents are crucial for the Retrieval Augmented Generation (RAG) process, allowing the LLM to pull in factual, domain-specific information to answer user queries accurately. By feeding the content of these files into the system, the AI becomes an expert on Miles of Smiles Car Rental Services.

Summary of each file's content found in this directory:

• general-information.txt: offers a snapshot of Miles of Smiles, including figures like employee numbers, international reach, fleet size with electric vehicle percentage and customer satisfaction rating
• list-of-cars.txt: provides a list of cars available for rent from Miles of Smiles such as Aston Martin, BMW, and Tesla.
• terms-of-use.txt: outlines the terms and conditions governing the use of Miles of Smiles car rental services, covering bookings, cancellations, vehicle usage restrictions, liability and legal governance.

The information from these files equips the AI agent to handle specific inquiries about the car rental service.

A Closer Look at the Code: Key Classes and Snippets

Now that we've covered the setup and configuration, let's examine some of the core Java classes from the io.jefrajames.booking package in the payara-car-booking example.

The Data Model: Booking.java and Customer.java

At the foundation, we have simple Plain Old Java Objects (POJOs) to represent our data. The Booking and Customer classes uses Lombok's @Data annotation for automatic generation of getters, setters, equals(), hashCode(), and toString() methods. Booking simply holds the state for a car booking, including customer details (referencing Customer class) and the car model.

@Data
@NoArgsConstructor
@AllArgsConstructor
public class Booking {
    private String bookingNumber;
    private LocalDate start;
    private LocalDate end;
    private Customer customer;
    private boolean canceled = false;
    private String carModel;
}

@Data
@NoArgsConstructor
@AllArgsConstructor
@EqualsAndHashCode(of = { "name", "surname" })
public class Customer {
    private String name;
    private String surname;
}

Handling Specific Scenarios: Custom Exceptions

Custom exceptions are used to clearly define specific error conditions related to booking operations: BookingAlreadyCanceledException, BookingCannotBeCanceledException and  BookingNotFoundException.

The Core Logic & AI Tools: BookingService.java

This is where much of the application's business logic resides and, crucially, where methods are exposed as Tools for the AI agent.

@ApplicationScoped
@Log
public class BookingService {
    private static final Map<String, Booking> BOOKINGS = new HashMap<>();

    static {
        // James Bond: hero customer!
        BOOKINGS.put("123-456", new Booking("123-456", LocalDate.now().plusDays(1), LocalDate.now().plusDays(7),
                new Customer("James", "Bond"), false, "Aston Martin"));
        BOOKINGS.put("234-567", new Booking("234-567", LocalDate.now().plusDays(10), LocalDate.now().plusDays(12),
                new Customer("James", "Bond"), false, "Renault"));
        BOOKINGS.put("345-678", new Booking("345-678", LocalDate.now().plusDays(14), LocalDate.now().plusDays(20),
                new Customer("James", "Bond"), false, "Porsche"));
        ...
    }
    ...
    @Tool("Get booking details given a booking number and customer name and surname")
    public Booking getBookingDetails(String bookingNumber, String name, String surname) {
        log.info("DEMO: Calling Tool-getBookingDetails: " + bookingNumber + " and customer: "
                + name + " " + surname);
        return checkBookingExists(bookingNumber, name, surname);
    }

    @Tool("Get all booking ids for a customer given his name and surname")
    public List<String> getBookingsForCustomer(String name, String surname) {
        log.info("DEMO: Calling Tool-getBookingsForCustomer: " + name + " " + surname);
        Customer customer = new Customer(name, surname);
        return BOOKINGS.values()
                .stream()
                .filter(booking -> booking.getCustomer().equals(customer))
                .map(Booking::getBookingNumber)
                .collect(Collectors.toList());
    }
    ...
    @Tool("Cancel a booking given its booking number and customer name and surname")
    public Booking cancelBooking(String bookingNumber, String name, String surname) {
        log.info("DEMO: Calling Tool-cancelBooking " + bookingNumber + " for customer: " + name
                + " " + surname);
        Booking booking = checkBookingExists(bookingNumber, name, surname);
        if (booking.isCanceled())
            throw new BookingCannotBeCanceledException(bookingNumber);
        checkCancelPolicy(booking);
        booking.setCanceled(true);
        return booking;
    }
}

BookingService.java is a CDI @ApplicationScoped bean. The key feature here is the use of LangChain4j's @Tool annotation. Methods like getBookingDetails(), getBookingsForCustomer(), and cancelBooking() are exposed to the AI agent. The Javadoc or the string value in the @Tool annotation helps the LLM understand the tool's purpose, parameters, and when to use it. The LLM can then decide to call these methods to fulfill a user's request.

The API Layer: CarBookingResource.java

This class exposes the AI functionalities via a JAX-RS REST API.

@ApplicationScoped
@Path("/car-booking")
public class CarBookingResource {

    @Inject
    private ChatAiService aiService;

    @Inject
    private FraudAiService fraudService;

    @GET
    @Produces(MediaType.TEXT_PLAIN)
    @Path("/chat")
    public String chatWithAssistant(
            @QueryParam("question") String question) {
        String answer;
        try {
            answer = aiService.chat(question);
        } catch (Exception e) {
            e.printStackTrace();
            answer = "My failure reason is:\n\n" + e.getMessage();
        }
        return answer;
    }

    @GET
    @Produces(MediaType.APPLICATION_JSON)
    @Path("/fraud")
    public FraudResponse detectFraudForCustomer(
            @QueryParam("name") String name,
            @QueryParam("surname") String surname) {
        return fraudService.detectFraudForCustomer(name, surname);
    }
}

CarBookingResource.java is a standard JAX-RS resource. It uses @Inject to get instances of ChatAiService and FraudAiService. These services are the interfaces to our AI agents, likely configured via SmallRye LLM (as seen in llm-config.properties) and powered by LangChain4j. The /chat endpoint simply takes a user's question, passes it to the ChatAiService, and returns the AI's response.

AI Service Definition: ChatAiService.java

This interface defines the contract for our main conversational AI agent. SmallRye LLM handles the implementation.

@SuppressWarnings("CdiManagedBeanInconsistencyInspection")
@RegisterAIService(tools = BookingService.class, chatMemoryName = "chat-ai-service-memory", chatLanguageModelName = "chat-model")
public interface ChatAiService {

    @SystemMessage("""
            You are a customer support agent of a car rental company named 'Miles of Smiles'.
            Before providing information about booking or canceling a booking, you MUST always check:
            booking number, customer name and surname.
            You should not answer to any request not related to car booking or Miles of Smiles company general information.
            When a customer wants to cancel a booking, you must check his name and the Miles of Smiles cancellation policy first.
            Any cancelation request must comply with cancellation policy both for the delay and the duration.
            Today is .
            """)
    // String chat(@V("question") @UserMessage String question);
    String chat(String question);

    default String chatFallback(String question) {
        return String.format(
                "Sorry, I am not able to answer your request %s at the moment. Please try again later.",
                question);
    }
}

ChatAiService.java is an interface, not a concrete implementation. The @RegisterAIService annotation from SmallRye LLM is pivotal. It tells SmallRye LLM to create a CDI bean that implements this interface, wiring it up with the specified tools (from BookingService.class), chat memory and language model configured in llm-config.properties. The @SystemMessage annotation provides the LLM with its persona, instructions, and context, including dynamic data like .

Preparing RAG Data: DocRagIngestor.java

This CDI bean is responsible for loading documents, creating embeddings, and populating the embedding store for RAG at application startup.

@Log
@ApplicationScoped
public class DocRagIngestor {

    @Produces
    private EmbeddingModel embeddingModel = new AllMiniLmL6V2EmbeddingModel();

    @Produces
    private InMemoryEmbeddingStore embeddingStore = new InMemoryEmbeddingStore<>();

    private File docs = new File(System.getProperty("docragdir"));

    private List<Document> loadDocs() {
        return loadDocuments(docs.getPath(), new TextDocumentParser());
    }

    public void ingest(@Observes @Initialized(ApplicationScoped.class) Object pointless) {
        long start = System.currentTimeMillis();
        EmbeddingStoreIngestor ingestor = EmbeddingStoreIngestor.builder()
                .documentSplitter(DocumentSplitters.recursive(300, 30))
                .embeddingModel(embeddingModel)
                .embeddingStore(embeddingStore)
                .build();
        List<Document> docs = loadDocs();
        ingestor.ingest(docs);
        log.info(String.format("DEMO %d documents ingested in %d msec", docs.size(),
                System.currentTimeMillis() - start));
    }

    public static void main(String[] args) {
        System.out.println(InMemoryEmbeddingStore.class.getInterfaces()[0]);
    }
}

DocRagIngestor.java is a crucial @ApplicationScoped CDI bean for the RAG functionality.

• It uses @Produces to programmatically create and make available the EmbeddingModel (here, AllMiniLmL6V2EmbeddingModel, a local sentence transformer model) and an InMemoryEmbeddingStore. These beans are then used by LangChain4j, likely corresponding to the lookup:default configurations seen in llm-config.properties.
• The ingest() method, triggered by the @Initialized(ApplicationScoped.class) event, runs at application startup. It loads documents from a directory specified by the docragdir system property (defaulting to docs-for-rag), splits them into manageable segments, generates embeddings using the embeddingModel, and stores them in the embeddingStore. This makes the document content searchable for RAG.

Custom LLM Configuration (Advanced): DummyLLConfig.java

This class demonstrates an advanced capability: providing a custom way to load configurations for SmallRye LLM, bypassing or augmenting the standard MicroProfile Config.

public class DummyLLConfig implements LLMConfig {
    Properties properties = new Properties();

    @Override
    public void init() {
        try (FileReader fileReader = new FileReader(System.getProperty("llmconfigfile"))) {
            properties.load(fileReader);
        } catch (IOException e) {
            throw new RuntimeException(e);
        }
    }

    @Override
    public Set<String> getBeanNames() {
        return properties.keySet().stream().map(Object::toString)
                .filter(prop -> prop.startsWith(PREFIX))
                .map(prop -> prop.substring(PREFIX.length() + 1, prop.indexOf(".", PREFIX.length() + 2)))
                .collect(Collectors.toSet());
    }

    @Override
    public <T> T getBeanPropertyValue(String beanName, String propertyName, Class<T> type) {
        String value = properties.getProperty(PREFIX + "." + beanName + "." + propertyName);
        if ( value==null)
            return null;
        if ( type==String.class)
            return (T) value;
        if ( type== Duration.class)
            return (T) Duration.parse(value);
        try {
            return type.getConstructor(String.class).newInstance(value);
        } catch (Exception e) {
            throw new IllegalArgumentException();
        }
    }

    @Override
    public Set<String> getPropertyNamesForBean(String beanName) {
        String configPrefix = PREFIX + "." + beanName + ".config.";
        return properties.keySet().stream().map(Object::toString)
                .filter(prop -> prop.startsWith(configPrefix))
                .map(prop -> prop.substring(configPrefix.length()))
                .collect(Collectors.toSet());
    }
}

DummyLLConfig.java implements SmallRye LLM's LLMConfig Service Provider Interface (SPI). This is an advanced feature allowing developers to provide LLM configurations from sources other than the standard MicroProfile Config mechanism (like microprofile-config.properties). In this example, it loads properties from a file path specified by the llmconfigfile system property. This could be used for more dynamic configuration loading, testing scenarios or when deploying in environments with specific configuration management needs. For most standard use cases, relying on microprofile-config.properties would be the primary approach.

Conclusions

We hope this journey through SmallRye LLM and the payara-car-booking example has shed a light on the exciting possibilities that arise when LLMs meet the robust world of Jakarta EE, especially within the Payara Platform. We've moved beyond the theoretical, diving deep into a practical demonstration that showcases just how accessible and powerful this integration can be.

The application is more than just a demo, it's a blueprint. It illustrates that integrating cutting-edge AI—from conversational chatbots and RAG-enhanced Q&A systems to intelligent fraud detection—into your enterprise Java applications is no longer a Herculean task. With SmallRye LLM on the Payara Platform, you can tap into the potential of LLMs without leaving the comfort and reliability of the Jakarta EE ecosystem.

Ready to Build the Future? Get started with Payara Platform Community now!