Connecting to Instaclustr Managed PostgreSQL® and Apache Kafka® from Payara Cloud

Published on 18 Mar 2024

This article provides a brief introduction to using Instaclustr’s managed PostgreSQL Database and Apache Kafka from Payara Cloud. It was created to assist entrants in the upcoming Payara Hackathon (you can still sign up!) spin-up and configure some potentially useful Instaclustr technologies for Jakarta EE applications.

Why would you want to use PostgreSQL or Kafka with Jakarta EE?

Here are a few potential reasons:

PostgreSQL is a reliable and scalable SQL database that supports JDBC.
PostgreSQL has good support for JSON data.
PostgreSQL started out life as an object-oriented database, and it has good support for Java data types. Here’s a blog I wrote on mappings from PostgreSQL datatypes to Java.
Apache Kafka is great for building scalable stream processing systems, and integrating heterogeneous source and sink systems. For example:
It has support for Kafka Streams;, here are 3 blogs I wrote about it: https://www.linkedin.com/pulse/kafka-streams-3-blogs-paul-brebner/
Kafka Connect enables you to integrate with many different technologies. You can run a Kafka connect cluster alongside the Kafka cluster in your trial, and use it to generate test data, or ingest data from interesting public sources.

1. Sign up for a Free Trial

Go to the “Free Trial” button on our https://instaclustr.com homepage, which takes you to: https://console2.instaclustr.com/signup

No payment details are required, it’s free, the trial is for the non-production SLA tier, a subset of our services and node sizes (only developer-sized nodes https://www.instaclustr.com/pricing/), for 30 days. After 30 days (if you haven’t signed up for a normal subscription) the clusters (and all the data) will be deleted.

2. PostgreSQL

There are four main steps: (1) Create a PostgreSQL cluster, (2) Allow access, (3) Get connection details and (4) configure Payara cloud to use the PostgreSQL cluster and test (not covered here, see relevant Payara documentation)

2.1 Create a PostgreSQL cluster using the Instaclustr console

Instructions are here: https://www.instaclustr.com/support/documentation/postgresql/getting-started-with-postgresql/getting-started-with-postgres-creating-a-postgres-cluster/

Summary (unless otherwise instructed, don’t change default values):

1. “Create Cluster”

2. Enter Cluster Name

3. Application Selection: PostgreSQL

4. Provider: AWS

5. No Enterprise Features

Next, PostgreSQL Setup

6. Select: Version: PostgreSQL 14.10 (Any version 15+ will require changes to permissions, see below)

7. Replication Mode: ASYNCHRONOUS

8. Data Centre Options: Data Centre: Select the region you would like to use.

Next, Next

9. Accept terms

10. Create Cluster

Your cluster will provision and become available after a few minutes.

2.2 Add Payara cloud IPs to the PostgreSQL cluster firewall

For general instructions see here.

Click on your PostgreSQl cluster in the console, and then Firewall Rules.

Add these IPs to the firewall and Save:

20.50.49.180

20.62.214.88

These are documented here.

2.3 Get Configuration Details

Documentation on connection to Instaclustr PostgreSQL with Java clients is here.

Specific configuration details are available for your PostgreSQL cluster under Connection Info.

Details include Public and Private IPs, Port number, Username, Password, and Database name.

The user, host, port and dbname are visible under the CLI example (dname=postgres).

Under the Java example you can see the JDBC URL, user and password.

The JDBC URL looks like this:

"jdbc:postgresql://IP:5432/postgres?targetServerType=primary"

If you choose Postgres version 15+, you might have to grant create permission on the schema public of your database to be able to execute statements.

See our documentation for further information.

In order to change permissions you will need to:

Add your IP address to the PostgreSQL firewall (see above).
Download the PostgreSQL CLI application: https://www.postgresql.org/download/

2.4 Configure Payara Cloud to use PostgreSQL

You may need a PostgreSQL JDBC driver: https://jdbc.postgresql.org/documentation/use/

You will need to configure the Payara Cloud data source to use the driver with the relevant connection details from 2.3, and test it.

Potentially relevant documentation. And a guide for PostgreSQL (sign up required).

3. Apache Kafka

3.1 Create Kafka Cluster

The process for creating a Kafka Cluster is very similar, see documentation here.

1. Create Cluster

2. Enter Cluster Name

3. Application Selection = Apache Kafka

4. Kafka Setup: Don’t change the defaults (in particular, don’t select “KRaft” mode, as you will be using the older ZooKeeper mode)

5. Data Centre Options: Data Centre: Select desired region.

6. Accept terms

7. Create Cluster

Wait a few minutes

3.2 Allow Access

General instructions.

Add these IPs to the firewall and Save:

20.50.49.180

20.62.214.88

3.3. Get Configuration Details

Generic Kafka + Java specific instructions are here.

Specific Kafka connection information is available from the Connection Info tab.

This has Public IPs, User Authorization (Enabled), Encryption (Disabled), etc.

The default username/password are displayed (for 5 days only!). Detailed information is under Examples, Producer/Consumer, Java.

Producer configurations have:

bootstrap.servers=IP1:9092,IP2:9092,3IP3:9092

key.serializer=org.apache.kafka.common.serialization.StringSerializer

value.serializer=org.apache.kafka.common.serialization.StringSerializer

security.protocol=SASL_PLAINTEXT

sasl.mechanism=SCRAM-SHA-256

sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \

username="user" \

password="password";

Consumer configuration is identical with the addition of a consumer group.id (In Kafka, multiple consumers can consume from a topic, for increased concurrency, if they are in the same group):

group.id=my-group

3.4 Configure Payara Cloud to use Kafka

Use the relevant information from 3.3. to configure Payara Cloud to use your Apache Kafka cluster. You can send and receive messages to/from Kafka using Cloud Connectors, JCA and MDBs. See this documentation.

Note that I couldn’t find where to configure Kafka user/password details.

4. Further Information

Instaclustr: https://www.instaclustr.com/
Payara Cloud: https://www.payara.fish/products/payara-cloud/
PostgreSQL: https://www.postgresql.org/
Apache Kafka: https://kafka.apache.org/
My blogs have lots of Kafka and PostgreSQL examples: https://www.instaclustr.com/blog/author/paul-brebner/
All Instaclustr PostgreSQL blogs: https://www.instaclustr.com/blog/category/technical/postgresql/
All Instaclustr Kafka blogs: https://www.instaclustr.com/blog/category/technical/kafka/