Kafka

logiccloud Control can connect to an Apache Kafka cluster to publish process variables to, and consume messages from, Kafka topics. To do this, a new connection of type Kafka must be created.

Create a new Kafka connection

Select Add Kafka connection and enter the broker information. After the connection has been created, select it in the outline tree to edit its configuration.

Connection settings (on the connection itself):

• Name: name of the connection
• Description: description, optional
• Brokers: comma-separated list of Kafka broker addresses (host:port)
• Enable Low Level Logging: verbose client logging for troubleshooting

Network and security options are configured on each producer and consumer (not on the connection):

• Enable Tls: connect to the broker over TLS (off by default)
• Enable Sasl + SASL mechanism: SASL authentication with the broker. Supported mechanisms are PLAIN, OAUTHBEARER, SCRAM-SHA-256, SCRAM-SHA-512 and GSSAPI (Kerberos), each with their corresponding user/password/token or Kerberos fields.

Note

TLS can be enabled, but client-side TLS certificate files are not yet configurable from the portal — only the Enable Tls toggle is exposed.

Sending and receiving data through Kafka

As with MQTT, a Kafka connection can work as a producer (publishes process variable values to a topic; maps Output variables) or as a consumer (reads messages from a topic into Input variables).

To create a producer or consumer, select the connection node in the tree and use its Add consumer or Add producer action.

A producer groups its variables into one or more payloads. Each payload publishes to its own configurable topic (with an optional partition key) and has its own trigger and payload format. The supported payload formats are rawPlainText, rawBinary, jsonValue, jsonDictionary and jsonObjectWithKey — the same JSON value / JSON dictionary / JSON-object-with-key model used by the MQTT connection.

Mapping process variables

After adding a producer or consumer, you can map process variables to the Kafka message payload.

• Variable: the program variable that will be mapped
• Data type: data type of the variable
• Value path: path of the value within the JSON payload (used by the jsonDictionary and jsonObjectWithKey formats)
• Key path / Key name: key path and key name within the payload (used by the jsonObjectWithKey format)
• Sub topic: per-mapping sub-topic (used by the rawPlainText, rawBinary and jsonValue formats)

The destination topic and the optional partition key are configured per payload (on the producer), not per mapping. The partition key is therefore set at the payload level, while the per-mapping fields above control how each variable is placed inside the message body.