topic Re: Consuming Kafka events using Workato in Workato Pros Discussion Board

Consuming Kafka events using Workato

krishna — Wed, 14 Jul 2021 07:10:43 GMT

We are planning to leverage workato to consume events from Kafka generated from our product. Did anyone worked on similar use cases and aware of any limitations or gotchas i need to be knowing before implementing it?

Re: Consuming Kafka events using Workato

steven-marissen — Wed, 14 Jul 2021 16:43:43 GMT

We are looking into a similar usecase with either Kafka (self managed) or Confluent Cloud (managed kafka) in a POC phase. We've been able to produce & consume messages using Workato.

I do think the confluent cloud connector could offer a broader configuration / features. Some examples are: consume only from a specific partition instead of all of them (=default). Also not sure how consumer groups would work with Workato (= concurrency greater then 1?).

Apart from that something on our to-do list to test rigorously is the behaviour:

- When a new version of the recipe is promoted, does it impact off-set?

- Impact of starting a clone on off-set

- Impact of an incident at Workato itself

- Handling errors (DLQ topic or something else)

- Monitoring ( x amount messages produced into topic = amount of messages consumed from topic? If not, how to identify delta's)

Re: Consuming Kafka events using Workato

triesen — Thu, 15 Jul 2021 23:29:20 GMT

We're also using the Confluent connector, and highly recommend Confluent 🙂 Workato + Kafka has been pretty straight-forward for us; the main pre-requisite is getting the schema defined correctly (if not already defined). Happy to collaborate on anything if its helpful, and +1 on Steven's point to see if Workato can add broader configuration / features.

Re: Consuming Kafka events using Workato

steven-marissen — Fri, 16 Jul 2021 01:28:41 GMT

Hi Tony,

We do experience an issue when we consume from a Confluent Cloud topic, we are able to pull messages from the topic into Workato but the consumer never appears in the confluent cloud data flow or under cluser -> clients -> consumer groups.

This is an essential piece for monitoring consumer lag etc.

Have you encountered this issue before?

Thanks in advance for your feedback.

Re: Consuming Kafka events using Workato

rak-9177 — Wed, 28 Jul 2021 04:40:34 GMT

We builded lot of integrations in Workato in kafka. Happy to share information.

Re: Consuming Kafka events using Workato

steven-marissen — Wed, 28 Jul 2021 15:00:53 GMT

Hi Rakesh,

Any pitfalls to look out for?

Re: Consuming Kafka events using Workato

rak-9177 — Sat, 31 Jul 2021 15:03:44 GMT

Steven Marissens- Yes, there are some like especially for recipes which are consuming huge events from kafka but there are alternatives to avoid this errors. Stopping consumer recipes will loose messages as it’s won’t store latest offset. For high volumes, don’t build huge logic in consumer recipe which will delay Workato jobs and it will reduce the consumption speed(number of batches/minute) from kafka even though if kafka byte size is increased. Workato charges based on tasks so if your integrations deals with high volume then it would be expensive over the time.expensive.

Re: Consuming Kafka events using Workato

rak-9177 — Sat, 31 Jul 2021 15:06:36 GMT

Re: Consuming Kafka events using Workato

rak-9177 — Sat, 31 Jul 2021 15:07:57 GMT

It’s very hard to debug issues if we consume in batches.

Re: Consuming Kafka events using Workato

steven-marissen — Sat, 31 Jul 2021 20:06:20 GMT

Excellent feedback Rakesh, much appreciated. My main concern now is the storage of offset, as it would mean unwanted behaviour when you upgrade the version of the consumer or in case of downtime at Workato. For me it ruins the possible usage of Kafka.

How do you deal with that? Or is this much rather a highly needed enhancement Workato should look into?

Re: Consuming Kafka events using Workato

rak-9177 — Wed, 11 Aug 2021 04:33:02 GMT

We found a bug and some design issues in kafka connector process and currently working with Workato product teams. Workato team is acknowledged and working on this. I can provide you more details n couple of days.

Re: Consuming Kafka events using Workato

steven-marissen — Thu, 12 Aug 2021 15:12:37 GMT

Again appreciated for your input. We've also noticed the same for Confluent Cloud connector which also needs to be enhanced for it to work well with CC. In it's current form no monitoring (or very limited) in CC can be done.

Keep us posted on the progress! 🙂

Re: Consuming Kafka events using Workato

rak-9177 — Fri, 13 Aug 2021 19:53:11 GMT

Sure definitely!

Re: Consuming Kafka events using Workato

shashankbyakod — Tue, 25 Nov 2025 07:54:41 GMT

I am facing issue with the Kafka connector integration, I wanted to setup kafka connections with plain text or json format of data. Hence, not relying on the Schema registry. But wile editing the recipe, i was not able to make changes on the script. Pls help here, whether we do have options to use non-avro schema based.

Re: Consuming Kafka events using Workato

devidmithen — Wed, 26 Nov 2025 06:32:16 GMT

Nice information.