cancel
Showing results for 
Search instead for 
Did you mean: 

Consuming Kafka events using Workato

krishna
Deputy Chef I
Deputy Chef I

We are planning to leverage workato to consume events from Kafka generated from our product. Did anyone worked on similar use cases and aware of any limitations or gotchas i need to be knowing before implementing it?

12 REPLIES 12

Hi Rakesh,

Any pitfalls to look out for?

Steven Marissens- Yes, there are some like especially for recipes which are consuming huge events from kafka but there are alternatives to avoid this errors. Stopping consumer recipes will loose messages as it’s won’t store latest offset. For high volumes, don’t build huge logic in consumer recipe which will delay Workato jobs and it will reduce the consumption speed(number of batches/minute) from kafka even though if kafka byte size is increased. Workato charges based on tasks so if your integrations deals with high volume then it would be expensive over the time.expensive.

rak-9177
Deputy Chef I
Deputy Chef I

Steven Marissens- Yes, there are some like especially for recipes which are consuming huge events from kafka but there are alternatives to avoid this errors. Stopping consumer recipes will loose messages as it’s won’t store latest offset. For high volumes, don’t build huge logic in consumer recipe which will delay Workato jobs and it will reduce the consumption speed(number of batches/minute) from kafka even though if kafka byte size is increased. Workato charges based on tasks so if your integrations deals with high volume then it would be expensive over the time.expensive.

Excellent feedback Rakesh, much appreciated. My main concern now is the storage of offset, as it would mean unwanted behaviour when you upgrade the version of the consumer or in case of downtime at Workato. For me it ruins the possible usage of Kafka.

How do you deal with that? Or is this much rather a highly needed enhancement Workato should look into?

rak-9177
Deputy Chef I
Deputy Chef I

It’s very hard to debug issues if we consume in batches.