Consuming Kafka events using Workato

[krishna]

We are planning to leverage workato to consume events from Kafka generated from our product. Did anyone worked on similar use cases and aware of any limitations or gotchas i need to be knowing before implementing it?

[steven-marissen]

Hi Rakesh,

Any pitfalls to look out for?

[rak-9177]

Steven Marissens- Yes, there are some like especially for recipes which are consuming huge events from kafka but there are alternatives to avoid this errors. Stopping consumer recipes will loose messages as it’s won’t store latest offset. For high volumes, don’t build huge logic in consumer recipe which will delay Workato jobs and it will reduce the consumption speed(number of batches/minute) from kafka even though if kafka byte size is increased. Workato charges based on tasks so if your integrations deals with high volume then it would be expensive over the time.expensive.

[rak-9177]

Steven Marissens- Yes, there are some like especially for recipes which are consuming huge events from kafka but there are alternatives to avoid this errors. Stopping consumer recipes will loose messages as it’s won’t store latest offset. For high volumes, don’t build huge logic in consumer recipe which will delay Workato jobs and it will reduce the consumption speed(number of batches/minute) from kafka even though if kafka byte size is increased. Workato charges based on tasks so if your integrations deals with high volume then it would be expensive over the time.expensive.

[steven-marissen]

Excellent feedback Rakesh, much appreciated. My main concern now is the storage of offset, as it would mean unwanted behaviour when you upgrade the version of the consumer or in case of downtime at Workato. For me it ruins the possible usage of Kafka.

How do you deal with that? Or is this much rather a highly needed enhancement Workato should look into?

[rak-9177]

It’s very hard to debug issues if we consume in batches.