Caching in the API Platform

[mppowe]

Has anyone used a caching strategy with a recipe that's running in the API Platfrom (so reachable via REST)? My recipe will be called via Javascript from a webpage so performance is key. It will be doing a simple query against Dynamics CRM and the data is pretty static, so I'd like to minimize those hits against CRM. One thought I had was to set a cache header in the HTTP response but I didn't see a way to customize that.

I tried doing a test send from Postman and passing:

Cache-Control: public, max-age=604800, immutable

But no matter what the response comes back as:

Cache-Control: max-age=0, private, must-revalidate

So then I wondered about dumping the data into a Lookup table and pulling from there until they "expire" and then pulling from CRM, maybe once per day, but since I can't control the timing (since people can hit the page anytime), there would always be a risk of timing where I might be refreshing the data in that second. I had hoped to have concurrency at 5 for performance reasons but I realize going to 1 might be a way, but I'd hate to have requests backed up behind others too much.

Curious how others have handled caching in Workato?

Thanks

[mroldanvega]

If you use a lookup table, the one to watch for is the limit of 10k rows. Not sure what your volume is, but I ran into issues with that in the past.

[ajorde]

If it's a whole list you could possibly also cache in a csv file in Box? I have a couple APIs that are hit frequently which rely on a combination of data from multiple Salesforce objects, so for those we query our data warehouse rather than Salesforce.

[gwilkinson]

For future consideration, Workato will be coming out with persistent Cloud storage, which will live beyond an individual job like Lookup tables do. It will be an enhancement or extension to Workato Collections. The row limit will be far greater than the current Collections row limit. I believe this enhancement is scheduled for sometime later this year.

[mppowe]

Gary,

Will queries be allowed against these?  For example, with Lookups I can’t do a search where a column is greater than a value.  For example, a date column is older than 3 days from now.  Will this enhancement allow more searching ability like this?

Thanks,

[gwilkinson]

Yes, I believe it should be SQL Lite like the current Collections. Perhaps someone from Workato Product team can confirm Deven Maru Ee Shan Sim