topic Re: Backing up Job History in Workato Pros Discussion Board

Backing up Job History

jblanchett — Fri, 16 Apr 2021 10:14:20 GMT

What are folks using to back up their workato job history? We aren't currently backing up our job history, but need to start doing so. I would imagine tools Datadog, Snowflake, etc would be good for this. What's everyone using? Do you like it? Let me know!

Re: Backing up Job History

deven-maru — Fri, 16 Apr 2021 10:50:18 GMT

Hi Joe, you can use the log streaming feature to backup the job history to external source like S3 or any other log service provider like Datadog, Splunk, Sumologic etc. The log streaming feature supports any cloud based logging service that has an HTTP based ingestion endpoint. Here in this example I've configured job history and user activity to stream to Datadog.

Re: Backing up Job History

jblanchett — Fri, 16 Apr 2021 12:11:28 GMT

Thanks Deven! I'm aware of the event streaming feature, just curious where folks are streaming to. I thought Datadog might be a popular option.

Re: Backing up Job History

kkrivonos — Fri, 16 Apr 2021 15:27:36 GMT

Hi, Joe!

There is also an alternative way for this, such as putting the assembled log payload to DWH. It can be done using callable recipe from any recipe you want with the payload you want to put in it and table name as parameters.

This seems more complicated way but if you already have a DWH, you don't need other tools for this

Re: Backing up Job History

deepak23shastri — Fri, 16 Apr 2021 19:59:25 GMT

Hey Joe, we use Splunk! its amazing and have setup alerts and dashboards too using built-in Splunk feature offered. Its easy setup and intutive too!

Re: Backing up Job History

chris-backus — Fri, 16 Apr 2021 20:42:56 GMT

I use Datadog, but I'm biased because I work there.

Re: Backing up Job History

deven-maru — Fri, 16 Apr 2021 22:41:29 GMT

Hi Deepak,

If you're able to share, can you give example of the type of alerts and dashboard you've set up for monitoring?

Thanks

Re: Backing up Job History

deepak23shastri — Sun, 18 Apr 2021 00:36:48 GMT

Hey Deven,

The thought behind creating the dashboard and alerts in Splunk was to enable non-Workato users to have real time view and aid the Ops to ensure smooth process across teams. There are other applications of these logs consolidation to determine the usage of Bot commands and consumers of the Bots, effectiveness etc…(This is work in progress 🙂).

The dashboard consists of:

A date range pick list that can used to fetch data for various time range and defaults to last 15 minutes when launched.

A universal search section that will allow you to search the details specific to Folders. Sample below:

Health section that show Success v/s Failure count for each business team/Project:

Health section that show Success v/s Failure count for each folder. The Green and Red bars are clickable and will redirect you to a detailed search page within Splunk.

Alert is triggered with consolidated report of all the failures noticed in previous 24 hours.

I hope this helps!!

Regards,

Deepak

Re: Backing up Job History

deven-maru — Sun, 18 Apr 2021 07:18:37 GMT

This is super helpful Deepak. Thanks so much for providing the details. Really appreciate it.

Best,

Deven