We are using Pagerduty notification to alert Tidal job failures. Once the Tidal job failed, right now the job will rerun multiple times and probably completed successfully, and we want to mark the alerts/notifications as resolved on Pagerduty to avoid manually re-check Tidal jobs.
The steps might be as below:
Scan pager duty incidents, for each incident:
- get tidal job name
- check if the job already succeeded for the indident date
- resolve the incident if the job succeeded
I’m wondering how should we enable this functionality? Or what’s the solution for this?
Can you please give me some instructions?