[proposal] provide DbtCloudRunJobOperator with an option re-poking the api endpoint if it's a http error #49424
Unanswered
keerthi-reddiwandla
asked this question in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
We host a dbt jobs DAG in MWAA using DbtCloudRunJobOperator. Sometimes the DbtCloudRunJobOperator get 404 error from the dbt cloud api endpoint even though it has triggered the dbt cloud job run.The task is marked as FAILED and it moves on to next task. It would be helpful to add options after providing following arguments:
1.retry_attempts: if the error is http error, you can provide certain number of retry attempts.
2.retry_after_seconds: Initiate the retry after certain number of seconds.
I can look into opening a PR if you can guide me in a general direction of making this change.
Beta Was this translation helpful? Give feedback.
All reactions