Repository navigation
EmrStepSensor to auto detect stepid's #11877
Description
Activity
Thanks for opening your first issue here! Be sure to follow the issue template!
- added a commit that references this issue
on Oct 27, 2020 - changed the title
[-]Mr Apache Airflow[/-][+]EmrStepSensor to auto detect stepid's[/+]on Oct 27, 2020 - locked and limited conversation to collaborators
on Sep 30, 2022
Description
Currently the EmrStepSensor only looks at one step at a time. Enhancing the EmrStepSensor to retrieve all steps on a specified cluster if no stepId was provided.
Use case / motivation
If no step_id was provided the EmrStepSensor wil retrieve all steps on a specified cluster.
This allows you to retrieve all steps and their states with a singe api call.
What happened
We ran into a throttle limit.
botocore.exceptions.ClientError: An error occurred (ThrottlingException) when calling the DescribeStep operation (reached max retries: 4): Rate exceeded
With the enhancement of the EmrStepSensor that is taken care of and also then the EmrStepSensor is more dynamic.