Kill Switches

Kill switch logic classes for stopping the execution of a BatchTask.

KillSwitch

Bases: ABC

Abstract base class for a kill switch.

Each invocation of the should_flip_switch method should advance the state of the kill switch and return if the kill switch should be activated. The raise_if_triggered method should raise a KillSwitchError if the kill switch has been activated.


class AnyFailedSwitch(KillSwitch):
    def should_flip_switch(self, state: State) -> bool:
        return state.is_failed() or state.is_crashed()

    def raise_if_triggered(self, state: State):
        if self.should_flip_switch(state):
            raise KillSwitchError("Failed task detected.", self)

raise_if_triggered(state) abstractmethod

Check a state and raise a KillSwitchError if the kill switch has been activated.

Raises:
  • KillSwitchError –

    If the kill switch has been activated.

should_flip_switch(state) abstractmethod

Check if this state should flip the kill switch.

Returns:
  • bool –

    True if the kill switch should be activated, False otherwise.

AnyFailedSwitch

Bases: KillSwitch

A kill switch that activates if any task fails.

raise_if_triggered(state)

Raise a KillSwitchError if the state is failed or crashed.

should_flip_switch(state)

Check if the state is failed or crashed.

CountSwitch

Bases: KillSwitch

A kill switch that activates after a certain number of tasks fail.

Parameters:
  • count (int) –

    The number of failed or crashed states that should trigger the kill switch.

raise_if_triggered(state)

Raise a KillSwitchError if the count exceeds the maximum.

should_flip_switch(state)

Increment the count if the state is failed or crashed and return if the count exceeds the maximum.

RateSwitch

Bases: KillSwitch

A kill switch that activates after the failure rate exceeds a certain threshold. Requires a minimum number of states to sample.

Parameters:
  • min_sample (int) –

    The minimum number of states to sample.

  • max_fail_rate (float) –

    The maximum frequency of failed or crashed states.

raise_if_triggered(state)

Raise a KillSwitchError if the failure rate equals or exceeds the maximum rate.

should_flip_switch(state)

Increment the count if the state is failed or crashed and return if the failure rate equals or exceeds the max rate.