Cancelling a Polly Retry policy

#.NET #C# #Polly

Cancelling a Polly Retry Policy

I really enjoy using the Polly C# library. If like me you have painful memories of trying to write code to retry requests over a patchy network connection, and then cache and expire the information you got back, then Polly is definitely worth a look. It brings some excellent features that allow you to implement highly configurable retry policies, exponential back-off, circuit breakers, caching with expiration and guaranteed cache timeout.

Microsoft have even baked it into HttpClientFactory in ASP.NET Core.

Why Cancel a Retry Policy?

I’ve been working on a WPF project that needs to send data to connected Bluetooth devices that invariably drop out of range, get switched off or run out of battery. In this instance I’ve found Polly’s WaitAndRetryAsync() retry policy to be a perfect way to handle these connections, and make sure that messages can be delivered reliably, even if the device is unavailable when the data is sent.

However there are instances where the data to be sent is time sensitive, or becomes out of date and a new message must be sent. In this case it would be useful to be able to cancel a currently running retry policy, and for this purpose Polly provides support for Cancellation Tokens. You can pass a CancellationToken into the retry policy, and either cancel the policy externally without waiting for the next retry, or from within the policy itself.

I sometimes find it a little difficult to visualise concepts like this, and it is far easier to pop the logic into a simple proof-of-concept application than try and understand while fitting it into an already complex project.

I stumbled upon an example of this scenario in GitHub issues, that for the life of me I can’t seem to find again. So for the purposes of my own “Googling it later”, and whoever else it may help, I’ve put together a simple example that:

Starts a retry policy with visible indicator of retry status
Allows the retry policy to be cancelled immediately by the user
Allows the retry policy to be cancelled from within the next retry

If you’d like to try the code, it’s also on Github.

An Example

First let’s put together a simple WPF app MainWindow with 3 buttons:

<Grid Margin="30">
        <Grid.ColumnDefinitions>
            <ColumnDefinition />
            <ColumnDefinition />
            <ColumnDefinition />
        </Grid.ColumnDefinitions>
        <Grid.RowDefinitions>
            <RowDefinition />
            <RowDefinition Height="Auto" />
        </Grid.RowDefinitions>
        <Button Content="Start Policy"
                Click="StartPolicy"
                Margin="10"/>
        <Button Content="Stop Policy Immediately"
                Click="CancelPolicyImmediately"
                Grid.Column="1"
                Margin="10"/>
        <Button Content="Stop Policy On Next Retry"
                Click="CancelOnNextRetry"
                Grid.Column="2"
                Margin="10"/>

        <TextBlock x:Name="tbStatus" 
                   Text="Idle" 
                   FontSize="16"
                   Margin="10,0"
                   Grid.Row="1" 
                   Grid.ColumnSpan="3" />
    </Grid>

Scaffold the code behind:

public partial class MainWindow : Window
{
    public MainWindow()
    {
        InitializeComponent();
    }

    private void StartPolicy(object sender, RoutedEventArgs e)
    {
        
    }

    private void CancelPolicyImmediately(object sender, RoutedEventArgs e)
    {

    }

    private void CancelOnNextRetry(object sender, RoutedEventArgs e)
    {
        
    }
}

MainWindow should now look a bit like this:

Create a Retry Policy

First things first, to simulate a real-life scenario, we need to create a Polly retry policy that we know is going to run for some time. So let’s create an awaitable task that will always error out, for example a DivideByZeroException:

private Task<int> DivideByZero()
{
    var zero = 0;
    return Task.FromResult(1 / zero);
}

Then add a WaitAndRetryForever policy to our StartPolicy() method:

private async void StartPolicy(object sender, RoutedEventArgs e)
{
    var retryPolicy = Policy
        .Handle<DivideByZeroException>()
        .WaitAndRetryForeverAsync(
            sleepDurationProvider: (retry, ts) => { return TimeSpan.FromSeconds(5); },
            onRetry: (Exception ex, int retry, TimeSpan ts, Context ctx) =>
            {
                UpdateStatus($"Retrying (reason: {ex.Message}) (retry count {retry})");
            });

    var policyResult = await retryPolicy.ExecuteAndCaptureAsync(() => DivideByZero());
    UpdateStatus($"Policy finished with Outcome type: {policyResult.Outcome}");
}

Here we create a Polly WaitAndRetryForever policy that calls our DivideByZero method and handles the Exception, retrying the call every 5 seconds. In our OnRetry handler we simply update the label in MainWindow to display the number of retry attempts made. Notice however, that onRetry is given the thrown exception as parameter, so we could log the error here and run other checks on the exception if we needed to.

We can capture the final result of the retry using the ExecuteAndCaptureAsync extension method on the Policy. If the final result was failure, this will return either the faulting exception, or the result of our wrapped method if it succeeded.

On our last line, we display the captured result of the Policy. In this example, our last line will of course never be reached, as DivideByZero will always throw, and we’ve asked Polly to retry forever.

Cancelling a Polly Retry Policy Immediately

To support cancellation of a retry policy “in flight”, we can provide a CancellationToken to the policy context and call it’s Cancel() method from wherever we want. In my example earlier, this is useful when the message being sent to a Bluetooth device has become stale or out of date and we don’t need it to be sent anymore, or the data has been updated and should be replaced.

To pass our CancellationToken to the Policy, we create a Polly Context and add the token as a named dictionary item. We can then store a reference to the CancellationTokenSource externally and call it’s Cancel method, or access the named token inside our retry policy.

At the top of the MainWindow class, add a private reference to the CancellationTokenSource:

1	private CancellationTokenSource _policyCTS;

Then update the StartPolicy() method as follows:

private async void StartPolicy(object sender, RoutedEventArgs e)
{
    var retryPolicy = Policy
        .Handle<DivideByZeroException>()
        .WaitAndRetryForeverAsync(
            sleepDurationProvider: (retry, ts) => { return TimeSpan.FromSeconds(5); },
            onRetry: (Exception ex, int retry, TimeSpan ts, Context ctx) =>
            {
                UpdateStatus($"Retrying (reason: {ex.Message}) (retry count {retry})");
            });

    _policyCTS = new CancellationTokenSource();
    var policyContext = new Context("RetryContext");

    policyContext.Add("CancellationTokenSource", _policyCTS);
    var policyResult = await retryPolicy.ExecuteAndCaptureAsync((ctx, ct) => DivideByZero(), policyContext, _policyCTS.Token);

    UpdateStatus($"Policy finished with Outcome type: {policyResult.Outcome}");
}

Notice we now instantiate the CancellationTokenSource, create a policy Context, and add the token source to it as a named key. We then overload ExecuteAndCaptureAsync with policyContext and the _policyCTS.Token cancellation token.

To see this in action, call the CancellationTokenSource.Cancel() method from within the CancelPolicyImmediately method on our MainWindow:

private void CancelPolicyImmediately(object sender, RoutedEventArgs e)
{
    _policyCTS.Cancel();
}

Pop a breakpoint at the call to UpdateStatus() within our policy start method, then run the app. Click “Start Policy”, you’ll see it retry a couple of times and print out the captured exception message.

Now click the “Stop Policy Immediately” button; you’ll see Visual Studio hit the breakpoint. This means that Polly has cancelled the retry policy and given us a nice PolicyResult instance to play with. We’ll also see the message:

Great!

Cancelling a Policy from inside the Retry

There may be occasions where we need to wait until the next retry to cancel the policy, for example if we need to check if an external condition has changed since the last retry.

For this purpose, we can use the named CancellationTokenSource we added to our Polly context.

In our example, we can simulate an external condition changing by adding a “should continue” flag to MainWindow that is checked within each retry.

At the start of MainWindow class, add

1	private bool _shouldRetry;

Then update the StartPolicy() method to set to true, so we can repeat the Policy start, then check on each retry whether the the flag is still true. If it is no longer true, then we can pull the named CancellationTokenSource from our Policy context (which Polly helpfully provides as part of the OnRetry() callback parameters) and call it’s Cancel() method.

private async void StartPolicy(object sender, RoutedEventArgs e)
{
    _shouldRetry = true;
    var retryPolicy = Policy
        .Handle<DivideByZeroException>()
        .WaitAndRetryForeverAsync(
            sleepDurationProvider: (retry, ts) => { return TimeSpan.FromSeconds(5); },
            onRetry: (Exception ex, int retry, TimeSpan ts, Context ctx) =>
            {
                UpdateStatus($"Retrying (reason: {ex.Message}) (retry count {retry})");
                if (!_shouldRetry)
                {
                    var cts = ctx["CancellationTokenSource"] as CancellationTokenSource;
                    cts.Cancel();
                }
            });

    _policyCTS = new CancellationTokenSource();
    var policyContext = new Context("RetryContext");

    policyContext.Add("CancellationTokenSource", _policyCTS);
    var policyResult = await retryPolicy.ExecuteAndCaptureAsync((ctx, ct) => DivideByZero(), policyContext, _policyCTS.Token);

    UpdateStatus($"Policy finished with Outcome type: {policyResult.Outcome}");
}

Now all we need to do is tell the Policy that it should cancel further retries when it reaches the next attempt. We can simulate a condition like this by updating our CancelOnNextRetry() click handler:

private void CancelOnNextRetry(object sender, RoutedEventArgs e)
{
    UpdateStatus("Policy cancellation requested for next retry..");
    _shouldRetry = false;
}

If we run the application now, start the policy, then click “Stop Policy on Next Retry” we’ll notice that our label briefly changes to tell us that the cancellation is requested, and then our Policy is marked as cancelled.

Conclusion

Polly really is a brilliant toolkit, one that I find myself using more and more.
Retry & polling logic is tricky (and boring) to write and debug, so if you have custom code within your projects that retries network requests, or implements long polling of any kind, do give it a try.

You can install Polly from Nuget and grab the source code from this post on Github.