Retrying Transient Exceptions

Exception handling is not always a straightforward implementation paradigm. At times you are faced with decisions such as to catch or not to as well as is it foolish to retry with same strategy and expect a different outcome.

Dealing with mobile network connections (especially cellular network connections – no matter what protocol the mobile interface suggests) is always a shady business. Even though everything is setup properly, sometimes the results are not as expected and you might endup with timeout expcetions, dns errors as well as server side errors (e.g. 400, 502 etc.)

Recently I was dealing a server side implementation on the new Azure Service Fabric platform. The biggest problem we were having was dealing with the realiable collections timing out because of the deadlock safeguards (i.e. 4sec timeout). Especially with replica failures and/or other failover/recovery type of situations these timeouts were unavoidable.

One option we analyzed was to use the Transient Error block implementation of Enterprise Library and Microsoft Patterns and Practices team ( Transient Fault Handling ). However this seemed like an overkill for a simple task such as this.

Instead, I decided go with a simpler extension method implementation.

public static async Task<T> WithRetryAsync<T>(this Func<Task<T>> executionTask,
    [Optional, DefaultParameterValue(3)] int retryCount)
{
    var trials = 0;

    Retry:

    try
    {
        var results = await executionTask();

        return results;
    }
    catch (Exception)
    {
        if (++ trials > retryCount) throw;

        goto Retry;
    }
}

In this implementation, we are using a Func to carry over our asynchronous delegate and try to execute it as long as the retry count allows us.

And the call to this function would look simmilar to:

// Using method group
Func<Task<int>> myFunction = CalculateAsync;
// Or using lamdba
//Func<Task<int>> myFunction = () => CalculateAsync();

await myFunction.WithRetryAsync(4);

Cancellation

But what if the retries are taking way too long and we want to cancel the task at hand. For this scenario, we can of course resort to CancellationToken. Expanding our method implementation a little bit:

public static async Task<T> WithRetryAsync<T>(this Func<Task<T>> executionTask,
    [Optional, DefaultParameterValue(3)] int retryCount,
    [Optional] CancellationToken cancellationToken)
{
// commented out for brevity
    }
    catch (Exception)
    {
        if (++trials > retryCount || cancellationToken.IsCancellationRequested) throw;

// ...
    }
}

So not if the cancellation token is signalled, the process is cancelled after the last trial and the original exception is thrown.

Progress

You are probably wondering what does progress have to do with a failing task. Well in some scenarios the number of retries is an indication of application or system health and can be part of telemetry data. The easiest way to incorporate the retry events into this execution was using the IProgress interface.

Again extending our method signature with another optional parameter (on a side note the use of Optional attribute can be exchanged with parameters with default values, the choice here was simply syntactic):

public static async Task<T> WithRetryAsync<T>(this Func<Task<T>> executionTask,
    [Optional, DefaultParameterValue(3)] int retryCount,
    [Optional] CancellationToken cancellationToken,
    [Optional] IProgress<int> retryCallback)

and the catch block would become simmilar to (using the new safe navigation operator of C#):

catch (Exception)
{
    if (++trials > retryCount || cancellationToken.IsCancellationRequested) throw;

    retryCallback?.Report(trials);

    goto Retry;
}

Now everytime a retry occurs the Progress callback is executed notifying the caller about the retry.

Transient Errors

This implementation can be further improved by implementing a similar policy check implementation of Transient Fault Handling block for evaluating the exception to be transient or not.

Happy coding everyone,

Advertisements