You may run into a situation where you need to deal with transient errors – errors that come and go. This is often true when you’re dealing with third-party systems and services, over the internet. It may return the expected result one minute and return a 503 status code in the next. We can try to handle this manually, by writing a loop around this but there is a more elegant approach – use the opensource Polly library. This utility gives you an elegant way to declare retry logic in your .NET applications. It’s feature-rich, providing a mechanism for simple retries but also for more sophisticated use-cases: circuit-breaker, timeout, bulkhead, rate-limiting and fallback patterns.
The Basics
You can acquire Polly via NuGet: dotnet add package Polly
With the package in place, let’s look at a simple retry example.
using Polly;
var mySimpleRetryPolicy = Policy.Handle<ApplicationException>().Retry(3);
mySimpleRetryPolicy.Execute(() =>
{
MethodWithException();
});
static string MethodWithException()
{
Console.WriteLine("MethodWithException running.");
throw new ApplicationException("Uh-oh");
}
Above, I’m creating a retry policy that states that every time an ApplicationException is encountered, retry the code that’s enclosed in the policy another 3 times. Next, I’m using that policy to execute some code, via a lambda, and within that block, I’m executing my MethodWithException method. Running the app produces the following output.
Here, you’ll see the “MethodWithException running” message printed 4 times. That method is executed once and then retried another 3 times due to the retry policy surrounding that method call. Finally, after the retries have elapsed, that exception is then bubbled up.
This is obviously a rudimentary example. However, what if the method call that you are making works without an exception in most circumstances but throws an error, now and then? You can use a simple retry policy like shown here so that it will retry that method call a few more times before giving up.
What if you want to leave a little pause between each retry? You can do so with the WaitAndRetry option.
using Polly;
var retryThriceWithTimeout = Policy.Handle<ApplicationException>().WaitAndRetry(3, x => TimeSpan.FromSeconds(2));
retryThriceWithTimeout.Execute(() =>
{
MethodWithException();
});
static string MethodWithException()
{
Console.WriteLine("MethodWithException running.");
throw new ApplicationException("Uh-oh");
}
What if you just wanted to keep retrying forever? You can do that too, like so:
using Polly;
var retryForeverWithTimeout = Policy.Handle<ApplicationException>()
.WaitAndRetryForever(x => TimeSpan.FromSeconds(2));
retryForeverWithTimeout.Execute(() =>
{
MethodWithException();
});
static string MethodWithException()
{
Console.WriteLine("MethodWithException running.");
throw new ApplicationException("Uh-oh");
}
Exponential Back-Off is a pattern that you’ll often run into, in the wild. This allows you to retry something in quick succession at first but space out the retries on each subsequent attempt. Here’s an example:
using Polly;
var exponentialBackoff = Policy.Handle<ApplicationException>()
.WaitAndRetry(5, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)));
exponentialBackoff.Execute(() =>
{
Console.WriteLine(DateTime.Now.ToLongTimeString());
MethodWithException();
});
static string MethodWithException()
{
Console.WriteLine("MethodWithException running.");
throw new ApplicationException("Uh-oh");
}
Here, I’m using the Math.Pow function to dynamically set the number of seconds Polly must wait prior to a particular retry iteration. Running the example above produces the following output:
Note that the wait duration doubles after each retry attempt.
Attempt 1: 2 ^ 1 = 2 seconds
Attempt 2: 2 ^ 2 = 4 seconds
Attempt 3: 2 ^ 3 = 8 seconds
Attempt 4: 2 ^ 4 = 16 seconds
Attempt 5: 2 ^ 5 = 32 seconds
Circuit-Breaker Pattern
This is a pattern loosely modeled after circuit-breakers of the electrical world. The circuit breaker built into the wiring of your house trips when the system is under heavy load, essentially shutting it down, to avoid an electrical fire. Similarly, this pattern allows you to break the circuit after a certain number of exceptions have occurred for a specified duration. A pattern such as this can alleviate the strain on the failing module/system by not flooding it with additional requests while it is clearly down/inoperable. Polly allows a handful of variations on this pattern with varying degrees of complexity. Let’s look at a very basic example of this.
using Polly;
var circuitBreaker = Policy
.Handle<ApplicationException>()
.CircuitBreaker(3, TimeSpan.FromSeconds(5));
var i = 1;
while (i < 51)
{
Console.WriteLine($"Attempt {i}");
i++;
try
{
circuitBreaker.Execute(() =>
{
Console.WriteLine($"Calling MethodWithException at {DateTime.Now.ToLongTimeString()}");
MethodWithException();
});
}
catch (Exception e)
{
Console.WriteLine(e.Message);
}
Thread.Sleep(TimeSpan.FromSeconds(1));
}
static string MethodWithException()
{
Console.WriteLine("MethodWithException running.");
throw new ApplicationException("Uh-oh");
}
In this example, I’m dictating that the circuit break after 3 failures and have it remain open for five seconds prior to allowing any additional invocations through. For the demonstration, I’ve created a loop of 50 iterations where I’m calling the MethodWithException, once every second. In the output below, you’ll see that although I’m calling the method 50 times, not all those invocations are passed through as the circuit breaks in between and forces a 5 second pause prior to invocations are resumed. Output looks like this:
Adding Polly Goodness to HttpClient
Do you use the built in HttpClient library for making external HTTP calls from within your .NET apps? Checkout the Microsoft.Extensions.Http.Polly package that brings Polly functionality to HttpClient without you having to set everything up manually. With this package in place, you can do things like:
using Polly;
using Polly.Extensions.Http;
var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();
builder.Services.AddHttpClient("MyHttpClient", (opts) =>
{
// base address, etc.
}).AddPolicyHandler(GetMyRetryPolicy());
app.MapGet("/", () => "Hello World!");
app.Run();
static IAsyncPolicy<HttpResponseMessage> GetMyRetryPolicy()
{
return HttpPolicyExtensions.HandleTransientHttpError()
.WaitAndRetryAsync(3, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)));
}
Above, I’m creating a named HttpClient named MyHttpClient. To this client, I’m attaching a retry policy that will retry each HTTP call up to 3 times with an exponential backoff between each call. Now, I can use this client anywhere in the app and it will execute this policy if it encounters an exceptions while making an HTTP call with it.
Closing Thoughts
Having retry logic in your applications makes it more robust and resilient in real-world scenarios where you’re at the mercy of foreign systems, services and the internet in between. Adding retries is also a good choice in microservices based systems where you’re constantly going across the wire to varying systems to service a single request. Polly gives you a good way to implement this in such a way that it doesn’t muddy up your code too much.
You can check out the code samples from this post from GitHub, here: