What is common among the mighty water dam, a humble electric fuse, and i-dunno-what, hysterix and hysterix-go? Take a moment to identify the similarities.
Yes. All of them are checks that prevent things from going out of control, by deploying a safeguarding measure if a certain threshold is crossed.
What is circuit breaking?
While reading about circuit breakers, I stumbled upon a book called Release It! by Michael T. Nygard. It has a brilliant introduction to the subject which I will reproduce below:
“Not too long ago, when electrical wiring was first being built into houses, many people fell victim to physics. The unfortunates would plug too many appliances into their circuit. Each appliance drew a certain amount of current. When current is resisted, it produces heat proportional to the square of the current times the resistance (I2R).
Because they lacked superconducting home wiring, this hidden coupling between electronic gizmos made the wires in the walls get hot, sometimes hot enough to catch fire. Pfft. No more house.
The fledgling energy industry found a partial solution to the problem of resistive heating, in the form of fuses. The entire purpose of an electrical fuse is to burn up before the house does. It is a component designed to fail first, thereby controlling the overall failure mode. This brilliant device worked well, except for two flaws. First, a fuse is a disposable, one-time use item; therefore, it is possible to run out of them. Second, residential fuses (in the United States) were about the same diameter as copper pennies. Together, these two flaws led many people to conduct experiments with homemade, high-current, low-resistance fuses (that is, a 3/4-inch disk of copper). Pfft. No more house.
Residential fuses have gone the way of the rotary dial telephone. Now, circuit breakers protect overeager gadget hounds from burning their houses down. The principle is the same: detect excess usage, fail first, and open the circuit. More abstractly, the circuit breaker exists to allow one subsystem (an electrical circuit) to fail (excessive current draw, possibly from a short-circuit) without destroying the entire system (the house). Furthermore, once the danger has passed, the circuit breaker can be reset to restore full function to the system.
You can apply the same technique to software by wrapping dangerous operations with a component that can circumvent calls when the system is not healthy. This differs from retries, in that circuit breakers exist to prevent operations rather than re-execute them.”
Where do you break the circuit?
Any method where you don’t or can’t control unexpected behaviors.
Imagine a call from your service to an external service. This can be an HTTP call, a gRPC call or a TCP call. You don’t always know what the other service will return. It can throw an error that was not part of the contract. It might be down, resulting in a connection timeout. The other service’s database might be down which might result in a read timeout.
Now, just like the fuse sacrifices itself and saves the house from burning, you would want to avoid cascading failures with compounding effects (software engineering, an equivalent of the burning house).
How do you break the circuit?
It’s quite simple.
Enclose your external call (or any call prone to failure) in an object. The object should have metrics that monitor success and failure.
When the failure goes beyond a certain threshold, the object should stop sending that external call and instead fall back to some other option (a default return or run in degraded mode, etc.). What you do in fallback, or as they say, when the circuit is “open”, depends on the type of business that your system is supporting.
If your system is in a social media company that recommends friends intelligently, you can fall back to the default option of showing a randomly selected set of profiles as suggestions rather than showing “Something went wrong”. You can also consider showing the loading screen to the user, which, as you would agree, is a bad user experience.
On the other hand, if your system accepts payments and your payment gateway is down, you should return a static message – “Payment gateway is down. Try back again in a few minutes.”
Once a circuit is open, your circuit breaker can wait for some time (wait time), and after that, it can check if the external service is up or not. In this state which is called the “half close” state, the circuit breaker makes a trial call. If the trial call succeeds, the circuit breaker returns to the “close” state, else it continues to be in the open state.
Can I make my own circuit breaker?
Yes.
But there are many circuit breaker frameworks available on GitHub. One popular example is Resilence4J in Java and hysterix-go in Golang.
I tried to implement it by creating an abstract class that contains the logic for circuit breaking. The class has an abstract method called start. The external call has to be implemented as an implementation of the “start” method.
In the example given below, I will try to demonstrate opening a circuit based on a sliding window of the last 10 calls.
Here’s an example of a circuit breaker class.
package com.company.my.circuitbreaker;
public abstract class MyCircuitBreakerImpl<T, K> {
private int total;
private int failure;
private double threshold;
private int WINDOW_SIZE = 10;
private boolean isCircuitOpen = false;
MyCircuitBreakerImpl(double threshold){
this.threshold = threshold;
}
K run(T request){
isCircuitOpen();
if(isCircuitOpen){
System.out.println("Circuit Open. So Returning default response");
return null;
}
try {
K response = start(request);
return response;
} catch (Exception e){
System.out.println("Exception occured. Increasing failure count");
failure = failure + 1;
}
return null;
}
private void isCircuitOpen() {
total = total + 1;
if(total == 10){
double failurePercent = failure / total * 100;
if(failurePercent > threshold){
System.out.println("Circuit Open");
isCircuitOpen = true;
}
total = 0;
failure = 0;
}
}
abstract K start(T request);
}
We will now implement the start method.
package com.company.my.circuitbreaker;
public class MyCircuitBreakerFailureImpl extends MyCircuitBreakerImpl<Integer, String> {
MyCircuitBreakerFailureImpl(double threshold) {
super(threshold);
}
@Override
String start(Integer request) {
if(request < 10) {
System.out.println("****** I am success request ******");
return "pong";
}
try {
Thread.sleep(1000);
System.out.println(" ***** I am failed method. I am called here. ********");
} catch (InterruptedException e) {
e.printStackTrace();
}
throw new RuntimeException("Ooops! Something went wrong");
}
}
Running this through the main method.
package com.company.my.circuitbreaker;
public class TestMain {
public static double THRESHOLD = 10;
public static void main(String[] args){
MyCircuitBreakerFailureImpl myCircuitBreakerFailure = new MyCircuitBreakerFailureImpl(THRESHOLD);
for(int i = 1; i < 100; i++){
myCircuitBreakerFailure.run(i);
}
}
}
Output of the code.
****** I am success request ******
****** I am success request ******
****** I am success request ******
****** I am success request ******
****** I am success request ******
****** I am success request ******
****** I am success request ******
****** I am success request ******
****** I am success request ******
***** I am failed method. I am called here. ********
Exception occured. Increasing failure count
***** I am failed method. I am called here. ********
Exception occured. Increasing failure count
***** I am failed method. I am called here. ********
Exception occured. Increasing failure count
***** I am failed method. I am called here. ********
Exception occured. Increasing failure count
***** I am failed method. I am called here. ********
Exception occured. Increasing failure count
***** I am failed method. I am called here. ********
Exception occured. Increasing failure count
***** I am failed method. I am called here. ********
Exception occured. Increasing failure count
***** I am failed method. I am called here. ********
Exception occured. Increasing failure count
***** I am failed method. I am called here. ********
Exception occured. Increasing failure count
***** I am failed method. I am called here. ********
Exception occured. Increasing failure count
Circuit Open
Circuit Open. So Returning default response
Circuit Open. So Returning default response
Circuit Open. So Returning default response
Circuit Open. So Returning default response
Circuit Open. So Returning default response
Circuit Open. So Returning default response
How to implement a circuit breaker in Go?
Check out the hystrix-go library here by KevinPike. There is a sample implementation demonstrating the use here.
How to implement a circuit breaker in Java?
Check out the resilence4j library here. Here’s the sample implementation demonstrating the use here.