A circuit breaker doesn’t just prevent cascading failures; it actively rewards services for being unhealthy.
Let’s say you have ServiceA that calls ServiceB. If ServiceB starts failing, ServiceA should stop calling it. That’s the basic idea. But a good circuit breaker does more. It doesn’t just stop calls; it also gives ServiceB a chance to recover without being hammered by requests it can’t handle, which would only make things worse.
Here’s a common Go implementation using the go-circuitbreaker library.
package main
import (
"fmt"
"log"
"net/http"
"time"
"github.com/sony/gobreaker"
)
// This is a mock function that simulates calling an external service.
// It will randomly fail to demonstrate the circuit breaker's behavior.
func callExternalService() (string, error) {
// Simulate network latency
time.Sleep(50 * time.Millisecond)
// Simulate failure 70% of the time
if time.Now().UnixNano()%10 < 7 {
return "", fmt.Errorf("external service is unavailable")
}
return "Success from external service", nil
}
func main() {
// Configure the circuit breaker
// ReadyToTrip: Threshold for consecutive errors to open the circuit.
// Timeout: How long the circuit stays open before transitioning to Half-Open.
// In Half-Open, it allows a few requests to test the service.
// If they succeed, the circuit closes. If they fail, it opens again.
// MaxErrors: The maximum number of errors allowed in a window before tripping.
// OnStateChange: A callback function that executes when the circuit breaker's state changes.
settings := gobreaker.Settings{
Name: "externalService",
ReadyToTrip: func(counts gobreaker.Counts) bool {
// Trip if 10 consecutive requests fail.
return counts.ConsecutiveFailures >= 10
},
OnStateChange: func(from, to gobreaker.State) {
log.Printf("Circuit breaker state changed from %s to %s", from, to)
},
Timeout: 1 * time.Minute, // After 1 minute, try again (Half-Open state)
}
cb := gobreaker.NewCircuitBreaker(settings)
// Simulate making requests over time
for i := 0; i < 30; i++ {
// Wrap the actual service call with the circuit breaker
result, err := cb.Execute(func() (interface{}, error) {
return callExternalService()
})
if err != nil {
// This error can be from the circuit breaker (e.g., "circuit breaker is open")
// or from the actual service call if the breaker is in a closed or half-open state.
log.Printf("Request %d: Failed: %v", i, err)
} else {
log.Printf("Request %d: Succeeded: %s", i, result)
}
time.Sleep(500 * time.Millisecond) // Wait a bit between requests
}
}
When callExternalService starts failing consistently, the cb.Execute call will eventually return an error without even attempting to call the underlying callExternalService function. This is the "open" state. After the Timeout duration (1 minute in this example), the breaker enters a "half-open" state. During this state, a single request is allowed to pass through. If that request succeeds, the breaker closes. If it fails, it immediately re-opens.
The most surprising thing about circuit breakers is that they don’t just protect the caller; they actively protect the callee from being overwhelmed when it’s already struggling. By preventing a flood of requests, a failing service gets a breathing room to recover. Without this, a minor glitch could cascade into a complete outage for both services.
What most people don’t realize is how the ReadyToTrip function is your primary dial for how quickly you want to stop calling a failing service. Setting counts.ConsecutiveFailures >= 10 means you’re willing to tolerate up to nine failures before slamming the door shut. Some systems might want to trip on counts.TotalFailures >= 50 within a certain time window, or a counts.FailureRatio >= 0.5. The gobreaker library focuses on consecutive failures by default, which is often a good starting point.
The next logical step is to integrate this with a distributed tracing system to visualize when your circuit breakers are opening and closing across multiple services.