Twemproxy is an essential tool for managing sharded Memcached deployments, but its configuration and behavior can be surprisingly subtle.
Let’s see Twemproxy in action. Imagine you have three Memcached instances running on 10.0.1.10:11211, 10.0.1.11:11211, and 10.0.1.12:11211. You want Twemproxy to manage these, distributing keys across them.
Here’s a minimal twemproxy.yml configuration:
beta: 1
listen: 10.0.0.5:22222
admin:
listen: 10.0.0.5:22223
item_limit: 10000
key_limit: 20000
servers:
- 10.0.1.10:11211
- 10.0.1.11:11211
- 10.0.1.12:11211
With this running, your application would connect to 10.0.0.5:22222 instead of directly to the Memcached servers. Twemproxy handles the sharding.
This setup solves a fundamental problem: as your cache needs grow, you can’t just keep adding more memory to a single Memcached instance. Eventually, you hit limits on single-server performance and capacity. Sharding splits your data across multiple Memcached instances, allowing for horizontal scaling. Twemproxy acts as the intelligent intermediary, abstracting away the complexity of which server holds which key.
Internally, Twemproxy uses consistent hashing to distribute keys. When a key arrives, Twemproxy calculates its hash and maps that hash to one of the configured servers. The beta parameter in the configuration influences the distribution algorithm, with beta: 1 being a common default that provides a good balance.
The listen directive specifies the address and port Twemproxy itself listens on for client connections. The admin section provides a separate port for monitoring Twemproxy’s internal state, including statistics about requests, cache hits/misses, and server health. item_limit and key_limit are crucial for preventing the admin interface from being overwhelmed by too much data.
Your application code interacts with Twemproxy using the standard Memcached protocol. For example, to set a key mykey with value myvalue and an expiration of 3600 seconds, your client would send:
set mykey 0 3600 7
myvalue
to 10.0.0.5:22222. Twemproxy then computes the hash for mykey, determines which Memcached server should hold it, and forwards the set command to that specific Memcached instance.
The servers list is where you define the backend Memcached instances. When Twemproxy starts or when the configuration is reloaded, it establishes connections to these servers. It continuously monitors their health; if a server becomes unresponsive, Twemproxy will temporarily stop sending traffic to it, effectively removing it from the pool until it recovers.
One of the most powerful, yet often overlooked, aspects of Twemproxy is its ability to rebalance shards without downtime. If you add a new Memcached server to the servers list, Twemproxy doesn’t immediately redistribute all existing keys. Instead, it gradually shifts keys that would newly hash to the new server over to it. This means you can scale your cache cluster up or down dynamically.
When you’re using Twemproxy, understanding the hash command is key to debugging. If you send hash mykey to the Twemproxy listen port (e.g., 10.0.0.5:22222), Twemproxy will respond with the hash value for mykey. This allows you to manually verify which server a given key should be routed to, based on the current consistent hash ring.
The admin interface is invaluable. Accessing http://10.0.0.5:22223/ will show you statistics for each server pool. You can see get_hits, get_misses, delete_hits, evictions, and importantly, server_ejected_connections. High numbers of server_ejected_connections often indicate network issues or overloaded backend Memcached servers.
The real magic of Twemproxy is its ability to manage multiple Memcached pools within a single configuration. You can define different listen ports, each pointing to a distinct set of servers and using potentially different hashing algorithms or beta values. This allows for sophisticated partitioning of your cache data based on application needs.
The timeout setting in the twemproxy.yml is not for client connection timeouts, but for how long Twemproxy will wait for a response from a backend Memcached server before considering it failed. This is a critical parameter for tuning resilience.
If you’re using Twemproxy with older versions of Memcached or specific client libraries, you might encounter issues related to multiget commands. Twemproxy generally handles these well, but it’s worth knowing that some client implementations might send multiget requests in ways that Twemproxy needs to reassemble and forward efficiently.
The next step in mastering Twemproxy is understanding how to configure multiple server pools for different use cases or data partitions.