Sharded Redis cluster in Kubernetes

Set up

Namespace

Create the namespace “redis” which will hold all the resources related to redis cluster.

ConfigMaps

Create first ConfigMap which is used as the configurations for redis cluster. Create a file called “redis-config.yaml” and add below ConfigMap definition to it.

There are key configurations to start the nodes in “cluser-enabled” mode, see comments in yaml for details.

File name : redis-config.yaml

Apply the ConfigMap yaml file,

Next, create another ConfiMap for the user ACL, this is a list of users that need to be created along with their permissions

File name : redis-acl.yaml

Users list created by ACL configuration.

• default : default user with admin previleges. Access should be restricted to administrators. Note : This is needed for all the administrative purpose, use a strong password.
• worker : user to be used by apps or clients connect to redis cluster.
• replication : user used by replicas to AUTH and communicate with master.

User Password

The password can be configured either in plain text or SHA-256 encrypted. “#” indicates its SHA-256 hash value, “>” will be used for plain password.

More details : redis ACL

Create the ConfigMap,

StatefulSet

This is the key part, in this file we ill mount the ConfigMaps created above, and create the StatefulSet replicas based on these configurations.

File name : redis.yaml

There is a config container under “initContainers”. This will be created and terminated before the redis container starts. The sole purpose of this is to create/copy the configurations required for redis nodes.

Note: Here we are using volumeClaimTemplate, which creates PersistentVolume(PV) and PersistentVolumeClaim(PVC) dynamically for each node. See the link below for details on “How to make the Redis cluster use a pre-defined PV and PVC”

Create the StatefulSet,

Headless Service

A headless service to connect to the 6 replicas ( redis nodes) of the StatefulSet.

Filename: redis-service.yaml

create the headless service,

At this point, if all went well, there will be 6 pods running ( redis-0 to redis-5) in “redis” namespace and a headless service as below,

If you check the log of any pod, it should show printing like below, key part is where it says “Ready to accept connections”

Start Cluster

Now that all the nodes running, we need to start the cluster, This will assign 3 pods as masters and other 3 as replicas of each master,

“exec” into any running redis pod,

execute the below command to create the cluster,

This will ask for a confirmation to accept default configuration, type “yes”. this will create a sharded cluster with 3 master and 3 replicas(salves)

Sample Output,

At this point, the cluster is created, and its ready to store data. To see the cluster info, “exec” into a pod, then use “redis-cli -c”

Now, any “Cluster-aware” client can connect to redis, ( ex: jedis ).

There are two ways, a client can connect.

• It can use the IP’s of the pods by using DNS of headless service
• Or, more easy way, it can use the FQDN(Fully Qualified Domain Names) to connect. (ex:redis-n.redis.redis.svc.cluster.local)

Note: use the user “worker” (or similar), with less ACL privileges in the apps to connect and use redis.

Thanks!