Sharded Redis cluster in Kubernetes

1_lOt-UA7KVUEnYa_AEVDHPg.jpeg

This is what we are going to create here in Kubernetes,

A “redis cluster” sharded across 3 master nodes, with a replica (slave) for each master.

List of Kubernetes(K8s) Resources we need to create

1. Namespace: A namespace dedicated for “Redis” cluster and its resources.

2. ConfigMaps: Two configmaps, one with Redis cluster configuration and another for the user (Access
control List)ACL file.

3. Service: A headless service which will be used to get the cluster nodes.

4. StatefulSet: A StatefulSet with 6 replicas ( 3 masters and 3 replicas(slaves)

Set up

Namespace

Create the namespace “redis” which will hold all the resources related to redis cluster.

                kubectl create namespace redis
            

ConfigMaps

Create first ConfigMap which is used as the configurations for redis cluster. Create a file called “redis-config.yaml” and add below ConfigMap definition to it.

There are key configurations to start the nodes in “cluser-enabled” mode, see comments in yaml for details.

File name : redis-config.yaml

                apiVersion: v1
kind: ConfigMap
metadata:
  name: redis-config
  namespace: redis
data:
  redis.conf: |
  
    # important configurations
    # ------------------------
    cluster-enabled yes
    
    # configuration file used by redis to store cluster info - do not create.
    # created by redis, just give a path with a filename
    cluster-config-file /conf/nodes-6379.conf
    
    bind 0.0.0.0
    
    # configurations for snaposhot and AOF
    # read more : https://redis.io/docs/manual/persistence/
    dbfilename dump.rdb
    dir /data
    appendonly yes
    appendfilename "appendonly.aof"
    
    # Enabled ACL based auth. 
    protected-mode yes
    
    # This is used by the replics nodes to communicate with master to replicate the data.
    # we are using a user called "replication" for this, and the a strong pwd for the same is given in masterauth
    masterauth `5$!DfwSJ.Y(d:@M
    masteruser replication
    
    # this is the second ConfiMap will be mounted to. it has the list of uses needed.
    aclfile /conf/acl/users.acl
    
    # port, each redis nodes will be used
    port 6379
    # More configurations are optional, if not provided, redis will consider default values ------
    # ------ More details on configuration : https://redis.io/docs/manual/config/ ------
            

Apply the ConfigMap yaml file,

                kubectl apply -f redis-config.yaml
            

Next, create another ConfiMap for the user ACL, this is a list of users that need to be created along with their permissions

File name : redis-acl.yaml

                apiVersion: v1
kind: ConfigMap
metadata:
  name: redis-acl
  namespace: redis
data:
  users.acl: |
    # user "default" is the default user ( act as admin ) and user "replication" is used by the nodes for the replication.
    user default on #8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918 ~* &* +@all
    user replication on #65cf6f5f48186a4a6c5de02f156f1642b3873451d9de1607147799023dbf4ef8 +psync +replconf +ping
    user worker on #87eba76e7f3164534045ba922e7770fb58bbd14ad732bbf5ba6f11cc56989e6e ~* &* +@all -@dangerous
            

Users list created by ACL configuration.

  • default : default user with admin previleges. Access should be restricted to administrators. Note : This is needed for all the administrative purpose, use a strong password.
  • worker : user to be used by apps or clients connect to redis cluster.
  • replication : user used by replicas to AUTH and communicate with master.

User Password

The password can be configured either in plain text or SHA-256 encrypted. “#” indicates its SHA-256 hash value, “>” will be used for plain password.

More details : redis ACL

Create the ConfigMap,

                kubectl apply -f redis_acl.yaml
            

StatefulSet

This is the key part, in this file we ill mount the ConfigMaps created above, and create the StatefulSet replicas based on these configurations.

File name : redis.yaml

                apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis
  namespace: redis
spec:
  serviceName: redis
  replicas: 6   # 6 replicas, 3 master and 3 replicas(slaves)
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      initContainers:
      - name: config
        image: redis:7.0.0-alpine
        command: [ "sh", "-c" ]
        args:
          - |
            if [ -f "/conf/redis.conf" ]; then
              echo "config exists /conf/redis.conf .. not creating new"
            else
              echo "config doesn't exist copying to /conf/redis.conf"
              cp /tmp/conf/redis.conf /conf/redis.conf
            fi
        volumeMounts:
        - name: storage
          subPath: conf
          mountPath: /conf
        - name: config
          mountPath: /tmp/conf/
      containers:
      - name: redis
        image: redis:7.0.0-alpine
        command: ["redis-server"]
        args: ["/conf/redis.conf"]
        resources:
          requests:
            memory: "100M"
          limits:
            memory: "2000M"
        ports:
        - containerPort: 6379
          name: redis
        volumeMounts:
        - name: storage
          mountPath: /data
          subPath: data
        - name: storage
          mountPath: /conf
          subPath: conf
        - name: config-acl
          mountPath: /conf/acl/
      volumes:
      - name: config
        configMap:
          name: redis-config
      - name: config-acl
        configMap:
          name: redis-acl
  volumeClaimTemplates:
  - metadata:
      name: storage
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 50Mi
            

There is a config container under “initContainers”. This will be created and terminated before the redis container starts. The sole purpose of this is to create/copy the configurations required for redis nodes.

Note: Here we are using volumeClaimTemplate, which creates PersistentVolume(PV) and PersistentVolumeClaim(PVC) dynamically for each node. See the link below for details on “How to make the Redis cluster use a pre-defined PV and PVC

Create the StatefulSet,

                kubectl apply -f redis.yaml
            

Headless Service

A headless service to connect to the 6 replicas ( redis nodes) of the StatefulSet.

Filename: redis-service.yaml

                apiVersion: v1
kind: Service
metadata:
  name: redis
  namespace: redis
spec:
  clusterIP: None # "None" make it a headless service. No cluster IP.
  ports:
  - port: 6379
    targetPort: 6379
    name: redis
  selector:
    app: redis
            

create the headless service,

                kubectl apply -f redis-service.yaml
            

At this point, if all went well, there will be 6 pods running ( redis-0 to redis-5) in “redis” namespace and a headless service as below,

                > kubectl get all -n redis
NAME          READY   STATUS    RESTARTS   AGE     IP             
pod/redis-0   1/1     Running   0          3m32s   192.168.70.9   
pod/redis-1   1/1     Running   0          3m13s   192.168.75.40  
pod/redis-2   1/1     Running   0          9s      192.168.67.74  
pod/redis-3   1/1     Running   0          3m6s    192.168.71.78  
pod/redis-4   1/1     Running   0          3m3s    192.168.72.149 
pod/redis-5   1/1     Running   0          2m59s   192.168.75.105
NAME            TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE     SELECTOR
service/redis   ClusterIP   None         <none>        6379/TCP   3m32s   app=redis
NAME                     READY   AGE     CONTAINERS   IMAGES
statefulset.apps/redis   6/6     3m33s   redis        redis:7.0.0-alpine
            

If you check the log of any pod, it should show printing like below, key part is where it says “Ready to accept connections”

                > kubectl logs -n redis pod/redis-0

1:C 05 May 2022 23:28:54.937 # oO0OoO0OoO0Oo Redis is starting 
oO0OoO0OoO0Oo

1:C 05 May 2022 23:28:54.937 # Redis version=7.0.0, bits=64, 
commit=00000000, modified=0, pid=1, just started

1:C 05 May 2022 23:28:54.937 # Configuration loaded

1:M 05 May 2022 23:28:54.937 * monotonic clock: POSIX clock_gettime

1:M 05 May 2022 23:28:54.939 * Node configuration loaded, I'm 
92dfcf6b6008822e98b032761768a981217b9883

1:M 05 May 2022 23:28:54.939 * Running mode=cluster, port=6379.

1:M 05 May 2022 23:28:54.939 # Server initialized

1:M 05 May 2022 23:28:54.942 * The AOF directory appendonlydir doesn't 
exist

1:M 05 May 2022 23:28:54.946 * SYNC append only file rewrite performed

1:M 05 May 2022 23:28:54.948 * Ready to accept connections
            

Start Cluster

Now that all the nodes running, we need to start the cluster, This will assign 3 pods as masters and other 3 as replicas of each master,

“exec” into any running redis pod,

                kubectl exec -it -n redis pod/redis-0 -- sh
            

execute the below command to create the cluster,

                # Note : "-a admin" in the command is the "default" user password 
given in redis-acl.yaml (raw form, not SHA-256 hash value)
# Imp : use a strong password.

/data # redis-cli --cluster create redis-
0.redis.redis.svc.cluster.local:6379 redis-
1.redis.redis.svc.cluster.local:6379 redis-
2.redis.redis.svc.cluster.local:6379 redis-
3.redis.redis.svc.cluster.local:6379 redis-
4.redis.redis.svc.cluster.local:6379 redis-
5.redis.redis.svc.cluster.local:6379 -a admin --cluster-replicas 1
            

This will ask for a confirmation to accept default configuration, type “yes”. this will create a sharded cluster with 3 master and 3 replicas(salves)

Sample Output,

                /data # redis-cli --cluster create redis-
0.redis.redis.svc.cluster.local:6379 redis-
1.redis.redis.svc.cluster.local:6379 redis-
2.redis.redis.svc.cluster.local:6379 redis-
3.redis.redis.svc.cluster.local:6379 redis-4

.redis.redis.svc.cluster.local:6379 redis-
5.redis.redis.svc.cluster.local:6379 -a admin --cluster-replicas 1

Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.

>>> Performing hash slots allocation on 6 nodes...

Master[0] -> Slots 0 - 5460

Master[1] -> Slots 5461 - 10922

Master[2] -> Slots 10923 - 16383

Adding replica redis-4.redis.redis.svc.cluster.local:6379 to redis-
0.redis.redis.svc.cluster.local:6379

Adding replica redis-5.redis.redis.svc.cluster.local:6379 to redis-
1.redis.redis.svc.cluster.local:6379

Adding replica redis-3.redis.redis.svc.cluster.local:6379 to redis-
2.redis.redis.svc.cluster.local:6379

M: 2e2ecf247ee64fabec434a3cbec12bb211824ce4 redis-
0.redis.redis.svc.cluster.local:6379

slots:[0-5460] (5461 slots) master

M: 4bf5dbf6146eff3355c796883338b9c1c91ba5f2 redis-
1.redis.redis.svc.cluster.local:6379

slots:[5461-10922] (5462 slots) master

M: 06bd7df09c25f9d4ea433c3b69293cd9adbed304 redis-
2.redis.redis.svc.cluster.local:6379

slots:[10923-16383] (5461 slots) master

S: 8abb9fda1a2659851f22b298b813d91b14851a8f redis-
3.redis.redis.svc.cluster.local:6379


replicates 06bd7df09c25f9d4ea433c3b69293cd9adbed304

S: 19cf9a0a04d7bd5182c306b94cf8f589d394577d redis-
4.redis.redis.svc.cluster.local:6379

replicates 2e2ecf247ee64fabec434a3cbec12bb211824ce4

S: c3eda2d867f5dd2f0781e6bb48627ca40b6066c7 redis-
5.redis.redis.svc.cluster.local:6379

replicates 4bf5dbf6146eff3355c796883338b9c1c91ba5f2

Can I set the above configuration? (type 'yes' to accept): yes

>>> Nodes configuration updated

>>> Assign a different config epoch to each node

>>> Sending CLUSTER MEET messages to join the cluster

Waiting for the cluster to join
.
>>> Performing Cluster Check (using node redis-
0.redis.redis.svc.cluster.local:6379)

M: 2e2ecf247ee64fabec434a3cbec12bb211824ce4 redis-
0.redis.redis.svc.cluster.local:6379

slots:[0-5460] (5461 slots) master

1 additional replica(s)

S: c3eda2d867f5dd2f0781e6bb48627ca40b6066c7 192.168.46.242:6379

slots: (0 slots) slave

replicates 4bf5dbf6146eff3355c796883338b9c1c91ba5f2

M: 06bd7df09c25f9d4ea433c3b69293cd9adbed304 192.168.48.65:6379

slots:[10923-16383] (5461 slots) master

1 additional replica(s)

M: 4bf5dbf6146eff3355c796883338b9c1c91ba5f2 192.168.57.250:6379

slots:[5461-10922] (5462 slots) master

1 additional replica(s)

S: 8abb9fda1a2659851f22b298b813d91b14851a8f 192.168.60.185:6379

slots: (0 slots) slave

replicates 06bd7df09c25f9d4ea433c3b69293cd9adbed304

S: 19cf9a0a04d7bd5182c306b94cf8f589d394577d 192.168.46.128:6379

slots: (0 slots) slave

replicates 2e2ecf247ee64fabec434a3cbec12bb211824ce4

[OK] All nodes agree about slots configuration.

>>> Check for open slots...

>>> Check slots coverage...

[OK] All 16384 slots covered.
            

At this point, the cluster is created, and its ready to store data. To see the cluster info, “exec” into a pod, then use “redis-cli -c”

                # Note : "-c" tells redis-cli that it is connecting to a cluster. "-a 
admin" is the default user password. 

/data # redis-cli -c -a admin

Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.

127.0.0.1:6379> CLUSTER NODES

17fbef584bbbeac241bd33aa8e9d03b35ab4c94d 192.168.57.250:6379@16379 
myself,master - 0 1651805328000 1 connected 0-5460

2690fabc3b37462a08a44148a000efb000d9841c 192.168.46.242:6379@16379 
master - 0 1651805330000 3 connected 10923-16383

e68bab90451817f0008bb8ec4c9dd10b849565e8 192.168.46.128:6379@16379 
master - 0 1651805330589 2 connected 5461-10922

6982044e1596b188d56c367d16df46d000275e62 192.168.33.147:6379@16379 
slave e68bab90451817f0008bb8ec4c9dd10b849565e8 0 1651805329000 2 connected

e893d2421ad3c6c1d35966b1dee9389352901363 192.168.48.65:6379@16379 
slave 2690fabc3b37462a08a44148a000efb000d9841c 0 1651805330990 3 connected

83370d400ad962ae28d202288bada3c16aa4dc79 192.168.47.226:6379@16379 
slave 17fbef584bbbeac241bd33aa8e9d03b35ab4c94d 0 1651805329985 1 connected
            

Now, any “Cluster-aware” client can connect to redis, ( ex: jedis ).

There are two ways, a client can connect.

  • It can use the IP’s of the pods by using DNS of headless service
  • Or, more easy way, it can use the FQDN(Fully Qualified Domain Names) to connect. (ex:redis-n.redis.redis.svc.cluster.local)

Note: use the user “worker” (or similar), with less ACL privileges in the apps to connect and use redis.

Thanks!


Pawfives by
Pawfives by:

Only registered users can post comments. Please, login or signup.

Start blogging about your favorite technologies and get more readers

Join other developers and claim your FAUN account now!

Avatar

Jithin Scaria

Enterprise Architect

@jithinscaria
Enterprise Architect — Kubernetes(K8s) | Full-stack Engineer | Data Engineer | Development — Testing — Integration — Deployment | Cloud | Project Management
Stats
17

Influence

503

Total Hits

1

Posts