This is what we are going to create here in Kubernetes,
A “redis cluster” sharded across 3 master nodes, with a replica (slave) for each master.
List of Kubernetes(K8s) Resources we need to create
1. Namespace: A namespace dedicated for “Redis” cluster and its resources.
2. ConfigMaps: Two configmaps, one with Redis cluster configuration and another for the user (Access
control List)ACL file.
3. Service: A headless service which will be used to get the cluster nodes.
4. StatefulSet: A StatefulSet with 6 replicas ( 3 masters and 3 replicas(slaves)
Set up
Namespace
Create the namespace “redis” which will hold all the resources related to redis cluster.
kubectl create namespace redis
ConfigMaps
Create first ConfigMap which is used as the configurations for redis cluster. Create a file called “redis-config.yaml” and add below ConfigMap definition to it.
There are key configurations to start the nodes in “cluser-enabled” mode, see comments in yaml for details.
File name : redis-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: redis-config
namespace: redis
data:
redis.conf: |
# important configurations
# ------------------------
cluster-enabled yes
# configuration file used by redis to store cluster info - do not create.
# created by redis, just give a path with a filename
cluster-config-file /conf/nodes-6379.conf
bind 0.0.0.0
# configurations for snaposhot and AOF
# read more : https://redis.io/docs/manual/persistence/
dbfilename dump.rdb
dir /data
appendonly yes
appendfilename "appendonly.aof"
# Enabled ACL based auth.
protected-mode yes
# This is used by the replics nodes to communicate with master to replicate the data.
# we are using a user called "replication" for this, and the a strong pwd for the same is given in masterauth
masterauth `5$!DfwSJ.Y(d:@M
masteruser replication
# this is the second ConfiMap will be mounted to. it has the list of uses needed.
aclfile /conf/acl/users.acl
# port, each redis nodes will be used
port 6379
# More configurations are optional, if not provided, redis will consider default values ------
# ------ More details on configuration : https://redis.io/docs/manual/config/ ------
Apply the ConfigMap yaml file,
kubectl apply -f redis-config.yaml
Next, create another ConfiMap for the user ACL, this is a list of users that need to be created along with their permissions
File name : redis-acl.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: redis-acl
namespace: redis
data:
users.acl: |
# user "default" is the default user ( act as admin ) and user "replication" is used by the nodes for the replication.
user default on #8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918 ~* &* +@all
user replication on #65cf6f5f48186a4a6c5de02f156f1642b3873451d9de1607147799023dbf4ef8 +psync +replconf +ping
user worker on #87eba76e7f3164534045ba922e7770fb58bbd14ad732bbf5ba6f11cc56989e6e ~* &* +@all -@dangerous
Users list created by ACL configuration.
- default : default user with admin previleges. Access should be restricted to administrators. Note : This is needed for all the administrative purpose, use a strong password.
- worker : user to be used by apps or clients connect to redis cluster.
- replication : user used by replicas to AUTH and communicate with master.
User Password
The password can be configured either in plain text or SHA-256 encrypted. “#” indicates its SHA-256 hash value, “>” will be used for plain password.
More details : redis ACL
Create the ConfigMap,
kubectl apply -f redis_acl.yaml
StatefulSet
This is the key part, in this file we ill mount the ConfigMaps created above, and create the StatefulSet replicas based on these configurations.
File name : redis.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: redis
namespace: redis
spec:
serviceName: redis
replicas: 6 # 6 replicas, 3 master and 3 replicas(slaves)
selector:
matchLabels:
app: redis
template:
metadata:
labels:
app: redis
spec:
initContainers:
- name: config
image: redis:7.0.0-alpine
command: [ "sh", "-c" ]
args:
- |
if [ -f "/conf/redis.conf" ]; then
echo "config exists /conf/redis.conf .. not creating new"
else
echo "config doesn't exist copying to /conf/redis.conf"
cp /tmp/conf/redis.conf /conf/redis.conf
fi
volumeMounts:
- name: storage
subPath: conf
mountPath: /conf
- name: config
mountPath: /tmp/conf/
containers:
- name: redis
image: redis:7.0.0-alpine
command: ["redis-server"]
args: ["/conf/redis.conf"]
resources:
requests:
memory: "100M"
limits:
memory: "2000M"
ports:
- containerPort: 6379
name: redis
volumeMounts:
- name: storage
mountPath: /data
subPath: data
- name: storage
mountPath: /conf
subPath: conf
- name: config-acl
mountPath: /conf/acl/
volumes:
- name: config
configMap:
name: redis-config
- name: config-acl
configMap:
name: redis-acl
volumeClaimTemplates:
- metadata:
name: storage
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 50Mi
There is a config container under “initContainers”. This will be created and terminated before the redis container starts. The sole purpose of this is to create/copy the configurations required for redis nodes.
Note: Here we are using volumeClaimTemplate, which creates PersistentVolume(PV) and PersistentVolumeClaim(PVC) dynamically for each node. See the link below for details on “How to make the Redis cluster use a pre-defined PV and PVC”
Create the StatefulSet,
kubectl apply -f redis.yaml
Headless Service
A headless service to connect to the 6 replicas ( redis nodes) of the StatefulSet.
Filename: redis-service.yaml
apiVersion: v1
kind: Service
metadata:
name: redis
namespace: redis
spec:
clusterIP: None # "None" make it a headless service. No cluster IP.
ports:
- port: 6379
targetPort: 6379
name: redis
selector:
app: redis
create the headless service,
kubectl apply -f redis-service.yaml
At this point, if all went well, there will be 6 pods running ( redis-0 to redis-5) in “redis” namespace and a headless service as below,
> kubectl get all -n redis
NAME READY STATUS RESTARTS AGE IP
pod/redis-0 1/1 Running 0 3m32s 192.168.70.9
pod/redis-1 1/1 Running 0 3m13s 192.168.75.40
pod/redis-2 1/1 Running 0 9s 192.168.67.74
pod/redis-3 1/1 Running 0 3m6s 192.168.71.78
pod/redis-4 1/1 Running 0 3m3s 192.168.72.149
pod/redis-5 1/1 Running 0 2m59s 192.168.75.105
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/redis ClusterIP None <none> 6379/TCP 3m32s app=redis
NAME READY AGE CONTAINERS IMAGES
statefulset.apps/redis 6/6 3m33s redis redis:7.0.0-alpine
If you check the log of any pod, it should show printing like below, key part is where it says “Ready to accept connections”
> kubectl logs -n redis pod/redis-0
1:C 05 May 2022 23:28:54.937 # oO0OoO0OoO0Oo Redis is starting
oO0OoO0OoO0Oo
1:C 05 May 2022 23:28:54.937 # Redis version=7.0.0, bits=64,
commit=00000000, modified=0, pid=1, just started
1:C 05 May 2022 23:28:54.937 # Configuration loaded
1:M 05 May 2022 23:28:54.937 * monotonic clock: POSIX clock_gettime
1:M 05 May 2022 23:28:54.939 * Node configuration loaded, I'm
92dfcf6b6008822e98b032761768a981217b9883
1:M 05 May 2022 23:28:54.939 * Running mode=cluster, port=6379.
1:M 05 May 2022 23:28:54.939 # Server initialized
1:M 05 May 2022 23:28:54.942 * The AOF directory appendonlydir doesn't
exist
1:M 05 May 2022 23:28:54.946 * SYNC append only file rewrite performed
1:M 05 May 2022 23:28:54.948 * Ready to accept connections
Start Cluster
Now that all the nodes running, we need to start the cluster, This will assign 3 pods as masters and other 3 as replicas of each master,
“exec” into any running redis pod,
kubectl exec -it -n redis pod/redis-0 -- sh
execute the below command to create the cluster,
# Note : "-a admin" in the command is the "default" user password
given in redis-acl.yaml (raw form, not SHA-256 hash value)
# Imp : use a strong password.
/data # redis-cli --cluster create redis-
0.redis.redis.svc.cluster.local:6379 redis-
1.redis.redis.svc.cluster.local:6379 redis-
2.redis.redis.svc.cluster.local:6379 redis-
3.redis.redis.svc.cluster.local:6379 redis-
4.redis.redis.svc.cluster.local:6379 redis-
5.redis.redis.svc.cluster.local:6379 -a admin --cluster-replicas 1
This will ask for a confirmation to accept default configuration, type “yes”. this will create a sharded cluster with 3 master and 3 replicas(salves)
Sample Output,
/data # redis-cli --cluster create redis-
0.redis.redis.svc.cluster.local:6379 redis-
1.redis.redis.svc.cluster.local:6379 redis-
2.redis.redis.svc.cluster.local:6379 redis-
3.redis.redis.svc.cluster.local:6379 redis-4
.redis.redis.svc.cluster.local:6379 redis-
5.redis.redis.svc.cluster.local:6379 -a admin --cluster-replicas 1
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica redis-4.redis.redis.svc.cluster.local:6379 to redis-
0.redis.redis.svc.cluster.local:6379
Adding replica redis-5.redis.redis.svc.cluster.local:6379 to redis-
1.redis.redis.svc.cluster.local:6379
Adding replica redis-3.redis.redis.svc.cluster.local:6379 to redis-
2.redis.redis.svc.cluster.local:6379
M: 2e2ecf247ee64fabec434a3cbec12bb211824ce4 redis-
0.redis.redis.svc.cluster.local:6379
slots:[0-5460] (5461 slots) master
M: 4bf5dbf6146eff3355c796883338b9c1c91ba5f2 redis-
1.redis.redis.svc.cluster.local:6379
slots:[5461-10922] (5462 slots) master
M: 06bd7df09c25f9d4ea433c3b69293cd9adbed304 redis-
2.redis.redis.svc.cluster.local:6379
slots:[10923-16383] (5461 slots) master
S: 8abb9fda1a2659851f22b298b813d91b14851a8f redis-
3.redis.redis.svc.cluster.local:6379
replicates 06bd7df09c25f9d4ea433c3b69293cd9adbed304
S: 19cf9a0a04d7bd5182c306b94cf8f589d394577d redis-
4.redis.redis.svc.cluster.local:6379
replicates 2e2ecf247ee64fabec434a3cbec12bb211824ce4
S: c3eda2d867f5dd2f0781e6bb48627ca40b6066c7 redis-
5.redis.redis.svc.cluster.local:6379
replicates 4bf5dbf6146eff3355c796883338b9c1c91ba5f2
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.
>>> Performing Cluster Check (using node redis-
0.redis.redis.svc.cluster.local:6379)
M: 2e2ecf247ee64fabec434a3cbec12bb211824ce4 redis-
0.redis.redis.svc.cluster.local:6379
slots:[0-5460] (5461 slots) master
1 additional replica(s)
S: c3eda2d867f5dd2f0781e6bb48627ca40b6066c7 192.168.46.242:6379
slots: (0 slots) slave
replicates 4bf5dbf6146eff3355c796883338b9c1c91ba5f2
M: 06bd7df09c25f9d4ea433c3b69293cd9adbed304 192.168.48.65:6379
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
M: 4bf5dbf6146eff3355c796883338b9c1c91ba5f2 192.168.57.250:6379
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
S: 8abb9fda1a2659851f22b298b813d91b14851a8f 192.168.60.185:6379
slots: (0 slots) slave
replicates 06bd7df09c25f9d4ea433c3b69293cd9adbed304
S: 19cf9a0a04d7bd5182c306b94cf8f589d394577d 192.168.46.128:6379
slots: (0 slots) slave
replicates 2e2ecf247ee64fabec434a3cbec12bb211824ce4
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
At this point, the cluster is created, and its ready to store data. To see the cluster info, “exec” into a pod, then use “redis-cli -c”
# Note : "-c" tells redis-cli that it is connecting to a cluster. "-a
admin" is the default user password.
/data # redis-cli -c -a admin
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> CLUSTER NODES
17fbef584bbbeac241bd33aa8e9d03b35ab4c94d 192.168.57.250:6379@16379
myself,master - 0 1651805328000 1 connected 0-5460
2690fabc3b37462a08a44148a000efb000d9841c 192.168.46.242:6379@16379
master - 0 1651805330000 3 connected 10923-16383
e68bab90451817f0008bb8ec4c9dd10b849565e8 192.168.46.128:6379@16379
master - 0 1651805330589 2 connected 5461-10922
6982044e1596b188d56c367d16df46d000275e62 192.168.33.147:6379@16379
slave e68bab90451817f0008bb8ec4c9dd10b849565e8 0 1651805329000 2 connected
e893d2421ad3c6c1d35966b1dee9389352901363 192.168.48.65:6379@16379
slave 2690fabc3b37462a08a44148a000efb000d9841c 0 1651805330990 3 connected
83370d400ad962ae28d202288bada3c16aa4dc79 192.168.47.226:6379@16379
slave 17fbef584bbbeac241bd33aa8e9d03b35ab4c94d 0 1651805329985 1 connected
Now, any “Cluster-aware” client can connect to redis, ( ex: jedis ).
There are two ways, a client can connect.
- It can use the IP’s of the pods by using DNS of headless service
- Or, more easy way, it can use the FQDN(Fully Qualified Domain Names) to connect. (ex:redis-n.redis.redis.svc.cluster.local)
Note: use the user “worker” (or similar), with less ACL privileges in the apps to connect and use redis.
Thanks!
Share with your friends and followers
Start blogging about your favorite technologies, reach more readers and earn rewards!
Join other developers and claim your FAUN account now!
Jithin Scaria
Enterprise Architect
@jithinscaria
User Popularity
29
Influence
2k
Total Hits
1
Posts
Read, Learn, Know, Teach
Hand curated newsletters for Developers, private Slack with like minded people, podcasts, job offers, news and more!
Only registered users can post comments. Please, login or signup.