The strength of steel: Build Redis cluster on well-controlled Linux environment

Referring back to article The structural strength of steel, I will share some practical examples to encourage architect colleagues to break out of the role of „boxes and lines“ - which is usually expected from us. All of us should know stones and bricks what our architecture is built from - still should not be the best bricklayer, but even not the worst!

This first article is an infrastructural task, building a Redis cluster in well-controlled Linux environment on a RHEL7 host.

The well-controlled in the article means that the user you use for installation has no unlimited 'sudoers' rights, neither it is not the root certainly. On the other hand, it is assumed the the host uses SELinux, the enhanced Linux security package. One more thing should be highlighted that Redis install pack should be downloaded beforehand to ensure, which version you are using!

The following guide describes those steps, which are described at redis.io for cluster preparation and extended by all SELinux configuration, which are needed to run Redis as a daemon. If you would simply try Redis starting it as a user process SELinux will let it fly, since the normal users are handled as unconfined_u user, that has all rights needed. On the other side, daemons are running using system_u user in SELinux, which is so limited: no access to port, directories and so.

First of all, the root user has to delegate the appropriate rights to the user. The setup below gives all the right to the archer user, which is used to execute the setup. The details of the commands will be described below, in the course of the installation.

The cluster we build will have 6 nodes, 3 master and 3 slave one on two Linux hosts, therefore most of the commands has to be executed on both Linux nodes except the ones are indicated to run only one place. The example uses archer, which you should replace by your own non-root user name in your environment.

The setup described here should be added using the visudo command. There are 5 different command aliases prepared to support the installation. After successful installation the root may revoke all, instead of REDISCLUSTER, which contains those commands, which are needed for normal operation.

Cmnd_Alias REDIS = /bin/systemctl start redis, /bin/systemctl enable redis, /bin/systemctl stop redis

Cmnd_Alias REDISCLUSTER = /bin/systemctl start redis.640[012], /bin/systemctl enable redis.640[012], /bin/systemctl stop redis.640[012], /bin/systemctl status redis.640[012]

Cmnd_Alias REDISINSTALL = /usr/bin/yum install redis-5.0.4-1.el7.remi.x86_64.rpm

Cmnd_Alias REDISCLUSTERINSTALL = /bin/mkdir /opt/redis-cluster, /bin/mkdir /opt/redis-cluster/640[012], /usr/bin/chown -R redis\:redis /opt/redis-cluster/640[012], /bin/cp /etc/redis.conf /etc/redis.6400.conf, /bin/cp /etc/redis.6400.conf /etc/redis.640[12].conf, /bin/vi /etc/redis.640[012].conf, /bin/cp /usr/lib/systemd/system/redis.service /usr/lib/systemd/system/redis.6400.service, /bin/cp /usr/lib/systemd/system/redis.6400.service /usr/lib/systemd/system/redis.640[12].service, /bin/vi /usr/lib/systemd/system/redis.640[012].service, /usr/bin/chown redis\:root /etc/redis.640[012].conf

Cmnd_Alias REDISCLUSTERSEMANAGE = /usr/bin/yum install policycoreutils-python, /usr/sbin/semanage port -a -t redis_port_t -p tcp [012]640[012], /usr/sbin/semanage port -l, /usr/sbin/semanage fcontext -l, /usr/sbin/semanage fcontext -a -s system_u -t redis_var_lib_t /opt/redis-cluster/640[012], /usr/sbin/restorecon -v /opt/redis-cluster/640[012], /usr/sbin/semanage fcontext -a -s system_u -t redis_var_lib_t /opt/redis-cluster/640[012]/*, /usr/sbin/restorecon -v /opt/redis-cluster/640[012]/*

archer    ALL=(root) NOPASSWD: REDIS, REDISCLUSTER, REDISINSTALL, REDISCLUSTERINSTALL, REDISCLUSTERSEMANAGE

Revoke the rights means to change this last statement to:
archer    ALL=(root) NOPASSWD: REDISCLUSTER

Now you can log in the server and start the setup.

The steps till the point of Start Redis Cluster are the same on both hosts, therefore execute them properly before continue. Following the Redis suggested topology, the cluster will use 3 Redis nodes per server, 3 of them will be master and 3 will be slave nodes.

1. Obtain Redis offline installer from RPM repository

2. Install the RPM

$ sudo /usr/bin/yum install redis-5.0.4-1.el7.remi.x86_64.rpm

The installer has a question about the installing size: Is this ok [y/d/N]: please respond with y

3. Start Redis service

$ sudo systemctl start redis

4. Test if Redis executes properly

$ redis-cli ping

The expected result is: PONG

If Redis works properly, stop the service, the default instance will not be used later:

$ sudo systemctl stop redis

SELinux preparations

Since our cluster will run on two hosts therefore 3 free ports are needed per each, which all should be allowed in SELinux to use. By default SELinux contains predefined Redis types and policies. To check the actual settings and to change them, we have to use semanage commands, which is the part of the policycoreutils-python package. To install, use the following command:

$ sudo yum install policycoreutils-python

The next step is to check if SELinux is enabled or not using the following command:

$ sestatus

If SELinux is enabled you will see similar message as below:

SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      31

The Current mode: line may contain permissing instead of enforcing, but that is fine.

In the case SELinux is disabled, the response will be that:

SELinux status:                 disabled

If SELinux is disabled, enable it first with your Linux admin and at the end of the installation you can disable or keep it enables as you wish. Since it gives extra security it is worthwhile to keep it up and running.

Now, check which ports are configured for Redis and then if the ports we would use is enabled.

$ sudo semanage port -l | grep redis

redis_port_t                   tcp      6379, 16379, 26379

$ sudo semanage port -l | grep 6400

$ sudo semanage port -l | grep 6401

$ sudo semanage port -l | grep 6402

It is expected that the last three commands will show nothing. In the case your server has definition for any of these ports suggested here to use, it sup to you to select any three what is free using the same commands, but the please replace the referred ports later in the article identically.

To allow the ports for Redis, execute the following commands, which allows communication, cluster and sentinel ports, either we will not use sentinel now:

$ sudo semanage port -a -t redis_port_t -p tcp 06400

$ sudo semanage port -a -t redis_port_t -p tcp 06401

$ sudo semanage port -a -t redis_port_t -p tcp 06402

$ sudo semanage port -a -t redis_port_t -p tcp 16400

$ sudo semanage port -a -t redis_port_t -p tcp 16401

$ sudo semanage port -a -t redis_port_t -p tcp 16402

$ sudo semanage port -a -t redis_port_t -p tcp 26400

$ sudo semanage port -a -t redis_port_t -p tcp 26401

$ sudo semanage port -a -t redis_port_t -p tcp 26402

Check if everything is fine, executing the Redis port listing command we used above:

$ sudo semanage port -l | grep redis

redis_port_t tcp 26402, 26401, 26400, 16402, 16401, 16400, 6402, 6401, 6400, 6379, 16379, 26379

File access configuration for Redis cluster

The cluster database nodes will be created under /opt/redis-cluster directory. To simplify monitoring tasks, the node directories will be the same as the communication port used by the nodes.

Although the directories and files will only be created later, the SELinux configuration should be done in advance. The first step is to check what file context is set up for Redis:

$ sudo semanage fcontext -l | grep redis

/var/lib/redis(/.*)? all files system_u:object_r:redis_var_lib_t:s0

/var/log/redis(/.*)? all files system_u:object_r:redis_log_t:s0

/var/run/redis(/.*)? all files system_u:object_r:redis_var_run_t:s0

/etc/redis-sentinel.* regular file system_u:object_r:redis_conf_t:s0

/usr/lib/systemd/system/redis.* regular file system_u:object_r:redis_unit_file_t:s0

/usr/bin/redis-server regular file system_u:object_r:redis_exec_t:s0

/etc/rc\.d/init\.d/redis regular file system_u:object_r:redis_initrc_exec_t:s0

The following commands gives access to the system_u SELinux user, which represents service processes, and labels them by redis_var_lib_t type to comply Redis related policies.

$ sudo semanage fcontext -a -s system_u -t redis_var_lib_t /opt/redis-cluster/6400

$ sudo semanage fcontext -a -s system_u -t redis_var_lib_t /opt/redis-cluster/6400/nodes.conf

$ sudo semanage fcontext -a -s system_u -t redis_var_lib_t /opt/redis-cluster/6400/appendonly.aof

$ sudo semanage fcontext -a -s system_u -t redis_var_lib_t /opt/redis-cluster/6400/dump.rdb

$ sudo semanage fcontext -a -s system_u -t redis_var_lib_t /opt/redis-cluster/6401

$ sudo semanage fcontext -a -s system_u -t redis_var_lib_t /opt/redis-cluster/6401/nodes.conf

$ sudo semanage fcontext -a -s system_u -t redis_var_lib_t /opt/redis-cluster/6401/appendonly.aof

$ sudo semanage fcontext -a -s system_u -t redis_var_lib_t /opt/redis-cluster/6401/dump.rdb

$ sudo semanage fcontext -a -s system_u -t redis_var_lib_t /opt/redis-cluster/6402

$ sudo semanage fcontext -a -s system_u -t redis_var_lib_t /opt/redis-cluster/6402/nodes.conf

$ sudo semanage fcontext -a -s system_u -t redis_var_lib_t /opt/redis-cluster/6402/appendonly.aof

$ sudo semanage fcontext -a -s system_u -t redis_var_lib_t /opt/redis-cluster/6402/dump.rdb

To check if the configuration was successful, execute the following command:

$ sudo semanage fcontext -l | grep redis

/var/lib/redis(/.*)? all files system_u:object_r:redis_var_lib_t:s0

/var/log/redis(/.*)? all files system_u:object_r:redis_log_t:s0

/var/run/redis(/.*)? all files system_u:object_r:redis_var_run_t:s0

/etc/redis-sentinel.* regular file system_u:object_r:redis_conf_t:s0

/usr/lib/systemd/system/redis.* regular file system_u:object_r:redis_unit_file_t:s0

/usr/bin/redis-server regular file system_u:object_r:redis_exec_t:s0

/etc/rc\.d/init\.d/redis regular file system_u:object_r:redis_initrc_exec_t:s0

/opt/redis-cluster/6400 all files system_u:object_r:redis_var_lib_t:s0

/opt/redis-cluster/6400/nodes.conf all files system_u:object_r:redis_var_lib_t:s0

/opt/redis-cluster/6400/appendonly.aof all files system_u:object_r:redis_var_lib_t:s0

/opt/redis-cluster/6400/dump.rdb all files system_u:object_r:redis_var_lib_t:s0

/opt/redis-cluster/6401 all files system_u:object_r:redis_var_lib_t:s0

/opt/redis-cluster/6401/nodes.conf all files system_u:object_r:redis_var_lib_t:s0

/opt/redis-cluster/6401/appendonly.aof all files system_u:object_r:redis_var_lib_t:s0

/opt/redis-cluster/6401/dump.rdb all files system_u:object_r:redis_var_lib_t:s0

/opt/redis-cluster/6402 all files system_u:object_r:redis_var_lib_t:s0

/opt/redis-cluster/6402/nodes.conf all files system_u:object_r:redis_var_lib_t:s0

/opt/redis-cluster/6402/appendonly.aof all files system_u:object_r:redis_var_lib_t:s0

/opt/redis-cluster/6402/dump.rdb all files system_u:object_r:redis_var_lib_t:s0

Now we have finished configuration of SELinux for the Redis Cluster.

Redis cluster preparation

1. Create Redis directories

$ sudo mkdir /opt/redis-cluster

$ sudo mkdir /opt/redis-cluster/6400

$ sudo mkdir /opt/redis-cluster/6401

$ sudo mkdir /opt/redis-cluster/6402

2. Change the ownership of the node directories to redis:redis

$ sudo chown -R redis:redis /opt/redis-cluster/6400

$ sudo chown -R redis:redis /opt/redis-cluster/6401

$ sudo chown -R redis:redis /opt/redis-cluster/6402

Set the SELinux label on the new directories prepared

$ sudo restorecon -v /opt/redis-cluster/6400

$ sudo restorecon -v /opt/redis-cluster/6400/*

$ sudo restorecon -v /opt/redis-cluster/6401

$ sudo restorecon -v /opt/redis-cluster/6401/*

$ sudo restorecon -v /opt/redis-cluster/6402

$ sudo restorecon -v /opt/redis-cluster/6402/*

4. Copy the default configuration file of the default instance, using the communication port as identifier in the file name:

$ sudo cp /etc/redis.conf /etc/redis.6400.conf

5. Edit the configuration file. The installer uses vi, the necessary command rights are described for this at the top.

$ sudo vi /etc/redis.6400.conf

The configuration file is a self-documented, and it is suggested to keep it as it is. The following changes should be done by searching for the necessary data then change as described.

6. Create the second node configuration file by copying the first you have just finished:

$ sudo cp /etc/redis.6400.conf /etc/redis.6401.conf

7. Edit the file, and change 6400 to 6401 for port, cluster-config-file, dir and pidfile attributes. Create the third node configuration file by copying the first just finished:

$ sudo cp /etc/redis.6400.conf /etc/redis.6402.conf

8. Edit the file, and change 6400 to 6402 for port, cluster-config-file, dir and pidfile attributes.

Prepare Redis node services

The service definitions will be created based on the default Redis service.

1. The initial step is to create the service for the first node:

$ sudo cp /usr/lib/systemd/system/redis.service /usr/lib/systemd/system/redis.6400.service

2. Edit the configuration file. The installer uses vi, the necessary command rights are described for this at the top.

$ sudo vi /usr/lib/systemd/system/redis.6400.service

The necessary changes described below, where the requested new values are bolded. Be aware to set the proper IP address on the Linux host you are!

[Unit]
Description=Redis cluster node 1 listening on port 6400
After=network.target

[Service]
ExecStart=/usr/bin/redis-server /etc/redis.6400.conf --supervised systemd
ExecStop=/usr/bin/redis-cli -h 192.168.1.160 -p 6400 shutdown
Type=notify
User=redis
Group=redis
RuntimeDirectory=redis6400
RuntimeDirectoryMode=0755

[Install]
WantedBy=multi-user.target

Save the file and exit.

3. Create the second service definition file by copying the first just finished:

$ sudo cp /usr/lib/systemd/system/redis.6400.service /usr/lib/systemd/system/redis.6401.service

Edit the file, and change 6400 to 6401 at all places.

Create the third service definition file by copying the first just finished:

$ sudo cp /usr/lib/systemd/system/redis.6400.service /usr/lib/systemd/system/redis.6402.service

Edit the file, and change 6400 to 6402 at all places.

4. Set the ownership of service defintions

$ sudo chown redis:root /etc/redis.6400.conf

$ sudo chown redis:root /etc/redis.6401.conf

$ sudo chown redis:root /etc/redis.6402.conf

4. Start and enable the nodes created.

Enable the nodes services

$ sudo systemctl enable redis.6400

$ sudo systemctl enable redis.6401

$ sudo systemctl enable redis.6402

Start the nodes

$ sudo systemctl start redis.6400

$ sudo systemctl start redis.6401

$ sudo systemctl start redis.6402

5. Check if all the nodes are running properly:

$ sudo systemctl status redis.6400

$ sudo systemctl status redis.6401

$ sudo systemctl status redis.6402

The expected result is for redis.6400 is:

redis.6400.service - Redis cluster node 1 listening on port 6400
   Loaded: loaded (/usr/lib/systemd/system/redis.6400.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2019-05-22 11:26:21 CEST; 24h ago
  Process: 501 ExecStop=/usr/bin/redis-cli -h 192.168.1.160 -p 6400 shutdown (code=exited, status=0/SUCCESS)
 Main PID: 619 (redis-server)
   CGroup: /system.slice/redis.6400.service
           └─619 /usr/bin/redis-server 192.168.1.160:6400 [cluster]

May 22 11:26:21 <FQDN of your Redis host> systemd[1]: Starting Redis cluster node 1 listening on po.....
May 22 11:26:21 <FQDN of your Redis host> systemd[1]: Started Redis cluster node 1 listening on por...0.
Hint: Some lines were ellipsized, use -l to show in full.

The result will be the same for all nodes in both Linux hosts, the important difference is, that the proper port and the node related file name will reflected. The IP address will be the host IP.

We have finished the steps that should be executed on both of the Linux hosts and we are ready to start the cluster!

The next commands should be run on any of the Linux hosts. We suggest running them on the first Linux host installed.

Start Redis Cluster - This command should be executed only one of your Linux hosts

The command to start the cluster is the following. The IP addresses has to be the Linux host IP-s, which Redis was bind to. Be aware that node 1-3 are represented using their ports 6400-6402

$ redis-cli --cluster create 192.168.1.160:6400 192.168.1.160:6401 192.168.1.160:6402 192.168.1.161:6400 192.168.1.161:6401 192.168.1.161:6402 --cluster-replicas 1

 

The command will show, how the cluster is planned to set up. The information looks like the following:

>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 192.168.1.161:6402 to 192.168.1.160:6400
Adding replica 192.168.1.160:6402 to 192.168.1.161:6400
Adding replica 192.168.1.161:6401 to 192.168.1.160:6401
M: adedb3800ea96404bb91d3f83f96a6c3452341a1 192.168.1.160:6400
   slots:[0-5460] (5461 slots) master
M: 52b956d477a515753ee1b33c33aa2ed8a873f176 192.168.1.160:6401
   slots:[10923-16383] (5461 slots) master
S: 2a248a05e3243d8d90057e6481f416dba164eb2d 192.168.1.160:6402
   replicates 5d180b7aff4523f8fc7c982fecf6c6f42d29c07a
M: 5d180b7aff4523f8fc7c982fecf6c6f42d29c07a 192.168.1.161:6400
   slots:[5461-10922] (5462 slots) master
S: c25ba99ae9fff6d15548980f6edb08fb99c985e7 192.168.1.161:6401
   replicates 52b956d477a515753ee1b33c33aa2ed8a873f176
S: ae77097206d66e315899a72038828c174d350eed 192.168.1.161:6402
   replicates adedb3800ea96404bb91d3f83f96a6c3452341a1
Can I set the above configuration? (type 'yes' to accept):

Answer yes to the question and press enter.

A progress will start and the result will be like this:

>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
....
>>> Performing Cluster Check (using node 192.168.1.160:6400)
M: adedb3800ea96404bb91d3f83f96a6c3452341a1 192.168.1.160:6400
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: ae77097206d66e315899a72038828c174d350eed 192.168.1.161:6402
   slots: (0 slots) slave
   replicates adedb3800ea96404bb91d3f83f96a6c3452341a1
M: 52b956d477a515753ee1b33c33aa2ed8a873f176 192.168.1.160:6401
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: c25ba99ae9fff6d15548980f6edb08fb99c985e7 192.168.1.161:6401
   slots: (0 slots) slave
   replicates 52b956d477a515753ee1b33c33aa2ed8a873f176
M: 5d180b7aff4523f8fc7c982fecf6c6f42d29c07a 192.168.1.161:6400
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: 2a248a05e3243d8d90057e6481f416dba164eb2d 192.168.1.160:6402
   slots: (0 slots) slave
   replicates 5d180b7aff4523f8fc7c982fecf6c6f42d29c07a
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

To ensure that the cluster really works, check its status:

$ redis-cli --cluster check 192.168.1.160:6400

The result of this command is the same what we saw at cluster creation.

 

Now you have a shiny 6 nodes Redis cluster. Enjoy!

Log in to comment

Architect Archers

Architect Archers are Enterprise Architect professionals and mentors.

Our mission is to build Your Archers team while defining, aiming and hit your Enterprise Targets.

© 2017 Architect Archers