Referring back to article The structural strength of steel, I will share some practical examples to encourage architect colleagues to break out of the role of „boxes and lines“ - which is usually expected from us. All of us should know stones and bricks what our architecture is built from - still should not be the best bricklayer, but even not the worst!

This first article is an infrastructural task, building a Redis cluster in well-controlled Linux environment on a RHEL7 host.

The well-controlled in the article means that the user you use for installation has no unlimited 'sudoers' rights, neither it is not the root certainly. On the other hand, it is assumed the the host uses SELinux, the enhanced Linux security package. One more thing should be highlighted that Redis install pack should be downloaded beforehand to ensure, which version you are using!

The following guide describes those steps, which are described at redis.io for cluster preparation and extended by all SELinux configuration, which are needed to run Redis as a daemon. If you would simply try Redis starting it as a user process SELinux will let it fly, since the normal users are handled as unconfined_u user, that has all rights needed. On the other side, daemons are running using system_u user in SELinux, which is so limited: no access to port, directories and so.

First of all, the root user has to delegate the appropriate rights to the user. The setup below gives all the right to the archer user, which is used to execute the setup. The details of the commands will be described below, in the course of the installation.

The cluster we build will have 6 nodes, 3 master and 3 slave one on two Linux hosts, therefore most of the commands has to be executed on both Linux nodes except the ones are indicated to run only one place. The example uses archer, which you should replace by your own non-root user name in your environment.

The setup described here should be added using the visudo command. There are 5 different command aliases prepared to support the installation. After successful installation the root may revoke all, instead of REDISCLUSTER, which contains those commands, which are needed for normal operation.

Revoke the rights means to change this last statement to:
archer    ALL=(root) NOPASSWD: REDISCLUSTER

Now you can log in the server and start the setup.

The steps till the point of Start Redis Cluster are the same on both hosts, therefore execute them properly before continue. Following the Redis suggested topology, the cluster will use 3 Redis nodes per server, 3 of them will be master and 3 will be slave nodes.

1. Obtain Redis offline installer from RPM repository

2. Install the RPM

The installer has a question about the installing size: Is this ok [y/d/N]: please respond with y

3. Start Redis service

4. Test if Redis executes properly

The expected result is: PONG

If Redis works properly, stop the service, the default instance will not be used later:

SELinux preparations

Since our cluster will run on two hosts therefore 3 free ports are needed per each, which all should be allowed in SELinux to use. By default SELinux contains predefined Redis types and policies. To check the actual settings and to change them, we have to use semanage commands, which is the part of the policycoreutils-python package. To install, use the following command:

The next step is to check if SELinux is enabled or not using the following command:

If SELinux is enabled you will see similar message as below:

The Current mode: line may contain permissing instead of enforcing, but that is fine.

In the case SELinux is disabled, the response will be that:

If SELinux is disabled, enable it first with your Linux admin and at the end of the installation you can disable or keep it enables as you wish. Since it gives extra security it is worthwhile to keep it up and running.

Now, check which ports are configured for Redis and then if the ports we would use is enabled.

It is expected that the last three commands will show nothing. In the case your server has definition for any of these ports suggested here to use, it sup to you to select any three what is free using the same commands, but the please replace the referred ports later in the article identically.

To allow the ports for Redis, execute the following commands, which allows communication, cluster and sentinel ports, either we will not use sentinel now:

Check if everything is fine, executing the Redis port listing command we used above:

File access configuration for Redis cluster

The cluster database nodes will be created under /opt/redis-cluster directory. To simplify monitoring tasks, the node directories will be the same as the communication port used by the nodes.

Although the directories and files will only be created later, the SELinux configuration should be done in advance. The first step is to check what file context is set up for Redis:

The following commands gives access to the system_u SELinux user, which represents service processes, and labels them by redis_var_lib_t type to comply Redis related policies.

To check if the configuration was successful, execute the following command:

Now we have finished configuration of SELinux for the Redis Cluster.

Redis cluster preparation

1. Create Redis directories

2. Change the ownership of the node directories to redis:redis

Set the SELinux label on the new directories prepared

4. Copy the default configuration file of the default instance, using the communication port as identifier in the file name:

$ sudo cp /etc/redis.conf /etc/redis.6400.conf

5. Edit the configuration file. The installer uses vi, the necessary command rights are described for this at the top.

$ sudo vi /etc/redis.6400.conf

The configuration file is a self-documented, and it is suggested to keep it as it is. The following changes should be done by searching for the necessary data then change as described.

6. Create the second node configuration file by copying the first you have just finished:

$ sudo cp /etc/redis.6400.conf /etc/redis.6401.conf

7. Edit the file, and change 6400 to 6401 for port, cluster-config-file, dir and pidfile attributes. Create the third node configuration file by copying the first just finished:

$ sudo cp /etc/redis.6400.conf /etc/redis.6402.conf

8. Edit the file, and change 6400 to 6402 for port, cluster-config-file, dir and pidfile attributes.

Prepare Redis node services

The service definitions will be created based on the default Redis service.

1. The initial step is to create the service for the first node:

$ sudo cp /usr/lib/systemd/system/redis.service /usr/lib/systemd/system/redis.6400.service

2. Edit the configuration file. The installer uses vi, the necessary command rights are described for this at the top.

$ sudo vi /usr/lib/systemd/system/redis.6400.service

The necessary changes described below, where the requested new values are bolded. Be aware to set the proper IP address on the Linux host you are!

Save the file and exit.

3. Create the second service definition file by copying the first just finished:

$ sudo cp /usr/lib/systemd/system/redis.6400.service /usr/lib/systemd/system/redis.6401.service

Edit the file, and change 6400 to 6401 at all places.

Create the third service definition file by copying the first just finished:

$ sudo cp /usr/lib/systemd/system/redis.6400.service /usr/lib/systemd/system/redis.6402.service

Edit the file, and change 6400 to 6402 at all places.

4. Set the ownership of service defintions

4. Start and enable the nodes created.

Enable the nodes services

Start the nodes

5. Check if all the nodes are running properly:

The expected result is for redis.6400 is:

redis.6400.service - Redis cluster node 1 listening on port 6400
   Loaded: loaded (/usr/lib/systemd/system/redis.6400.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2019-05-22 11:26:21 CEST; 24h ago
  Process: 501 ExecStop=/usr/bin/redis-cli -h 192.168.1.160 -p 6400 shutdown (code=exited, status=0/SUCCESS)
 Main PID: 619 (redis-server)
   CGroup: /system.slice/redis.6400.service
           └─619 /usr/bin/redis-server 192.168.1.160:6400 [cluster]

May 22 11:26:21 <FQDN of your Redis host> systemd[1]: Starting Redis cluster node 1 listening on po.....
May 22 11:26:21 <FQDN of your Redis host> systemd[1]: Started Redis cluster node 1 listening on por...0.
Hint: Some lines were ellipsized, use -l to show in full.

The result will be the same for all nodes in both Linux hosts, the important difference is, that the proper port and the node related file name will reflected. The IP address will be the host IP.

We have finished the steps that should be executed on both of the Linux hosts and we are ready to start the cluster!

The next commands should be run on any of the Linux hosts. We suggest running them on the first Linux host installed.

Start Redis Cluster - This command should be executed only one of your Linux hosts

The command to start the cluster is the following. The IP addresses has to be the Linux host IP-s, which Redis was bind to. Be aware that node 1-3 are represented using their ports 6400-6402

$ redis-cli --cluster create 192.168.1.160:6400 192.168.1.160:6401 192.168.1.160:6402 192.168.1.161:6400 192.168.1.161:6401 192.168.1.161:6402 --cluster-replicas 1

 

The command will show, how the cluster is planned to set up. The information looks like the following:

>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 192.168.1.161:6402 to 192.168.1.160:6400
Adding replica 192.168.1.160:6402 to 192.168.1.161:6400
Adding replica 192.168.1.161:6401 to 192.168.1.160:6401
M: adedb3800ea96404bb91d3f83f96a6c3452341a1 192.168.1.160:6400
   slots:[0-5460] (5461 slots) master
M: 52b956d477a515753ee1b33c33aa2ed8a873f176 192.168.1.160:6401
   slots:[10923-16383] (5461 slots) master
S: 2a248a05e3243d8d90057e6481f416dba164eb2d 192.168.1.160:6402
   replicates 5d180b7aff4523f8fc7c982fecf6c6f42d29c07a
M: 5d180b7aff4523f8fc7c982fecf6c6f42d29c07a 192.168.1.161:6400
   slots:[5461-10922] (5462 slots) master
S: c25ba99ae9fff6d15548980f6edb08fb99c985e7 192.168.1.161:6401
   replicates 52b956d477a515753ee1b33c33aa2ed8a873f176
S: ae77097206d66e315899a72038828c174d350eed 192.168.1.161:6402
   replicates adedb3800ea96404bb91d3f83f96a6c3452341a1
Can I set the above configuration? (type 'yes' to accept):

Answer yes to the question and press enter.

A progress will start and the result will be like this:

>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
....
>>> Performing Cluster Check (using node 192.168.1.160:6400)
M: adedb3800ea96404bb91d3f83f96a6c3452341a1 192.168.1.160:6400
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: ae77097206d66e315899a72038828c174d350eed 192.168.1.161:6402
   slots: (0 slots) slave
   replicates adedb3800ea96404bb91d3f83f96a6c3452341a1
M: 52b956d477a515753ee1b33c33aa2ed8a873f176 192.168.1.160:6401
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: c25ba99ae9fff6d15548980f6edb08fb99c985e7 192.168.1.161:6401
   slots: (0 slots) slave
   replicates 52b956d477a515753ee1b33c33aa2ed8a873f176
M: 5d180b7aff4523f8fc7c982fecf6c6f42d29c07a 192.168.1.161:6400
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: 2a248a05e3243d8d90057e6481f416dba164eb2d 192.168.1.160:6402
   slots: (0 slots) slave
   replicates 5d180b7aff4523f8fc7c982fecf6c6f42d29c07a
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

To ensure that the cluster really works, check its status:

$ redis-cli --cluster check 192.168.1.160:6400

The result of this command is the same what we saw at cluster creation.

 

Now you have a shiny 6 nodes Redis cluster. Enjoy!