Redis

Redis-Server

通过bitnami/redis:7.4.1实现 redis 主从集群

version: '3.8'

name: 'redis-service'

services:
  redis-master:
    image: 'bitnami/redis:7.4.1'
    container_name: redis-master
    hostname: redis-master
    expose:
      - '6379'
    environment:
      - REDIS_REPLICATION_MODE=master
      - REDIS_PASSWORD=OeLFo2b3GVXKj0Ren1Dh
      - REDIS_MASTER_PORT_NUMBER=6379
    networks:
      redis:
        ipv4_address: 192.168.0.2
  redis-replica:
    image: 'bitnami/redis:7.4.1'
    deploy:
      replicas: 2
    depends_on:
      - redis-master
    environment:
      - REDIS_REPLICATION_MODE=slave
      - REDIS_MASTER_HOST=192.168.0.2
      - REDIS_MASTER_PORT_NUMBER=6379
      - REDIS_MASTER_PASSWORD=OeLFo2b3GVXKj0Ren1Dh
      - REDIS_PASSWORD=OeLFo2b3GVXKj0Ren1Dh
    networks:
      redis:

  
networks:
  redis:
    driver: bridge
    name: redis-net
    ipam:
      config:
        - subnet: 192.168.0.0/24
          gateway: 192.168.0.1

Redis-Sentinel

为保证redis集群高可用，添加redis sentinel监控master节点。

version: '3.8'
name: 'redis-sentinel-cluster'

services:
  redis-sentinel:
    image: 'bitnami/redis-sentinel:6.2.16'
    deploy:
      replicas: 3
    environment:
      REDIS_SENTINEL_PORT_NUMBER: '26379'
      REDIS_MASTER_HOST: '192.168.0.2'
      REDIS_MASTER_PORT_NUMBER: '6379'
      REDIS_SENTINEL_QUORUM: '2'  # 在进行故障转移之前，必须同意主节点失效的 Sentinel 实例的最小数量。
      REDIS_MASTER_PASSWORD: 'OeLFo2b3GVXKj0Ren1Dh'
      REDIS_SENTINEL_DOWN_AFTER_MILLISECONDS: '5000'
      REDIS_SENTINEL_FAILOVER_TIMEOUT: '5000'
      REDIS_MASTER_SET: 'mymaster'
    networks:
      - redis

networks:
  redis:
    name: 'redis-net'
    external: true

测试后发现，redis sentinel 能在master节点失效后选举新master节点，但是在docker-compose中只暴露了master节点的6379端口。于是在redis-net网络中添加nginx反代。

nginx反代（不可用）

nginx-reverse-proxy:
    image: 'bitnami/nginx:1.27'
    volumes:
      - ./nginx.conf:/opt/bitnami/nginx/conf/stream_server_blocks/my_server_block.conf
    environment:
      NGINX_ENABLE_STREAM: yes
    ports:
      - '6379:6379'
    networks:
      redis:
    depends_on:
      - redis-replica

添加反代后，发现slave节点不能向master节点转发命令（记错了，这个是redis分片集群的功能，分片节点计算键的插槽，该插槽不归该节点管就转发给该管这个插槽的节点）。放弃反代，直接将测试代码接入redis-net网络。

问题

发现容器内无法访问外部网络

(.venv) ronie@ronie:~/PycharmProjects/evilbox$ docker attach 81
/ # ping www.baidu.com
PING www.baidu.com (180.101.50.188): 56 data bytes
^C
--- www.baidu.com ping statistics ---
2 packets transmitted, 0 packets received, 100% packet loss

• 发现docker0网卡down且未被分配IP。

  (.venv) ronie@ronie:~/PycharmProjects/evilbox$ ip addr show docker0
  3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
      link/ether 02:42:e2:f4:ca:4c brd ff:ff:ff:ff:ff:ff
  

  猜测可能是root mode docker 和 rootless mode docker 冲突。停止root mode docker。

  (.venv) ronie@ronie:~/PycharmProjects/evilbox$ ip addr show docker0
  3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
      link/ether 02:42:f3:75:a1:fd brd ff:ff:ff:ff:ff:ff
      inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
         valid_lft forever preferred_lft forever
  

  分配了IP，但仍然是down（应该是down，docker0在root mode下使用）。

• 检查容器网络设置 docker inspect 89，发现容器被设置了代理？？？不知道什么时候修改的，该回来。Proxy configuration | Docker Docs

  "Env": [
      "NO_PROXY=localhost, 127.0.0.1, 10.96.0.0/12, 192.168.59.0/24, 192.168.49.0/24, 192.168.39.0/24",
      "no_proxy=localhost, 127.0.0.1, 10.96.0.0/12, 192.168.59.0/24, 192.168.49.0/24, 192.168.39.0/24",
      "HTTP_PROXY=http://127.0.0.1:2081",
      "http_proxy=http://127.0.0.1:2081",
      "HTTPS_PROXY=http://127.0.0.1:2081",
      "https_proxy=http://127.0.0.1:2081"
  ],

• 注意到4404 /proc/self/exe --state-dir=/run/user/1000/dockerd-rootless --net=slirp4netns ，发现docker在rootless模式下是不用docker0这个网卡的。rootlesskit/docs/network.md at v2.0.0 · rootless-containers/rootlesskit

• Rootless mode | Docker Docs

  Ping does not work when /proc/sys/net/ipv4/ping_group_range is set to 1 0:

  (.venv) ronie@ronie:/proc/sys/net/ipv4$ pwd
  /proc/sys/net/ipv4
  (.venv) ronie@ronie:/proc/sys/net/ipv4$ cat ping_group_range 
  1       0

• Rootless mode | Docker Docs

  On some distributions, ping does not work by default.

  Add net.ipv4.ping_group_range = 0 2147483647 to /etc/sysctl.conf (or /etc/sysctl.d) and run sudo sysctl --system to allow using ping.

• rootless docker 网络 | 今天我学了什么

总的来说是代理问题。

client

import typing
from redis.sentinel import Sentinel
from redis import Redis
import redis
import string
import random

class RedisClient(Redis):
    def __init__(
            self,
            sentinels: typing.Union[typing.List[typing.Tuple[str, int]], Sentinel],
            **kwargs
    ):
        if isinstance(sentinels, Sentinel):
            self.sentinels = sentinels
        else:
            self.sentinels = Sentinel(sentinels, socket_timeout=0.1)
        self.client_kwargs = kwargs
        host, port = self.sentinels.discover_master("mymaster")
        super().__init__(host=host, port=port, **kwargs)

    def renew(self):
        host, port = self.sentinels.discover_master("mymaster")
        super().__init__(host=host, port=port, **self.client_kwargs)
        return self

    def __getattribute__(self, item):
        attr = super().__getattribute__(item)
        if item != 'ensure_client' and callable(attr):
            attr = self.ensure_client(attr)
        return attr

    @staticmethod
    def ensure_client(func):
        def wrap(*args, **kwargs):
            self = args[0] if args else None
            if self is None:
                return func(*args, **kwargs)
            for i in range(3):
                try:
                    ret = func(*args, **kwargs)
                    break
                except redis.exceptions.ConnectionError:
                    self.renew()
            else:
                raise redis.exceptions.ConnectionError
            return ret
        return wrap

if __name__ == '__main__':
    r = RedisClient(
        sentinels=[
            ('redis-sentinel-cluster-sentinel-1', 26379),
            ('redis-sentinel-cluster-sentinel-2', 26379),
            ('redis-sentinel-cluster-sentinel-3', 26379),
        ],
        password='OeLFo2b3GVXKj0Ren1Dh'
    )
    r.lpush("test", "".join(random.choices(string.printable, k=20)))
    print(r.rpop("test"))