Feeding the Werewolves

It was decided not to use YugabyteDB at this stage. Setup is quite complicated, maintenance costs are unknown and compatibility is also questionable.

We stay by PostgreSQL for now. There will be two database to switch over. We will accept short read-only window during the database switch.

I leave the document here for future reference. But be aware that this is not used in PDM.control for now.

Distributed PostgreSQL for Modern Apps Run your business-critical applications on a distributed database with built-in resilience, seamless scalability, and flexible geo-distribution using PostgreSQL-compatible and Cassandra-inspired APIs.

This article describes how to deploy YugabyteDB as highly available cluster. There must be odd number of masters (3 for HA cluster) and any number of t-servers. The master saves metadata and coordinates cluster functions. T-servers store actual data and perform R/W operations. Three can be also specialized RO t-servers for long running analytics queries.

It seems that DB servers prefer bind-mount on the host directories for data storage then volumes. This the case here.

Before deploying the stack create a data directory on each node.

mkdir -p /data/yugabyte/master /data/yugabyte/tserver
chown -R root:root /data/yugabyte
chmod -R 755 /data/yugabyte
Example of 3 node installation
version: "3.8"

services:
  yb-master-1:
    image: yugabytedb/yugabyte:2025.2.2.1-b1
    hostname: yb-master-1
    command:
      - /home/yugabyte/bin/yb-master
      - --fs_data_dirs=/mnt/disk0
      - --master_addresses=yb-master-1:7100,yb-master-2:7100,yb-master-3:7100
      - --rpc_bind_addresses=0.0.0.0:7100
      - --server_broadcast_addresses=yb-master-1:7100
      - --webserver_interface=0.0.0.0
      - --webserver_port=7000
      - --replication_factor=3
    volumes:
      - /data/yugabyte/master:/mnt/disk0
    stop_grace_period: 60s  # Gives the DB 1 minute to shut down safely
    networks:
      - yb-net
    ports:
      # Master HTTP UI
      - target: 7000
        published: 7000
        protocol: tcp
        mode: host
      # Master RPC port used for master-to-master and cluster internal communication
      - target: 7100
        published: 7100
        protocol: tcp
        mode: host
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.labels.app.yugabytedb.master == 1

  yb-master-2:
    image: yugabytedb/yugabyte:2025.2.2.1-b1
    hostname: yb-master-2
    command:
      - /home/yugabyte/bin/yb-master
      - --fs_data_dirs=/mnt/disk0
      - --master_addresses=yb-master-1:7100,yb-master-2:7100,yb-master-3:7100
      - --rpc_bind_addresses=0.0.0.0:7100
      - --server_broadcast_addresses=yb-master-2:7100
      - --webserver_interface=0.0.0.0
      - --webserver_port=7000
      - --replication_factor=3
    stop_grace_period: 60s
    volumes:
      - /data/yugabyte/master:/mnt/disk0
    networks:
      - yb-net
    ports:
      - target: 7000
        published: 7000
        protocol: tcp
        mode: host
      - target: 7100
        published: 7100
        protocol: tcp
        mode: host
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.labels.app.yugabytedb.master == 2

  yb-master-3:
    image: yugabytedb/yugabyte:2025.2.2.1-b1
    hostname: yb-master-3
    command:
      - /home/yugabyte/bin/yb-master
      - --fs_data_dirs=/mnt/disk0
      - --master_addresses=yb-master-1:7100,yb-master-2:7100,yb-master-3:7100
      - --rpc_bind_addresses=0.0.0.0:7100
      - --server_broadcast_addresses=yb-master-3:7100
      - --webserver_interface=0.0.0.0
      - --webserver_port=7000
      - --replication_factor=3
    volumes:
      - /data/yugabyte/master:/mnt/disk0
    stop_grace_period: 60s
    networks:
      - yb-net
    ports:
      - target: 7000
        published: 7000
        protocol: tcp
        mode: host
      - target: 7100
        published: 7100
        protocol: tcp
        mode: host
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.labels.app.yugabytedb.master == 3

  yb-tserver-1:
    image: yugabytedb/yugabyte:2025.2.2.1-b1
    hostname: yb-tserver-1
    command:
      - /home/yugabyte/bin/yb-tserver
      - --fs_data_dirs=/mnt/disk0
      - --tserver_master_addrs=yb-master-1:7100,yb-master-2:7100,yb-master-3:7100
      - --rpc_bind_addresses=0.0.0.0:9100
      - --server_broadcast_addresses=yb-tserver-1:9100
      - --webserver_interface=0.0.0.0
      - --webserver_port=9000
      - --pgsql_proxy_bind_address=0.0.0.0:5433
      - --cql_proxy_bind_address=0.0.0.0:9042
      - --redis_proxy_bind_address=0.0.0.0:6379
    volumes:
      - /data/yugabyte/tserver:/mnt/disk0
    stop_grace_period: 120s  # Gives the T-server 2 minutes to shut down safely
    networks:
      - yb-net
    ports:
      # TServer HTTP UI
      - target: 9000
        published: 9000
        protocol: tcp
        mode: host
      # TServer RPC port used for internal cluster communication
      - target: 9100
        published: 9100
        protocol: tcp
        mode: host
      # YSQL / PostgreSQL-compatible client port
      - target: 5433
        published: 5433
        protocol: tcp
        mode: host
      # YCQL / Cassandra-compatible client port
      - target: 9042
        published: 9042
        protocol: tcp
        mode: host
      # YEDIS / Redis-compatible client port
      - target: 6379
        published: 6379
        protocol: tcp
        mode: host
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.labels.app.yugabytedb.tserver == 1

  yb-tserver-2:
    image: yugabytedb/yugabyte:2025.2.2.1-b1
    hostname: yb-tserver-2
    command:
      - /home/yugabyte/bin/yb-tserver
      - --fs_data_dirs=/mnt/disk0
      - --tserver_master_addrs=yb-master-1:7100,yb-master-2:7100,yb-master-3:7100
      - --rpc_bind_addresses=0.0.0.0:9100
      - --server_broadcast_addresses=yb-tserver-2:9100
      - --webserver_interface=0.0.0.0
      - --webserver_port=9000
      - --pgsql_proxy_bind_address=0.0.0.0:5433
      - --cql_proxy_bind_address=0.0.0.0:9042
      - --redis_proxy_bind_address=0.0.0.0:6379
    volumes:
      - /data/yugabyte/tserver:/mnt/disk0
    stop_grace_period: 120s
    networks:
      - yb-net
    ports:
      - target: 9000
        published: 9000
        protocol: tcp
        mode: host
      - target: 9100
        published: 9100
        protocol: tcp
        mode: host
      - target: 5433
        published: 5433
        protocol: tcp
        mode: host
      - target: 9042
        published: 9042
        protocol: tcp
        mode: host
      - target: 6379
        published: 6379
        protocol: tcp
        mode: host
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.labels.app.yugabytedb.tserver == 2

  yb-tserver-3:
    image: yugabytedb/yugabyte:2025.2.2.1-b1
    hostname: yb-tserver-3
    command:
      - /home/yugabyte/bin/yb-tserver
      - --fs_data_dirs=/mnt/disk0
      - --tserver_master_addrs=yb-master-1:7100,yb-master-2:7100,yb-master-3:7100
      - --rpc_bind_addresses=0.0.0.0:9100
      - --server_broadcast_addresses=yb-tserver-3:9100
      - --webserver_interface=0.0.0.0
      - --webserver_port=9000
      - --pgsql_proxy_bind_address=0.0.0.0:5433
      - --cql_proxy_bind_address=0.0.0.0:9042
      - --redis_proxy_bind_address=0.0.0.0:6379
    volumes:
      - /data/yugabyte/tserver:/mnt/disk0
    stop_grace_period: 120s
    networks:
      - yb-net
    ports:
      - target: 9000
        published: 9000
        protocol: tcp
        mode: host
      - target: 9100
        published: 9100
        protocol: tcp
        mode: host
      - target: 5433
        published: 5433
        protocol: tcp
        mode: host
      - target: 9042
        published: 9042
        protocol: tcp
        mode: host
      - target: 6379
        published: 6379
        protocol: tcp
        mode: host
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.labels.app.yugabytedb.tserver == 3

networks:
  yb-net:
    driver: overlay
    attachable: true

1. Maintenance

1.1. Migration to new node

This is a scenario when e.g. OS is upgraded by replacing the node. According to information from ChatGPT first should be migrated T-server and then master.

Migration of T-server should be relatively simple.

Migration the master is a bit more complicated. It is necessary to tell the DB that the master will be disconnected.

Exec shell in on of master container which will not be disconnected.

# Set variable with master to not type them over and over
# But it seems that the command works also without --master_addresses $MASTERS
[root@yb-master-1 yugabyte]# export MASTERS=yb-master-1:7100,yb-master-2:7100,yb-master-3:7100
# List available masters
[root@yb-master-1 yugabyte]# /home/yugabyte/bin/yb-admin --master_addresses $MASTERS list_all_masters
Master UUID                             RPC Host/Port           State           Role    Broadcast Host/Port
34dc0c5fb0704a9f8c17865aa191d1f4        yb-master-1:7100        ALIVE           FOLLOWER        yb-master-1:7100
35450a46e6774bfab367f115eb72c1d6        yb-master-2:7100        ALIVE           FOLLOWER        yb-master-2:7100
ce192fa446c741128e9117e29320dc20        yb-master-3:7100        ALIVE           LEADER          yb-master-3:7100
# Remove one master
[root@yb-master-1 yugabyte]# /home/yugabyte/bin/yb-admin --master_addresses $MASTERS change_master_config REMOVE_SERVER yb-master-3 7100
# Add new master
[root@yb-master-1 yugabyte]# /home/yugabyte/bin/yb-admin --master_addresses $MASTERS change_master_config ADD_SERVER yb-master-3 7100