一、前言

前几天有幸接触了MinIO，不知所以然，经过查询MinIO是一个对象存储。究竟有什么魅力，让这么多人对它情有独钟。

二、MinIO概述

MinIO 是在 Apache License v2.0 下发布的高性能对象存储。它是与 Amazon S3 云存储服务兼容的 API。使用 MinIO 构建用于机器学习、分析和应用的高性能基础设施数据工作负载。MinIO 从根本上与众不同,专为企业和私有云设计。MinIO生产部署涵盖了全球。MinIO是全球使用最多和下载量最大的对象存储服务系统，还是全世界增长最快的对象存储系统。

官方文档：http://docs.minio.org.cn/docs/(中文文档)

主要有以下几个特点：

三、MinIO单机部署(docker 单节点单驱动)

MinIO强烈推荐生产集群由 minimum 4 个组成 minio server 节点在服务器池。

3.1 准备工作

$ mkdir -p /data/minio/data
$ mkdir -p /data/minio/config
$ docker run -d \
   -p 9000:9000 \
   -p 9090:9090 \
   --name minio \
   -v /data/data:/data \
   -v /data/minio/config:/root/.minio \
   -e "MINIO_ROOT_USER=minioadmin" \
   -e "MINIO_ROOT_PASSWORD=minioadmin" \
   quay.io/minio/minio:RELEASE.2022-09-25T15-44-53Z server /data --console-address ":9090

参数说明：

docker run启动 MinIO 容器
-p将本地端口绑定到容器端口
-name为容器创建一个名称
9000表示MinIO服务地址，其上传调用的就是这个服务地址
9090表示MinIO的Web Console地址，Console监听的是一个动态的端口, 可以通过 --console-address ":port" 指定静态端口
/mnt/data/minio/data:/data表示将MiniIO的数据挂载到宿主机上
/mnt/data/minio/config:/root/.minio表示将MiniIO的配置文件挂载到宿主机上
MINIO_ROOT_USER=minioadmin表示MinIO部署的root用户的用户名(accessKey)，不写默认的用户名就是minioadmin
MINIO_ROOT_PASSWORD=minioadmin表示MinIO部署的root用户的密码(secretKey)，不写默认的密码就是minioadmin

3.2 部署

提前拉取镜像，docker pull quay.io/minio/minio:latest 这里采用最新稳定版镜像。

$ docker run \
>    -p 9000:9000 \
>    -p 9090:9090 \
>    --name minio \
>    -v /data/data:/data \
>    -v /data/minio/config:/root/.minio \
>    -e "MINIO_ROOT_USER=minioadmin" \
>    -e "MINIO_ROOT_PASSWORD=minioadmin" \
>    quay.io/minio/minio:RELEASE.2022-09-25T15-44-53Z server /data --console-address ":9090"
Unable to find image 'quay.io/minio/minio:latest' locally
latest: Pulling from minio/minio
d5d2e87c6892: Pull complete 
008dba906bf6: Pull complete 
e632dfd7f0e0: Pull complete 
bb2a22a09061: Pull complete 
19b7bd2519a4: Pull complete 
a2d565af0f6d: Pull complete 
f0067029ce19: Downloading [============>                                      ]  10.65MB/42.53MB

安装成功后如下：

$  docker ps
CONTAINER ID   IMAGE                 COMMAND                  CREATED         STATUS         PORTS                                                                                  NAMES
1aa0bf931300   quay.io/minio/minio   "/usr/bin/docker-ent…"   6 seconds ago   Up 5 seconds   0.0.0.0:9000->9000/tcp, :::9000->9000/tcp, 0.0.0.0:9090->9090/tcp, :::9090->9090/tcp   minio
[root@xiaohezi ~]#

3.3 登录Web Console

浏览器中输入http://ip:9090/，用户名密码登录系统。

四、MinIO单机部署（二进制单节点单驱动）

4.1 创建数据目录

$ mkdir /data/minio/data -p

4.2 下载二进制包

下载地址：https://www.minio.org.cn/download.shtml#/linux

$ wget https://dl.min.io/server/minio/release/linux-amd64/minio
$ chmod +x minio
$ mv minio /usr/local/bin/
$ minio  --version
minio version RELEASE.2022-09-25T15-44-53Z (commit-id=877bd95fa312c5282c3aa0b73c75af43af9c5914)
Runtime: go1.18.6 linux/amd64
License: GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
Copyright: 2015-2022 MinIO, Inc.

4.3 设置环境变量

配置用户名密码

$ export MINIO_ROOT_USER=minioadmin
$ export MINIO_ROOT_PASSWORD=minioadmin

4.4 创建服务启动文件（可选）

vim  /usr/lib/systemd/system/minio.service
[Unit]
Description=Minio service
Documentation=https://docs.minio.io/
[Service]
WorkingDirectory=/data/minio/run/
ExecStart=/usr/local/bin/minio --console-address ":9090" /data/minio/data
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target

4.5 启动

$ minio server --console-address ":9090" /data/minio/data
或者:
$ systemctl daemon-reload
$ systemctl start minio.service && systemctl status minio.service
$ systemctl enable  minio.service

$ minio server --console-address ":9090" /data/minio/data
WARNING: Detected Linux kernel version older than 4.0.0 release, there are some known potential performance problems with this kernel version. MinIO recommends a minimum of 4.x.x linux kernel version for best performance
Formatting 1st pool, 1 set(s), 1 drives per set.
WARNING: Host local has more than 0 drives of set. A host failure will result in data becoming unavailable.
WARNING: Detected default credentials 'minioadmin:minioadmin', we recommend that you change these values with 'MINIO_ROOT_USER' and 'MINIO_ROOT_PASSWORD' environment variables
MinIO Object Storage Server
Copyright: 2015-2022 MinIO, Inc.
License: GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
Version: RELEASE.2022-09-25T15-44-53Z (go1.18.6 linux/amd64)
Status:         1 Online, 0 Offline. 
API: http://192.168.0.4:9000  http://172.17.0.1:9000  http://127.0.0.1:9000       
RootUser: minioadmin 
RootPass: minioadmin 
Console: http://192.168.0.4:9090 http://172.17.0.1:9090 http://127.0.0.1:9090    
RootUser: minioadmin 
RootPass: minioadmin 
Command-line: https://docs.min.io/docs/minio-client-quickstart-guide
   $ mc alias set myminio http://192.168.0.4:9000 minioadmin minioadmin
Documentation: https://docs.min.io

通过上面可以看到用户名密码为minioadmin，默认启动的API端口是9000，API 默认端口可通过--address IP:PORT来指定；还有一个Web Console的端口，并且Console监听的是一个动态的端口, 可以通过 --console-address ":port" 指定静态端口。

默认的配置目录是{HOME}/.minio，可以通过 --config-dir 命令自定义配置目录

$ minio server -config-dir /data/minio/config  --console-address ":9090" --address ":9000" /data/minio/data

后台启动

nohup minio server -config-dir /data/minio/config  --console-address ":9090" --address ":9000" /data/minio/data >/data/minio/minio.log 2>&1 &

五、MinIO单机部署(docker-compose

单节点单驱动)

5.1 准备工作

编写docker-compose-minio.yaml文件，内容如下：

version: '3.7'
services:
  minio:
    image: quay.io/minio/minio:RELEASE.2022-09-25T15-44-53Z
    container_name: minio
    hostname: minio
    restart: always
    command: server --console-address ":9090" /data
    environment:
      MINIO_ROOT_USER: minioadmin
      MINIO_ROOT_PASSWORD: minioadmin
      TZ: Asia/Shanghai
    volumes:
      - /data/minio/data:/data
      - /data/minio/config:/root/.minio
    ports:
      - "9000:9000"
      - "9090:9090"
    networks:
      - pk_net
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3
networks:
  pk_net:
    external: true

自定义docker网络

### 预先创建一个自定义的网络pk_net,此处的10.139可以自定义，不冲突即可
$ sudo docker network create --driver bridge --subnet 10.139.0.0/16 --gateway 10.139.0.1 pk_net

参数说明：healthcheck表示健康探测 server --console-address ":9090" /data 表示指定的minio服务下面挂载的目标磁盘为/data，并且指定Web Console的端口

5.2 部署

$ docker-compose -f docker-compose-minio.yaml up -d

六、MinIO单机部署(docker-compose

单节点多驱动)

6.1 准备工作

挂载的磁盘增加到四个,单机版部署也可挂载多个磁盘，单个服务挂载超过(等于)4个磁盘，自动启动纠删码模式，可以预防磁盘损坏的情况下，导致文件不丢失。

调整docker-compose-minio.yaml文件，内容如下：

version: '3.7'
services:
  minio:
    image: quay.io/minio/minio:RELEASE.2022-09-25T15-44-53Z
    container_name: minio
    hostname: minio
    restart: always
    command: server --console-address ":9090" http://minio/data{1...4}
    environment:
      MINIO_ROOT_USER: minioadmin
      MINIO_ROOT_PASSWORD: minioadmin
      TZ: Asia/Shanghai
    volumes:
      - /data/minio/data1:/data1
      - /data/minio/data2:/data2
      - /data/minio/data3:/data3
      - /data/minio/data4:/data4
      - /data/minio/config:/root/.minio
    ports:
      - "9000:9000"
      - "9090:9090"
    networks:
      - pk_net
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3
networks:
  pk_net:
    external: true

参数说明：healthcheck表示健康探测 http://minio/data{1...4} 表示指定的minio服务下面挂载的目标磁盘为/data1、/data2、/data3和/data4 单机版部署也可挂载多个磁盘，单个服务挂载超过(等于)4个磁盘，自动启动纠删码模式，可以预防磁盘损坏的情况下，导致文件不丢失。

6.2 部署

$ docker-compose -f docker-compose-minio.yaml up -d

七、MinIO分布式部署(docker-compose模拟

多节点多驱动)

分布式 Minio 可以让你将多块硬盘（甚至是在不同的机器上）组成一个对象存储服务。由于硬盘分布在不同的节点上，分布式 Minio 避免了单点故障。

数据保护：分布式 Minio 采用纠删码来防范多个节点宕机和位衰减 bit rot。分布式 Minio 至少需要四块硬盘，使用分布式 Minio 自动引入了纠删码功能。MinIO 使用纠删码 erasure code 和校验和 checksum 来保护数据免受硬件故障和无声数据损坏。即便您丢失一半数量 (N/2) 的硬盘，依然可以使用。纠删码是一种恢复丢失和损坏数据的数学算法，MinIO 采用 Reed-Solomon code 将对象拆分成 N/2 数据和 N/2 奇偶校验块。这就意味着如果是 12 块硬盘，一个对象会被分成 6 个数据块、6 个奇偶校验块，你可以丢失任意 6 块（不管其是否存放的数据还是奇偶校验块），你仍可以从剩下的盘中的数据进行恢复。

高可用：单机节点存在单点故障，相反，如果是一个有 N 块硬盘的分布式 MinIO，只要有 N/2 块硬盘在线，你的数据就是安全的。不过你需要有 N/2 + 1 个硬盘来创建新的对象。例如，一个 16 个节点的 MinIO 集群，每个节点 16 块硬盘，就算 8 台服务器宕机，这个集群仍然是可读的，不过需要有9台服务器才能写数据

一致性：MinIO 在分布式和单机模式下，所有读写操作都严格遵守 read-adter-write 一致性模型

7.1 准备工作

调整docker-compose-minio.yaml文件，内容如下：

version: '3.7'
# Settings and configurations that are common for all containers
x-minio-common: &minio-common
  image: quay.io/minio/minio:RELEASE.2022-09-25T15-44-53Z
  command: server --console-address ":9090" http://minio{1...4}/data{1...4}
  expose:
    - "9000"
    - "9090"
  environment:
    MINIO_ROOT_USER: minioadmin
    MINIO_ROOT_PASSWORD: minioadmin
    TZ: Asia/Shanghai
  healthcheck:
    test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
    interval: 30s
    timeout: 20s
    retries: 3
    networks:
      - pk_net
# 启动4个minio服务器实例的docker容器
# 使用nginx反向代理，负载均衡，通过端口9000可以访问
# it through port 9000.
services:
  minio1:
    <<: *minio-common
    hostname: minio1
    container_name: minio1
    volumes:
      - /data1/data1:/data1
      - /data1/data2:/data2
      - /data1/data3:/data3
      - /data1/data4:/data4
  minio2:
    <<: *minio-common
    hostname: minio2
    container_name: minio2   
    volumes:
      - /data2/data1:/data1
      - /data2/data2:/data2
      - /data2/data3:/data3
      - /data2/data4:/data4
  minio3:
    <<: *minio-common
    hostname: minio3
    container_name: minio3
    volumes:
      - /data3/data1:/data1
      - /data3/data2:/data2
      - /data3/data3:/data3
      - /data3/data4:/data4
  minio4:
    <<: *minio-common
    hostname: minio4
    container_name: minio4
    volumes:
      - /data4/data1:/data1
      - /data4/data2:/data2
      - /data4/data3:/data3
      - /data4/data4:/data4
  nginx:
    image: nginx:1.19.2-alpine
    hostname: nginx
    container_name: nginx
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
    ports:
      - "9000:9000"
      - "9090:9090"
    depends_on:
      - minio1
      - minio2
      - minio3
      - minio4
networks:
  pk_net:
    external: true

参数说明：x-minio-common表示所有容器的设置和配置启动4个minio服务器实例的docker容器使用nginx反向代理，负载均衡，通过端口9000可以访问

nginx.conf内容如下：

user  nginx;
worker_processes  auto;
error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;
events {
    worker_connections  4096;
}
http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
    access_log  /var/log/nginx/access.log  main;
    sendfile        on;
    keepalive_timeout  65;
    # include /etc/nginx/conf.d/*.conf;
    upstream minio {
        server minio1:9000;
        server minio2:9000;
        server minio3:9000;
        server minio4:9000;
    }
    upstream console {
        ip_hash;
        server minio1:9090;
        server minio2:9090;
        server minio3:9090;
        server minio4:9090;
    }
    server {
        listen       9000;
        listen  [::]:9000;
        server_name  localhost;
        # To allow special characters in headers
        ignore_invalid_headers off;
        # Allow any size file to be uploaded.
        # Set to a value such as 1000m; to restrict file size to a specific value
        client_max_body_size 0;
        # To disable buffering
        proxy_buffering off;
        proxy_request_buffering off;
        location / {
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_connect_timeout 300;
            # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            chunked_transfer_encoding off;
            proxy_pass http://minio;
        }
    }
    server {
        listen       9090;
        listen  [::]:9090;
        server_name  localhost;
        # To allow special characters in headers
        ignore_invalid_headers off;
        # Allow any size file to be uploaded.
        # Set to a value such as 1000m; to restrict file size to a specific value
        client_max_body_size 0;
        # To disable buffering
        proxy_buffering off;
        proxy_request_buffering off;
        location / {
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-NginX-Proxy true;
            # This is necessary to pass the correct IP to be hashed
            real_ip_header X-Real-IP;
            proxy_connect_timeout 300;
            # To support websocket
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            chunked_transfer_encoding off;
            proxy_pass http://console;
        }
    }
}

7.2 部署

$ docker-compose -f docker-compose-minio.yaml up -d
Creating minio4 ... done
Creating minio2 ... done
Creating minio3 ... done
Creating minio1 ... done
Creating test_nginx_1 ... done
$ docker ps
CONTAINER ID   IMAGE                                              COMMAND                  CREATED         STATUS                            PORTS                                                                                          NAMES
a6bda0af2d24   nginx:1.19.2-alpine                                "/docker-entrypoint.…"   7 seconds ago   Up 6 seconds                      0.0.0.0:9000->9000/tcp, :::9000->9000/tcp, 80/tcp, 0.0.0.0:9090->9090/tcp, :::9090->9090/tcp   test_nginx_1
239c4bdeb151   quay.io/minio/minio:RELEASE.2022-09-25T15-44-53Z   "/usr/bin/docker-ent…"   9 seconds ago   Up 6 seconds (health: starting)   9000/tcp, 9090/tcp                                                                             minio1
0c5050be7fd3   quay.io/minio/minio:RELEASE.2022-09-25T15-44-53Z   "/usr/bin/docker-ent…"   9 seconds ago   Up 6 seconds (health: starting)   9000/tcp, 9090/tcp                                                                             minio2
e65601ce11c0   quay.io/minio/minio:RELEASE.2022-09-25T15-44-53Z   "/usr/bin/docker-ent…"   9 seconds ago   Up 7 seconds (health: starting)   9000/tcp, 9090/tcp                                                                             minio3
f75733c2da78   quay.io/minio/minio:RELEASE.2022-09-25T15-44-53Z   "/usr/bin/docker-ent…"   9 seconds ago   Up 7 seconds (health: starting)   9000/tcp, 9090/tcp                                                                             minio4

八、MinIO分布式部署(多节点多驱动)--生产推荐

在单机上部署只能保证磁盘损坏的情况下，文件不丢失，并不能解决单点故障的问题，所以我们下面为了避免单点故障导致服务不可用，把minio服务改成真正分布式部署。MinIO强烈推荐生产集群由 minimum 4 个组成 minio server 节点在服务器池。可以采用docker-compose部署，也可采用二进制部署，最后使用nginx进行负载均衡就行了。这里我们还是采用docker-compose为例。

8.1 准备工作

服务器1-4如下：

	IP	部署服务	备注
服务器1	192.168.0.1	minio	挂载4块磁盘
服务器2	192.168.0.2	minio	挂载4块磁盘
服务器3	192.168.0.3	minio	挂载4块磁盘
服务器4	192.168.0.4	minio	挂载4块磁盘
服务器5	192.168.0.5	nginx	nginx进行负载均衡

docker-compose-minio.yaml文件，内容如下：

version: '3.7'
services:
  minio:
    image: quay.io/minio/minio:RELEASE.2022-09-25T15-44-53Z
    container_name: minio
    hostname: minio
    restart: always
    command: server --console-address ":9090" http://192.168.0.1:9000/data{1...4}
    environment:
      MINIO_ROOT_USER: minioadmin
      MINIO_ROOT_PASSWORD: minioadmin
      TZ: Asia/Shanghai
    volumes:
      - /minio1/data1:/data1
      - /minio2/data2:/data2
      - /minio3/data3:/data3
      - /minio4/data4:/data4
      - /data/minio/config:/root/.minio
    ports:
      - "9000:9000"
      - "9090:9090"
    networks:
      - pk_net
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3
networks:
  pk_net:
    external: true

服务器1-4都采用以上docker-compose文件部署。/minio1/data1、/minio2/data2、/minio3/data2、/minio4/data2表示分别挂载的四块磁盘

docker-compose-nginx.yaml文件，内容如下：

version: '3.7'
services:
  nginx:
    image: nginx:1.19.2-alpine
    hostname: nginx
 container_name: nginx
    restart: always
    environment:
      TZ: Asia/Shanghai
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
    ports:
      - "9000:9000"
      - "9090:9090"
    networks:
      - pk_net
networks:
  pk_net:
    external: true

nginx.conf内容如下：

user  nginx;
worker_processes  auto;
error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;
events {
    worker_connections  4096;
}
http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
    access_log  /var/log/nginx/access.log  main;
    sendfile        on;
    keepalive_timeout  65;
    # include /etc/nginx/conf.d/*.conf;
    upstream minio {
        server 192.168.0.1:9000;
        server 192.168.0.2:9000;
        server 192.168.0.3:9000;
        server 192.168.0.4:9000;
    }
    upstream console {
        ip_hash;
        server 192.168.0.1:9090;
        server 192.168.0.2:9090;
        server 192.168.0.3:9090;
        server 192.168.0.4:9090;
    }
    server {
        listen       9000;
        listen  [::]:9000;
        server_name  localhost;
        # To allow special characters in headers
        ignore_invalid_headers off;
        # Allow any size file to be uploaded.
        # Set to a value such as 1000m; to restrict file size to a specific value
        client_max_body_size 0;
        # To disable buffering
        proxy_buffering off;
        proxy_request_buffering off;
        location / {
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_connect_timeout 300;
            # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            chunked_transfer_encoding off;
            proxy_pass http://minio;
        }
    }
    server {
        listen       9090;
        listen  [::]:9090;
        server_name  localhost;
        # To allow special characters in headers
        ignore_invalid_headers off;
        # Allow any size file to be uploaded.
        # Set to a value such as 1000m; to restrict file size to a specific value
        client_max_body_size 0;
        # To disable buffering
        proxy_buffering off;
        proxy_request_buffering off;
        location / {
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-NginX-Proxy true;
            # This is necessary to pass the correct IP to be hashed
            real_ip_header X-Real-IP;
            proxy_connect_timeout 300;
            # To support websocket
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            chunked_transfer_encoding off;
            proxy_pass http://console;
        }
    }
}

8.2 部署

分别部署四台minio服务

$ docker-compose -f docker-compose-minio.yaml up -d

部署nginx服务

$ docker-compose -f docker-compose-nginx.yaml up -d

九、优化

nginx也存在单节点故障，可以进行高可用配置。

非云上：

使用 Keepalived 和 HAproxy 负载minio
使用 Keepalived 和 Nginx 负载minio

云上：

使用 Keepalived 和 HAproxy 负载minio
使用 Keepalived 和 Nginx 负载minio
可以直接使用云上的lb，比如阿里云slb，腾讯云elb，青云lb等负载minio（推荐）

ps:有些云可能不支持keepalived

你好，MinIO