报错信息:
harbor-db容器重启报错:initdb: error: directory “/var/lib/postgresql/data/pg13“ exists but is not empty
现象:
在重启docker服务或者通过docker-compose重启harbor服务时,harbor-db容器都一直处于Restarting状态,无法恢复UP状态。
Harbor版本:v2.3.2
查看日志:
1.查看harbor所有容器启动日志路径
[root@k8s-master harbor]# vim harbor.yml ... 112 # Log configurations 113 log: 114 # options are debug, info, warning, error, fatal 115 level: info 116 # configs for logs in local storage 117 local: 118 # Log files are rotated log_rotate_count times before being removed. If count is 0, old versions are remov ed rather than rotated. 119 rotate_count: 50 120 # Log files are rotated only if they grow bigger than log_rotate_size bytes. If size is followed by k, the size is assumed to be in kilobytes. 121 # If the M is used, the size is in megabytes, and if G is used, the size is in gigabytes. So size 100, siz e 100k, size 100M and size 100G 122 # are all valid. 123 rotate_size: 200M 124 # The directory on your host that store log 125 location: /var/log/harbor //Harbor所有容器运行的日志存放路径
2.查看harbor-db容器启动日志
[root@k8s-master harbor]# cat /var/log/harbor/postgresql.log | tail -20 ... Oct 13 08:14:22 172.23.0.1 postgresql[28777]: This user must also own the server process. Oct 13 08:14:22 172.23.0.1 postgresql[28777]: Oct 13 08:14:22 172.23.0.1 postgresql[28777]: The database cluster will be initialized with locales Oct 13 08:14:22 172.23.0.1 postgresql[28777]: COLLATE: en_US.UTF-8 Oct 13 08:14:22 172.23.0.1 postgresql[28777]: CTYPE: en_US.UTF-8 Oct 13 08:14:22 172.23.0.1 postgresql[28777]: MESSAGES: C Oct 13 08:14:22 172.23.0.1 postgresql[28777]: MONETARY: C Oct 13 08:14:22 172.23.0.1 postgresql[28777]: NUMERIC: C Oct 13 08:14:22 172.23.0.1 postgresql[28777]: TIME: C Oct 13 08:14:22 172.23.0.1 postgresql[28777]: The default text search configuration will be set to "english". Oct 13 08:14:22 172.23.0.1 postgresql[28777]: Oct 13 08:14:22 172.23.0.1 postgresql[28777]: Data page checksums are disabled. Oct 13 08:14:22 172.23.0.1 postgresql[28777]: Oct 13 08:14:22 172.23.0.1 postgresql[28777]: initdb: error: directory "/var/lib/postgresql/data/pg13" exists but is not empty Oct 13 08:14:22 172.23.0.1 postgresql[28777]: If you want to create a new database system, either remove or empty Oct 13 08:14:22 172.23.0.1 postgresql[28777]: the directory "/var/lib/postgresql/data/pg13" or run initdb Oct 13 08:14:22 172.23.0.1 postgresql[28777]: with an argument other than "/var/lib/postg
解决:
1.从上面日志中的提示directory "/var/lib/postgresql/data/pg13" exists but is not empty可以清楚/var/lib/postgresql/data/pg13这个目录存在,并且不为空,但是在宿主机上找不到这个目录。
[root@k8s-master harbor]# ll /var/lib/postgresql/data/pg13 ls: 无法访问/var/lib/postgresql/data/pg13: 没有那个文件或目录
2.于是想到查看一下docker-compose的yaml文件,查看一下harbor-db容器的挂载目录的信息。
[root@k8s-master harbor]# vim docker-compose.yml ... postgresql: image: goharbor/harbor-db:v2.3.2 container_name: harbor-db restart: always cap_drop: - ALL cap_add: - CHOWN - DAC_OVERRIDE - SETGID - SETUID volumes: - /home/harbor/data/database:/var/lib/postgresql/data:z //挂载目录信息 networks: harbor: dns_search: . env_file: - ./common/config/db/env depends_on: - log logging: driver: "syslog" options: syslog-address: "tcp://127.0.0.1:1514" tag: "postgresql" ...
3.从docker-compose.yaml文件中可以看到宿主机上的/home/harbor/data/database目录挂载到容器中的/var/lib/postgresql/data目录下,那会不会是/home/harbor/data/database目录不为空呢?删除掉它试试。
[root@k8s-master harbor]# rm -rf /home/harbor/data/database
4.删除/home/harbor/data/database目录后将harbor运行的容器删除,在重新创建并启动。
[root@k8s-master harbor]# docker-compose down -v Stopping harbor-jobservice ... done Stopping nginx ... done Stopping harbor-core ... done Stopping redis ... done Stopping harbor-portal ... done Stopping harbor-db ... done Stopping registry ... done Stopping registryctl ... done Stopping harbor-log ... done Removing harbor-jobservice ... done Removing nginx ... done Removing harbor-core ... done Removing redis ... done Removing harbor-portal ... done Removing harbor-db ... done Removing registry ... done Removing registryctl ... done Removing harbor-log ... done Removing network harbor_harbor [root@k8s-master harbor]# docker-compose up -d Creating nginx ... done Creating registryctl ... Creating registry ... Creating harbor-db ... Creating redis ... Creating harbor-portal ... Creating harbor-core ... Creating harbor-jobservice ... Creating nginx ... [root@k8s-master harbor]# docker-compose ps Name Command State Ports -------------------------------------------------------------------------------------- harbor-core /harbor/entrypoint.sh Up harbor-db /docker-entrypoint.sh Up harbor-jobservice /harbor/entrypoint.sh Up harbor-log /bin/sh -c /usr/local/bin/ ... Up 127.0.0.1:1514->10514/tcp harbor-portal nginx -g daemon off; Up nginx nginx -g daemon off; Up 0.0.0.0:80->8080/tcp redis redis-server /etc/redis.conf Up registry /home/harbor/entrypoint.sh Up registryctl /home/harbor/start.sh Up
可以看到harbor-db已经启动成功,恢复正常了。
原因:
1.出现如上的原因很大可能是之前在这台机器上安装过Harbor服务,第一次安装的时候在挂载目录下创建了数据,再次安装时需要将之前的脏数据都清空才可以;
2.还遇到过一种情况,那就是在清理了挂载目录的脏数据之后重启docker服务或者使用docker-compose重启Harbor服务时,harbor-db容器还是一直处于Restarting状态,这种情况下可以将Harbor所有容器都down -v删除掉,修改harbor.yaml中data_volume数据挂载目录,再重新up -d(成功过)。或者换一个Harbor的版本,实测Harbor v2.1.1版本可以(之前是Harbor v2.3.2)。