ubuntu系统使用docker gitlab 磁盘空间满的问题及解决
作者：一只海星
问题
发现git上传不了，登录服务器，table补齐命令都报错，发现根目录磁盘使用率100%
解决问题
问题排查：
1.使用排查docker使用空间
docker system df
未见异常
2.清理系统缓存
sudo apt autoremove rm -rf ~/.cache/thumbnails/* sudo rm -rf ~/.local/share/Trash/* ....
等等一顿操作下来，还是100%,reboot
发现界面都进不去了！
屮！
3.进入终端
Ctrl+Alt+F2
登录终端
4.查找大文件夹
sudo du -hsx /* | sort -rh | head -10
一层层找进去，在/var/lib/docker/containers目录发发现了一个日志文件*-json.log占了几百个G
我屮艸芔茻！！！！
删掉，重启，系统启动正常
but，docker gitlab还是起不来，一直starting
一直报错…
[+] Running 2/1
⠿ Network docker_gitlab_default Created 0.1s
⠿ Container gitlab Created 0.1s
Attaching to gitlab
gitlab | Thank you for using GitLab Docker Image!
gitlab | Current version: gitlab-ce=11.1.4-ce.0
gitlab |
gitlab | Configure GitLab for your system by editing /etc/gitlab/gitlab.rb file
gitlab | And restart this container to reload settings.
gitlab | To do it use docker exec:
gitlab |
gitlab | docker exec -it gitlab vim /etc/gitlab/gitlab.rb
gitlab | docker restart gitlab
gitlab |
gitlab | For a comprehensive list of configuration options please see the Omnibus GitLab readme
gitlab | https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.md
gitlab |
gitlab | If this container fails to start due to permission problems try to fix it by executing:
gitlab |
gitlab | docker exec -it gitlab update-permissions
gitlab | docker restart gitlab
gitlab |gitlab | Preparing services...
gitlab | Starting services...
gitlab | Configuring GitLab package...
gitlab | /opt/gitlab/embedded/bin/runsvdir-start: line 24: ulimit: pending signals: cannot modify limit: Operation not permitted
gitlab | /opt/gitlab/embedded/bin/runsvdir-start: line 37: /proc/sys/fs/file-max: Read-only file system
gitlab | JSON file with existing configuration not found inside /opt/gitlab/embedded/nodes.
gitlab | Skipping config check.
gitlab | gitlab preinstall:
gitlab | gitlab preinstall: This node does not appear to be running a database
gitlab | gitlab preinstall: Skipping version check, if you think this is an error exit now
gitlab | gitlab preinstall:
gitlab | Configuring GitLab...
使用docker exec -it gitlab update-permissions等操作，然并nuan~
找到最后持续输出的打印如下：
gitlab | ==> /var/log/gitlab/postgresql/current <==
gitlab | 2023-02-16_07:24:22.32111 FATAL: lock file "postmaster.pid" is empty
gitlab | 2023-02-16_07:24:22.32112 HINT: Either another server is starting, or the lock file is the remnant of a previous server startup crash.
gitlab | 2023-02-16_07:24:23.32501 FATAL: lock file "postmaster.pid" is empty
gitlab | 2023-02-16_07:24:23.32502 HINT: Either another server is starting, or the lock file is the remnant of a previous server startup crash.
gitlab | 2023-02-16_07:24:24.32886 FATAL: lock file "postmaster.pid" is empty
gitlab | 2023-02-16_07:24:24.32887 HINT: Either another server is starting, or the lock file is the remnant of a previous server startup crash.
gitlab | 2023-02-16_07:24:25.33276 FATAL: lock file "postmaster.pid" is empty
gitlab | 2023-02-16_07:24:25.33278 HINT: Either another server is starting, or the lock file is the remnant of a previous server startup crash.
进入docker 容器，在根目录find这个文件postmaster.pid,删之…，重启容器，一切正常！
切记
磁盘空间满先不要不要乱删东西！！！
一劳永逸解决问题
更改docker日志全局设置
在主机修改docker配置问题，只对新建docker容器有效。
# 排查Docker容器日志输出情况 docker inspect --format {{.LogPath}} 容器名/容器ID
vim /etc/docker/daemon.json
{
"registry-mirrors": [
"https://mirror.ccs.tencentyun.com",
"https://10odyesl.mirror.aliyuncs.com"
],
"log-driver":"json-file",
"log-opts": {"max-size":"300m", "max-file":"3"}
}
systemctl restart docker
systemctl daemon-reload 更改容器配置（docker-compose）
修改 docker-compose.yml，增加如下配置。yml分级按两个空格缩进，可以使用在线yml格式校验
logging: driver: "json-file" options: max-size: "50m" max-file: "2
查看配置：docker inspect -f ‘{{.HostConfig.LogConfig}}’ 容器名xxx
总结
以上为个人经验，希望能给大家一个参考，也希望大家多多支持脚本之家。