docker环境部署Hadoop集群教程
作者：Zhu758
文章介绍了在Docker环境下部署Hadoop集群的步骤,包括ocker-compose.yml配置文件创建、Docker容器启动集群验证集群状态数据持久化测试NamenodeDatanatannode数据持久化测试容器故障场景验证持久化效果
需求
在docker环境部署Hadoop集群，用于预生产环境测试。
操作流程
创建文件config
CORE-SITE.XML_fs.default.name=hdfs://namenode CORE-SITE.XML_fs.defaultFS=hdfs://namenode HDFS-SITE.XML_dfs.namenode.name.dir=/opt/hadoop/data/nn HDFS-SITE.XML_dfs.datanode.data.dir=/opt/hadoop/data/dn HDFS-SITE.XML_dfs.namenode.rpc-address=namenode:8020 HDFS-SITE.XML_dfs.replication=1 MAPRED-SITE.XML_mapreduce.framework.name=yarn MAPRED-SITE.XML_yarn.app.mapreduce.am.env=HADOOP_MAPRED_HOME=/opt/hadoop MAPRED-SITE.XML_mapreduce.map.env=HADOOP_MAPRED_HOME=/opt/hadoop MAPRED-SITE.XML_mapreduce.reduce.env=HADOOP_MAPRED_HOME=/opt/hadoop YARN-SITE.XML_yarn.resourcemanager.hostname=resourcemanager YARN-SITE.XML_yarn.nodemanager.pmem-check-enabled=false YARN-SITE.XML_yarn.nodemanager.delete.debug-delay-sec=600 YARN-SITE.XML_yarn.nodemanager.vmem-check-enabled=false YARN-SITE.XML_yarn.nodemanager.aux-services=mapreduce_shuffle CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.maximum-applications=10000 CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.maximum-am-resource-percent=0.1 CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.resource-calculator=org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.root.queues=default CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.root.default.capacity=100 CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.root.default.user-limit-factor=1 CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.root.default.maximum-capacity=100 CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.root.default.state=RUNNING CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.root.default.acl_submit_applications=* CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.root.default.acl_administer_queue=* CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.node-locality-delay=40 CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.queue-mappings= CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.queue-mappings-override.enable=false
创建文件docker-compose.yaml
volumes:
hadoop-nn-data:
name: hadoop-nn-data
hadoop-dn-data:
name: hadoop-dn-data
services:
namenode:
image: zhuyifeiruichuang/hadoop:3.1.1
hostname: namenode
container_name: hadoop-namenode
command: ["sh", "-c", "sudo mkdir -p /opt/hadoop/data/nn && sudo chown -R hadoop:hadoop /opt/hadoop/data && if [ ! -d /opt/hadoop/data/nn/current ]; then echo 'Formatting NameNode...' && hdfs namenode -format; fi && hdfs namenode"]
ports:
- 9870:9870
- 9000:9000
env_file:
- ./config
environment:
ENSURE_NAMENODE_DIR: "/opt/hadoop/data/nn"
volumes:
- hadoop-nn-data:/opt/hadoop/data/nn
restart: unless-stopped
healthcheck:
test: ["CMD-SHELL", "curl -f http://localhost:9870 || exit 1"]
interval: 10s
timeout: 5s
retries: 3
start_period: 60s
datanode:
image: zhuyifeiruichuang/hadoop:3.1.1
container_name: hadoop-datanode
command: ["sh", "-c", "sudo mkdir -p /opt/hadoop/data/dn && sudo chown -R hadoop:hadoop /opt/hadoop/data && hdfs datanode"]
env_file:
- ./config
ports:
- 9866:9866
- 9864:9864
volumes:
- hadoop-dn-data:/opt/hadoop/data/dn
depends_on:
- namenode
restart: unless-stopped
healthcheck:
test: ["CMD-SHELL", "curl -f http://localhost:9864 || exit 1"]
interval: 10s
timeout: 5s
retries: 3
start_period: 40s
resourcemanager:
image: zhuyifeiruichuang/hadoop:3.1.1
hostname: resourcemanager
container_name: hadoop-resourcemanager
command: ["yarn", "resourcemanager"]
ports:
- 8088:8088
- 8032:8032
env_file:
- ./config
depends_on:
- namenode
restart: unless-stopped
healthcheck:
test: ["CMD-SHELL", "curl -f http://127.0.0.1:8088 || exit 1"]
interval: 10s
timeout: 5s
retries: 3
start_period: 60s
nodemanager:
hostname: nodemanager
image: zhuyifeiruichuang/hadoop:3.1.1
container_name: hadoop-nodemanager
command: ["yarn", "nodemanager"]
env_file:
- ./config
ports:
- 8042:8042
depends_on:
- resourcemanager
- namenode
restart: unless-stopped
healthcheck:
test: ["CMD-SHELL", "curl -f http://localhost:8042 || exit 1"]
interval: 10s
timeout: 5s
retries: 3
start_period: 40s启动集群
docker compose up -d
查询集群状态
docker compose ps -a
数据持久化测试
以下方法用于证明数据持久化生效，可确保容器故障后不丢失数据。
测试namenode
以下方法证明namenode数据持久化生效。
# 1. 进入 NameNode 容器 docker exec -it hadoop-namenode bash # 2. 在 HDFS 中创建测试目录 hdfs dfs -mkdir /test_persistence # 3. 创建一个包含时间戳的测试文件，写入 HDFS echo "This is a test file created at $(date)" > /tmp/test_file.txt hdfs dfs -put /tmp/test_file.txt /test_persistence/namenode_test.txt # 4. 退出容器 exit # 5. 使用 down/up 重建 Namenode 服务 # 注意：这里只操作 Namenode docker compose stop namenode docker compose rm -f namenode # 6. 重新启动集群（这会重建并启动 Namenode） docker compose up -d namenode # 7. 进入 NameNode 容器 docker exec -it hadoop-namenode bash # 8. 尝试列出并查看刚才创建的文件 hdfs dfs -ls /test_persistence hdfs dfs -cat /test_persistence/namenode_test.txt # 9. 退出容器 exit
测试datanode
以下方法证明datanode数据持久化生效
模拟容器故障场景
确认容器状态正常
docker ps -a
在容器内写入测试数据
docker exec -it hadoop-namenode bash echo "验证DataNode数据持久化：容器重启后数据不丢失" > persist-test.txt hdfs dfs -put persist-test.txt / hdfs dfs -ls / hdfs dfs -cat /persist-test.txt exit
删除容器，确认容器真实删除
docker compose down docker ps -a
重建容器，确认容器正常
docker compose up -d docker compose ps -a
验证数据状态，
docker exec -it hadoop-namenode bash hdfs dfs -ls / hdfs dfs -cat /persist-test.txt exit
证明常规启停容器时，数据持久化生效。
模拟破坏删除容器，重建容器并测试数据，
docker rm -f hadoop-datanode docker compose up -d datanode docker exec -it hadoop-namenode hdfs dfs -cat /persist-test.txt
证明破坏删除容器时，数据持久化生效。
模拟数据卷故障场景
备份数据卷，确认备份成功
cp -r hadoop_hadoop-dn-data/ /opt/
删除容器，
docker compose down
删除数据卷，模拟数据卷被破坏，
docker volume rm hadoop_hadoop-dn-data
重建容器，
docker compose up -d
可见容器存在故障，已找不到数据，
删除容器，删除重建重启时的无数据的卷，新建空数据卷。注意：必须先有数据卷，才能再把数据复制过去，否则新建数据卷会使旧数据混乱。
docker compose down docker volume rm hadoop_hadoop-dn-data docker volume create hadoop_hadoop-dn-data
将备份数据转移到新数据卷，
cp -r hadoop_hadoop-dn-data/ /data/docker/volumes/
创建容器
docker compose up -d
验证数据状态
docker exec -it hadoop-namenode-1 hdfs dfs -cat /persist-test.txt
证明数据卷被破坏时，可通过数据备份恢复，实现数据持久化。
总结
以上为个人经验，希望能给大家一个参考，也希望大家多多支持脚本之家。