docker使用dockercompose编排一个Hadoop集群方式
作者：大鹏blog
这篇文章主要介绍了docker使用dockercompose编排一个Hadoop集群方式,具有很好的参考价值,希望对大家有所帮助,如有错误或未考虑完全的地方,望不吝赐教
使用dockercompose编排一个Hadoop集群
本文使用docker-compose 最简单的方式编排一个Hadoop集群
运行脚本文件
version: "3"
services:
namenode:
image: bde2020/hadoop-namenode:2.0.0-hadoop3.1.2-java8
container_name: namenode
ports:
- 9870:9870
volumes:
- hadoop_namenode:/hadoop/dfs/name
environment:
- CLUSTER_NAME=test
env_file:
- ./hadoop.env
datanode:
image: bde2020/hadoop-datanode:2.0.0-hadoop3.1.2-java8
container_name: datanode
volumes:
- hadoop_datanode:/hadoop/dfs/data
environment:
SERVICE_PRECONDITION: "namenode:9870"
env_file:
- ./hadoop.env
resourcemanager:
image: bde2020/hadoop-resourcemanager:2.0.0-hadoop3.1.2-java8
container_name: resourcemanager
environment:
SERVICE_PRECONDITION: "namenode:9870 datanode:9864"
env_file:
- ./hadoop.env
nodemanager1:
image: bde2020/hadoop-nodemanager:2.0.0-hadoop3.1.2-java8
container_name: nodemanager
environment:
SERVICE_PRECONDITION: "namenode:9870 datanode:9864 resourcemanager:8088"
env_file:
- ./hadoop.env
historyserver:
image: bde2020/hadoop-historyserver:2.0.0-hadoop3.1.2-java8
container_name: historyserver
environment:
SERVICE_PRECONDITION: "namenode:9870 datanode:9864 resourcemanager:8088"
volumes:
- hadoop_historyserver:/hadoop/yarn/timeline
env_file:
- ./hadoop.env
volumes:
hadoop_namenode:
hadoop_datanode:
hadoop_historyserver:
运行方式
docker-compose up
运行
wordcount job make wordcount
相关的访问地址
Namenode: http://<dockerhadoop_IP_address>:9870/dfshealth.html#tab-overview History server: http://<dockerhadoop_IP_address>:8188/applicationhistory Datanode: http://<dockerhadoop_IP_address>:9864/ Nodemanager: http://<dockerhadoop_IP_address>:8042/node Resource manager: http://<dockerhadoop_IP_address>:8088/
配置环境变量
配置参数可以在hadoop.env文件中指定
也可以作为特定服务（如namenode、datanode等）的环境变量指定：
可用配置包括
/etc/hadoop/core-site.xml CORE_CONF /etc/hadoop/hdfs-site.xml HDFS_CONF /etc/hadoop/yarn-site.xml YARN_CONF /etc/hadoop/httpfs-site.xml HTTPFS_CONF /etc/hadoop/kms-site.xml KMS_CONF /etc/hadoop/mapred-site.xml MAPRED_CONF
总结
以上为个人经验，希望能给大家一个参考，也希望大家多多支持脚本之家。