docker

关注公众号 jb51net

关闭
首页 > 网站技巧 > 服务器 > 云和虚拟化 > docker > docker 部署Flink

docker 部署Flink和传统部署的详细过程

作者：❀͜͡傀儡师

Flink是一个开源的分布式流处理框架,支持状态管理、事件时间、窗口模型等特性,可以通过下载、解压、配置和启动来使用,提供了多种运行方式,包括Docker和SQLClient,本文介绍docker部署Flink和传统部署的详细过程,感兴趣的朋友跟随小编一起看看吧

简单介绍

Flink 是一个开源的、分布式的、高性能的流处理框架，主要特性有：状态管理与容错，事件时间与水位线，先进的窗口模型，轻量级的分布式快照，端到端的精确一次交付，流批一体与统一的 API，复杂的流处理拓扑与高级 API（侧输出，CEP：复杂事件处理库，Table API & SQL）等
下载地址: https://mirrors.aliyun.com/apache/flink/ z根据需求选版本

下载安装包

https://mirrors.aliyun.com/apache/flink/flink-1.20.3/flink-1.20.3-bin-scala_2.12.tgz

解压

tra -zxf flink-1.20.3-bin-scala_2.12.tgz

修改配置文件

jobmanager.rpc.address: master
jobmanager.rpc.port: 6123
jobmanager.bind-host: 0.0.0.0
jobmanager.memory.process.size: 1600m
taskmanager.bind-host: 0.0.0.0
taskmanager.host: master
taskmanager.memory.process.size: 1728m
taskmanager.numberOfTaskSlots: 2
parallelism.default: 2
jobmanager.execution.failover-strategy: region
rest.address: localhost
rest.bind-address: 0.0.0.0

启动

bin/start-cluster.sh

登录首页： http://192.168.115.102:8081

Docker Run

拉取镜像

x86架构
docker pull registry.cn-hangzhou.aliyuncs.com/qiluo-images/flink
arm架构
docker pull registry.cn-hangzhou.aliyuncs.com/qiluo-images/linux_arm64_flink
  1. 创建网络
docker network create flink-network
  1. 启动JobManager
docker run -d \
  --name=flink-jobmanager \
  --network=flink-network \
  -p 8081:8081 \
  -e FLINK_PROPERTIES="jobmanager.rpc.address: flink-jobmanager" \
  -v /data/flink:/opt/flink/data \
  registry.cn-hangzhou.aliyuncs.com/qiluo-images/flink \
  jobmanager
  1. 启动TaskManager
docker run -d \
  --name=flink-taskmanager \
  --network=flink-network \
  -e FLINK_PROPERTIES="jobmanager.rpc.address: flink-jobmanager" \
  -v /data/flink:/opt/flink/data \
  registry.cn-hangzhou.aliyuncs.com/qiluo-images/flink \
  taskmanager

带依赖的完整版本
docker-compose-full.yml

version: "3.8"
services:
  jobmanager:
    image: registry.cn-hangzhou.aliyuncs.com/qiluo-images/flink
    container_name: jobmanager
    ports:
      - "8081:8081"
      - "6123:6123"
    command: jobmanager
    environment:
      - FLINK_PROPERTIES=jobmanager.rpc.address: jobmanager
    volumes:
      - ./data/jobmanager:/opt/flink/data
      - ./logs/jobmanager:/opt/flink/log
      - ./apps:/opt/flink/apps
    networks:
      - flink-network
  taskmanager:
    image: flink:1.17.2
    container_name: taskmanager
    depends_on:
      - jobmanager
    command: taskmanager
    scale: 2
    environment:
      - FLINK_PROPERTIES=jobmanager.rpc.address: jobmanager
      - TASK_MANAGER_NUMBER_OF_TASK_SLOTS=4
    volumes:
      - ./data/taskmanager:/opt/flink/data
      - ./logs/taskmanager:/opt/flink/log
      - ./apps:/opt/flink/apps
    networks:
      - flink-network
  zookeeper:
    image: registry.cn-hangzhou.aliyuncs.com/qiluo-images/zookeeper
    container_name: zookeeper
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
    ports:
      - "2181:2181"
    networks:
      - flink-network
  kafka:
    image: registry.cn-hangzhou.aliyuncs.com/qiluo-images/kafka
    container_name: kafka
    depends_on:
      - zookeeper
    environment:
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092
      - ALLOW_PLAINTEXT_LISTENER=yes
    ports:
      - "9092:9092"
    networks:
      - flink-network
networks:
  flink-network:
    driver: bridge
volumes:
  flink-data:

使用SQL Client的完整环境
docker-compose-sql.yml

version: "3.8"
services:
  jobmanager:
    image: registry.cn-hangzhou.aliyuncs.com/qiluo-images/flink
    container_name: flink-jobmanager
    ports:
      - "8081:8081"
      - "6123:6123"
    command: jobmanager
    environment:
      - FLINK_PROPERTIES=
          jobmanager.rpc.address: jobmanager
          taskmanager.numberOfTaskSlots: 4
          parallelism.default: 2
    volumes:
      - ./data/jobmanager:/opt/flink/data
      - ./logs/jobmanager:/opt/flink/log
    networks:
      - flink-network
  taskmanager:
    image: registry.cn-hangzhou.aliyuncs.com/qiluo-images/flink
    container_name: flink-taskmanager
    depends_on:
      - jobmanager
    command: taskmanager
    scale: 2
    environment:
      - FLINK_PROPERTIES=
          jobmanager.rpc.address: jobmanager
          taskmanager.numberOfTaskSlots: 4
          parallelism.default: 2
    volumes:
      - ./data/taskmanager:/opt/flink/data
      - ./logs/taskmanager:/opt/flink/log
    networks:
      - flink-network
  sql-client:
    image: registry.cn-hangzhou.aliyuncs.com/qiluo-images/flink
    container_name: flink-sql-client
    depends_on:
      - jobmanager
    command: |
      bash -c "
      echo 'Waiting for Flink cluster to be ready...'
      while ! nc -z jobmanager 8081; do
        sleep 1
      done
      echo 'Starting SQL Client...'
      ./bin/sql-client.sh"
    volumes:
      - ./sql-scripts:/opt/flink/sql-scripts
    networks:
      - flink-network
    tty: true
    stdin_open: true
networks:
  flink-network:
    driver: bridge

到此这篇关于docker 部署Flink和传统部署的详细过程的文章就介绍到这了,更多相关docker 部署Flink内容请搜索脚本之家以前的文章或继续浏览下面的相关文章希望大家以后多多支持脚本之家！

您可能感兴趣的文章:
阅读全文