代码仓库
会同步代码到 GitHub
https://github.com/turbo-duck/flink-demo
本节内容
在学习和测试阶段，需要快速启动一套Flink供我们学习测试。使用常规的安装的方式太费劲了，利用Docker可以快速启动一套完整的系统，方便我们的进行测试！
这里放一些我平常常用的 docker-compose 的文件。供大家学习和测试！
这些都是我们之前要用到的内容：
Flink 部署
Kafka 部署
Redis 部署
Flink 单节点
1个 job manager
1个 task manager
version: '2' services: jobmanager: image: 10.10.52.8/flink/flink expose: - "6123" ports: - "8081:8081" command: jobmanager environment: - JOB_MANAGER_RPC_ADDRESS=jobmanager taskmanager: image: 10.10.52.8/flink/flink expose: - "6121" - "6122" depends_on: - jobmanager command: taskmanager links: - "jobmanager:jobmanager" environment: - JOB_MANAGER_RPC_ADDRESS=jobmanager
启动方式
docker-compose up -d
查看日志
docker-compose logs -f
运行效果
Flink高可用
通过 zookeeper 做的高可用
- 2个 job manager
- 3个 task manager
- 1个 zookeeper
version: '2' services: zookeeper: image: zookeeper:3.5 ports: - "2181:2181" jobmanager1: image: 10.10.52.8/flink/flink expose: - "6123" ports: - "8081:8081" command: jobmanager environment: - JOB_MANAGER_RPC_ADDRESS=jobmanager1 - ZOOKEEPER_SERVERS=zookeeper:2181 - HIGH_AVAILABILITY=zookeeper - HIGH_AVAILABILITY_ZOOKEEPER_QUORUM=zookeeper:2181 - HIGH_AVAILABILITY_STORAGE_DIR=file:///flink/ha/ jobmanager2: image: 10.10.52.8/flink/flink expose: - "6123" ports: - "8082:8081" command: jobmanager environment: - JOB_MANAGER_RPC_ADDRESS=jobmanager2 - ZOOKEEPER_SERVERS=zookeeper:2181 - HIGH_AVAILABILITY=zookeeper - HIGH_AVAILABILITY_ZOOKEEPER_QUORUM=zookeeper:2181 - HIGH_AVAILABILITY_STORAGE_DIR=file:///flink/ha/ taskmanager1: image: 10.10.52.8/flink/flink expose: - "6121" - "6122" depends_on: - jobmanager1 - jobmanager2 command: taskmanager links: - "jobmanager1:jobmanager1" - "jobmanager2:jobmanager2" - "zookeeper:zookeeper" environment: - JOB_MANAGER_RPC_ADDRESS=jobmanager1 - ZOOKEEPER_SERVERS=zookeeper:2181 - HIGH_AVAILABILITY=zookeeper - HIGH_AVAILABILITY_ZOOKEEPER_QUORUM=zookeeper:2181 taskmanager2: image: 10.10.52.8/flink/flink expose: - "6121" - "6122" depends_on: - jobmanager1 - jobmanager2 command: taskmanager links: - "jobmanager1:jobmanager1" - "jobmanager2:jobmanager2" - "zookeeper:zookeeper" environment: - JOB_MANAGER_RPC_ADDRESS=jobmanager1 - ZOOKEEPER_SERVERS=zookeeper:2181 - HIGH_AVAILABILITY=zookeeper - HIGH_AVAILABILITY_ZOOKEEPER_QUORUM=zookeeper:2181 taskmanager3: image: 10.10.52.8/flink/flink expose: - "6121" - "6122" depends_on: - jobmanager1 - jobmanager2 command: taskmanager links: - "jobmanager1:jobmanager1" - "jobmanager2:jobmanager2" - "zookeeper:zookeeper" environment: - JOB_MANAGER_RPC_ADDRESS=jobmanager1 - ZOOKEEPER_SERVERS=zookeeper:2181 - HIGH_AVAILABILITY=zookeeper - HIGH_AVAILABILITY_ZOOKEEPER_QUORUM=zookeeper:2181
启动方式
docker-compose up -d
查看日志
docker-compose logs -f
Flink多节点
可以不做高可用，直接 1个 JOB 对应 3个 TASK 节点。
但是如果JOB节点挂掉了，TASK节点将无法工作！！！
- 1个 job manager
- 3个 task manager
version: '2' services: jobmanager: image: 10.10.52.8/flink/flink expose: - "6123" ports: - "8081:8081" command: jobmanager environment: - JOB_MANAGER_RPC_ADDRESS=jobmanager taskmanager1: image: 10.10.52.8/flink/flink expose: - "6121" - "6122" depends_on: - jobmanager command: taskmanager links: - "jobmanager:jobmanager" environment: - JOB_MANAGER_RPC_ADDRESS=jobmanager taskmanager2: image: 10.10.52.8/flink/flink expose: - "6121" - "6122" depends_on: - jobmanager command: taskmanager links: - "jobmanager:jobmanager" environment: - JOB_MANAGER_RPC_ADDRESS=jobmanager taskmanager3: image: 10.10.52.8/flink/flink expose: - "6121" - "6122" depends_on: - jobmanager command: taskmanager links: - "jobmanager:jobmanager" environment: - JOB_MANAGER_RPC_ADDRESS=jobmanager
Kafka 部署
注意！KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://10.10.5.68:9092 要修改为自己的IP！
version: "3" services: zookeeper: image: 'bitnami/zookeeper:latest' ports: - '2181:2181' environment: - ALLOW_ANONYMOUS_LOGIN=yes kafka: image: 'bitnami/kafka:2.8.0' ports: - '9092:9092' - '9999:9999' environment: - KAFKA_BROKER_ID=1 - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092 # 注意这里的IP 改成自己的！！！ - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://10.10.5.68:9092 - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181 - ALLOW_PLAINTEXT_LISTENER=yes - KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE=true - KAFKA_CFG_LOG_RETENTION_HOURS=6 - JMX_PORT=9999 depends_on: - zookeeper
Redis 部署
version: "3" services: redis: image: "redis:latest" container_name: "redis_container" ports: - "6379:6379"
