要在 Flink SQL 中使用 RocksDB 存储引擎,需要进行以下环境配置:
在 Flink 的配置文件(例如 flink.properties 或 yarn-site.xml)中,设置以下参数来启用 RocksDB 存储引擎:
streamExecutionEnvironment.defaultJobManagerAddress=ip:port
streamExecutionEnvironment.defaultTaskManagerAddress=ip:port
streamExecutionEnvironment.jobManagerAddress=ip:port
streamExecutionEnvironment.taskManagerAddress=ip:port
streamExecutionEnvironment.jobManagerHttpAddress=ip:port
streamExecutionEnvironment.jobManagerRmiAddress=ip:port
streamExecutionEnvironment.jobManagerPasswordFile=/path/to/password/file
streamExecutionEnvironment.jobManagerUser=userName
streamExecutionEnvironment.taskManagerUser=userName
streamExecutionEnvironment.taskManagerPassword=password
streamExecutionEnvironment.jobManagerSecurityProtocol=SSL
streamExecutionEnvironment.taskManagerSecurityProtocol=SSL
streamExecutionEnvironment.jobManagerSaslMechanism=GSSAPI
streamExecutionEnvironment.taskManagerSaslMechanism=GSSAPI
streamExecutionEnvironment.jobManagerSaslJaasConfig=org.apache.flink.api.common.security.GSSSVCAuthenticatorConfiguration@54148e91
streamExecutionEnvironment.taskManagerSaslJaasConfig=org.apache.flink.api.common.security.GSSSVCAuthenticatorConfiguration@54148e91
streamExecutionEnvironment.hdfs.rootDirectory=/path/to/hdfs/directory
streamExecutionEnvironment.hdfs.defaultFS=hdfs://namenode:port
streamExecutionEnvironment.hdfs.jobSubmissionDirectory=/path/to/job/directory
streamExecutionEnvironment.hdfs.jobHistoryDirectory=/path/to/job/history/directory
streamExecutionEnvironment.hdfs.jobCheckpointDirectory=/path/to/job/checkpoint/directory
streamExecutionEnvironment.hdfs.jobCheckpointBackupDirectory=/path/to/job/checkpoint/backup/directory
streamExecutionEnvironment.hdfs.jobLogDirectory=/path/to/job/log/directory
streamExecutionEnvironment.hdfs.fileSystem=hdfs://namenode:port
streamExecutionEnvironment.hdfs.jobManager.webPort=port
streamExecutionEnvironment.hdfs.jobManager.webAddress=ip:port
streamExecutionEnvironment.hdfs.jobManager.administrationWebPort=port
streamExecutionEnvironment.hdfs.jobManager.administrationWebAddress=ip:port
streamExecutionEnvironment.hdfs.jobHistory.webPort=port
streamExecutionEnvironment.hdfs.jobHistory.webAddress=ip:port
streamExecutionEnvironment.hdfs.jobHistory.administrationWebPort=port
streamExecutionEnvironment.hdfs.jobHistory.administrationWebAddress=ip:port
streamExecutionEnvironment.hdfs.jobCheckpointWebPort=port
streamExecutionEnvironment.hdfs.jobCheckpointWebAddress=ip:port
streamExecutionEnvironment.hdfs.jobCheckpoint.administrationWebPort=port
streamExecutionEnvironment.hdfs.jobCheckpoint.administrationWebAddress=ip:port
streamExecutionEnvironment.hdfs.jobManager.webPasswordFile=/path/to/password/file
streamExecutionEnvironment.hdfs.jobManager.administrationWebPasswordFile=/path/to/password/file
streamExecutionEnvironment.hdfs.jobHistory.webPasswordFile=/path/to/password/file
streamExecutionEnvironment.hdfs.jobHistory.administrationWebPasswordFile=/path/to/password/file
streamExecutionEnvironment.hdfs.jobCheckpoint.webPasswordFile=/path/to/password/file
streamExecutionEnvironment.hdfs.jobCheckpoint.administrationWebPasswordFile=/path/to/password/file
streamExecutionEnvironment.jobGraphStorePort=port
streamExecutionEnvironment.jobGraphStoreAddress=ip:port
streamExecutionEnvironment.jobGraphStorePasswordFile=/path/to/password/file
streamExecutionEnvironment.jobGraphStoreUser=userName
streamExecutionEnvironment.jobGraphStorePassword=password
streamExecutionEnvironment.jobGraphStoreConfiguration=/path/to/jobgraphstore/config
streamExecutionEnvironment
在 Flink SQL 的配置文件(例如 sql.properties 或 sql-yarn-site.xml)中,设置以下参数来启用 RocksDB 存储引擎:
stream.sql.catalogImplementation=org.apache.flink.table.catalog.CatalogImpl
stream.sql.catalog.type=RocksDB
stream.sql.catalog.rocksdb.path=/path/to/rocksdb/db
stream.sql.catalog.rocksdb.tablePrefix=my_table_
stream.sql.catalog.rocksdb.tableType=GLOBAL
stream.sql.catalog.rocksdb.columnFamilyOptions.maxVersions=10
stream.sql.catalog.rocksdb.columnFamilyOptions.timeToLiveSeconds=3600
stream.sql.catalog.rocksdb.columnFamilyOptions.maxTimeToLiveSeconds=7200
stream.sql.catalog.rocksdb.columnFamilyOptions.writeBufferSizeBytes=4096
stream.sql.catalog.rocksdb.columnFamilyOptions.blockCacheEnabled=true
stream.sql.catalog.rocksdb.columnFamilyOptions.blockCacheSizeBytes=2048000
stream.sql.catalog.rocksdb.columnFamilyOptions.flushOnPut=true
stream.sql.catalog.rocksdb.columnFamilyOptions.compactionStrategyClassName=com.rocksdb.memtable.DefaultMemTableConfiguration@545a0a85
stream.sql.catalog.rocksdb.columnFamilyOptions.compactionStrategyProperties.put泱Threshold=5000
stream.sql.catalog.rocksdb.columnFamilyOptions.compactionStrategyProperties.sstFileSize=1073741824
stream.sql.catalog.rocksdb.columnFamilyOptions.compactionStrategyProperties.level0FileNumThreshold=10
stream.sql.catalog.rocksdb.columnFamilyOptions.compactionStrategyProperties.level0SlowdownWriteThreshold=60
stream.sql.catalog.rocksdb.columnFamilyOptions.compactionStrategyProperties.level0StopWriteThreshold=90
stream.sql.catalog.rocksdb.columnFamilyOptions.compactionStrategyProperties.minFilesForCompaction=4
stream.sql.catalog.rocksdb.columnFamilyOptions.compactionStrategyProperties.maxFilesForCompaction=8
stream.sql.catalog.rocksdb.columnFamilyOptions.compactionStrategyProperties.maxCompactionBytes=268435456
stream.sql.catalog.rocksdb.columnFamilyOptions.compactionStrategyProperties.flushL0OnNativeWrite=true
stream.sql.catalog.rocksdb.columnFamilyOptions.compactionStrategyProperties.maxSSTFilesInLevel0=8
stream.sql.catalog.rocksdb.columnFamilyOptions.compactionStrategyProperties.minFilesInLevel1=2
stream.sql.catalog.rocksdb.columnFamilyOptions.compactionStrategyProperties.minFilesInLevel2=4
stream.sql.catalog.rocksdb.columnFamilyOptions.compactionStrategyProperties.minFilesInLevel3=8
stream.sql.catalog.rocksdb.columnFamilyOptions.compactionStrategyProperties.minFilesInLevel4=16
stream.sql.catalog.rocksdb.columnFamilyOptions.compactionStrategyProperties.minFilesInLevel5=32
stream.sql.catalog.rocksdb.columnFamilyOptions.compactionStrategyProperties.minFilesInLevel6=64
stream.sql.catalog.rocksdb.columnFamilyOptions.compactionStrategyProperties.minFilesInLevel7=128
stream.sql.catalog.rocksdb.columnFamilyOptions.compactionStrategyProperties.maxFileSizeForLevel=1073741824
stream.sql.catalog.rocksdb.columnFamilyOptions.compactionStrategyProperties.writeBufferSizeBytes=4096
stream.sql.catalog.rocksdb.column
版权声明:本文内容由阿里云实名注册用户自发贡献,版权归原作者所有,阿里云开发者社区不拥有其著作权,亦不承担相应法律责任。具体规则请查看《阿里云开发者社区用户服务协议》和《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容,填写侵权投诉表单进行举报,一经查实,本社区将立刻删除涉嫌侵权内容。
实时计算Flink版是阿里云提供的全托管Serverless Flink云服务,基于 Apache Flink 构建的企业级、高性能实时大数据处理系统。提供全托管版 Flink 集群和引擎,提高作业开发运维效率。