开发者社区 > 大数据与机器学习 > 实时计算 Flink > 正文

flinksql 环境如何配置使用rocksdb啊

flinksql 环境如何配置使用rocksdb啊

展开
收起
游客3oewgrzrf6o5c 2022-07-12 16:02:57 632 0
1 条回答
写回答
取消 提交回答
  • CSDN全栈领域优质创作者,万粉博主;InfoQ签约博主;华为云享专家;华为Iot专家;亚马逊人工智能自动驾驶(大众组)吉尼斯世界纪录获得者

    要在 Flink SQL 中使用 RocksDB 存储引擎,需要进行以下环境配置:

    在 Flink 的配置文件(例如 flink.properties 或 yarn-site.xml)中,设置以下参数来启用 RocksDB 存储引擎:

    streamExecutionEnvironment.defaultJobManagerAddress=ip:port
    streamExecutionEnvironment.defaultTaskManagerAddress=ip:port
    streamExecutionEnvironment.jobManagerAddress=ip:port
    streamExecutionEnvironment.taskManagerAddress=ip:port
    streamExecutionEnvironment.jobManagerHttpAddress=ip:port
    streamExecutionEnvironment.jobManagerRmiAddress=ip:port
    streamExecutionEnvironment.jobManagerPasswordFile=/path/to/password/file
    streamExecutionEnvironment.jobManagerUser=userName
    streamExecutionEnvironment.taskManagerUser=userName
    streamExecutionEnvironment.taskManagerPassword=password
    streamExecutionEnvironment.jobManagerSecurityProtocol=SSL
    streamExecutionEnvironment.taskManagerSecurityProtocol=SSL
    streamExecutionEnvironment.jobManagerSaslMechanism=GSSAPI
    streamExecutionEnvironment.taskManagerSaslMechanism=GSSAPI
    streamExecutionEnvironment.jobManagerSaslJaasConfig=org.apache.flink.api.common.security.GSSSVCAuthenticatorConfiguration@54148e91
    streamExecutionEnvironment.taskManagerSaslJaasConfig=org.apache.flink.api.common.security.GSSSVCAuthenticatorConfiguration@54148e91
    streamExecutionEnvironment.hdfs.rootDirectory=/path/to/hdfs/directory
    streamExecutionEnvironment.hdfs.defaultFS=hdfs://namenode:port
    streamExecutionEnvironment.hdfs.jobSubmissionDirectory=/path/to/job/directory
    streamExecutionEnvironment.hdfs.jobHistoryDirectory=/path/to/job/history/directory
    streamExecutionEnvironment.hdfs.jobCheckpointDirectory=/path/to/job/checkpoint/directory
    streamExecutionEnvironment.hdfs.jobCheckpointBackupDirectory=/path/to/job/checkpoint/backup/directory
    streamExecutionEnvironment.hdfs.jobLogDirectory=/path/to/job/log/directory
    streamExecutionEnvironment.hdfs.fileSystem=hdfs://namenode:port
    streamExecutionEnvironment.hdfs.jobManager.webPort=port
    streamExecutionEnvironment.hdfs.jobManager.webAddress=ip:port
    streamExecutionEnvironment.hdfs.jobManager.administrationWebPort=port
    streamExecutionEnvironment.hdfs.jobManager.administrationWebAddress=ip:port
    streamExecutionEnvironment.hdfs.jobHistory.webPort=port
    streamExecutionEnvironment.hdfs.jobHistory.webAddress=ip:port
    streamExecutionEnvironment.hdfs.jobHistory.administrationWebPort=port
    streamExecutionEnvironment.hdfs.jobHistory.administrationWebAddress=ip:port
    streamExecutionEnvironment.hdfs.jobCheckpointWebPort=port
    streamExecutionEnvironment.hdfs.jobCheckpointWebAddress=ip:port
    streamExecutionEnvironment.hdfs.jobCheckpoint.administrationWebPort=port
    streamExecutionEnvironment.hdfs.jobCheckpoint.administrationWebAddress=ip:port
    streamExecutionEnvironment.hdfs.jobManager.webPasswordFile=/path/to/password/file
    streamExecutionEnvironment.hdfs.jobManager.administrationWebPasswordFile=/path/to/password/file
    streamExecutionEnvironment.hdfs.jobHistory.webPasswordFile=/path/to/password/file
    streamExecutionEnvironment.hdfs.jobHistory.administrationWebPasswordFile=/path/to/password/file
    streamExecutionEnvironment.hdfs.jobCheckpoint.webPasswordFile=/path/to/password/file
    streamExecutionEnvironment.hdfs.jobCheckpoint.administrationWebPasswordFile=/path/to/password/file
    streamExecutionEnvironment.jobGraphStorePort=port
    streamExecutionEnvironment.jobGraphStoreAddress=ip:port
    streamExecutionEnvironment.jobGraphStorePasswordFile=/path/to/password/file
    streamExecutionEnvironment.jobGraphStoreUser=userName
    streamExecutionEnvironment.jobGraphStorePassword=password
    streamExecutionEnvironment.jobGraphStoreConfiguration=/path/to/jobgraphstore/config
    streamExecutionEnvironment
    
    • 3、配置 SQL

    在 Flink SQL 的配置文件(例如 sql.properties 或 sql-yarn-site.xml)中,设置以下参数来启用 RocksDB 存储引擎:

    stream.sql.catalogImplementation=org.apache.flink.table.catalog.CatalogImpl
    stream.sql.catalog.type=RocksDB
    stream.sql.catalog.rocksdb.path=/path/to/rocksdb/db
    stream.sql.catalog.rocksdb.tablePrefix=my_table_
    stream.sql.catalog.rocksdb.tableType=GLOBAL
    stream.sql.catalog.rocksdb.columnFamilyOptions.maxVersions=10
    stream.sql.catalog.rocksdb.columnFamilyOptions.timeToLiveSeconds=3600
    stream.sql.catalog.rocksdb.columnFamilyOptions.maxTimeToLiveSeconds=7200
    stream.sql.catalog.rocksdb.columnFamilyOptions.writeBufferSizeBytes=4096
    stream.sql.catalog.rocksdb.columnFamilyOptions.blockCacheEnabled=true
    stream.sql.catalog.rocksdb.columnFamilyOptions.blockCacheSizeBytes=2048000
    stream.sql.catalog.rocksdb.columnFamilyOptions.flushOnPut=true
    stream.sql.catalog.rocksdb.columnFamilyOptions.compactionStrategyClassName=com.rocksdb.memtable.DefaultMemTableConfiguration@545a0a85
    stream.sql.catalog.rocksdb.columnFamilyOptions.compactionStrategyProperties.put泱Threshold=5000
    stream.sql.catalog.rocksdb.columnFamilyOptions.compactionStrategyProperties.sstFileSize=1073741824
    stream.sql.catalog.rocksdb.columnFamilyOptions.compactionStrategyProperties.level0FileNumThreshold=10
    stream.sql.catalog.rocksdb.columnFamilyOptions.compactionStrategyProperties.level0SlowdownWriteThreshold=60
    stream.sql.catalog.rocksdb.columnFamilyOptions.compactionStrategyProperties.level0StopWriteThreshold=90
    stream.sql.catalog.rocksdb.columnFamilyOptions.compactionStrategyProperties.minFilesForCompaction=4
    stream.sql.catalog.rocksdb.columnFamilyOptions.compactionStrategyProperties.maxFilesForCompaction=8
    stream.sql.catalog.rocksdb.columnFamilyOptions.compactionStrategyProperties.maxCompactionBytes=268435456
    stream.sql.catalog.rocksdb.columnFamilyOptions.compactionStrategyProperties.flushL0OnNativeWrite=true
    stream.sql.catalog.rocksdb.columnFamilyOptions.compactionStrategyProperties.maxSSTFilesInLevel0=8
    stream.sql.catalog.rocksdb.columnFamilyOptions.compactionStrategyProperties.minFilesInLevel1=2
    stream.sql.catalog.rocksdb.columnFamilyOptions.compactionStrategyProperties.minFilesInLevel2=4
    stream.sql.catalog.rocksdb.columnFamilyOptions.compactionStrategyProperties.minFilesInLevel3=8
    stream.sql.catalog.rocksdb.columnFamilyOptions.compactionStrategyProperties.minFilesInLevel4=16
    stream.sql.catalog.rocksdb.columnFamilyOptions.compactionStrategyProperties.minFilesInLevel5=32
    stream.sql.catalog.rocksdb.columnFamilyOptions.compactionStrategyProperties.minFilesInLevel6=64
    stream.sql.catalog.rocksdb.columnFamilyOptions.compactionStrategyProperties.minFilesInLevel7=128
    stream.sql.catalog.rocksdb.columnFamilyOptions.compactionStrategyProperties.maxFileSizeForLevel=1073741824
    stream.sql.catalog.rocksdb.columnFamilyOptions.compactionStrategyProperties.writeBufferSizeBytes=4096
    stream.sql.catalog.rocksdb.column
    
    2023-08-12 17:15:46
    赞同 展开评论 打赏

实时计算Flink版是阿里云提供的全托管Serverless Flink云服务,基于 Apache Flink 构建的企业级、高性能实时大数据处理系统。提供全托管版 Flink 集群和引擎,提高作业开发运维效率。

相关电子书

更多
HBase in Practise- 性能、监控和问题排查 立即下载
HBase in Practise: 性能、监控和问题排查 立即下载
MaxCompute Logview参数详解和问题排查(废弃) 立即下载