Storage Developer Conference 2020总结

简介: Storage Developer Conference 2020总结

Storage Developer Conference 2020有两篇关于PM适配PG的议题。一篇是之前总结过的memhive,一篇为NVM WAL BUFFER


Memhive架构


Memhive的架构如下所示,大致分为四部分:基于PM的持久cachewal文件置于PM、数据文件置于PMManager。有两种使用模式:Persistent Cache和表文件、WAL文件都在PM模式。


Persistent Cache下,buffer描述符和buffer blockPM上通过pmdk函数pmem_map_file方式映射到内存,并具有持久性。Cache Manager在关键时间点执行pmem_memcpy_nodrain()pmem_flush()CPU CACHEbuffer描述符和buffer block数据刷写到PM。但是数据还是需要刷写到硬盘的。这里有个问题,持久内存上的数据什么时候刷写到磁盘,仍然是按照PG原有机制异步刷新?


带来的好处:

1、PG重启时,减小free list的更新。我的理解是,buffer blocks位于PM本身具有持久性,重启后不需要再从磁盘加载到buffer blocks,从而避免了加载数据后需要更新free list的动作。这样启动后就天然具有数据预热功能,即启动后热数据就可以直接访问。

2、双重模式:

   Always persistent:对于buffer内容和选择的描述符都执行CPU CACHE flush/drain。计划内和非计划内的服务重启都是持久性的。(正常关闭和异常宕机重启?)

Selective persistencebuffer/meta更新后不执行flush/drain。只有计划内的重启才具持久性。

3、仅对持久化有意义的buffer进行优化:生存周期很短的cache buffer不执行flush/drain(例如vacuumCOPY IN等)

WAL和表数据在PM情况下:

1、wal位于PM有两种模式,性能模式:以fsdax模式的namespaceXLOG刷写路径由pmem_memcpy_xxx()替代原来的write调用;LocalDIMM)冗余模式:sector 类型的namespace,在sectorLVM mirror

2、表数据文件(索引和表)在PM上,PMsector类型,当DB大小<=PM大小是,这些数据cacheDRAM

3、PGsectorfsdax类型下都可复制。

总结下,两种模式下的配置。

Persistent cache+WAL PM

1)local redundancyLVM mirror on sectorWAL+fsdax(cache) namespacenon-interleaved DIMMS

2)PerformancefsdaxWAL+CACHE),interleaved DIMMS(这种情况应该是pmdk持久化cache+pmdk wal

3)表数据文件在现有的DAS/SAN存储上

表数据文件+WAL on PM:

1) local redundancy: LVM mirror on sector namespaceWAL+表数据)

2) 性能:LVM on sector(表数据)+fsdaxWAL,应该是pmdk替换)

3) Cache on DRAM

文件布局:

 

总体上对于数据上的优化:buffer描述符和buffer blockWAL的优化。但是他这几种模式下哪种最优呢?持久化buffer描述符、buffer block前后性能有多大提升呢?期待Memhive提供更多资料,能够开源最好了~~


Takashi Menjo的NVM WAL BUFFER架构


单层的WAL架构,创建并映射buffer文件作为非易失性WAL buffer,将WAL日志直接写入buffer,避免两次拷贝。提交时,刷写CPU cache,保证WAL持久化到PMXLOG FLUSH时,即提交时执行CPU CACHE FLUSH而不是每次xlog insert时。

目录
相关文章
|
6月前
|
自然语言处理 供应链 中间件
什么是 Intelligence Enterprise 的 Business network?
什么是 Intelligence Enterprise 的 Business network?
46 0
如何启用 SAP Business Technology Platform 上的 Mobile Services
如何启用 SAP Business Technology Platform 上的 Mobile Services
116 0
如何启用 SAP Business Technology Platform 上的 Mobile Services
|
JavaScript 前端开发 vr&ar
Use Augmented Reality technology to bring enhanced customer experience
Use Augmented Reality technology to bring enhanced customer experience
89 0
Use Augmented Reality technology to bring enhanced customer experience
6 Major Changes in Science and Technology Reshaping the Retail Industry
As the transformation of the retail industry deepens, consumers are now at the center of the entire industry ecosystem.
3779 0
6 Major Changes in Science and Technology Reshaping the Retail Industry
|
定位技术
Esri and Alibaba Cloud to Bring Enhanced Location Intelligence Technology to Cloud Users
Esri and Alibaba Cloud are working together to combine Esri’s excellence in GIS with Alibaba Cloud’s world-class Cloud Computing capabilities.
1356 0
Esri and Alibaba Cloud to Bring Enhanced Location Intelligence Technology to Cloud Users
SAP New Intelligent Business Applications
SAP New Intelligent Business Applications        
747 0
|
前端开发
Introduction to SAP Extended Warehouse Management
Introduction to SAP Extended Warehouse Management 转自:http://searchsap.
1350 0
|
安全
The Cloud Security Ecosystem: Technical, Legal, Business and Management Issues
http://www.amazon.com/Cloud-Security-Ecosystem-Technical-Management-ebook/dp/B00ZC90H02/ref=mt_kindle?_encoding=UTF8&me= http://www.
781 0