之前收集了一些存储产品,最近又重新整理了一下,对他们进行了简单的分类。每个对存储的分类可能不仅相同,我的分类完全按照自己的喜好来分,如和您的分类不同,仅供参考。只是做了搜集和分类,少量产品加了写介绍,希望以后有时间,加更多更详细的介绍。
[TOC]
1.存储引擎
1.1 Hash Table
1.1.1 dbm (database manager)
https://en.wikipedia.org/wiki/Dbm
The dbm library stores arbitrary data by use of a single key (a primary key) in fixed-size buckets and uses hashing techniques to enable fast retrieval of the data by key.
1.2 btree
1.2.1 berkerlydb
https://en.wikipedia.org/wiki/Berkeley_DB
http://baike.baidu.com/view/1281930.htm
Key/value数据模型
Berkeley DB最初开发的目的是以新的HASH访问算法来代替旧的hsearch函数和大量的dbm实现(如AT&T的dbm,Berkeley的 ndbm,GNU项目的gdbm)
oracle
Written in C, java
BTREE, HASH, QUEUE, RECNO storage
https://www.oracle.com/database/berkeley-db/db.html
http://www.oracle.com/technetwork/database/database-technologies/berkeleydb/overview/index.html
1.2.2 LMDB (Lightning Memory-Mapped Database)
LMDB is a Btree-based database management library modeled loosely on the BerkeleyDB API, but much simplified. The entire database is exposed in a memory map, and all data fetches return data directly from the mapped memory, so no malloc's or memcpy's occur during data fetches.
https://symas.com/products/lightning-memory-mapped-database/
1.2.3 BoltDB
Bolt is a pure Go key/value store inspired by Howard Chu's LMDB project.
https://github.com/boltdb/bolt
1.3 LSM
1.3.1 LevelDB
https://github.com/google/leveldb
C++, google
1.3.2 RocksDB
https://github.com/facebook/rocksdb/
C++/java
1.3.3GoLevelDB
https://github.com/syndtr/goleveldb
1.3.4gorocksdb
https://github.com/tecbot/gorocksdb
Go wrapper for RocksDB
1.3.5 levigo
Go wrapper for LevelDB
https://github.com/jmhodges/levigo
1.3.6 mongo-rocks
RocksDB Storage Engine Module for MongoDB
https://github.com/mongodb-partners/mongo-rocks
C++
1.4 LSH
1.4.1 bitcask
https://github.com/basho/bitcask
采用bitcask模型的有:beandb, Riak
erlang
日志结构的key/value存储系统Bitcas
http://blog.chinaunix.net/uid-20196318-id-154750.html
Bitcask存储模型
http://blog.csdn.net/qq910894904/article/details/37756377
1.5 FractalTree
1.5.1 PerconaFT
https://github.com/percona/PerconaFT
https://www.percona.com/doc/percona-server-for-mongodb/perconaft.html
https://github.com/percona/PerconaFT/wiki
Mysql存储引擎之TokuDB以及它的数据结构Fractal tree(分形树)
http://www.fxysw.com/thread-5061-1-1.html
https://en.wikipedia.org/wiki/Fractal_tree_index
TokuDB的索引结构–分形树的实现
http://www.cnblogs.com/chaosheng/p/5250037.html
1.6 dbm系列
1.6.1 QDBM (Quick DataBase Manager)
http://sourceforge.net/projects/qdbm/
http://qdbm.sourceforge.net/benchmark.pdf
used by nmdb
1.6.2 ndbm (New Database Manager)
https://en.wikipedia.org/wiki/Ndbm
http://infolab.stanford.edu/~ullman/dbsi/win98/ndbm.html
ndbm (standing for New Database Manager) is a Berkeley produced version from 1986 of the AT&T dbm database.
ndbm stores arbitrary data by use of a single key in fixed-size buckets.
1.6.3 SDBM (Substitute Database Manager)
Substitute Database Manager
https://github.com/davidar/sdbm
http://www.cse.yorku.ca/~oz/sdbm.bun
1.6.4 GDBM (GNU Database Manager)
GNU Database Manager
http://www.gnu.org.ua/software/gdbm/
1.6.5 tdb (Trivial Database)
Trivial Database
http://sourceforge.net/projects/tdb/
1.6.6 CDB
1.6.7 TinyCDB
Tiny Constant Database
http://www.corpit.ru/mjt/tinycdb.html
1.7 双类型
1.7.1 Wiredtiger (btree, LSM)
C语言的
WiredTiger 本身也支持 LSM option (默认是 btree )
mangodb
https://github.com/wiredtiger/wiredtiger
http://source.wiredtiger.com/2.8.0/architecture.html#cache
1.7.2 Tokyo Cabinet and Kyoto Cabinet (B+tree,hash table)
http://baike.baidu.com/view/2640411.htm
http://fallabs.com/tokyocabinet/
https://en.wikipedia.org/wiki/Tokyo_Cabinet_and_Kyoto_Cabinet
Tokyo Cabinet and Kyoto Cabinet are two libraries of routines for managing key-value databases.
Kyoto Cabinet is the designated successor of Tokyo Cabinet
Tokyo Cabinet 是一个DBM的实现
Tokyo Cabinet features on-disk B+ trees and hash tables for key-value storage
Used by nmdb,Kyoto TreeDB
1.7.3 RaptorDB key value store (B+ 树 或者 MurMur 哈希索引)
一个很小的、快速的嵌入式 NoSQL 存储模块,使用 B+ 树 或者 MurMur 哈希索引
Implemented in .NET
http://raptordbkv.codeplex.com/
http://www.codeproject.com/Articles/190504/RaptorDB
http://www.codeproject.com/Articles/316816/RaptorDB-The-Key-Value-Store-V
1.8 SQL引擎类
1.8.1 InnoDB
http://www.oschina.net/p/innodb/
http://baike.baidu.com/view/1238935.htm
https://en.wikipedia.org/wiki/InnoDB
1.9 document类
1.9.1 RaptorDB document store
http://www.codeproject.com/Articles/375413/RaptorDB-the-Document-Store
2 嵌入式
https://en.wikipedia.org/wiki/Embedded_database
2.1 SQLite
https://en.wikipedia.org/wiki/SQLite
2.2 UnQLite
盘点移动开发中最流行的5个数据库
http://www.evget.com/article/2014/11/21/21843.html
3 单机存储
3.1 单值KV存储
3.1.1 Memcache
3.1.2 nmdb
https://blitiri.com.ar/p/nmdb/
use qdbm, berkeley db, tokyo cabinet or tdb as database backends
C语言
3.1.3 Memcachedb
C语言,新浪
a distributed key-value storage system designed for persistent
It conforms to memcache protocol
uses Berkeley DB as a storing backend
write 18868 w/s
read 44444 r/s
using replication for master/slave
6 policy for replication:
http://memcachedb.org/memcachedb-guide-1.0.pdf
3.1.4 Kyoto Tycoon
http://fallabs.com/kyototycoon/
C/C++,FAL Labs
Kyoto Tycoon is a lightweight database server with auto expiration mechanism, which is useful to handle cache data and persistent data of various applications. Kyoto Tycoon is also a package of network interface to the DBM called Kyoto Cabinet.
3.1.5 ThruDB
建立在Apache Thrift framework下的简单服务
支持多个数据存储后端,包括BerkeleyDB、Disk、MySQL,还拥有Memcache和Spread集成
http://code.google.com/p/thrudb/
Thrudb is a set of simple services built on top of the Apache Thrift framework that provides indexing and document storage services
3.2 结构化KV存储
3.2.1 Redis
3.2.2 ssdb
https://github.com/ideawu/ssdb
http://ssdb.io/docs/replication.html
http://ssdb.io/docs/config.html
https://github.com/ideawu/ssdb/blob/master/ssdb.conf
采用ssd,使用leveldb作为存储引擎,兼容redis接口
C/C++
3.2.3 ssdb-rocks
https://github.com/ideawu/ssdb-rocks
ssdb的另一个版本,采用ssd,使用rocksdb作为存储引擎,兼容redis接口
3.3.4 ardb
https://github.com/yinqiwen/ardb
redis-protocol compatible persistent nosql, it support multiple storage engines as backend like Google's LevelDB, Facebook's RocksDB, OpenLDAP's LMDB, WiredTiger, the default backend is Facebook's RocksDB.
C++
http://yinqiwen.github.io/ardb/2014/08/23/ardbintroduction/
3.2.5 (reborndb)QDB
兼容redis协议
Rocksdb and LevelDB
https://github.com/reborndb/qdb
3.2.6 Pika
Pika 的存储引擎, 基于Rocksdb 修改. 封装Hash, List, Set, Zset等数据结构
https://github.com/Qihoo360/pika
http://git.oschina.net/baotiao/pika
http://www.jianshu.com/p/d4f23120cbe4
首发丨360开源的类Redis存储系统:Pika
首发丨360开源的类Redis存储系统:Pika
https://media.weibo.cn/article?id=2309403974295628970629
3.2.7 LedisDB
A high performance NoSQL like Redis powered by Go
LevelDB, goleveldb, LMDB, RocksDB, BoltDB or Memory
https://github.com/siddontang/ledisdb
3.3 文档型
3.3.1 SisoDB
C#编写的,专门提供给SQL Server面向文档的db-provider
3.4 SQL
3.4.1 MySQL
3.4.2 innostore
https://github.com/basho/innostore
Innostore is a simple Erlang API to Embedded InnoDB
4 单机存储的proxy集群方案
4.1 KV/Redis类
4.1.1 Twenproxy
http://www.oschina.net/p/twemproxy
https://github.com/twitter/twemproxy
静态的分布式Redis方案
4.1.2 Reborndb
https://github.com/reborndb/reborn/blob/master/doc/tutorial_zh.md
4.1.3 Codis
http://www.oschina.net/p/codis
https://github.com/wandoulabs/codis
https://github.com/CodisLabs/codis
4.1.4 Netflix Dynomite
https://github.com/Netflix/dynomite
基于dynamo的思想
Dynomite: NetFlix对dynamo的开源通用实现
http://www.infoq.com/cn/news/2014/11/dynomite-netflix-dynamo
Netflix open sources Dynomite to make any datastore distributed
https://gigaom.com/2014/11/03/netflix-open-sources-dynomite-to-make-any-datastore-distributed/
A generic dynamo implementation for different k-v storage engines
inspired by Dynamo whitepaper
4.1.5 dbcached
http://code.google.com/p/dbcached/
a distributed key-value memory caching system for QDBM or Berkeley DB base on Memcached and NMDB
4.2 SQL类
4.2.1 Mycat
MyCat:开源分布式数据库中间件
http://mp.weixin.qq.com/s?__biz=MzAwNjMxNjQzNA==&mid=208187004&idx=1&sn=60aba39c148711e95f00ec7ca2e13bb1&scene=0#rd
4.2.2 MySQL Fabric
4.2.3 TDDL
4.2.4 Cobar
4.2.5 Atlas
4.2.6 Heisenberg
4.2.7 Vitess
5 KV存储
5.1 riak
http://www.oschina.net/p/riak/
采用bitcask模型
Riak的实现是基于Amazon的Dynamo论文
erlang
Riak是以 Erlang 编写的一个高度可扩展的分布式数据存储,Riak的实现是基于Amazon的Dynamo论文,Riak的设计目标之一就是高可用。
http://docs.basho.com/riak/kv/2.2.3/
5.2 beandb
https://github.com/douban/beansdb
https://github.com/douban/beanseye
采用bitcask模型
distributed key-value storage system
took the ideas from Amazon's Dynamo
5.3 Project Voldemort
http://www.project-voldemort.com/voldemort/
Voldemort is a distributed key-value storage system
http://www.project-voldemort.com/voldemort/design.html
https://github.com/voldemort/voldemort
BDB-JE, MySQL, Read-Only
5.4 Scalaris
分布式key value
erlang
supported the ACID properties for multi-key transactions
http://code.google.com/p/scalaris/
5.5 Aeospike
www.aerospike.com
http://www.aerospike.com/technologies/
http://www.aerospike.com/docs/
http://www.aerospike.com/docs/architecture/index.html
5.6 Tair
http://code.taobao.org/p/tair/
http://www.oschina.net/p/tair/
https://github.com/alibaba/tair
c/c++, 自研的mdb,fdb
5.7 dynomite
https://github.com/moonpolysoft/dynomite/wiki
http://www.oschina.net/p/dynomite/
6 文档型存储
6.1 MongoDB
6.2 CouchDB
CouchDB是文档型存储
6.3 Membase / Couchbase
http://www.oschina.net/p/membase
http://www.oschina.net/p/couchbase-server
https://en.wikipedia.org/wiki/Couchbase_Server
http://www.couchbase.com/wiki/display/couchbase/Home
面向文档的 NoSQL 数据库管理系统
6.4 SequoiaDB
一个类mongodb的文档型存储
http://www.sequoiadb.com/cn/
6.5 RavenDB
http://ravendb.net/
a .net document database built on the Windows ESENT storage system
支持Linq,可以使用C#的Linq语法查询数据
6.6 OrientDB
虽然是文档型数据库,但是它的关系管理方式却和图形数据库相类似
http://www.orientechnologies.com/
Distributed Graph Database with the flexibility of Documents
是兼具文挡数据库的灵活性和图形数据库管理链接 能力的可深层次扩展的文档-图形数据库管理系统。可选无模式、全模式或混合模式下。支持许 多高级特性,诸如ACID事务、快速索引,原生和SQL查询功能。可以JSON格式导入、导出文档。若不执行昂贵的JOIN操作的话,如同关系数据库可在 几毫秒内可检索数以百记的链接文档图
Even if it is a document-based database, the relationships are managed as in graph databases with direct connections between records.
It supports schema-less, schema-full and schema-mixed modes.
6.7 RethinkDB
https://en.wikipedia.org/wiki/RethinkDB
RethinkDB 1.14 (Brazil) 发布,分布式数据库
http://www.oschina.net/news/54771/rethinkdb-1-14-brazil
7 列式存储
7.1 HBase
7.2 Cassandra
7.3 Accumulo
https://accumulo.apache.org/
Apache Accumulo is based on Google's BigTable design and is built on top of Apache Hadoop, Zookeeper, and Thrift.
7.4 Hypertable
http://hypertable.org/
an open source database system inspired by publications on the design of Google's BigTable.
Hypertable runs on top of a distributed file system such as the Apache HDFS, GlusterFS or the Kosmos File System (KFS).
7.5 Scylla
http://www.scylladb.com/
https://github.com/scylladb/scylla
8 NewSQL
8.1 Actordb
http://m.oschina.net/p/actordb
https://github.com/biokoda/actordb
8.2 Cockroachdb
Go语言
https://www.cockroachlabs.com/
https://github.com/cockroachdb/cockroach
https://groups.google.com/forum/#!forum/cockroachdb-users
https://groups.google.com/forum/#!forum/cockroach-db
https://www.cockroachlabs.com/docs/
Design Documents
https://github.com/cockroachdb/cockroach/blob/master/docs/design.md
8.3 FoundationDB
https://github.com/FoundationDB
8.4 Oceanbase
https://github.com/alibaba/oceanbase
8.5 SnappyData
8.6 TiDB
https://github.com/pingcap/tidb
Go 语言
https://github.com/pingcap/tikv
Rust语言
https://github.com/pingcap/tidb/blob/master/docs/QUICKSTART.md
从零开始写分布式数据库
https://github.com/ngaut/builddatabase
9 图数据库
9.1 Neo4j
9.2 Infinite Graph
http://www.objectivity.com/products/infinitegraph/
an enterprise distributed graph database
10 File存储
10.1 Ceph
Ceph:一个 Linux PB 级分布式文件系统
http://www.ibm.com/developerworks/cn/linux/l-ceph/
10.2 FastDFS
http://code.google.com/p/fastdfs/
10.3 HDFS
10.4 MogileFs
10.5 MooseFS
a fault tolerant, network distributed file system
10.6 TFS
10.7 GlusterFS
GlusterFS:异地备份(Geo-replication)源码分析
http://blog.chinaunix.net/uid-22166872-id-4392777.html
Gluster Geo-replication工作原理[转载]
http://blog.163.com/szy8706@yeah/blog/static/62713185201363163131800/
http://disclu.blogspot.com/2012/11/gluster-geo-replication.html
10.8 kosmosfs
http://code.google.com/p/kosmosfs/
11 In-Memory 存储
11.1 Redis cluster
11.2 Mysql cluster
11.3 Gemfire/Gemde
11.4 VoltDB
内存数据库
12 私有存储
12.1 Amazon
12.1.1 Amazon Dynamo
Key-value
http://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf
Amazon’s Dynamo paper
Merkle trees
Gossiping of membership
Gossiped synchronization of partitions
12.2 Google
12.2.1 BigTable
Bigtable: A Distributed Storage System for Structured Data
http://static.googleusercontent.com/media/research.google.com/en//archive/bigtable-osdi06.pdf
12.2.2 F1
F1: A Distributed SQL Database That Scales
http://research.google.com/pubs/pub41344.html
http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/41344.pdf
12.2.3 Spanner
http://research.google.com/archive/spanner.html
Spanner: Google’s Globally-Distributed Database
Exclusive: Inside Google Spanner, the Largest Single Database on Earth
http://www.wired.com/2012/11/google-spanner-time/all/
解析全球级分布式数据库Google Spanner
http://www.csdn.net/article/2012-09-19/2810132-google-spanner-next-database-datacenter
12.2.4 Megastore
Megastore: Providing Scalable, Highly Available Storage for Interactive Services
https://research.google.com/pubs/pub36971.html
https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/36971.pdf
12.3 Baidu
12.3.1 Baidu Mola
Key-value
12.3.2 Baidu BDRP
12.3.3 Baidu DDBS
12.4 腾讯
单机MySQL到NoSQL集群 腾讯存储进阶路
http://tech.it168.com/a2017/0322/3105/000003105672.shtml
腾讯十多个人管理一万多台NoSQL存储服务器的秘密
http://m.techweb.com.cn/article/2016-01-06/2253310.shtml
12.4.1 腾讯CKV
腾讯CKV海量分布式存储系统
http://www.csdn.net/article/2014-03-11/2818723
12.4.2 QuorumKV
微信PaxosStore内存云揭秘:十亿Paxos/分钟的挑战
12.6 京东
12.6.1 京东JIMDB
解读JIMDB 京东分布式缓存与高速KV存储
http://m.chinabyte.com/storage/463/13344963_mi.shtml
解读JIMDB 京东分布式缓存与高速KV存储
http://m.it168.com/article_1720792.html
12.7 滴滴
12.7.1 滴滴Rockstable
滴滴高性能KV存储系统实践
http://m.it168.com/article_3091989.html
12.8 美团
12.8.1 Cellar
基于tair研发的新一代KV存储服务
美团cellar讲座笔记
http://blog.leanote.com/post/yuannight/cellar%E8%AE%B2%E5%BA%A7%E7%AC%94%E8%AE%B0
12.9 360
12.9.1 360 Bada
Key-value
360自研分布式存储系统Bada的架构设计和应用
http://www.chinacloud.cn/wap.aspx?cid=16&nid=21103
1.9.2 HustStore
https://github.com/Qihoo360/huststore
HustStore 360高性能分布式存储服务
http://www.oschina.net/p/huststore?fromerr=ug3CEPNP
13 云产品
13.1 Amazon DynamoDB
13.2 AWS Aurora
https://aws.amazon.com/cn/rds/aurora/
2.3 Google云
2.4 阿里云
2.5 美团云
Mangix:分布式对象存储
Mangix: 美团云分布式对象存储系统