ceph monitor功能的兼容性管理-阿里云开发者社区

开发者社区> 开发与运维> 正文

ceph monitor功能的兼容性管理

简介: 软件需要兼容旧版本 --------------- ceph是在一个不断进化的软件,会增加功能也会取消旧的功能,如何在ceph monitor的不同版本间保持兼容或者防止不兼容而产生错误,是需要认真思考的问题。而我们阅读代码时往往忽略了兼容性管理的代码,而重点关心它的业务代码,所以这里特别写一篇文章谈谈兼容性的管理。 ceph monitor兼容主要包括两方面: * 通讯时检查对方

软件需要兼容旧版本

ceph是在一个不断进化的软件,会增加功能也会取消旧的功能,如何在ceph monitor的不同版本间保持兼容或者防止不兼容而产生错误,是需要认真思考的问题。而我们阅读代码时往往忽略了兼容性管理的代码,而重点关心它的业务代码,所以这里特别写一篇文章谈谈兼容性的管理。

ceph monitor兼容主要包括两方面:

  • 通讯时检查对方的兼容性
  • 代码访问本机数据时的兼容性

Messenger的创建和功能位features的设置

ceph使用原生的Messenger的方式来通讯,在通讯开始前需要创建一个Messenger, 例如ceph monitor创建monitor之间通讯的messenger:

Messenger *msgr = Messenger::create(g_ceph_context, g_conf->ms_type,
                       entity_name_t::MON(rank),
                       "mon",
                       0);

ceph在建立连接时会告诉对方自己是什么类型的节点,主要有几种类型: monitor, osd, mds, client,上面的代码就说明自己是一个Monitor,
而任何一个Messenger都会准对某一种节点设置一个Policy, Policy的内容与兼容有关的是:

 /// Specify features supported locally by the endpoint.
uint64_t features_supported;
/// Specify features any remotes must have to talk to this endpoint.
int64_t features_required;

其中features_support表示本节点支持的功能,features_required表示对方必须具备的功能,每个功能一个bit位。缺省的Policy把featuires_supported设置成当前代码支持的全部功能。即:CEPH_FEATURES_ALL。

而ceph为monitor之间设置的Policy:

msgr->set_policy(entity_name_t::TYPE_MON,
                    Messenger::Policy::lossless_peer_reuse(                      
                        supported,                                               
                        CEPH_FEATURE_UID |                                       
                        CEPH_FEATURE_MON_SINGLE_PAXOS)); 

初始设置的features_required仅仅包含了CEPH_FEATUIRE_UID和CEPH_FEATURE_MON_SINGLE_PAXOS,而features_supported则是全部功能, monitor之间的兼容性是在随后的通讯过程中逐渐被检测的。

通讯时的兼容性检查

在连接建立时就检查对方的功能位设置,在ceph messenger通讯协议中,双方提供给对方支持的功能集,并且与本地Policy中设置的必需的功能位进行比较,例如:

ceph的simple messenger中,connect发起方会提供supported features:

    while (1) {
         delete authorizer;                                                          
         authorizer = msgr->get_authorizer(peer_type, false);                        
         bufferlist authorizer_reply;                                                
                                                                                 
         ceph_msg_connect connect;                                                   
         connect.features = policy.features_supported;   

当接收到应答后,又会验证对方支持的功能:

     if (reply.tag == CEPH_MSGR_TAG_READY ||
        reply.tag == CEPH_MSGR_TAG_SEQ) {                                       
        uint64_t feat_missing = policy.features_required & ~(uint64_t)reply.features;
        if (feat_missing) {                                                       
             ldout(msgr->cct,1) << "missing required features " << std::hex << feat_missing << std::dec << dendl;
             goto fail_locked;                                                           
       } 

查看reply的feature与本地必需的功能,如果缺少就会失败。

一些内部兼容性的表示方法

ceph使用一个CompatSet的数据结构来表示功能集合:

    struct CompatSet {
        struct Feature {
            uint64_t id;
            string name;
    
            Feature(uint64_t _id, const char *_name) : id(_id), name(_name) {}
            Feature(uint64_t _id, const string& _name) : id(_id), name(_name) {}
        };
    
        struct FeatureSet {
            uint64_t mask;                                                              
            map <uint64_t,string> names;  
        };

        FeatureSet compat;
          FeatureSet ro_compat; 
          FeatureSet incompat;
    };

mask中的每一位代表代表一个功能, 兼容测试主要判断是否可读可写。
测试是否可读是通过readable成员函数来实现:

bool readable(CompatSet const& other) const {
    return !((other.incompat.mask ^ incompat.mask) & other.incompat.mask);
}

这个函数的意思是如果我的incompat不能全部包含对方的位域,我就无法读取对方数据。

测试是否可写是用writable成员函数来实现:

bool writeable(CompatSet const& other) const {
   return readable(other) &&
        !((other.ro_compat.mask ^ ro_compat.mask) & other.ro_compat.mask);        
}

这个函数的意思就是:除了readable,我在ro_compat全部的位域包含了对方的位域才能写数据。

ceph monitor的内部兼容的保护

  • 每个Monitor被创建时,都需要初始化本地数据,其中mkfs函数是被调用的重要一环。首先Monitor::mkfs会在本地写入一个兼容性集合,用以记录用当前代码生成这些数据库的时候,具备什么功能:
int Monitor::mkfs()
{
   MonitorDBStore::TransactionRef t(new MonitorDBStore::Transaction);
                                                                                 
   // verify cluster fsid
   int r = check_fsid();
   if (r < 0 && r != -ENOENT)
     return r;
                                                                                 
   bufferlist magicbl;
   magicbl.append(CEPH_MON_ONDISK_MAGIC);
   magicbl.append("\n");
   t->put(MONITOR_NAME, "magic", magicbl);
                                                                                                                                                        
   features = get_initial_supported_features();    <<<<==============
   write_features(t); 
  • 当Monitor启动时,代码检查本地文件系统上的数据结构是否兼容当前代码,注意因为本地文件不被修改的情况下,
    ceph程序依然可以被升级或者用其他方法替换,所以程序启动检查本地数据是否兼容是必须的, ceph_mon.cc会调用check_features(),它检查本地数据文件格式是否和当前代码兼容:
int Monitor::check_features(MonitorDBStore *store)                              
{                                                                               
   CompatSet required = get_supported_features();                                
   CompatSet ondisk;                                                             
                                                                                  
    read_features_off_disk(store, &ondisk);                                       
                                                                                  
    if (!required.writeable(ondisk)) {                                            
      CompatSet diff = required.unsupported(ondisk);                              
      generic_derr << "ERROR: on disk data includes unsupported features: " << diff << dendl;
      return -EPERM;                                                              
    }                                                                             
                                                                                  
    return 0;                                                                     
}               
 

而get_supported_features()就是当前Monitor代码能支持的所有功能,read_features_off_disk()则把write_features()的数据读出来,我们看到它用writable()测试当前代码是否有能力可以写本地文件系统上的数据。

read_features_off_disk读出write_feature()生成的数据:

void Monitor::read_features_off_disk(MonitorDBStore *store, CompatSet *features)
{                                                                               
  bufferlist featuresbl;                                                        
  store->get(MONITOR_NAME, COMPAT_SET_LOC, featuresbl);                         
  if (featuresbl.length() == 0) {                                               
    generic_dout(0) << "WARNING: mon fs missing feature list.\n"                
            << "Assuming it is old-style and introducing one." << dendl;        
    //we only want the baseline ~v.18 features assumed to be on disk.           
    //If new features are introduced this code needs to disappear or            
    //be made smarter.                                                          
    *features = get_legacy_features();                                          
                                                                                
    bufferlist bl;                                                              
    features->encode(bl);                                                       
    MonitorDBStore::TransactionRef t(new MonitorDBStore::Transaction);          
    t->put(MONITOR_NAME, COMPAT_SET_LOC, bl);                                   
    store->apply_transaction(t);                                                
  } else {                                                                      
    bufferlist::iterator it = featuresbl.begin();                               
    features->decode(it);                                                       
  }                                                                             
}                                                  

作为一种特殊情况,如果数据是旧版的ceph monitor生成的,因为旧版没有写features到本地文件系统,所以read_features_off_disk会调用get_legacy_features()函数得到旧版本ceph monitor的功能集,这只是一个简单的构造:

CompatSet Monitor::get_legacy_features()                                        
{                                                                               
   CompatSet::FeatureSet ceph_mon_feature_compat;                                
   CompatSet::FeatureSet ceph_mon_feature_ro_compat;                             
   CompatSet::FeatureSet ceph_mon_feature_incompat;                              
   ceph_mon_feature_incompat.insert(CEPH_MON_FEATURE_INCOMPAT_BASE);             
   return CompatSet(ceph_mon_feature_compat, ceph_mon_feature_ro_compat,         
            ceph_mon_feature_incompat);                                          
}  
  • Monitor运行时检查和设置features

一旦ceph_mon.cc决定运行Monitor, 首先会调用成员函数preinit(), 而preinit的一项工作就时调用read_features()把
本地文件中记录的feature读入到成员变量features中:

void Monitor::read_features()
{
   read_features_off_disk(store, &features);
   dout(10) << "features " << features << dendl;
                                                                                 
   apply_compatset_features_to_quorum_requirements();
   dout(10) << "required_features " << required_features << dendl;
 }

当然它不会忘记按照本地数据中保存的功能位,要求monitor paxos集群的所有法人必须有对应的功能位:

void Monitor::apply_compatset_features_to_quorum_requirements()                 
{                                                                               
  required_features = 0;                                                        
  if (features.incompat.contains(CEPH_MON_FEATURE_INCOMPAT_OSD_ERASURE_CODES)) {
    required_features |= CEPH_FEATURE_OSD_ERASURE_CODES;                        
  }                                                                             
  if (features.incompat.contains(CEPH_MON_FEATURE_INCOMPAT_OSDMAP_ENC)) {       
    required_features |= CEPH_FEATURE_OSDMAP_ENC;                               
  }                                                                             
  if (features.incompat.contains(CEPH_MON_FEATURE_INCOMPAT_ERASURE_CODE_PLUGINS_V2)) {
    required_features |= CEPH_FEATURE_ERASURE_CODE_PLUGINS_V2;                  
  }                                                                             
  dout(10) << __func__ << " required_features " << required_features << dendl;  
}         

required_features的设置,可以防止不兼容的Monitor构成一个paxos集群,有几个地方通过required_features阻断这些不兼容的monitor之间的通讯:

收到一个探测包,发现对方不能提供相关的功能位,则阻断通讯:

void Monitor::handle_probe_probe(MMonProbe *m)                                  
{                                                                               
   MMonProbe *r;                                                                 
                                                                                 
   dout(10) << "handle_probe_probe " << m->get_source_inst() << *m               
        << " features " << m->get_connection()->get_features() << dendl;         
   uint64_t missing = required_features & ~m->get_connection()->get_features();  
   if (missing) {                                                                
       dout(1) << " peer " << m->get_source_addr() << " missing features "         
            << missing << dendl;                                                    
       if (m->get_connection()->has_feature(CEPH_FEATURE_OSD_PRIMARY_AFFINITY)) {  
           MMonProbe *r = new MMonProbe(monmap->fsid, MMonProbe::OP_MISSING_FEATURES,
                    name, has_ever_joined);                                      
           m->required_features = required_features;                                 
           m->get_connection()->send_message(r);                                     
     }
     goto out;
}

获取数据复制的cookie时的,发现对方不能提供相关的功能位,阻断通讯:

void Monitor::handle_sync_get_cookie(MMonSync *m)                               
{                                                                               
   if (is_synchronizing()) {                                                     
     _sync_reply_no_cookie(m);                                                   
     return;                                                                     
   }                                                                             
                                                                                 
   assert(g_conf->mon_sync_provider_kill_at != 1);                               
                                                                                 
   // make sure they can understand us.                                          
   if ((required_features ^ m->get_connection()->get_features()) &               
       required_features) {              <<=======================
       dout(5) << " ignoring peer mon." << m->get_source().num()                   
          << " has features " << std::hex                                         
          << m->get_connection()->get_features()                                  
          << " but we require " << required_features << std::dec << dendl;        
     return;                                                                     
   }           
  • paxos集群形成时的功能集兼容

一个接收到要求投票的请求的选举器,检查功能位是否兼容:通过获得当前Monitor对兼容性的要求和对方能支持的功能集的比较来决定是否还要继续:

void Elector::handle_propose(MMonElection *m)                                   
{                                                                               
...     
   uint64_t required_features = mon->get_required_features();                    
   dout(10) << __func__ << " required features " << required_features            
            << ", peer features " << m->get_connection()->get_features()         
            << dendl;                                                            
   if ((required_features ^ m->get_connection()->get_features()) &               
       required_features) {                                                      
     dout(5) << " ignoring propose from mon" << from                             
         << " without required features" << dendl;                               
     nak_old_peer(m);                                                            
     return;      

一个选举器在接收到选举应答时,检查功能位是否兼容:通过获得当前Monitor对兼容性的要求和对方能支持的功能集的比较来决定是否还要继续:

void Elector::handle_ack(MMonElection *m)                                       
{                                                                               
  ...
  uint64_t required_features = mon->get_required_features();                    
  if ((required_features ^ m->get_connection()->get_features()) &               
      required_features) {                                                      
    dout(5) << " ignoring ack from mon" << from                                 
        << " without required features" << dendl;                               
    m->put();                                                                   
    return;                                                                     
  }         

一个提出选举的Monitor,会在选举过程中收集与各个monitor连接时对方提供的features, 记录在案,在获胜后,会求出这些Monitor共同支持的功能集:

void Elector::victory()                                                         
{                                                                               
  leader_acked = -1;                                                            
  electing_me = false;                                                          
                                                                                
  uint64_t features = CEPH_FEATURES_ALL;                                        
  set<int> quorum;                                                              
  for (map<int, uint64_t>::iterator p = acked_me.begin(); p != acked_me.end();  
       ++p) {                                                                   
    quorum.insert(p->first);                                                    
    features &= p->second;                                                      
  }
  
  ...
  mon->win_election(epoch, quorum, features, cmds, cmdsize, &copy_classic_mons);                 

最后得到的features变量包含这些monitor共同支持的集合,同时把这个features传给Monitor类记录在案。

而Monitor::win_election把features保存在qurum_features后调用finish_election, finish_election调用
apply_quorum_to_compatset_features(), apply_quorum_to_compatset_features就是把paxos集群中的monitor的共同的功能集合保存在本地文件中,以备下次ceph mon启动时通过read_features读回来:

void Monitor::apply_quorum_to_compatset_features()                              
{                                                                               
   CompatSet new_features(features);                                             
   if (quorum_features & CEPH_FEATURE_OSD_ERASURE_CODES) {                       
    new_features.incompat.insert(CEPH_MON_FEATURE_INCOMPAT_OSD_ERASURE_CODES);  
   }                                                                             
   if (quorum_features & CEPH_FEATURE_OSDMAP_ENC) {                              
     new_features.incompat.insert(CEPH_MON_FEATURE_INCOMPAT_OSDMAP_ENC);         
   }                                                                             
   if (quorum_features & CEPH_FEATURE_ERASURE_CODE_PLUGINS_V2) {                 
     new_features.incompat.insert(CEPH_MON_FEATURE_INCOMPAT_ERASURE_CODE_PLUGINS_V2);
   }                                                                             
   if (new_features.compare(features) != 0) {                                    
     CompatSet diff = features.unsupported(new_features);                        
     dout(1) << __func__ << " enabling new quorum features: " << diff << dendl;  
     features = new_features;                                                    
                                                                                 
     MonitorDBStore::TransactionRef t(new MonitorDBStore::Transaction);          
     write_features(t);     <<<<<<<<<<<<<<保存
     store->apply_transaction(t);                                                
                                                                                 
    apply_compatset_features_to_quorum_requirements();                          
   }                                                                             
}                    

总结

ceph monitor在通讯初始化时,声明需要最小的功能集,它绕开了Messenger中对required_features的过分依赖,而是在通讯建立后动态地检查是否兼容。

版权声明:本文中所有内容均属于阿里云开发者社区所有,任何媒体、网站或个人未经阿里云开发者社区协议授权不得转载、链接、转贴或以其他方式复制发布/发表。申请授权请邮件developerteam@list.alibaba-inc.com,已获得阿里云开发者社区协议授权的媒体、网站,在转载使用时必须注明"稿件来源:阿里云开发者社区,原文作者姓名",违者本社区将依法追究责任。 如果您发现本社区中有涉嫌抄袭的内容,欢迎发送邮件至:developer2020@service.aliyun.com 进行举报,并提供相关证据,一经查实,本社区将立刻删除涉嫌侵权内容。

分享:
开发与运维
使用钉钉扫一扫加入圈子
+ 订阅

集结各类场景实战经验,助你开发运维畅行无忧

其他文章