E-MapReduce gateway host文件记录太多
已解决E-MapReduce gateway host文件记录太多
展开
收起
1
条回答
写回答
写回答
-
开发者社区问答官方账号官方回答采纳回答问题原因：主集群扩容时，会往gateway下发新增host的任务，但缩容时不会。这个问题已经修复，但存量节点可能也会有提出问题的现象。操作方案： 可在gateway机器上，用ecm-agent python3 跑如下脚本：from ecm_agent.hosts.hosts import Hosts, HostsEntryimport shutilimport osimport randomimport commandsimport reHostEntryPattern1 = 'emr-worker-(\d+)$'HostEntryPattern2 = 'emr-worker-(\d+).cluster-\d+$'class HostsFileUtil:TEMP_HOST_FILE = "/var/lib/ecm-agent/data/host_tmp_"def __init__(self):self.temporary_hosts = Nonedef cleanup_host_file(self):try:host_suffix = self._ensure_temporary_hosts()print("records before: " + str(len(self.temporary_hosts.entries)))new_entrys = []for entry in self.temporary_hosts.entries:if entry.names:entry.names = self._cleanup_host_names(entry.names)new_entry = HostsEntry(entry_type=entry.entry_type, address=entry.address, names=entry.names, comment=entry.comment)new_entrys.append(new_entry)print("records before: " + str(len(new_entrys)))self.temporary_hosts.entries = new_entrysself.temporary_hosts.write()self._dump_temporary_hosts(host_suffix)except Exception, e:print("cleanup_host_entry failed, " + str(e))def _ensure_temporary_hosts(self):host_suffix = str(random.randint(0, 1000))if self.temporary_hosts is None:if not os.path.exists(self.TEMP_HOST_FILE + host_suffix):with open(self.TEMP_HOST_FILE + host_suffix, "w+") as temp_hosts:passshutil.copyfile("/etc/hosts", self.TEMP_HOST_FILE + host_suffix)self.temporary_hosts = Hosts(path=self.TEMP_HOST_FILE + host_suffix)if not self.temporary_hosts:raise Exception('init hosts object failed')return host_suffixdef _dump_temporary_hosts(self, host_suffix):code, result = commands.getstatusoutput("mv " + self.TEMP_HOST_FILE + host_suffix + " /etc/hosts")if code != 0:print(result)exit(1)def _cleanup_host_names(self, names):result = []temp_names_short = []temp_names_long = []for name in names:matchObj = re.match(HostEntryPattern1, name)if matchObj:temp_names_short.append([int(matchObj.group(1)), matchObj.group()])continuematchObj = re.match(HostEntryPattern2, name)if matchObj:temp_names_long.append([int(matchObj.group(1)), matchObj.group()])continueresult.append(name)if temp_names_short:temp_names_short = sorted(temp_names_short, key=lambda x:x[0])result.insert(0, temp_names_short[-1][1])if temp_names_long:temp_names_long = sorted(temp_names_long, key=lambda x:x[0])result.insert(0, temp_names_long[-1][1])return resultif __name__ == "__main__":hostsUtil = HostsFileUtil()hostsUtil.cleanup_host_file()2024-05-31 18:04:58赞同 1 展开评论 打赏
关联地址：
版权声明：本文内容由阿里云实名注册用户自发贡献，版权归原作者所有，阿里云开发者社区不拥有其著作权，亦不承担相应法律责任。具体规则请查看《阿里云开发者社区用户服务协议》和《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容，填写侵权投诉表单进行举报，一经查实，本社区将立刻删除涉嫌侵权内容。
相关问答
