1.1.6原地升级2.3.4 升级后compact不结束,经排查是因为Scan其中一个rowkey一直不结束进入死循环导致,但是我不知道怎么解决?
在一个HFile中一直循环
===========各cell 在hfile中 的trailer、blockBuffer、curBlock、nextIndexedKey 信息==========
cell=350ef10764bc4af960b6b468479aff1e_10032279726411/d:directory/1637112759915/Put/vlen=0/seqid=1555467268
reader.trailer=fileinfoOffset=251902094, loadOnOpenDataOffset=251901646, dataIndexCount=5, metaIndexCount=0, totalUncomressedBytes=616627871, entryCount=3831303, compressionCodec=LZ4, uncompressedDataIndexSize=623813, numDataIndexLevels=2, firstDataBlockOffset=0, lastDataBlockOffset=251756307, comparatorClassName=org.apache.hadoop.hbase.CellComparatorImpl, encryptionKey=NONE, majorVersion=3, minorVersion=3
blockBuffer=SingleByteBuff[pos=64486, lim=75102, cap= 75102]
curBlock=[blockType=DATA, fileOffset=153871189, headerSize=33, onDiskSizeWithoutHeader=31376, uncompressedSizeWithoutHeader=75102, prevBlockOffset=153846578, isUseHBaseChecksum=true, checksumType=CRC32C, bytesPerChecksum=16384, onDiskDataSizeWithHeader=31401, getOnDiskSizeWithHeader=31409, totalChecksumBytes=8, isUnpacked=true, buf=[SingleByteBuff[pos=0, lim=75143, cap= 75143]], dataBeginsWith=\x00\x00\x00E\x00\x00\x00\x00\x00/350eef5defca3e7c5bc6e3, fileContext=[usesHBaseChecksum=true, checksumType=CRC32C, bytesPerChecksum=16384, blocksize=65536, encoding=NONE, includesMvcc=true, includesTags=false, compressAlgo=LZ4, compressTags=false, cryptoContext=[cipher=NONE keyHash=NONE], name=93cc194d62fc419792991613332b990d, cellComparator=org.apache.hadoop.hbase.CellComparatorImpl@7cbeac65], nextBlockOnDiskSize=26690]
nextIndexedKey=4//LATEST_TIMESTAMP/Maximum/vlen=0/mvcc=0
cell=350ef10764bc4af960b6b468479aff1e_10032279726411/d:product_ext_nick/1637057318457/Put/vlen=27/seqid=1554882684
reader.trailer=fileinfoOffset=251902094, loadOnOpenDataOffset=251901646, dataIndexCount=5, metaIndexCount=0, totalUncomressedBytes=616627871, entryCount=3831303, compressionCodec=LZ4, uncompressedDataIndexSize=623813, numDataIndexLevels=2, firstDataBlockOffset=0, lastDataBlockOffset=251756307, comparatorClassName=org.apache.hadoop.hbase.CellComparatorImpl, encryptionKey=NONE, majorVersion=3, minorVersion=3
blockBuffer=SingleByteBuff[pos=64568, lim=75102, cap= 75102]
curBlock=[blockType=DATA, fileOffset=153871189, headerSize=33, onDiskSizeWithoutHeader=31376, uncompressedSizeWithoutHeader=75102, prevBlockOffset=153846578, isUseHBaseChecksum=true, checksumType=CRC32C, bytesPerChecksum=16384, onDiskDataSizeWithHeader=31401, getOnDiskSizeWithHeader=31409, totalChecksumBytes=8, isUnpacked=true, buf=[SingleByteBuff[pos=0, lim=75143, cap= 75143]], dataBeginsWith=\x00\x00\x00E\x00\x00\x00\x00\x00/350eef5defca3e7c5bc6e3, fileContext=[usesHBaseChecksum=true, checksumType=CRC32C, bytesPerChecksum=16384, blocksize=65536, encoding=NONE, includesMvcc=true, includesTags=false, compressAlgo=LZ4, compressTags=false, cryptoContext=[cipher=NONE keyHash=NONE], name=93cc194d62fc419792991613332b990d, cellComparator=org.apache.hadoop.hbase.CellComparatorImpl@7cbeac65], nextBlockOnDiskSize=26690]
nextIndexedKey=4//LATEST_TIMESTAMP/Maximum/vlen=0/mvcc=0
cell=350ef10764bc4af960b6b468479aff1e_10032279726411/d:store_info_ex_pc/1637125253154/Put/vlen=10329/seqid=1555632639
reader.trailer=fileinfoOffset=251902094, loadOnOpenDataOffset=251901646, dataIndexCount=5, metaIndexCount=0, totalUncomressedBytes=616627871, entryCount=3831303, compressionCodec=LZ4, uncompressedDataIndexSize=623813, numDataIndexLevels=2, firstDataBlockOffset=0, lastDataBlockOffset=251756307, comparatorClassName=org.apache.hadoop.hbase.CellComparatorImpl, encryptionKey=NONE, majorVersion=3, minorVersion=3
blockBuffer=SingleByteBuff[pos=64684, lim=75102, cap= 75102]
curBlock=[blockType=DATA, fileOffset=153871189, headerSize=33, onDiskSizeWithoutHeader=31376, uncompressedSizeWithoutHeader=75102, prevBlockOffset=153846578, isUseHBaseChecksum=true, checksumType=CRC32C, bytesPerChecksum=16384, onDiskDataSizeWithHeader=31401, getOnDiskSizeWithHeader=31409, totalChecksumBytes=8, isUnpacked=true, buf=[SingleByteBuff[pos=0, lim=75143, cap= 75143]], dataBeginsWith=\x00\x00\x00E\x00\x00\x00\x00\x00/350eef5defca3e7c5bc6e3, fileContext=[usesHBaseChecksum=true, checksumType=CRC32C, bytesPerChecksum=16384, blocksize=65536, encoding=NONE, includesMvcc=true, includesTags=false, compressAlgo=LZ4, compressTags=false, cryptoContext=[cipher=NONE keyHash=NONE], name=93cc194d62fc419792991613332b990d, cellComparator=org.apache.hadoop.hbase.CellComparatorImpl@7cbeac65], nextBlockOnDiskSize=26690]
nextIndexedKey=4//LATEST_TIMESTAMP/Maximum/vlen=0/mvcc=0
cell=350ef10764bc4af960b6b468479aff1e_10032279726411/d:vender_name/1637112759915/Put/vlen=0/seqid=1555467268
reader.trailer=fileinfoOffset=251902094, loadOnOpenDataOffset=251901646, dataIndexCount=5, metaIndexCount=0, totalUncomressedBytes=616627871, entryCount=3831303, compressionCodec=LZ4, uncompressedDataIndexSize=623813, numDataIndexLevels=2, firstDataBlockOffset=0, lastDataBlockOffset=251756307, comparatorClassName=org.apache.hadoop.hbase.CellComparatorImpl, encryptionKey=NONE, majorVersion=3, minorVersion=3
blockBuffer=SingleByteBuff[pos=0, lim=65594, cap= 65594]
curBlock=[blockType=DATA, fileOffset=153902598, headerSize=33, onDiskSizeWithoutHeader=26657, uncompressedSizeWithoutHeader=65594, prevBlockOffset=153871189, isUseHBaseChecksum=true, checksumType=CRC32C, bytesPerChecksum=16384, onDiskDataSizeWithHeader=26682, getOnDiskSizeWithHeader=26690, totalChecksumBytes=8, isUnpacked=true, buf=[SingleByteBuff[pos=0, lim=65635, cap= 65635]], dataBeginsWith=\x00\x00\x00G\x00\x00\x00\x00\x00/350ef10764bc4af960b6b4, fileContext=[usesHBaseChecksum=true, checksumType=CRC32C, bytesPerChecksum=16384, blocksize=65536, encoding=NONE, includesMvcc=true, includesTags=false, compressAlgo=LZ4, compressTags=false, cryptoContext=[cipher=NONE keyHash=NONE], name=93cc194d62fc419792991613332b990d, cellComparator=org.apache.hadoop.hbase.CellComparatorImpl@7cbeac65], nextBlockOnDiskSize=27336]
nextIndexedKey=null
-------------------------分割线-----------------------
当指针执行至列为vender_name 的数据时: generalizedSeek()-》NonLazyKeyValueScanner.doRealSeek()-》scanner.reseek(kv)-》reseekAtOrAfter(hfs, key)=》s.reseekTo(k)=》seekTo(key, false)-》BlockWithScanInfo blockWithScanInfo = indexReader.loadDataBlockWithScanInfo() 触发非正常流程:
==== loadDataBlockWithScanInfo before==== seekKey为:key=350ef10764bc4af960b6b468479aff1e_10032279726411,qualifie=vender_name,time=-9223372036854775808
HFileScanner为:StoreFileScanner[HFileScanner for reader reader=hdfs://snake/hbase/data/realtime_search/WD_MERGED_DATA/2fc7bd24c7d6d8d500f04b2480660de9/d/93cc194d62fc419792991613332b990d, compression=lz4, cacheConf=cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, firstKey=Optional[34f5c0cb717779d9ede9a8b80e46d4d4_10039682866610/d:product_ext_nick/1637109574351/Put/seqid=0], lastKey=Optional[351eb49327ecd6099a11efe0d1347724_10020558744609/d:commentcount/1637134716442/Put/seqid=0], avgKeyLen=70, avgValueLen=76, entries=3831303, length=251907468, cur=350ef10764bc4af960b6b468479aff1e_10032279726411/d:vender_name/1637112759915/Put/vlen=0/seqid=1555467268]
reader.trailer 为:fileinfoOffset=251902094, loadOnOpenDataOffset=251901646, dataIndexCount=5, metaIndexCount=0, totalUncomressedBytes=616627871, entryCount=3831303, compressionCodec=LZ4, uncompressedDataIndexSize=623813, numDataIndexLevels=2, firstDataBlockOffset=0, lastDataBlockOffset=251756307, comparatorClassName=org.apache.hadoop.hbase.CellComparatorImpl, encryptionKey=NONE, majorVersion=3, minorVersion=3
blockBuffer为:SingleByteBuff[pos=0, lim=65594, cap= 65594]
curBlock为:[blockType=DATA, fileOffset=153902598, headerSize=33, onDiskSizeWithoutHeader=26657, uncompressedSizeWithoutHeader=65594, prevBlockOffset=153871189, isUseHBaseChecksum=true, checksumType=CRC32C, bytesPerChecksum=16384, onDiskDataSizeWithHeader=26682, getOnDiskSizeWithHeader=26690, totalChecksumBytes=8, isUnpacked=true, buf=[SingleByteBuff[pos=0, lim=65635, cap= 65635]], dataBeginsWith=\x00\x00\x00G\x00\x00\x00\x00\x00/350ef10764bc4af960b6b4, fileContext=[usesHBaseChecksum=true, checksumType=CRC32C, bytesPerChecksum=16384, blocksize=65536, encoding=NONE, includesMvcc=true, includesTags=false, compressAlgo=LZ4, compressTags=false, cryptoContext=[cipher=NONE keyHash=NONE], name=93cc194d62fc419792991613332b990d, cellComparator=org.apache.hadoop.hbase.CellComparatorImpl@7cbeac65], nextBlockOnDiskSize=27336]
====loadDataBlockWithScanInfo after==== 返回预期之外的block, 返回d:store_info_ex_pc所在的block 返回信息如下: block=[blockType=DATA, fileOffset=153871189, headerSize=33, onDiskSizeWithoutHeader=31376, uncompressedSizeWithoutHeader=75102, prevBlockOffset=153846578, isUseHBaseChecksum=true, checksumType=CRC32C, bytesPerChecksum=16384, onDiskDataSizeWithHeader=31401, getOnDiskSizeWithHeader=31409, totalChecksumBytes=8, isUnpacked=true, buf=[SingleByteBuff[pos=0, lim=75143, cap= 75143]], dataBeginsWith=\x00\x00\x00E\x00\x00\x00\x00\x00/350eef5defca3e7c5bc6e3, fileContext=[usesHBaseChecksum=true, checksumType=CRC32C, bytesPerChecksum=16384, blocksize=65536, encoding=NONE, includesMvcc=true, includesTags=false, compressAlgo=LZ4, compressTags=false, cryptoContext=[cipher=NONE keyHash=NONE], name=93cc194d62fc419792991613332b990d, cellComparator=org.apache.hadoop.hbase.CellComparatorImpl@7cbeac65], nextBlockOnDiskSize=26690] nextIndexedKey=4//LATEST_TIMESTAMP/Maximum/vlen=0/mvcc=0
所以当indexReader.loadDataBlockWithScanInfo()=》seekTo(key, false)=》s.reseekTo(k)=》reseekAtOrAfter(hfs, key)=》scanner.reseek(kv) 在scanner.reseek(kv) 执行 setCurrentCell(hfs.getCell()); 取到的cell 为350ef10764bc4af960b6b468479aff1e_10032279726411/d:vender_name/1637112759915/Put/vlen=0/seqid=1555467268 导致往后的逻辑一直此逻辑
版权声明:本文内容由阿里云实名注册用户自发贡献,版权归原作者所有,阿里云开发者社区不拥有其著作权,亦不承担相应法律责任。具体规则请查看《阿里云开发者社区用户服务协议》和《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容,填写侵权投诉表单进行举报,一经查实,本社区将立刻删除涉嫌侵权内容。