如何实现通过crc64校验数据传输的完整性？

背景


数据在客户端和服务器之间传输时有可能会出错。OSS现在支持对各种方式上传的object返回其crc64值，客户端可以和本地计算的crc64值做对比，从而完成数据完整性的验证。

• OSS对新上传的object进行crc64的计算，并将结果存储为object的元信息存储，随后在返回的response header 中增加x-oss-hash-crc64ecma头部，表示其crc64值，该64位CRC根据 ECMA-182标准 计算得出。
• 对于crc64上线之前就已经存在于OSS上的object, OSS不会对其计算crc64值，所以获取此类object时不会返回其crc64值。

• Put Object / Append Object / Post Object / Multipart upload part 均会返回对应的crc64值，客户端可以在上传完成后拿到服务器返回的crc64值和本地计算的数值进行校验。
• Multipart Complete时，如果所有的part都有crc64值，则会返回整个object的crc64值；否则，比如有crc64上线之前就已经上传的part，则不返回crc64值。
• Get Object / Head Object / Get ObjectMeta 都会返回对应的crc64值（如有）。客户端可以在Get Object完成后，拿到服务器返回的crc64值和本地计算的数值进行校验。[backcolor=transparent]注意，range get请求返回的将会是整个object的crc64值。
• Copy相关的操作，如Copy Object / Upload Part Copy, 新生成的object/Part不保证具有crc64值。

1. [backcolor=transparent]import[backcolor=transparent] oss2
2. [backcolor=transparent]from[backcolor=transparent] oss2[backcolor=transparent].[backcolor=transparent]models [backcolor=transparent]import[backcolor=transparent] [backcolor=transparent]PartInfo
3. [backcolor=transparent]import[backcolor=transparent] os
4. [backcolor=transparent]import[backcolor=transparent] crcmod
5. [backcolor=transparent]import[backcolor=transparent] random
6. [backcolor=transparent]import[backcolor=transparent] [backcolor=transparent]string
8. [backcolor=transparent]do_crc64 [backcolor=transparent]=[backcolor=transparent] crcmod[backcolor=transparent].[backcolor=transparent]mkCrcFun[backcolor=transparent]([backcolor=transparent]0x142F0E1EBA9EA3693L[backcolor=transparent],[backcolor=transparent] initCrc[backcolor=transparent]=[backcolor=transparent]0L[backcolor=transparent],[backcolor=transparent] xorOut[backcolor=transparent]=[backcolor=transparent]0xffffffffffffffffL[backcolor=transparent],[backcolor=transparent] rev[backcolor=transparent]=[backcolor=transparent]True[backcolor=transparent])
10. [backcolor=transparent]def[backcolor=transparent] check_crc64[backcolor=transparent]([backcolor=transparent]local_crc64[backcolor=transparent],[backcolor=transparent] oss_crc64[backcolor=transparent],[backcolor=transparent] msg[backcolor=transparent]=[backcolor=transparent]"check crc64"[backcolor=transparent]):
11. [backcolor=transparent]if[backcolor=transparent] local_crc64 [backcolor=transparent]!=[backcolor=transparent] oss_crc64[backcolor=transparent]:
12. [backcolor=transparent]print[backcolor=transparent] [backcolor=transparent]"{0} check crc64 failed. local:{1}, oss:{2}."[backcolor=transparent].[backcolor=transparent]format[backcolor=transparent]([backcolor=transparent]msg[backcolor=transparent],[backcolor=transparent] local_crc64[backcolor=transparent],[backcolor=transparent] oss_crc64[backcolor=transparent])
13. [backcolor=transparent]return[backcolor=transparent] [backcolor=transparent]False
14. [backcolor=transparent]else[backcolor=transparent]:
15. [backcolor=transparent]print[backcolor=transparent] [backcolor=transparent]"{0} check crc64 ok."[backcolor=transparent].[backcolor=transparent]format[backcolor=transparent]([backcolor=transparent]msg[backcolor=transparent])
16. [backcolor=transparent]return[backcolor=transparent] [backcolor=transparent]True
18. [backcolor=transparent]def[backcolor=transparent] random_string[backcolor=transparent]([backcolor=transparent]length[backcolor=transparent]):
19. [backcolor=transparent]return[backcolor=transparent] [backcolor=transparent]''[backcolor=transparent].[backcolor=transparent]join[backcolor=transparent]([backcolor=transparent]random[backcolor=transparent].[backcolor=transparent]choice[backcolor=transparent]([backcolor=transparent]string[backcolor=transparent].[backcolor=transparent]lowercase[backcolor=transparent])[backcolor=transparent] [backcolor=transparent]for[backcolor=transparent] i [backcolor=transparent]in[backcolor=transparent] range[backcolor=transparent]([backcolor=transparent]length[backcolor=transparent]))
21. [backcolor=transparent]bucket [backcolor=transparent]=[backcolor=transparent] oss2[backcolor=transparent].[backcolor=transparent]Bucket[backcolor=transparent]([backcolor=transparent]oss2[backcolor=transparent].[backcolor=transparent]Auth[backcolor=transparent]([backcolor=transparent]access_key_id[backcolor=transparent],[backcolor=transparent] access_key_secret[backcolor=transparent]),[backcolor=transparent] endpoint[backcolor=transparent],[backcolor=transparent] bucket_name[backcolor=transparent])

1. [backcolor=transparent]content [backcolor=transparent]=[backcolor=transparent] random_string[backcolor=transparent]([backcolor=transparent]1024[backcolor=transparent])
2. [backcolor=transparent]key [backcolor=transparent]=[backcolor=transparent] [backcolor=transparent]'normal-key'
4. [backcolor=transparent]result [backcolor=transparent]=[backcolor=transparent] bucket[backcolor=transparent].[backcolor=transparent]put_object[backcolor=transparent]([backcolor=transparent]key[backcolor=transparent],[backcolor=transparent] content[backcolor=transparent])
5. [backcolor=transparent]oss_crc64 [backcolor=transparent]=[backcolor=transparent] result[backcolor=transparent].[backcolor=transparent]headers[backcolor=transparent].[backcolor=transparent]get[backcolor=transparent]([backcolor=transparent]'x-oss-hash-crc64ecma'[backcolor=transparent],[backcolor=transparent] [backcolor=transparent]''[backcolor=transparent])
6. [backcolor=transparent]local_crc64 [backcolor=transparent]=[backcolor=transparent] str[backcolor=transparent]([backcolor=transparent]do_crc64[backcolor=transparent]([backcolor=transparent]content[backcolor=transparent]))
7. [backcolor=transparent]check_crc64[backcolor=transparent]([backcolor=transparent]local_crc64[backcolor=transparent],[backcolor=transparent] oss_crc64[backcolor=transparent],[backcolor=transparent] [backcolor=transparent]"put object"[backcolor=transparent])

1. [backcolor=transparent]result [backcolor=transparent]=[backcolor=transparent] bucket[backcolor=transparent].[backcolor=transparent]get_object[backcolor=transparent]([backcolor=transparent]key[backcolor=transparent])
2. [backcolor=transparent]oss_crc64 [backcolor=transparent]=[backcolor=transparent] result[backcolor=transparent].[backcolor=transparent]headers[backcolor=transparent].[backcolor=transparent]get[backcolor=transparent]([backcolor=transparent]'x-oss-hash-crc64ecma'[backcolor=transparent],[backcolor=transparent] [backcolor=transparent]''[backcolor=transparent])
3. [backcolor=transparent]local_crc64 [backcolor=transparent]=[backcolor=transparent] str[backcolor=transparent]([backcolor=transparent]do_crc64[backcolor=transparent]([backcolor=transparent]result[backcolor=transparent].[backcolor=transparent]resp[backcolor=transparent].[backcolor=transparent]read[backcolor=transparent]()))
4. [backcolor=transparent]check_crc64[backcolor=transparent]([backcolor=transparent]local_crc64[backcolor=transparent],[backcolor=transparent] oss_crc64[backcolor=transparent],[backcolor=transparent] [backcolor=transparent]"get object"[backcolor=transparent])

1. [backcolor=transparent]part_info_list [backcolor=transparent]=[backcolor=transparent] [backcolor=transparent][]
3. [backcolor=transparent]key [backcolor=transparent]=[backcolor=transparent] [backcolor=transparent]"multipart-key"
4. [backcolor=transparent]result [backcolor=transparent]=[backcolor=transparent] bucket[backcolor=transparent].[backcolor=transparent]init_multipart_upload[backcolor=transparent]([backcolor=transparent]key[backcolor=transparent])
5. [backcolor=transparent]upload_id [backcolor=transparent]=[backcolor=transparent] result[backcolor=transparent].[backcolor=transparent]upload_id
7. [backcolor=transparent]part_1 [backcolor=transparent]=[backcolor=transparent] random_string[backcolor=transparent]([backcolor=transparent]1024[backcolor=transparent] [backcolor=transparent]*[backcolor=transparent] [backcolor=transparent]1024[backcolor=transparent])
8. [backcolor=transparent]result [backcolor=transparent]=[backcolor=transparent] bucket[backcolor=transparent].[backcolor=transparent]upload_part[backcolor=transparent]([backcolor=transparent]key[backcolor=transparent],[backcolor=transparent] upload_id[backcolor=transparent],[backcolor=transparent] [backcolor=transparent]1[backcolor=transparent],[backcolor=transparent] part_1[backcolor=transparent])
9. [backcolor=transparent]oss_crc64 [backcolor=transparent]=[backcolor=transparent] result[backcolor=transparent].[backcolor=transparent]headers[backcolor=transparent].[backcolor=transparent]get[backcolor=transparent]([backcolor=transparent]'x-oss-hash-crc64ecma'[backcolor=transparent],[backcolor=transparent] [backcolor=transparent]''[backcolor=transparent])
10. [backcolor=transparent]local_crc64 [backcolor=transparent]=[backcolor=transparent] str[backcolor=transparent]([backcolor=transparent]do_crc64[backcolor=transparent]([backcolor=transparent]part_1[backcolor=transparent]))
11. [backcolor=transparent]#check 上传的 part 1数据是否完整
12. [backcolor=transparent]check_crc64[backcolor=transparent]([backcolor=transparent]local_crc64[backcolor=transparent],[backcolor=transparent] oss_crc64[backcolor=transparent],[backcolor=transparent] [backcolor=transparent]"upload_part object 1"[backcolor=transparent])
13. [backcolor=transparent]part_info_list[backcolor=transparent].[backcolor=transparent]append[backcolor=transparent]([backcolor=transparent]PartInfo[backcolor=transparent]([backcolor=transparent]1[backcolor=transparent],[backcolor=transparent] result[backcolor=transparent].[backcolor=transparent]etag[backcolor=transparent],[backcolor=transparent] len[backcolor=transparent]([backcolor=transparent]part_1[backcolor=transparent])))
15. [backcolor=transparent]part_2 [backcolor=transparent]=[backcolor=transparent] random_string[backcolor=transparent]([backcolor=transparent]1024[backcolor=transparent] [backcolor=transparent]*[backcolor=transparent] [backcolor=transparent]1024[backcolor=transparent])
16. [backcolor=transparent]result [backcolor=transparent]=[backcolor=transparent] bucket[backcolor=transparent].[backcolor=transparent]upload_part[backcolor=transparent]([backcolor=transparent]key[backcolor=transparent],[backcolor=transparent] upload_id[backcolor=transparent],[backcolor=transparent] [backcolor=transparent]2[backcolor=transparent],[backcolor=transparent] part_2[backcolor=transparent])
17. [backcolor=transparent]oss_crc64 [backcolor=transparent]=[backcolor=transparent] result[backcolor=transparent].[backcolor=transparent]headers[backcolor=transparent].[backcolor=transparent]get[backcolor=transparent]([backcolor=transparent]'x-oss-hash-crc64ecma'[backcolor=transparent],[backcolor=transparent] [backcolor=transparent]''[backcolor=transparent])
18. [backcolor=transparent]local_crc64 [backcolor=transparent]=[backcolor=transparent] str[backcolor=transparent]([backcolor=transparent]do_crc64[backcolor=transparent]([backcolor=transparent]part_2[backcolor=transparent]))
19. [backcolor=transparent]#check 上传的 part 2数据是否完整
20. [backcolor=transparent]check_crc64[backcolor=transparent]([backcolor=transparent]local_crc64[backcolor=transparent],[backcolor=transparent] oss_crc64[backcolor=transparent],[backcolor=transparent] [backcolor=transparent]"upload_part object 2"[backcolor=transparent])
21. [backcolor=transparent]part_info_list[backcolor=transparent].[backcolor=transparent]append[backcolor=transparent]([backcolor=transparent]PartInfo[backcolor=transparent]([backcolor=transparent]2[backcolor=transparent],[backcolor=transparent] result[backcolor=transparent].[backcolor=transparent]etag[backcolor=transparent],[backcolor=transparent] len[backcolor=transparent]([backcolor=transparent]part_2[backcolor=transparent])))
23. [backcolor=transparent]result [backcolor=transparent]=[backcolor=transparent] bucket[backcolor=transparent].[backcolor=transparent]complete_multipart_upload[backcolor=transparent]([backcolor=transparent]key[backcolor=transparent],[backcolor=transparent] upload_id[backcolor=transparent],[backcolor=transparent] part_info_list[backcolor=transparent])
24. [backcolor=transparent]oss_crc64 [backcolor=transparent]=[backcolor=transparent] result[backcolor=transparent].[backcolor=transparent]headers[backcolor=transparent].[backcolor=transparent]get[backcolor=transparent]([backcolor=transparent]'x-oss-hash-crc64ecma'[backcolor=transparent],[backcolor=transparent] [backcolor=transparent]''[backcolor=transparent])
25. [backcolor=transparent]local_crc64 [backcolor=transparent]=[backcolor=transparent] str[backcolor=transparent]([backcolor=transparent]do_crc64[backcolor=transparent]([backcolor=transparent]part_2[backcolor=transparent],[backcolor=transparent] do_crc64[backcolor=transparent]([backcolor=transparent]part_1[backcolor=transparent])))
26. [backcolor=transparent]#check 最终oss上的object和本地文件是否一致
27. [backcolor=transparent]check_crc64[backcolor=transparent]([backcolor=transparent]local_crc64[backcolor=transparent],[backcolor=transparent] oss_crc64[backcolor=transparent],[backcolor=transparent] [backcolor=transparent]"complete object"[backcolor=transparent])