- 概要
- 技术点
- 最佳实践
- 改进
- 参考
概要
所谓断点下载,其实是客户端在从网络上下载资源时,由于某种原因中断下载。再次开启下载时可以从已经下载的部分开始继续下载未完成的部分,从而节省时间和流量。
应用场景:当我们在手机端使用视频软件下载视频时,下载期间网络模式从WIFI切换到移动网络,默认App都会中断下载。当再次切换到WIFI网络时,由用户手动重新开启下载任务,此时就用到了断点下载。
优点:节省时间和流量。
技术点
HTTP1.1中新增了Range头的支持,用于指定获取数据的范围。Range的格式一般分为以下几种:
-
Range: bytes=100-
从 101 bytes 之后开始传,一直传到最后。 -
Range: bytes=100-200
指定开始到结束这一段的长度,记住 Range 是从 0 计数 的,所以这个是要求服务器从 101 字节开始传,一直到 201 字节结束。这个一般用来特别大的文件分片传输,比如视频。 -
Range: bytes=-100
如果范围没有指定开始位置,就是要服务器端传递倒数 100 字节的内容。而不是从 0 开始的 100 字节。 -
Range: bytes=0-100, 200-300
也可以同时指定多个范围的内容,这种情况不是很常见。
另外断点续传时需要验证服务器上面的文件是否发生变化,此时用到了If-Match
头。If-Match
对应的是Etag
的值。
客户端在发起请求时在Header中携带Range
,If-Match
,OSS服务器在收到请求后会验证'If-Match'中的Etag值,如果不匹配,则会返回412 precondition 状态码。
OSS的服务器针对getObject这个开放API已经支持了Range
, If-Match
,If-None-Match
,If-Modified-Since
,If-Unmodified-Since
,所有我们能够在移动端实践OSS资源的断点下载功能。
最佳实践
首先来看一下流程图
效果图:
这里以iOS下载实现思路为例,参考代码如下 仅供参考,切不可用于生产!
#import "DownloadService.h"
#import "OSSTestMacros.h"
@implementation DownloadRequest
@end
@implementation Checkpoint
- (instancetype)copyWithZone:(NSZone *)zone {
Checkpoint *other = [[[self class] allocWithZone:zone] init];
other.etag = self.etag;
other.totalExpectedLength = self.totalExpectedLength;
return other;
}
@end
@interface DownloadService()<NSURLSessionTaskDelegate, NSURLSessionDataDelegate>
@property (nonatomic, strong) NSURLSession *session; //网络会话
@property (nonatomic, strong) NSURLSessionDataTask *dataTask; //数据请求任务
@property (nonatomic, copy) DownloadFailureBlock failure; //请求出错
@property (nonatomic, copy) DownloadSuccessBlock success; //请求成功
@property (nonatomic, copy) DownloadProgressBlock progress; //下载进度
@property (nonatomic, copy) Checkpoint *checkpoint; //检查节点
@property (nonatomic, copy) NSString *requestURLString; //文件资源地址,用于下载请求
@property (nonatomic, copy) NSString *headURLString; //文件资源地址,用于head请求
@property (nonatomic, copy) NSString *targetPath; //文件存储路径
@property (nonatomic, assign) unsigned long long totalReceivedContentLength; //已下载大小
@property (nonatomic, strong) dispatch_semaphore_t semaphore;
@end
@implementation DownloadService
- (instancetype)init
{
self = [super init];
if (self) {
NSURLSessionConfiguration *conf = [NSURLSessionConfiguration defaultSessionConfiguration];
conf.timeoutIntervalForRequest = 15;
NSOperationQueue *processQueue = [NSOperationQueue new];
_session = [NSURLSession sessionWithConfiguration:conf delegate:self delegateQueue:processQueue];
_semaphore = dispatch_semaphore_create(0);
_checkpoint = [[Checkpoint alloc] init];
}
return self;
}
+ (instancetype)downloadServiceWithRequest:(DownloadRequest *)request {
DownloadService *service = [[DownloadService alloc] init];
if (service) {
service.failure = request.failure;
service.success = request.success;
service.requestURLString = request.sourceURLString;
service.headURLString = request.headURLString;
service.targetPath = request.downloadFilePath;
service.progress = request.downloadProgress;
if (request.checkpoint) {
service.checkpoint = request.checkpoint;
}
}
return service;
}
/**
* head文件信息,取出来文件的etag和本地checkpoint中保存的etag进行对比,并且将结果返回
*/
- (BOOL)getFileInfo {
__block BOOL resumable = NO;
NSURL *url = [NSURL URLWithString:self.headURLString];
NSMutableURLRequest *request = [[NSMutableURLRequest alloc]initWithURL:url];
[request setHTTPMethod:@"HEAD"];
NSURLSessionDataTask *task = [[NSURLSession sharedSession] dataTaskWithRequest:request completionHandler:^(NSData * _Nullable data, NSURLResponse * _Nullable response, NSError * _Nullable error) {
if (error) {
NSLog(@"获取文件meta信息失败,error : %@", error);
} else {
NSHTTPURLResponse *httpResponse = (NSHTTPURLResponse *)response;
NSString *etag = [httpResponse.allHeaderFields objectForKey:@"Etag"];
if ([self.checkpoint.etag isEqualToString:etag]) {
resumable = YES;
} else {
resumable = NO;
}
}
dispatch_semaphore_signal(self.semaphore);
}];
[task resume];
dispatch_semaphore_wait(self.semaphore, DISPATCH_TIME_FOREVER);
return resumable;
}
/**
* 用于获取本地文件的大小
*/
- (unsigned long long)fileSizeAtPath:(NSString *)filePath {
unsigned long long fileSize = 0;
NSFileManager *dfm = [NSFileManager defaultManager];
if ([dfm fileExistsAtPath:filePath]) {
NSError *error = nil;
NSDictionary *attributes = [dfm attributesOfItemAtPath:filePath error:&error];
if (!error && attributes) {
fileSize = attributes.fileSize;
} else if (error) {
NSLog(@"error: %@", error);
}
}
return fileSize;
}
- (void)resume {
NSURL *url = [NSURL URLWithString:self.requestURLString];
NSMutableURLRequest *request = [[NSMutableURLRequest alloc]initWithURL:url];
[request setHTTPMethod:@"GET"];
BOOL resumable = [self getFileInfo]; // 如果resumable为NO,则证明不能断点续传,否则走续传逻辑。
if (resumable) {
self.totalReceivedContentLength = [self fileSizeAtPath:self.targetPath];
NSString *requestRange = [NSString stringWithFormat:@"bytes=%llu-", self.totalReceivedContentLength];
[request setValue:requestRange forHTTPHeaderField:@"Range"];
} else {
self.totalReceivedContentLength = 0;
}
if (self.totalReceivedContentLength == 0) {
[[NSFileManager defaultManager] createFileAtPath:self.targetPath contents:nil attributes:nil];
}
self.dataTask = [self.session dataTaskWithRequest:request];
[self.dataTask resume];
}
- (void)pause {
[self.dataTask cancel];
self.dataTask = nil;
}
- (void)cancel {
[self.dataTask cancel];
self.dataTask = nil;
[self removeFileAtPath: self.targetPath];
}
- (void)removeFileAtPath:(NSString *)filePath {
NSError *error = nil;
[[NSFileManager defaultManager] removeItemAtPath:self.targetPath error:&error];
if (error) {
NSLog(@"remove file with error : %@", error);
}
}
#pragma mark - NSURLSessionDataDelegate
- (void)URLSession:(NSURLSession *)session task:(NSURLSessionTask *)task
didCompleteWithError:(nullable NSError *)error {
NSHTTPURLResponse *httpResponse = (NSHTTPURLResponse *)task.response;
if ([httpResponse isKindOfClass:[NSHTTPURLResponse class]]) {
if (httpResponse.statusCode == 200) {
self.checkpoint.etag = [[httpResponse allHeaderFields] objectForKey:@"Etag"];
self.checkpoint.totalExpectedLength = httpResponse.expectedContentLength;
} else if (httpResponse.statusCode == 206) {
self.checkpoint.etag = [[httpResponse allHeaderFields] objectForKey:@"Etag"];
self.checkpoint.totalExpectedLength = self.totalReceivedContentLength + httpResponse.expectedContentLength;
}
}
if (error) {
if (self.failure) {
NSMutableDictionary *userInfo = [NSMutableDictionary dictionaryWithDictionary:error.userInfo];
[userInfo oss_setObject:self.checkpoint forKey:@"checkpoint"];
NSError *tError = [NSError errorWithDomain:error.domain code:error.code userInfo:userInfo];
self.failure(tError);
}
} else if (self.success) {
self.success(@{@"status": @"success"});
}
}
- (void)URLSession:(NSURLSession *)session dataTask:(NSURLSessionDataTask *)dataTask didReceiveResponse:(NSURLResponse *)response completionHandler:(void (^)(NSURLSessionResponseDisposition))completionHandler
{
NSHTTPURLResponse *httpResponse = (NSHTTPURLResponse *)dataTask.response;
if ([httpResponse isKindOfClass:[NSHTTPURLResponse class]]) {
if (httpResponse.statusCode == 200) {
self.checkpoint.totalExpectedLength = httpResponse.expectedContentLength;
} else if (httpResponse.statusCode == 206) {
self.checkpoint.totalExpectedLength = self.totalReceivedContentLength + httpResponse.expectedContentLength;
}
}
completionHandler(NSURLSessionResponseAllow);
}
- (void)URLSession:(NSURLSession *)session dataTask:(NSURLSessionDataTask *)dataTask didReceiveData:(NSData *)data {
NSFileHandle *fileHandle = [NSFileHandle fileHandleForWritingAtPath:self.targetPath];
[fileHandle seekToEndOfFile];
[fileHandle writeData:data];
[fileHandle closeFile];
self.totalReceivedContentLength += data.length;
if (self.progress) {
self.progress(data.length, self.totalReceivedContentLength, self.checkpoint.totalExpectedLength);
}
}
@end
上面展示的是从网络接收数据的处理逻辑,DownloadService是下载逻辑的核心。在URLSession:dataTask:didReceiveData中将接收到的网络数据按照追加的方式写入到文件中,并更新下载进度。在URLSession:task:didCompleteWithError判断下载任务是否完成,然后将结果返回给上层业务。URLSession:dataTask:didReceiveResponse:completionHandler代理方法中将object相关的信息,比如etag用于断点续传是的precheck,content-length用于计算下载进度。
#import <Foundation/Foundation.h>
typedef void(^DownloadProgressBlock)(int64_t bytesReceived, int64_t totalBytesReceived, int64_t totalBytesExpectToReceived);
typedef void(^DownloadFailureBlock)(NSError *error);
typedef void(^DownloadSuccessBlock)(NSDictionary *result);
@interface Checkpoint : NSObject<NSCopying>
@property (nonatomic, copy) NSString *etag; // 资源的etag值
@property (nonatomic, assign) unsigned long long totalExpectedLength; //文件总大小
@end
@interface DownloadRequest : NSObject
@property (nonatomic, copy) NSString *sourceURLString; // 用于下载的url
@property (nonatomic, copy) NSString *headURLString; // 用于获取文件原信息的url
@property (nonatomic, copy) NSString *downloadFilePath; // 文件的本地存储地址
@property (nonatomic, copy) DownloadProgressBlock downloadProgress; // 下载进度
@property (nonatomic, copy) DownloadFailureBlock failure; // 下载成功的回调
@property (nonatomic, copy) DownloadSuccessBlock success; // 下载失败的回调
@property (nonatomic, copy) Checkpoint *checkpoint; // checkpoint,用于存储文件的etag
@end
@interface DownloadService : NSObject
+ (instancetype)downloadServiceWithRequest:(DownloadRequest *)request;
/**
* 启动下载
*/
- (void)resume;
/**
* 暂停下载
*/
- (void)pause;
/**
* 取消下载
*/
- (void)cancel;
@end
上面这一部分是DownloadRequest的定义。
- (void)initDownloadURLs {
OSSPlainTextAKSKPairCredentialProvider *pCredential = [[OSSPlainTextAKSKPairCredentialProvider alloc] initWithPlainTextAccessKey:OSS_ACCESSKEY_ID secretKey:OSS_SECRETKEY_ID];
_mClient = [[OSSClient alloc] initWithEndpoint:OSS_ENDPOINT credentialProvider:pCredential];
// 生成用于get请求的带签名的url
OSSTask *downloadURLTask = [_mClient presignConstrainURLWithBucketName:@"aliyun-dhc-shanghai" withObjectKey:OSS_DOWNLOAD_FILE_NAME withExpirationInterval:1800];
[downloadURLTask waitUntilFinished];
_downloadURLString = downloadURLTask.result;
// 生成用于head请求的带签名的url
OSSTask *headURLTask = [_mClient presignConstrainURLWithBucketName:@"aliyun-dhc-shanghai" withObjectKey:OSS_DOWNLOAD_FILE_NAME httpMethod:@"HEAD" withExpirationInterval:1800 withParameters:nil];
[headURLTask waitUntilFinished];
_headURLString = headURLTask.result;
}
- (IBAction)resumeDownloadClicked:(id)sender {
_downloadRequest = [DownloadRequest new];
_downloadRequest.sourceURLString = _downloadURLString; // 设置资源的url
_downloadRequest.headURLString = _headURLString;
NSString *documentPath = NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES).firstObject;
_downloadRequest.downloadFilePath = [documentPath stringByAppendingPathComponent:OSS_DOWNLOAD_FILE_NAME]; //设置下载文件的本地保存路径
__weak typeof(self) wSelf = self;
_downloadRequest.downloadProgress = ^(int64_t bytesReceived, int64_t totalBytesReceived, int64_t totalBytesExpectToReceived) {
// totalBytesReceived是当前客户端已经缓存了的字节数,totalBytesExpectToReceived是总共需要下载的字节数。
dispatch_async(dispatch_get_main_queue(), ^{
__strong typeof(self) sSelf = wSelf;
CGFloat fProgress = totalBytesReceived * 1.f / totalBytesExpectToReceived;
sSelf.progressLab.text = [NSString stringWithFormat:@"%.2f%%", fProgress * 100];
sSelf.progressBar.progress = fProgress;
});
};
_downloadRequest.failure = ^(NSError *error) {
__strong typeof(self) sSelf = wSelf;
sSelf.checkpoint = error.userInfo[@"checkpoint"];
};
_downloadRequest.success = ^(NSDictionary *result) {
NSLog(@"下载成功");
};
_downloadRequest.checkpoint = self.checkpoint;
NSString *titleText = [[_downloadButton titleLabel] text];
if ([titleText isEqualToString:@"download"]) {
[_downloadButton setTitle:@"pause" forState: UIControlStateNormal];
_downloadService = [DownloadService downloadServiceWithRequest:_downloadRequest];
[_downloadService resume];
} else {
[_downloadButton setTitle:@"download" forState: UIControlStateNormal];
[_downloadService pause];
}
}
- (IBAction)cancelDownloadClicked:(id)sender {
[_downloadButton setTitle:@"download" forState: UIControlStateNormal];
[_downloadService cancel];
}
这一部分是在上层业务的调用。暂停或取消上传时均能从failure回调中获取到checkpoint,重启下载时可以将checkpoint传到DownloadRequest,然后DownloadService内部会使用checkpoint做一致性的校验。
关于安卓里面实现对OSS Object的断点下载可以参考 基于okhttp3实现的断点下载功能实现 这个开源工程。这里列出如何使用这个示例工程下载OSS资源。
//1. 首先使用sdk获取object的下载链接
String signedURLString = ossClient.presignConstrainedObjectURL(bucket, object, expires);
//2. 添加下载任务
mDownloadManager = DownloadManager.getInstance();
mDownloadManager.add(signedURLString, new DownloadListner() {
@Override
public void onFinished() {
Toast.makeText(MainActivity.this, "下载完成!", Toast.LENGTH_SHORT).show();
}
@Override
public void onProgress(float progress) {
pb_progress1.setProgress((int) (progress * 100));
tv_progress1.setText(String.format("%.2f", progress * 100) + "%");
}
@Override
public void onPause() {
Toast.makeText(MainActivity.this, "暂停了!", Toast.LENGTH_SHORT).show();
}
@Override
public void onCancel() {
tv_progress1.setText("0%");
pb_progress1.setProgress(0);
btn_download1.setText("下载");
Toast.makeText(MainActivity.this, "下载已取消!", Toast.LENGTH_SHORT).show();
}
});
//3.开启下载
mDownloadManager.download(signedURLString);
//4.暂停下载
mDownloadManager.cancel(signedURLString);
//5.继续下载
mDownloadManager.download(signedURLString);
改进
HTTP1.1里的If-Range,我们来看针对该header的描述:
If-Range HTTP 请求头字段用来使得 Range 头字段在一定条件下起作用:当字段值中的条件得到满足时,Range 头字段才会起作用,同时服务器回复206 部分内容状态码,以及Range 头字段请求的相应部分;如果字段值中的条件没有得到满足,服务器将会返回 200 OK 状态码,并返回完整的请求资源。
字段值中既可以用 Last-Modified 时间值用作验证,也可以用ETag标记作为验证,但不能将两者同时使用。
If-Range 头字段通常用于断点续传的下载过程中,用来自从上次中断后,确保下载的资源没有发生改变。
使用If-Range
的优点:客户端只需要一次网络请求,而前面所讲的If-Unmodified-Since
或者If-Match
在条件判断失败时,会返回412前置条件检查失败状态码,客户端不得不再开启一个请求来获取资源。
目前OSS Server还不支持If-Range
字段。而在iOS中使用NSURLSessionDownloadTask
实现断点下载时会发送If-Range
给服务器以确认文件是否发生过变化。所以目前还不能使用NSURLSessionDownloadTask
去实现断点下载。
参考
https://developer.mozilla.org/zh-CN/docs/Web/HTTP/Status/412