【最佳实践】OSS开源工具ossutil-调节并发参数[英文版]-阿里云开发者社区

开发者社区> 阿里云存储服务> 正文

【最佳实践】OSS开源工具ossutil-调节并发参数[英文版]

简介: 前言 前面支持一个国外客户,使用ossutil上传文件到OSS,一直调整不好--jobs和--parallel参数。因此给客户写了一篇简单的英文文档。 用户可从这里获取ossutil。 官网: https://help.

前言

前面支持一个国外客户,使用ossutil上传文件到OSS,一直调整不好--jobs和--parallel参数。因此给客户写了一篇简单的英文文档。

用户可从这里获取ossutil。

Concurrency Control

--jobs or --parallel

--jobs controls the amount of concurrency tasks between multi-files.
--parallel controls the amount of concurrency tasks within a file.
  • By default, ossutil will calculate the parallel num according to file size, if file size > 100MB.
  • So --parallel is useless to small file.
  • The file size to use multipart upload can be specified by --bigfile-threshold, the default value is 100MB (104857600).
  • Therefore when batch upload/download/copy files, the total concurrency tasks num

     ```concurrency = jobs * parallel```
  • The two option can be specified by user, if the performance of default setting is poor, user can adjust the two options.

Note:

  • If parallel and jobs are too big, because of the switching between threads, the performance of upload/download/copy may decline, so please set the options according to your machine condition.
  • If need performance tuning, user can set the two options to small numbers at first and increase them step by step.
  • If parallel and jobs are too big, in case of limited machine resources, error "EOF" may occur due to the network transfer too slow, in this situation, please reduce the --jobs and --parallel num.

Best Practice

1. For single small file

e.g., test1.log,

  • file-size = 50MB
  • And user expect to upload in MPU (MultiPart Upload)

Then need to specify --bigfile-threshold less than 50MB, such as, 20MB (20971520),

ossutil cp test1.log oss://<bucket_name> --bigfile-threshold=20971520

Then, ossutil will upload test1.log in MPU.

2. For single big file

e.g., test2.log,

  • file-size = 150MB
  • And user expect to upload in MPU

Then no need to specify --bigfile-threshold because file-size is more than the default value (100MB).

ossutil cp test2.log oss://<bucket_name>

Then, ossutil will upload test2.log in MPU.

3. For multi files

Of course, --bigfile-threshold will affect each file for this case.

3.1 If --jobs and parallel not specified

e.g.,
ossutil cp <local_dir> oss://<bucket_name>

Then concurrency will be calculated automatically, and

  • If ossutil version <= 1.4.0
    max parallel is 15. By default jobs = 5, so
    max concurrency = 75
  • If ossutil version = 1.4.1
    max parallel is 12. By default jobs = 3, so
    max concurrency = 36

3.2 If --jobs and parallel specified

e.g.,
ossutil cp <local_dir> oss://<bucket_name> --jobs=3 --parallel=8

Then,
concurrency = jobs * parallel = 24

版权声明:本文内容由阿里云实名注册用户自发贡献,版权归原作者所有,阿里云开发者社区不拥有其著作权,亦不承担相应法律责任。具体规则请查看《阿里云开发者社区用户服务协议》和《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容,填写侵权投诉表单进行举报,一经查实,本社区将立刻删除涉嫌侵权内容。

分享:

阿里云存储基于飞天盘古2.0分布式存储系统,产品多种多样,充分满足用户数据存储和迁移上云需求。

官方博客
链接