业务需要将大文件切分成一块一块的小文件进行传输,接收方需要将一块一块的文件组装起来,起到快速上传的功能。代码如下:
classes= {AppApplication.class}) (SpringRunner.class) (publicclassFileReadTest { privatestaticfinalintCHUNK_SIZE=8*1024*1024; // 8MB chunk sizeprivatestaticfinalintMAX_CONCURRENT_UPLOADS=10; // Maximum number of concurrent uploadspublicvoidtestFileUploadList() throwsIOException, NoSuchAlgorithmException, ExecutionException, InterruptedException { StringfilePath="C:\\tools\\openjdk-17.0.2_windows-x64_bin.zip"; ExecutorServiceexecutorService=Executors.newFixedThreadPool(MAX_CONCURRENT_UPLOADS); List<Future<String>>futureList=newArrayList<>(); try (InputStreamis=newFileInputStream(filePath)) { Filefile=newFile(filePath); longfileSize=file.length(); intchunkIndex=0; intbytesRead=0; byte[] buffer=newbyte[CHUNK_SIZE]; intreadSum=0; while ((bytesRead=is.read(buffer)) !=-1&& (fileSize-readSum) >0) { readSum+=bytesRead; Stringmd5=DigestUtils.md5Hex(buffer); Stringnewurl=""; StringbytesStr=chunkIndex*CHUNK_SIZE+"-"+Math.min((chunkIndex+1) *CHUNK_SIZE-1, fileSize-1) +"/"+fileSize; FileTransRequestfileTransRequest=newFileTransRequest(newurl,buffer,bytesStr,chunkIndex,bytesRead,null,md5); Future<String>future=executorService.submit(fileTransRequest); futureList.add(future); chunkIndex++; if(fileSize-readSum<CHUNK_SIZE){ buffer=newbyte[(int)(fileSize-readSum)]; }else{ buffer=newbyte[CHUNK_SIZE]; } } } if(futureList.size()>0){ for(Future<String>future:futureList){ Stringres=future.get(); if(StringUtils.hasText(res)){ System.out.println(res); } } } executorService.shutdown(); } }
publicclassFileTransRequestimplementsCallable<String> { privateStringurl; privatebyte[] buffer; privateStringbytesStr; privateintchunkIndex; privateintchunkSize; privateMap<String, String>httpHeaderMap; privateStringmd5; publicFileTransRequest(Stringurl, byte[] buffer, StringbytesStr, intchunkIndex, intchunkSize, Map<String, String>httpHeaderMap, Stringmd5){ this.url=url; this.buffer=buffer; this.bytesStr=bytesStr; this.chunkIndex=chunkIndex; this.chunkSize=chunkSize; this.httpHeaderMap=httpHeaderMap; this.md5=md5; } publicStringcall() throwsException { returnchunkIndex+"=="+chunkSize+"==="+md5+">>>"+DigestUtils.md5Hex(buffer); } }
可以在FileTransRequest类中实现文件的网络传输的功能,在FileReadTest类中可以实现处理并发结束以后的文件合并以及将结果存储起来的功能,使用到了apache common-codec的获取文件分块的md5的技术,可以在服务器端对文件的完整性做校验。