MappedByteBuffer是java nio引入的文件内存映射方案,读写性能极高。在NIO中主要用到普通的输入流,带缓冲的输入流,RandomAccessFile和MappedByteBuffer。
@Test public void mmapTest() throws Exception { FileChannel fileChannel = null; try { String filePath = "D:\\temp\\avatar.jpg"; fileChannel = new RandomAccessFile(new File(filePath), "rw").getChannel(); MappedByteBuffer mappedByteBuffer = fileChannel.map(FileChannel.MapMode.READ_WRITE, 0, fileChannel.size()); byte[] bytes = new byte[(int) fileChannel.size()]; mappedByteBuffer.get(bytes); print(bytes); } catch (Exception e) { e.printStackTrace(); } finally { if (fileChannel != null) { try { fileChannel.close(); } catch (IOException e) { e.printStackTrace(); } } } } public static void print(byte[] bytes) { for (int i = 0; i < bytes.length; i++) { System.out.printf("0x%02X ", bytes[i]); if ((i + 1) % 8 == 0 || i + 1 == bytes.length) { System.out.print("\n"); } } }
MapMode mode:内存映像文件访问的方式,共三种:
1. MapMode.READ_ONLY:只读,试图修改得到的缓冲区将导致抛出异常。
2. MapMode.READ_WRITE:读/写,对得到的缓冲区的更改最终将写入文件;但该更改对映射到同一文件的其他程序不一定是可见的。
3. MapMode.PRIVATE:私用,可读可写,但是修改的内容不会写入文件,只是buffer自身的改变,这种能力称之为”copy on write”。
通过内存映射的方法访问硬盘上的文件,效率要比read和write系统调用高,这是为什么?
read()是系统调用,首先将文件从硬盘拷贝到内核空间的一个缓冲区,再将这些数据拷贝到用户空间,实际上进行了两次数据拷贝;
map()也是系统调用,但没有进行数据拷贝,当缺页中断发生时,直接将文件从硬盘拷贝到用户空间,只进行了一次数据拷贝。
所以,采用内存映射的读写效率要比传统的read/write性能高。
总结
1. MappedByteBuffer使用虚拟内存,因此分配(map)的内存大小不受JVM的-Xmx参数限制,但是也是有大小限制的。
2. 如果当文件超出1.5G限制时,可以通过position参数重新map文件后面的内容。
3. MappedByteBuffer在处理大文件时的确性能很高,但也存在一些问题,如内存占用、文件关闭不确定,被其打开的文件只有在垃圾回收的才会被关闭,而且这个时间点是不确定的。
javadoc中也提到:A mapped byte buffer and the file mapping that it represents remain valid until the buffer itself is garbage-collected.*
MappedByteBuffer效率最高,所消耗的时间最少【网上看到的示例,结果好像不准,只是记录对文件不同的读法】
@Test void mmapTest() throws Exception { String name = "D:\\temp\\avatar.jpg"; Path filename = Paths.get(name); long start, crcValue, end; System.out.println("Input Stream:"); start = System.currentTimeMillis(); crcValue = checksumInputStream(filename); end = System.currentTimeMillis(); System.out.println("CRC:"+Long.toHexString(crcValue)); System.out.println((end - start) + " milliseconds"); System.out.println("-----------------"); System.out.println("Buffered Input Stream:"); start = System.currentTimeMillis(); crcValue = checksumBufferedInputStream(filename); end = System.currentTimeMillis(); System.out.println("CRC:"+Long.toHexString(crcValue)); System.out.println((end - start) + " milliseconds"); System.out.println("-----------------"); System.out.println("Random Access File:"); start = System.currentTimeMillis(); crcValue = checksumRandomAccessFile(filename); end = System.currentTimeMillis(); System.out.println("CRC:"+Long.toHexString(crcValue)); System.out.println((end - start) + " milliseconds"); System.out.println("-----------------"); System.out.println("Mapped File:"); start = System.currentTimeMillis(); crcValue = checksumMappedFile(filename); end = System.currentTimeMillis(); System.out.println("CRC:"+Long.toHexString(crcValue)); System.out.println((end - start) + " milliseconds"); } public long checksumInputStream(Path filename) throws IOException { try (InputStream in = Files.newInputStream(filename)) { CRC32 crc = new CRC32(); int c; while ((c = in.read()) != -1) crc.update(c); return crc.getValue(); } } public static long checksumBufferedInputStream(Path filename) throws IOException { try (InputStream in = new BufferedInputStream(Files.newInputStream(filename))) { CRC32 crc = new CRC32(); int c; while ((c = in.read()) != -1) crc.update(c); return crc.getValue(); } } public static long checksumRandomAccessFile(Path filename) throws IOException { try (RandomAccessFile file = new RandomAccessFile(filename.toFile(), "r")) { long length = file.length(); CRC32 crc = new CRC32(); for (long p = 0; p < length; p++) { file.seek(p); int c = file.readByte(); crc.update(c); } return crc.getValue(); } } public static long checksumMappedFile(Path filename) throws IOException { try (FileChannel channel = FileChannel.open(filename)) { CRC32 crc = new CRC32(); int length = (int) channel.size(); MappedByteBuffer buffer = channel.map(FileChannel.MapMode.READ_ONLY, 0, length); for (int p = 0; p < length; p++) { int c = buffer.get(p); crc.update(c); } return crc.getValue(); } }