开发者学堂课程【大数据实时计算框架 Spark 快速入门：Spark textFile 和排序-1】学习笔记，与课程紧密联系，让用户快速学习知识。
课程地址：https://developer.aliyun.com/learning/course/100/detail/1694
Spark textFile 和排序-1
内容简介：
一、数据序列化
二、Java 序列化
三、Kryo 序列化
一、Data Serialization（数据序列化）
Serlalzation plays an Important role In the performance of any distributed application. Formats that are slow to serialize objects into, or consume a large number 0f bytes, will greatly slow down the computation. Often, this will be the first thing you should tune to optimize a Spark application.Spark aims to strike a balance between convenience (allowing you to work with any Java type in your operations) and performance. It provides two serialization libraries:
译文：服务化在任何分布式应用程序的性能中起着重要的作用。将对象序列化为Into 或消耗大量字节的格式将大大降低计算速度。通常，这将是优化 Spark 应用程序的第一件事。
Spark 的目标是在方便性(允许您在操作中使用任何 Java 类型)和性能之间取得平衡。它提供了两个分类图书馆。
二、Java 序列化
Java serialization: By default, Spark serializes objects using Java's objectoutputStream framework, and can work with any class you createthat implements java. io. Serializable. You can also control the performance of your serialization more closely by extendingjava. io. Externalizable. Java seralization Is flexible but often quite slow, and leads to large serallzed ormats for many classes.
译文：
Java 序列化:默认情况下，Spark 使用 Java 的 objectoutputStream 框架序列化对象，并且可以使用您创建的实现 Java 的任何类。还可以通过扩展 java 来更密切地控制序列化的性能。外置 Java 服务器化是灵活的，但通常非常慢，并导致了许多类的大的服务器化格式。
三、Kryo 序列化
Kryo serialization: Spark can also use the Kryo library(version 2) to serialize objects more quickly. Kryo is significantly faster and more compact than Java serialization (often as much as 10x), but does not support all Serializable types and requires you to register the classes you'l l use in the program in advance for best performance.
译文：Kryo 序列化: Spark 还可以使用 Kryo 库](版本2)来更快地序列化对象。Kryo比 Java 序列化更快、更紧凑(通常高10倍)，但并不支持所有可序列化的类型，并且需要注册类。你可以提前在程序中使用In来获得最佳性能。
You can switch to using Kryo by initializing your job with a SparkConf and calling conf set("spark. serializer","org. apache. spark. serializer. KryoSerializer"), This setting configures the serializer used for not only shuffling data between worker nodesbut also when serializing RDDs to disk. The only reason Kryo is not the default Is because 0f the custom registration requirement, but we recommend trying it in any network-Intensive application.
Spark automatically includes Kryo serializers for the many commonly-used core Scala classes covered in the AIIScalaRegistrar from the Twitterchill library.
To register your own custom classes with Kryo, use the registerKryoClasses method.
val conf= new SparkConf(). setMaster(...). setAppName(...)
conf. registerKryoC1 asses (Array(classof [MyC1ass1], c1ass0f MyC1ass2]))
val sc = new SparkContext(conf)
译文：您可以通过使用 Spark Conf 初始化作业并调用conf来切换到使用 Kryo。
此设置不仅配置用于在辅助节点之间交换数据的序列化器，而且还配置用于序列化时的数据交换器 RDDs 磁盘。Kryo 不是默认的唯一原因是因为of的自定义注册要求，但我们建议尝试它在任何网络密集型应用中。Spark 自动包含了许多常用核心Scala 类的 Kryo 序列化器，这些类包含在 Twitter chill 的 AIIScalaRegistrar 中要用 Kryo 注册自己的定制类，请使用 registerKryoclasses 方法。
The Kryo documentation describes more advanced registration options, such as adding custom serialization code.if your objects are large, you may also need to increase the spark. kryoserializer. buffer config. This value needs to be large enough to hold the largest object you will serialize.
Finally, if you don't register your custom classes, Kryo will still work, but it will have to store the full class name with each object. which is wasteful.
译文：Kryo 文档描述了更高级的注册选项，比如添加自定义序列化代码。
当你的对象很大时你可能还需要增加火花。Kypo srialize .buffer 缓冲区配置这个Value需要足够大的空间来容纳要序列化的最大对象。最后，如果您不注册您的自定义类，Kryo 将仍然工作，但它将不得不与每个对象存储完整的类名，这是浪费。
