我正试图用org.slf4j.Logger spark。如果我写如下,我将得到错误non-static field cannot be referenced from a static context。因为该方法main是静态的,但是logger非静态的。
import org.apache.spark.api.java.*;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.function.Function;
import org.apache.spark.broadcast.Broadcast;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class simpleApp {
private final Logger logger = LoggerFactory.getLogger(getClass());
public static void main(String[] args) {
String logFile = "/user/beibei/zhaokai/spark_java/a.txt"; // Should be some file on your system
SparkConf conf = new SparkConf().setAppName("Simple Application");
JavaSparkContext sc = new JavaSparkContext(conf);
JavaRDD<String> logData = sc.textFile(logFile).cache();
logger.info("loading graph from cache");
long numAs = logData.filter(new Function<String, Boolean>() {
public Boolean call(String s) { return s.contains("a"); }
}).count();
long numBs = logData.filter(new Function<String, Boolean>() {
public Boolean call(String s) { return s.contains("t"); }
}).count();
System.out.println("Lines with a: " + numAs + ", lines with t: " + numBs);
}
}
但是,如果我写的如下。我会得到另一个
错误线程“main”中的异常org.apache.spark.SparkException:任务不可序列化。
因为Class of Class simpleApp不可序列化。
import org.apache.spark.api.java.*;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.function.Function;
import org.apache.spark.broadcast.Broadcast;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class simpleApp {
private final Logger logger = LoggerFactory.getLogger(getClass());
public static void main(String[] args) {
new simpleApp().start();
}
private void start() {
String logFile = "/path/a.txt"; // Should be some file on your system
SparkConf conf = new SparkConf().setAppName("Simple Application");
JavaSparkContext sc = new JavaSparkContext(conf);
JavaRDD<String> logData = sc.textFile(logFile).cache();
logger.info("loading graph from cache");
long numAs = logData.filter(new Function<String, Boolean>() {
public Boolean call(String s) { return s.contains("a"); }
}).count();
long numBs = logData.filter(new Function<String, Boolean>() {
public Boolean call(String s) { return s.contains("t"); }
}).count();
System.out.println("Lines with a: " + numAs + ", lines with t: " + numBs);
}
}
如果我想使用其他类似的软件包org.slf4j.Logger,我会遇到同样的问题吗?
可能有几个选项....我会提供由spark 提供的org.apache.spark.internal.Logging。
Doc说:
/**
private def isLog4j12(): Boolean = {
// This distinguishes the log4j 1.2 binding, currently
// org.slf4j.impl.Log4jLoggerFactory, from the log4j 2.0 binding, currently
// org.apache.logging.slf4j.Log4jLoggerFactory
val binderClass = StaticLoggerBinder.getSingleton.getLoggerFactoryClassStr
"org.slf4j.impl.Log4jLoggerFactory".equals(binderClass)
}
如果你想使用spark提供的api自己做同样的事情,你可以模仿相同的。
注意:在上面的方法中。要调整日志记录级别,请使用sc.setLogLevel(newLevel)。对于SparkR,请使用setLogLevel(newLevel)。
版权声明:本文内容由阿里云实名注册用户自发贡献,版权归原作者所有,阿里云开发者社区不拥有其著作权,亦不承担相应法律责任。具体规则请查看《阿里云开发者社区用户服务协议》和《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容,填写侵权投诉表单进行举报,一经查实,本社区将立刻删除涉嫌侵权内容。