开发者社区> 问答> 正文

如何在spark中使用org.slf4j.Logger?

我正试图用org.slf4j.Logger spark。如果我写如下,我将得到错误non-static field cannot be referenced from a static context。因为该方法main是静态的,但是logger非静态的。

import org.apache.spark.api.java.*;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.function.Function;
import org.apache.spark.broadcast.Broadcast;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class simpleApp {

private final Logger logger = LoggerFactory.getLogger(getClass());

public static void main(String[] args) {
    String logFile = "/user/beibei/zhaokai/spark_java/a.txt"; // Should be some file on your system
    SparkConf conf = new SparkConf().setAppName("Simple Application");
    JavaSparkContext sc = new JavaSparkContext(conf);
    JavaRDD<String> logData = sc.textFile(logFile).cache();

    logger.info("loading graph from cache");

    long numAs = logData.filter(new Function<String, Boolean>() {
        public Boolean call(String s) { return s.contains("a"); }
    }).count();

    long numBs = logData.filter(new Function<String, Boolean>() {
        public Boolean call(String s) { return s.contains("t"); }
    }).count();

    System.out.println("Lines with a: " + numAs + ", lines with t: " + numBs);
}

}
但是,如果我写的如下。我会得到另一个

错误线程“main”中的异常org.apache.spark.SparkException:任务不可序列化。

因为Class of Class simpleApp不可序列化。

import org.apache.spark.api.java.*;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.function.Function;
import org.apache.spark.broadcast.Broadcast;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class simpleApp {

private final Logger logger = LoggerFactory.getLogger(getClass());

public static void main(String[] args) {
    new simpleApp().start();
}

private void start() {
    String logFile = "/path/a.txt"; // Should be some file on your system
    SparkConf conf = new SparkConf().setAppName("Simple Application");
    JavaSparkContext sc = new JavaSparkContext(conf);
    JavaRDD<String> logData = sc.textFile(logFile).cache();

    logger.info("loading graph from cache");

    long numAs = logData.filter(new Function<String, Boolean>() {
        public Boolean call(String s) { return s.contains("a"); }
    }).count();

    long numBs = logData.filter(new Function<String, Boolean>() {
        public Boolean call(String s) { return s.contains("t"); }
    }).count();

    System.out.println("Lines with a: " + numAs + ", lines with t: " + numBs);
}

}
如果我想使用其他类似的软件包org.slf4j.Logger,我会遇到同样的问题吗?

展开
收起
社区小助手 2018-12-05 14:05:35 5140 0
1 条回答
写回答
取消 提交回答
  • 社区小助手是spark中国社区的管理员,我会定期更新直播回顾等资料和文章干货,还整合了大家在钉群提出的有关spark的问题及回答。

    可能有几个选项....我会提供由spark 提供的org.apache.spark.internal.Logging。

    Doc说:

    /**

    • Utility trait for classes that want to log data. Creates a SLF4J logger for the class and allows
    • logging messages at different levels using methods that only evaluate parameters lazily if the
    • log level is enabled.
      */

    private def isLog4j12(): Boolean = {
    // This distinguishes the log4j 1.2 binding, currently
    // org.slf4j.impl.Log4jLoggerFactory, from the log4j 2.0 binding, currently
    // org.apache.logging.slf4j.Log4jLoggerFactory
    val binderClass = StaticLoggerBinder.getSingleton.getLoggerFactoryClassStr
    "org.slf4j.impl.Log4jLoggerFactory".equals(binderClass)
    }
    如果你想使用spark提供的api自己做同样的事情,你可以模仿相同的。

    注意:在上面的方法中。要调整日志记录级别,请使用sc.setLogLevel(newLevel)。对于SparkR,请使用setLogLevel(newLevel)。

    2019-07-17 23:18:21
    赞同 展开评论 打赏
问答分类:
问答地址:
问答排行榜
最热
最新

相关电子书

更多
低代码开发师(初级)实战教程 立即下载
冬季实战营第三期:MySQL数据库进阶实战 立即下载
阿里巴巴DevOps 最佳实践手册 立即下载