StreamingPro 可以暴露出原生API给大家使用

我们知道StreamingPro 是一个完全SQL/Script化的，基于Spark平台的一套生产力工具。但是不可避免的，我们可能希望直接操作SqlContext或者使用原生的DataFrame API。 这里我们通过script 让大家支持这个功能：

{
        "name": "batch.script.df",
        "params": [
          {
            "script": "context.sql(\"select a as t from test\").registerTempTable(\"finalOutputTable\")",
            "source": "-"
          }
        ]
      }

        

          
        
        
        

          

          AI 代码解读
        
      
      

这里给大家一个完整的例子：

{
  "batch-console": {
    "desc": "测试",
    "strategy": "spark",
    "algorithm": [],
    "ref": [],
    "compositor": [
      {
        "name": "batch.sources",
        "params": [
          {
            "path": "file:///tmp/hdfsfile/abc.txt",
            "format": "json",
            "outputTable": "test"

          }
        ]
      },
      {
        "name": "batch.script.df",
        "params": [
          {
            "script": "context.sql(\"select a as t from test\").registerTempTable(\"finalOutputTable\")",
            "source": "-"
          }
        ]
      },
      {
        "name": "batch.outputs",
        "params": [
          {
            "name":"jack",
            "format": "console",
            "path": "-",
            "inputTableName": "finalOutputTable",
            "mode":"Overwrite"
          }
        ]
      }
    ],
    "configParams": {
    }
  }
}

        

          
        
        
        

          

          AI 代码解读
        
      
      

另外，对于输出，我们也可以控制文件数目：

{
        "name": "batch.outputs",
        "params": [
          {
            "name": "jack",
            "format": "json",
            "path": "file:///tmp/batch-console",
            "outputFileNum": "3",
            "inputTableName": "finalOutputTable",
            "mode": "Overwrite"
          }
        ]
      }

        

          
        
        
        

          

          AI 代码解读