Apache Spark alone, by default, generates a lot of information in its logs. Spark Streaming creates a metric ton more (in fairness, there’s a lot going on). So, how do we lower that gargantuan wall of text to something more manageable?
One way is to lower the log level for the Spark Context, which is retrieved from the Streaming Context. Simply:
|val conf = new SparkConf().setAppName(appName) // run on cluster|
|val ssc = new StreamingContext(conf, Seconds(5))|
|val sc = ssc.sparkContext|
Pretty easy, right?