How to restrict the createunionstream. about azure-event-hubs-spark

[CodingCat]

sorry, I didn't understand the question here, you do not want to save to file in a shutdown hook , but why you have

sys.ShutdownHookThread {
println("Gracefully stopping Spark Streaming Application")
println("*****************************************************########################################################################################################################")
//val conf = new SparkConf().setAppName("Merging Files")
// val sc= new SparkContext(conf)
// sc.textFile("/EventCheckpoint_0.1/ParquetEvent/part*").coalesce(1).saveAsTextFile("/EventCheckpoint_0.1/SingleFile")
// sc.stop()
streamingContext.stop(true, true)
println("Application stopped")
}

[ankushreddy]

@CodingCat Hi sorry for the confusion I have modified the code I mean removed all the comments.
If we look at println("COUNT" +dataString.count()) this is not getting executed until I press ctrl+c.

Thank You for your help.

[CodingCat]

how you detect this line is not executed? can you post a screenshot of your streaming UI?

[CodingCat]

and we also have other questions

• how you start the application

• how many resources you allocate to the application

@arijitt

[ankushreddy]

Am printing a dummy line just to debug.
I started the application in local. yarn client and cluster mode.
in yarn client and cluster mode I have used 2 executors and 5 executor cores. driver memory as 2GB and executor memory as 2G.
As this is a streaming application.
Am expecting it should write the data in the batches for every 10 seconds.

[CodingCat]

wait...you have 10 partitions in eventhubs, and you only give 2 executors and 5 executor cores, that means you have 10 cores in total

because you are using receiver-based connection, these 10 cores would be used to run 10 receivers and no resources left for your data processing tasks. Once you killed the application, those receivers are killed and some pending data processing tasks are started (but quickly killed since the whole app is down)

Are you using Spark 2.0+? if yes, you can try direct dstream integration of Spark and Event Hubs which is much faster and resource-efficient

[ankushreddy]

Hi @CodingCat when ever there is a single record in the stream it is taking forever to finish the process. and rest of the streaming process jobs are accumulating in queue.

Please find the screenshot and let me know if am missing anything.

Thanks You,
Ankush Reddy.

[CodingCat]

as I said, it is due to the fact that there is no resources left for your processing....you can either increase the number of cores or use direct dstream

[CodingCat]

update on this? otherwise I will close it

[ankushreddy]

I am using createDirectStream.

Could you please let me know if we have any documentation on the differences between union stream and direct stream.

Thanks,
Ankush.

[CodingCat]

We are uploading docs but still waiting for approval...basically it is the same high level idea of the diff beween kafka receiver based and direct dstream based integration

[CodingCat]

update on this? otherwise, I will close it

[ankushreddy]

@CodingCat Thank You for your inputs Am closing this.