Comments (13)
sorry, I didn't understand the question here, you do not want to save to file in a shutdown hook , but why you have
sys.ShutdownHookThread {
println("Gracefully stopping Spark Streaming Application")
println("*****************************************************########################################################################################################################")
//val conf = new SparkConf().setAppName("Merging Files")
// val sc= new SparkContext(conf)
// sc.textFile("/EventCheckpoint_0.1/ParquetEvent/part*").coalesce(1).saveAsTextFile("/EventCheckpoint_0.1/SingleFile")
// sc.stop()
streamingContext.stop(true, true)
println("Application stopped")
}
from azure-event-hubs-spark.
@CodingCat Hi sorry for the confusion I have modified the code I mean removed all the comments.
If we look at println("COUNT" +dataString.count()) this is not getting executed until I press ctrl+c.
Thank You for your help.
from azure-event-hubs-spark.
how you detect this line is not executed? can you post a screenshot of your streaming UI?
from azure-event-hubs-spark.
and we also have other questions
-
how you start the application
-
how many resources you allocate to the application
from azure-event-hubs-spark.
Am printing a dummy line just to debug.
I started the application in local. yarn client and cluster mode.
in yarn client and cluster mode I have used 2 executors and 5 executor cores. driver memory as 2GB and executor memory as 2G.
As this is a streaming application.
Am expecting it should write the data in the batches for every 10 seconds.
from azure-event-hubs-spark.
wait...you have 10 partitions in eventhubs, and you only give 2 executors and 5 executor cores, that means you have 10 cores in total
because you are using receiver-based connection, these 10 cores would be used to run 10 receivers and no resources left for your data processing tasks. Once you killed the application, those receivers are killed and some pending data processing tasks are started (but quickly killed since the whole app is down)
Are you using Spark 2.0+? if yes, you can try direct dstream integration of Spark and Event Hubs which is much faster and resource-efficient
from azure-event-hubs-spark.
Hi @CodingCat when ever there is a single record in the stream it is taking forever to finish the process. and rest of the streaming process jobs are accumulating in queue.
Please find the screenshot and let me know if am missing anything.
Thanks You,
Ankush Reddy.
from azure-event-hubs-spark.
as I said, it is due to the fact that there is no resources left for your processing....you can either increase the number of cores or use direct dstream
from azure-event-hubs-spark.
update on this? otherwise I will close it
from azure-event-hubs-spark.
I am using createDirectStream.
Could you please let me know if we have any documentation on the differences between union stream and direct stream.
Thanks,
Ankush.
from azure-event-hubs-spark.
We are uploading docs but still waiting for approval...basically it is the same high level idea of the diff beween kafka receiver based and direct dstream based integration
from azure-event-hubs-spark.
update on this? otherwise, I will close it
from azure-event-hubs-spark.
@CodingCat Thank You for your inputs Am closing this.
from azure-event-hubs-spark.
Related Issues (20)
- Structured streaming job hangs after a while
- EH - Trigger once HOT 1
- This library or kafka
- Using Kafka driver for Spark the throughput is 50-80 times faster
- AAD Authentication is terminated after running for a couple of minutes HOT 1
- Job for consumption of Event Hub messages aborts on Databricks (request seqNo less than received seqNo) HOT 3
- Package Support for Scala 2.13 or 3+ HOT 4
- PySpark job doesnt stop on stopping query
- Batch read from eventhubs throws duration format error
- Checkpoint write to blob storage using hadoop configuration HOT 3
- Missing tag for v2.3.22
- Spark-scala api references for Azure eventhub schema registry
- maxEventsPerTrigger is not working
- Spark streaming kubernetes - Fails to recover from chechpoint. Cannot find endpoint: spark://PartitionPerformanceReceiver
- EventHub Writer fails due to Throttling of EventHub, configuration settings have no impact. HOT 1
- ReceiverDisconnectedException even if using different consumer groups HOT 1
- The auth docs are wrong - causing service unavailability issues
- Azure EventHub - PySpark Failed to configure SaslClientAuthenticator works when using Confluent cloud
- Support for Spark v3.3.0 HOT 1
- java.util.concurrent.TimeoutException: Futures timed out after [5 minutes] HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from azure-event-hubs-spark.