Comments (4)
Hi, @shailendraksharma , what you provided is the log from ApplicationMaster, would you post the log from Container side?
from azure-event-hubs-spark.
if you're a customer of HDInsight ,would you please communicate through our working email, your colleague shall have our contact
from azure-event-hubs-spark.
Thanks for help i think i figure out the issue. while saving small json data it works fine but saving lengthy json causing errors. one such error is following while inserting data into sql db:
17/01/22 19:48:01 INFO DStreamGraph: Clearing checkpoint data for time 1485114434000 ms
17/01/22 19:48:01 WARN AzureFileSystemThreadPoolExecutor: Disabling threads for Delete operation as thread count 0 is <= 1
17/01/22 19:48:01 INFO AzureFileSystemThreadPoolExecutor: Time taken for Delete operation is: 216 ms with threads: 0
17/01/22 19:48:01 WARN TaskSetManager: Lost task 3.0 in stage 99.0 (TID 584, 10.0.0.14): com.microsoft.sqlserver.jdbc.SQLServerException: Incorrect syntax near 's'.
at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:216)
at com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1515)
at com.microsoft.sqlserver.jdbc.SQLServerStatement.doExecuteStatement(SQLServerStatement.java:792)
at com.microsoft.sqlserver.jdbc.SQLServerStatement$StmtExecCmd.doExecute(SQLServerStatement.java:689)
at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:5696)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:1715)
at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:180)
at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:155)
at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeUpdate(SQLServerStatement.java:642)
at com.microsoft.spark.streaming.examples.common.DataFrameExtensions$ExtendedDataFrame$$anonfun$insertToAzureSql$1$$anonfun$apply$1.apply(DataFrameExtensions.scala:48)
at com.microsoft.spark.streaming.examples.common.DataFrameExtensions$ExtendedDataFrame$$anonfun$insertToAzureSql$1$$anonfun$apply$1.apply(DataFrameExtensions.scala:39)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at com.microsoft.spark.streaming.examples.common.DataFrameExtensions$ExtendedDataFrame$$anonfun$insertToAzureSql$1.apply(DataFrameExtensions.scala:38)
at com.microsoft.spark.streaming.examples.common.DataFrameExtensions$ExtendedDataFrame$$anonfun$insertToAzureSql$1.apply(DataFrameExtensions.scala:33)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$28.apply(RDD.scala:902)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$28.apply(RDD.scala:902)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1899)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1899)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
at org.apache.spark.scheduler.Task.run(Task.scala:86)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
trying to figure out this one.
from azure-event-hubs-spark.
Hey, thanks for your interest in using Spark and EventHubs :) If this issue is still ongoing, please re-open a new issue! I'll be actively checking for issues, and would be happy to help out. Thanks!
from azure-event-hubs-spark.
Related Issues (20)
- Spark starts crashing when some of the eventhub partitions go down.
- Structured streaming job hangs after a while
- EH - Trigger once HOT 1
- This library or kafka
- Using Kafka driver for Spark the throughput is 50-80 times faster
- AAD Authentication is terminated after running for a couple of minutes HOT 1
- Job for consumption of Event Hub messages aborts on Databricks (request seqNo less than received seqNo) HOT 3
- Package Support for Scala 2.13 or 3+ HOT 4
- PySpark job doesnt stop on stopping query
- Batch read from eventhubs throws duration format error
- Checkpoint write to blob storage using hadoop configuration HOT 3
- Missing tag for v2.3.22
- Spark-scala api references for Azure eventhub schema registry
- maxEventsPerTrigger is not working
- Spark streaming kubernetes - Fails to recover from chechpoint. Cannot find endpoint: spark://PartitionPerformanceReceiver
- EventHub Writer fails due to Throttling of EventHub, configuration settings have no impact. HOT 1
- ReceiverDisconnectedException even if using different consumer groups HOT 1
- The auth docs are wrong - causing service unavailability issues
- Azure EventHub - PySpark Failed to configure SaslClientAuthenticator works when using Confluent cloud
- Support for Spark v3.3.0 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from azure-event-hubs-spark.