spark read long datatype error about elasticsearch-hadoop

[jbaiera]

spark.option("es.field.read.empty.as.null", no)

This field means that an empty string will not be treated as a null value, and thus will attempt to read the empty string as a numeric value. This defaults to true normally and thus the empty string will be parsed as a null value. I would 1) reenable the setting if you need this logic and 2) be careful when storing string data in long types since that can easily lead to confusing errors from the parsing code if they are not valid numerics

[zzzjx329]

tks, I attempted to set this option to true, but I encountered another error.
at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$Serializer.apply(ExpressionEncoder.scala:213)
at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$Serializer.apply(ExpressionEncoder.scala:195)
at scala.collection.Iterator$sanon$10.next(Iterator.scala:461)
at org.apache.spark.sql.catalyst.expressions.Generatedclass$GeneratedIteratorforCodegenstagel.processNext(Unknown Source
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator. java:43)
at org.apache.spark.sql.execution.wholestageCodegenExec$$anon$1.hasNext(wholestageCodegenExec.scala:755)
at org.apache.spark.sql.execution.SparkPlan.$anon fun$getByteArrayRdd$1(SparkPlan.scala:345)
at org.apache.spark.rdd.RDD.$anon fun$mapPartitionsInternal$2(RDD.scala:897)
at org.apache.spark.rdd.RDD.Sanon fun$mapPartitionsInternal$2sadapted(RDD.scala:897
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.compute0rReadCheckpoint(RDD.scala:373)
at org.apache.spark.rdd.RDD.iterator (RDD.scala:337)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90
at org.apache.spark.scheduler. Task.run(Task.scala:131)
at org. apache. spark.executor. Executor$TaskRunner. sanonfunsrun$3(Executor.scala:516
at org.apache.spark.util.utilss.trywithSafeFinally(Utils.scala:1604)
at org.apache.spark.executor.Executor$TaskRunner. run(Executor.scala:519)
at java.util.concurrent.ThreadPoolExecutor. runworker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$worker.run(ThreadPoolExecutor.java:624)
at java. lang.Thread.run(Thread.java:750)
Caused by: java.lang.RuntimeException: scala.Nones is not a valid external type for schema of bigint
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.If 0$(Unknown Source)
at org.apache, spark.sql.catalyst.expressions.GeneratedClasssSpecificUnsafeProjection.writeFields 5$(Unknown Source'
at org.apache.spark.sql.catalyst.expressions.Generatedclass$SpecificUnsafeProjection.apply(Unknown Source)
at org.apache, spark.sql.catalyst.encoders.ExoressionEncodertSerializer.apply(ExpressionEncoder,scala:209)
. 19 more

At first, I set this parameter to true because I reported the above error

[zzzjx329]

Actually, I want to know how to read the data to avoid program errors when dirty data is known to exist, such as through parameter control or any other means.