Comments (4)
Can you give us an example where a swift public container is?
from stocator.
Done.
from stocator.
@djalova I am getting complaints that it's not working anymore. To reproduce it - you need a public container with some objects. Then use Spark with
val d = sc.binaryFiles('swift2d://dal.objectstorage.open.softlayer.com/v1/AUTH_/somedata/*', minPartitions=20)
Getting
Py4JJavaError: An error occurred while calling o31.binaryFiles.
: java.lang.NullPointerException
at com.ibm.stocator.fs.swift.auth.JossAccount.getAccessURL(JossAccount.java:116)
at com.ibm.stocator.fs.swift.SwiftAPIClient.getAccessURI(SwiftAPIClient.java:588)
at com.ibm.stocator.fs.ObjectStoreFileSystem.getUri(ObjectStoreFileSystem.java:104)
at org.apache.hadoop.fs.FileSystem.makeQualified(FileSystem.java:469)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(FileInputFormat.java:500)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(FileInputFormat.java:469)
at org.apache.spark.SparkContext$$anonfun$binaryFiles$1.apply(SparkContext.scala:898)
at org.apache.spark.SparkContext$$anonfun$binaryFiles$1.apply(SparkContext.scala:893)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.SparkContext.withScope(SparkContext.scala:687)
at org.apache.spark.SparkContext.binaryFiles(SparkContext.scala:893)
at org.apache.spark.api.java.JavaSparkContext.binaryFiles(JavaSparkContext.scala:256)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:507)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:237)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:280)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:214)
at java.lang.Thread.run(Thread.java:785)
It worked fine in the past, seems something is broken in the recent code.
@djalova can you please check it? Thanks a lot
from stocator.
Resolved.
from stocator.
Related Issues (20)
- FileNotFoundException while accessing objects from hive-style partitioned data stored on an OCS cluster HOT 2
- Upgrade guava from 27.0-jre to 29.0-jre #277
- Upgrade packages that have security vulnerabilities. HOT 4
- Support for Spark 3.x? HOT 5
- Catalog table doesn't work with HIVE-style partitioning if there is no empty object ending with '/' HOT 2
- Using bucket endpoint having virtual-host pattern HOT 1
- HTTP Client Version Warning HOT 1
- Pandas load files written in COS with stocator
- Upgrade hadoop-version from 2.7.3 to 3.2.3
- Upgrade version of jackson_databind and commons-codec as per twistlock scan HOT 2
- Upgrade Amazon SDK HOT 5
- Upgrade version of jackson_databind as per twistlock scan HOT 1
- appstatus file rename failed for spark rolling logs HOT 5
- Spark History Server unable to read rolling logs from COS using stocator
- Double slash in COS parquet path is being added HOT 1
- build failing with java classes not found HOT 6
- support reading credentials from kubernetes secrets
- Get Bucket not found HOT 3
- Upgrade guava and ibm-cos-sdk (to get latest jackson) to fix twistlock issue findings
- Upgrade commons-validator library (to get latest commons-beanutils) to fix twistlock issue findings HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from stocator.