Comments (5)
@djalova Can you check it please?
from stocator.
@gilv In addition to the above, instead of doing the head at all, would it make sense to try the PUT of the object, catch the exception for the non-existing container and create the container, and then redo the object PUT? The check first approach typically involves two calls but can be three. The try and recover approach is typically one call but can be three. The check first approach is simpler code.
from stocator.
@michaelfactor it depends on what is the 1st operation. If it's PUT - then your suggestion is fine. But it may be HEAD, or GET, etc.. so we will need to try-catch many operation in Stocator. It will be complicated change, since there is no single point when there is HEAD or GET.
from stocator.
@gilv Agreed completely -- this is a performance-simplicity tradeoff and the recovery on the HEAD or GET will be different for the PUT. Not sure you need to autocreate on HEAD or GET of an object since if the container doesn't exist, obviously the object doesn't. But this is obviously a performance optimization and we need to understand its value in practice before rushing to make the change.
from stocator.
@gilv I think all operations start out by calling FileSystem.exists() whether its a PUT or GET operation. In the case where auto create is off should we just throw a new exception when a container is not present?
from stocator.
Related Issues (20)
- FileNotFoundException while accessing objects from hive-style partitioned data stored on an OCS cluster HOT 2
- Upgrade guava from 27.0-jre to 29.0-jre #277
- Upgrade packages that have security vulnerabilities. HOT 4
- Support for Spark 3.x? HOT 5
- Catalog table doesn't work with HIVE-style partitioning if there is no empty object ending with '/' HOT 2
- Using bucket endpoint having virtual-host pattern HOT 1
- HTTP Client Version Warning HOT 1
- Pandas load files written in COS with stocator
- Upgrade hadoop-version from 2.7.3 to 3.2.3
- Upgrade version of jackson_databind and commons-codec as per twistlock scan HOT 2
- Upgrade Amazon SDK HOT 5
- Upgrade version of jackson_databind as per twistlock scan HOT 1
- appstatus file rename failed for spark rolling logs HOT 5
- Spark History Server unable to read rolling logs from COS using stocator
- Double slash in COS parquet path is being added HOT 1
- build failing with java classes not found HOT 6
- support reading credentials from kubernetes secrets
- Get Bucket not found HOT 3
- Upgrade guava and ibm-cos-sdk (to get latest jackson) to fix twistlock issue findings
- Upgrade commons-validator library (to get latest commons-beanutils) to fix twistlock issue findings HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from stocator.