Comments (2)
Good to know. Thank you.
from kedro.
Hi @6Hhcy
It's a good question!
Kedro uses s3fs
, which uses boto
library to access S3. Boto is not thread-safe indeed https://boto3.amazonaws.com/v1/documentation/api/latest/guide/resources.html?highlight=multithreading#multithreading-multiprocessing - but only if you are trying to reuse the same Session object.
All Kedro S3 datasets maintain separate instances of S3FileSystem
, which means separate boto sessions, so it's safe.
It's probably not great in terms of performance, and if you work with hundreds of S3 data sets in parallel, or thousands of small S3 datasets sequentially - the pipeline might run quite long and even fail on connection errors, but you are totally safe with a few dozens of them.
from kedro.
Related Issues (20)
- ci: Nightly build failure on `develop` HOT 3
- ci: Nightly build failure on `main` HOT 1
- Update `kedro new` hint and docs to clarify how to provide a project tools selection to `--tools`
- ci: Nightly build failure on `develop` HOT 2
- DatasetAlreadyExistsError thrown when using ThreadRunner, dataset factories HOT 4
- Maintenance of documentation versions is complex HOT 5
- Consider removing micropackaging HOT 2
- Improve Developer Experience
- Improve logging experience
- %load_node truncates import statements HOT 2
- ci: Nightly build failure on `main` HOT 1
- Upgrade Pluggy depdendency version (<1.4) - Preventing upgrade of Pytest 8.1 that requires pluggy >=1.4 HOT 1
- Monthly issue metrics report
- Update CONTRIBUTING.md and other instructions with new usage of Discussions vs Issues
- Release `kedro` 0.19.4 HOT 3
- Can't build docs in starter - need to update sphinx version HOT 2
- Improve `kedro jupyter setup` with options from `ipykernel install` HOT 1
- Kedro new starter CLI : user_input.lower() HOT 4
- Deprecate (mark for future removal) `get_pkg_version` from the public API HOT 5
- Decouple starters from framework in tool selection flow
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kedro.