Comments (4)
I can read 1000 files if I specify 1001 as 4th parameter in the getBucket method. But how can I know how many files are in a bucket upfront? If I pass a higher number of (number of files in the folder) +1 it won't work
$this->connector->getBucket($this->bucket, $this->folder.'/', null, 1001);
from s3.
If you do not set $maxKeys it will cause the S3 call to run recursively until S3 reports there are no more entries with the given prefix. It does not hang indefinitely, it takes time to list thousands of files.
If you do set $maxKeys to a non–null value it will only list that many items and stop. This values has to be less than or equal to 1000 because that's the Amazon S3 API limit.
What you want (list all files without spending more time listing the files) is not possible. The Amazon S3 API will ONLY return up to 1000 files. If you want to list more files you have to do ceil(n / 1000) requests to S3. If you have 99,999 files you need to do 100 requests. Each request takes around 2–3 seconds so you'd have to wait about 5 minutes to get the full listing.
from s3.
Could you please provide an example of how reading more than 1001 files could be done? How do you know in advance how many files are in a folder in order to do ceil(n / 1000) ?
from s3.
Look, man. I can tell you how things work, I cannot understand it for you. I already told you that you need to leave $maxKeys as null. I also explained that in this case the library itself will AUTOMATICALLY make the required number of requests. I also explained that even though you said it “hangs indefinitely” when you do that it actually doesn't, it makes this many number of requests which take time.
This is the second issue you are filing because you seem unable to read the docblock headers and keep on asking for someone to understand it for you. Sorry, this doesn't work like that. You really need to understand it yourself. The library is very simple and open source, you can read the code and follow it with a debugger. I will have to lock this conversation since there's only so many ways I can say the same, simple thing.
from s3.
Related Issues (20)
- Support for Digital Ocean Spaces? HOT 3
- Documentation without using Composer HOT 14
- getAuthenticatedURL with HTTPS? HOT 3
- Solved: PHP Config Issue - small file upload works, files over 2MB fail HOT 8
- Question: Syntax for Creating Folders HOT 1
- Check if an object exists on S3 HOT 3
- Error::setCode() throws TypeError HOT 4
- getBucket() and file timestamp HOT 5
- getBucket() $maxKeys usage HOT 1
- Cannot upload empty files - Missing input parameters HOT 3
- Presigned upload URLS HOT 3
- getAuthenticatedURL does not honor setUseLegacyPathStyle HOT 1
- getObject cannot get XML files HOT 2
- PHP warnings after last update HOT 5
- Any way of testing from localhost? [Question] HOT 1
- Receiving SignatureDoesNotMatch errors after upgrading from 2.0.0 HOT 2
- Upgrading from version 2.0 HOT 6
- Add an option to switch between `X-Amz-Date` and `Date` HOT 1
- Support for DreamObject s3 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from s3.