Comments (17)
I think he's asking if there's a way to chunk data while exporting one excel file?
from fast-excel.
@rap2hpoutre I think maybe it's time to revisit chunking for the package? People will use this package because it's fast, and their needs will gravitate towards bigger data sets.
from fast-excel.
@markdieselcore Yes you are right! Not sure about the implementation though. Maybe we could follow the same pattern as for SheetCollection
, something like ChunkCollection
(then the export method would process chunks by chunks)...
from fast-excel.
Another (more readable, with "real" chunks) example:
function usersGenerator() {
yield from User::chunk(200, function($users) {
foreach($users as $user) {
yield $user;
}
});
}
// Export consumes only a few MB
(new FastExcel(usersGenerator()))->export('test.xlsx');
from fast-excel.
Fixed, available in v1.3.0 π
from fast-excel.
Showing Error: Can use "yield from" only with arrays and Traversables. Can you help?
from fast-excel.
@ehsanhoushmand @alexbabich1990 please go to https://dev.to/rahulprgrmr/comment/f1lk
from fast-excel.
Thank you!
Not sure it will answer your question, but you can chunk data and export multiple excels files:
$users = User::all();
$chunks = $users->chunk(1000);
foreach($chunks as $id => $chunk) {
(new FastExcel($users))->export("file-$id.xlsx");
}
(it will create N
files).
from fast-excel.
Hi, Thanks for the reply,
as maekdieselcore said, I want to export a one sheet excel from chunks of data, in a foreach chunk add to excel loop.
this way I can export large number of rows. without exhausting memory.
from fast-excel.
if all fails I will have to split the chunks into different files. I like the speed of your package, I guess I have to compromise something.
from fast-excel.
@rap2hpoutre hello, export big data into one excel file by chunk , is that ok ? i need this method
from fast-excel.
@mengdodo this is not developed yet, but it seems important, so I will try to do something about that!
from fast-excel.
from fast-excel.
I just created a draft PR for this issue: https://github.com/rap2hpoutre/fast-excel/pull/112/files
The idea is to use generators, which seems to have been build for this exact kind of need. With my (quick) tests, for about 100k lines, it only use ~5M of RAM versus ~50M without chunks. π
To make it work, you have to create a generator function using yield
then pass it to FastExcel
:
// Generator function
function getUsersOneByOne() {
// build your chunks as you want (200 chunks of 10 in this example)
for ($i = 0; $i < 200; $i++) {
$users = DB::table('users')->skip($i * 10)->take(10)->get();
// Yield user one by one
foreach($users as $user) {
yield $user;
}
}
}
// Export consumes only a few MB
(new FastExcel(getUsersOneByOne()))->export('test.xlsx');
Minimal (shorter) example:
function myGenerator() {
$users = get_users_as_you_want();
foreach($users as $user) {
yield $user;
}
}
(new FastExcel(myGenerator()))->export('test.xlsx');
What do you think about this implementation @mengdodo @ArturoGasca @Elshaden @markdieselcore ? Could it solve your problem?
from fast-excel.
Article here: https://dev.to/rap2hpoutre/export-10m-rows-in-xlsx-with-laravel-without-memory-issues-6bk
from fast-excel.
I have the same problem
from fast-excel.
For what is it worth, the following also solves the yield from
issue with using the chunks
method.
Using the lazy
method did the trick for me.
function resultsGenerator(Builder $query) {
yield from $query->lazy();
}
(new FastExcel(resultsGenerator($query))->export(....
from fast-excel.
Related Issues (20)
- How to export large dataset from API?
- How to Change cell width while downloading excel? HOT 1
- How to import with matching columns? HOT 2
- The empty cells in excel is considered as string (""), which should be NULL.
- The empty cells in excel is considered as string (""), which should be NULL. HOT 2
- How to ignore sep=; when importing CSV HOT 1
- php 8.2 not supporting HOT 4
- Call to undefined method Rap2hpoutre\FastExcel\Facades\FastExcel::import() in laravel 10 job although the method exists HOT 2
- download filename can't show utf-8 encode
- Version 5.3.0 doesn't have the update as description
- Laravel ^10.10 - does not calculate excel formulas during import (openspout/openspout depenency problem) HOT 1
- not download format xlsx HOT 2
- autowrap
- How to have multiple sheets and also give specific column names to the same?
- Crashing when huge data is need to be downloaded
- startRow does not work
- Contact Details HOT 1
- how to recover the header of an imported excel file
- Excel File Treats Cell Content Starting with +,-,=,@ as Formula
- DateTime values are lost during export
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fast-excel.