Comments (6)
from pgfutter.
from pgfutter.
The latter, so it's one JSON object per line, but millions of lines. I see the pgfutter process using one (of eight) cores to 100% and the Postgresql (on the same machine), although running on several processes, using on average another core, so there are basically 6 cores doing nothing.
from pgfutter.
OH, I have to mention, I have a fast SSD, that still is far of beeing on its limit.
from pgfutter.
It's cool to hear someone else is importing large JSON files as well.
I used it before to import few hundred GBs with 500 million lines of JSON objects - but there I also run the processes in parallel.
The only thing is, it seems to run only on one thread, which leaves a lot of resources unused during the import process. Would be cool, if pgfutter would automatically split up the import file into junks which would get imported on multiple threads in parallel!
Is there a chance to implement this?
It might transactional failure harder. When the copy stream fails I can not guarantee that no data is inserted. But perhaps it's not an issue because I wrap everything into a transaction (might be that the copy operation is a transaction on its own though).
This is where one could optimize https://github.com/lukasmartinelli/pgfutter/blob/master/json.go#L28
Not sure whether I will get around this soon - it is interesting because optimizing is fun :)
For you I recommend to try import with multiple processes and check whether that makes it faster?
from pgfutter.
When the copy stream fails I can not guarantee that no data is inserted
That would not be a huge issue for me, since I always make sure that the table is empty before the import.
This is where one could optimize
Unfortunately I'm not at all a GO-programmer, so I'm completely unable to work directly on the code.
optimizing is fun :)
100% agreed! That's why I was asking...
For you I recommend to try import with multiple processes and check whether that makes it faster?
For this I would need to split up the import file by myself. Since I'm not controlling the program that creates the JSON file, I can only try to use another 3rd-party tool to do the split, but since it is a 16GB-gzipped-json, this probably is also not easy. Do you have a recommendation, or tool which I could try to achieve this?
from pgfutter.
Related Issues (20)
- Import text values as null instead of blank HOT 2
- Unable to import JSON
- No longer builds HOT 2
- --table doesn't exist as an option HOT 1
- index out of range error HOT 8
- build failures on master HOT 2
- unable to build with go1.14.2 HOT 1
- cannot understand how to use from the documentation HOT 3
- pq: copyin statement has already been closed HOT 1
- help needed syntax
- Can make it work with amd64.exe or GO source HOT 2
- Pgfutter not working with PostgreSQL13 HOT 5
- Pgfutter command not found error in Ubuntu bash HOT 1
- index out of range error HOT 1
- Does nothing and shows no output HOT 6
- pgfutter doesn't correctly honor --table names HOT 1
- bsdthread_register error HOT 2
- header row case conversion
- panic: runtime error: index out of range
- fail importing a tsv file with more than 1000 columns
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pgfutter.