Comments (7)
A log with the complete error:
https://dl.dropboxusercontent.com/u/20756661/sqlite2postgres_blob.log
from pgloader.
I think it's a problem with the unique concept of manifest typing used in SQLite3, wherein I have the following quite strange result:
SQLITE> (let ((statement (sqlite:prepare-statement *sqlite-db*
"select file_data from files limit 1")))
(sqlite:step-statement statement)
(prog1
(sqlite-ffi:sqlite3-column-type (sqlite::handle statement) 0)
(sqlite:finalize-statement statement)))
:TEXT
Note that it's not the driver's fault:
sqlite> select typeof(file_data) from files limit 1;
text
I will see later what can be done here, given that SQLite happily returns text
rather than blob
as soon as the data itself fits into the text datatype rules, or something. Hopefully there's a way to force SQLite into returning blob
data without having to parse and copy the text output into a PostgreSQL bytea
representation.
That said it might be that just storing the string characters as their ascii bytes into the bytea is all we need to do here.
from pgloader.
Aha. After some tests, your data is base64 encoded, so returned as text for that reason.
from pgloader.
I tried rebuilding pgloader, with make pgloader: but it seems to need an extra DEP:
Fatal MISSING-DEPENDENCY:
Component #:BASE64 not found, required by #<SYSTEM "pgloader">
from pgloader.
Sorry about that, went too fast again. Should be ok now (did compile a binary and used it against the new regression test here).
from pgloader.
It works now. But when converting a very big DB (sqlite with 400Mb on Blob in a table) I run into memory shortage issues. Is there a smart way around this? (it's not a big issue for me, I can clean up the table before doing the convert)
from pgloader.
You can try the batch settings (see Batch behaviour options in the Reference documentation), or even try compiling with CCL (at http://ccl.clozure.com/) if you feel that way.
from pgloader.
Related Issues (20)
- An unhandled error condition has been signalled: on #<BASIC-TCP-STREAM UTF-8 (SOCKET/6) #x302002CF61DD> :
- Illegal :UTF-8 character starting at position 1 HOT 1
- Cannot understand migration error reasons HOT 8
- Migration from MariaDB to PostgreSQL HOT 1
- Problem SQLITE-TYPE-NAME
- debian package version 3.6.9-1 mismatch with sources version 3.6.7 HOT 1
- MSSQL "Connection to the database failed for an unknown reason" but other tools can connect HOT 1
- URL paramter options=endpoint not working HOT 1
- pgloader repeatedly disconnecting from db during migration attempt HOT 3
- Feature: Configuration File for Type Conversion overrides?
- MySQL connection recognition failed
- can you provide a stable version as default?
- missing `pgloader-bundle-3.6.10.tgz` for 3.6.10 release HOT 2
- Option to migrate table without its data
- no tables are found in geopackage/sqlite file HOT 1
- Issue with pgloader migration from MySQL MariaDB to Supabase HOT 2
- MySQL conversion syntax error at or near "\" HOT 2
- SQLite - primary keys not transfered, unique index creation fails, reset sequences fail
- sqlite to postgresql (Windows + Docker) HOT 1
- Casting errors when using Redshift as a data source
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pgloader.