ecraft / couchdb_to_sql Goto Github PK

Right now, the current implementation will fail with an Errno::ENOENT if the file does not exist. This is unnecessarily awkward. We should fail gracefully if the file is missing.

The problem right now is that the latest changes done by @afinne are Postgres-only, since they use UPSERT etc, which is not available in SQLite.

Either we:

• make the code handle both scenarios, with if-statements or similar, or
• change Travis to use Postgres for its test runs.

Since #8, we support JRuby, but our test suite can only run on MRI because of dependencies on sqlite.

This issue exists to remind us to fix this. On JRuby, we should probably use H2, Derby or some similar database, with a low number of dependencies.

This gem is all we need to get running with Derby. It seems to quite popular, and there is a Sequel adapter for it. My take would be that we start with this one and see if we can get it working reasonably easy.

Since our gem has significant changes from the original couch_tap implementation, we (me and @afinne) discussed and decided to rename it to couchdb_to_sql - a name which clearly tells what it does, and also clearly distinguishes it from @samlown's great original work, from which we are derived.

The module name will be CouchdbToSql, in line with Rubygems: Name your Gem

At the moment, SQL inserts are being performed one row at a time. This has significant performance impacts. We should look into changing this to use Sequel's great multi_insert support to speed it up.

One of the challenges here is that we receive the data from CouchDB in very small chunks (6 rows per chunk). I wonder if that could be tweaked somehow...

This indicates an error; all documents should match a document. First step would be to log to the console.

Should we even raise an error and halt further processing?

We use rspec much more internally, so the cost of having to maintain tests written in a non-standard testing framework is greater than zero. It would be worthwhile to refactor the tests to rspec sometime.