fauna-labs / faunadb-importer Goto Github PK

Right now it's confusing. I was thinking about something simpler and wait for feedback. Like:

Completed: 42% Errors: 2 RPS: 123 Latency(client/server): 12.2/5.2

For testing purposes I tried to import a very simple JSON file into the developer version of faunadb:

.\bin\faunadb-importer.bat import-file --endpoints http://localhost:8443 --secret ABCDEF --class simple test.json

Here is my import file (test.json):

[{
	"name": "ABC",
	"address": {
		"city": "Wien",
		"country": "AT",
		"street": "Testgasse 4",
		"zip": "1210"
	}
},
{
	"name": "DEF",
	"address": {
		"city": "Wien",
		"country": "AT",
		"street": "Testgasse 10",
		"zip": "1010"
	}
}
]

The error:

[2017-09-21 22:02:24] validation failed: Instance data is not valid. at line: 1, column: 1: [{ "name": "ABC", "address": { "city": "Wien", "country": "AT", "street": "Testgasse 4", "zip": "1210" } }, { "name": "DEF", "address": { "city": "Wien", "country": "AT", "street": "Testgasse 10", "zip": "1010" } }]

It should contain a directory file structure as well as executables for linux a windows.

Measure:

• General execution time
• Memory utilization
• Id cache serialization overhead

For slow networks, we expect timeouts and other networks related issues to happen. We could retry batches in those case. Monix has a good helper for that: https://monix.io/docs/2x/eval/circuit-breaker.html

Version 1.2.0 of scala driver will allow us to:

• Upgrade scala to 2.12.x
• Upgrade jackson versio to 1.8 (which has some performance improvements)

There is a bug on the scala driver that needs to be fixed: fauna/faunadb-jvm#110

Once that is in we're going to be able to use the status code from a http error to decide when to retry for the http code 413. We're currently relying exception message, which itself has a typo.

• Apply backpressure when service is unavailable
• Handle errors when the request is too large due to huge batch sizes
• Implement a split and retry strategy for batches in order to find the error for a single instance rather than marking the whole batch as failed

Hi

We are looking to import csv with arrays of refs, is there a way to do this with the importer?

Examples code with a ref to another file - ref is currently a single ref but we have records which have multiple refs.

data/test1.csv:
  class: test1
  fields:
    - name: id
      type: ref

    - name: name
      type: string

  ignoredFields:
    - id

data/test2.csv:
  class: test2
  fields:
    - name: user_id
      type: ref(test1) # We would like this to be an array of brands

Thank you in advance!

Hi,
I have a file with lines of JSON like
{
"id": "3924",
"name": "Mary"
}.

I can upload it with no problems but the "id" field that I mark as ref using the parameter
--format "id:ref"
is ignored and the record gets a different ref (generated automatically) like "198480451673784842"

Ref
q.Ref("classes/my-class/198480451673784842")
Class
q.Ref("classes/my-class")
TS
1525544559283937
Data
{
"id": "3924",
"name": "Mary"
}

Am I missing something?
Thank you

• Seems to have a nicer API
• Less dependency footprint
• Potentially better performance for our use case

Not sure if we need to support java 7. Maybe it's work exploring to see how hard that would be.

If errors occured during import, the next import attempt will failed too, because the temp working file (cache) is not cleared, i need to manually delete the file to make it works.

logs/errors.log already give users a good idea on what went wrong but, there are cases where that will not be sufficient. We need a place to programmatically recover missing entries.

• Usage instructions
• Explain field syntax
• Explain yaml file syntax
• Explain constraints (like, do not remove a line from the import file if you're using row number as ids)