dalenguyen / firestore-import-export

An application that can help you to backup and restore from Cloud Firestore | Firebase

Firestore Import Export

A script that help to export and import in Cloud Firestore

** This repo is not maintained. Please try the backup and restore from Firestore package instead!

Requirements

You need NODE or something that can run JAVASCRIPT (JS) file.

Get serviceAccount JSON file from Project Setting > SERVICE ACCOUNTS in Firebase Console

Change the databaseURL when initializeApp with your own

Setting Up

Download or clone this repository

git clone https://github.com/dalenguyen/firestore-import-export.git

Install NPM packages

npm install

Export database from Firestore

This will help you create a backup of your collection and subcollection from Firestore to a JSON file name firestore-export.json

node export.js <your-collection-name> <sub-collection-name-(optional)>

Import database to Firestore

This will import a collection to Firestore will overwrite any documents in that collection with matching id's to those in your json. If you have date type in your JSON, please add the field to the command line. The date and geo arguments is optional.

node import.js import-to-firestore.json date=date geo=Location

If you have date type in your JSON, please add to your command line

Sample from import-to-firestore.json. "test" will be the collection name. The date type will have _seconds and _nanoseconds in it.

{
  "test" : {
    "first-key" : {
      "email"   : "[email protected]",
      "website" : "dalenguyen.me",
      "custom"  : {
        "firstName" : "Dale",
        "lastName"  : "Nguyen"
      },
      "date": {
        "_seconds":1534046400,
        "_nanoseconds":0
      },
      "Location": {
        "_latitude": 49.290683,
        "_longitude": -123.133956
      }
    },
    "second-key" : {
      "email"   : "[email protected]",
      "website" : "google.com",
      "custom"  : {
        "firstName" : "Harry",
        "lastName"  : "Potter"
      },
      "date": {
        "_seconds":1534262435,
        "_nanoseconds":0
      },
      "Location": {
        "_latitude": 49.290683,
        "_longitude": -123.133956
      }
    }
  }
}

Thanks to @fed239, you can use YAML files instead of JSON files in order to import to firestore.

If you have any recommendation or question, please create an issue. Thanks,

firestore-import-export's Issues

node import.js import-to-firestore.json module.js:557 throw err; can not find module './serviceAccountKey.json'

Cannot find module './serviceAccountKey.json'

unable to solve this

Function name mispelled

line 43, 48 "udpateCollection"

udpate -> update

:)

Better "timestamp" handling

I don't use this package directly, but it helped me a lot when I was building my own import export tool. I recently started adding update timestamps into my firestore data using the FieldValue.serverTimestamp() sentinel. The problem is, when I exported the data and reimported it, the field was no longer a Timestamp object, but rather some unusable json junk... I was able to fix this using some pretty basic replacer/reviver functions in my JSON.parse and JSON.stringify calls:

const replacer = (key, value) => {
    if (typeof value === "object" && value instanceof admin.firestore.Timestamp) {
        return value.toDate().toISOString();
    }
    return value;
}
const isoRegex = /^\d{4}-[01]\d-[0-3]\dT[0-2]\d:[0-5]\d:[0-5]\d\.\d+Z$/
const reviver = (key: any, value: any) => {
    if(typeof value === 'string' && value.match(isoRegex)) {
        return admin.firestore.Timestamp.fromDate(new Date(value));
    }
    return value;
}

Then you just need to add the second argument to your stringify and parse calls:

// during export
... JSON.stringify(data, replacer) ...
// during import
... JSON.parse(data, reviver) ...

It wouldn't fix timestamps previously exported, but you could always adjust the reviver to account for objects that look like {"_seconds":1223456,"_nanoseconds":38575} -- that seems to be how the default stringify call was serializing the objects.

There are two Timestamp fields in documents and importing the collection only handling first field.

I have tried providing date arguments twice in command lines for both fields. But it's only properly handling the first field "product_adding_timestamp"

The commands I tried :
node import.js firestore-export.json date=product_adding_timestamp, product_updating_timestamp
node import.js firestore-export.json date=product_adding_timestamp, product_updating_timestamp

Here's the snippet of JSON data for both fields :

"product_adding_timestamp": {
               "_seconds": 1570476289,
               "_nanoseconds": 16000000
           }
"product_updating_timestamp": {
               "_seconds": 1570476289,
               "_nanoseconds": 16000000
           }

Auto generate firestore document ID

@dalenguyen Thanks for contributing this library for data import. I managed to use it to upload my data.

{
  "collectionName" : {
    "ID00001" : {
      "venue" : "Super Ring"
    }
  }
}

It would be to be able to automatic generate the key because I am uploading new data. At the moment, I am trying to generate the ID outside this import utility.

References Support

Thank you for the tool. Really useful!

Reading through the exported that, it seems to get everything right about the reference paths.
But when I try to import the same json, the references are just set as objects.

Is there any special way to arrange the data to be able to set that data as an actual reference?

Cheers

Sub collections as nested object

In an attempt to migrate data from Firestore to the RealtimeDB I was hoping the collections of my data would be placed inside of an object. It seems trivial to accomplish and maybe I'm the only one with this use case so I will be working on this. (First PR coming through so bear with me!)

why is it not working with sub collections?. only downloading root collection data

Timestamps not being properly handled?

My original database uses Firestore's timestamp type.

The export script creates:
"endDate": { "_seconds": 1577318400, "_nanoseconds": 0 }

Then if I import, the newly imported DB no longer has the timestamp field, instead it has a map with the above structure.

Where the file is saved?

In command line it showed that The file was saved but I couldn't find it

Error: Failed to load gRPC binary module because it was not installed for the current system

how can i get it out

timestamp

How do you convert the server timestamp to datetime while exporting from firestore

thanks for the help.

How to import data as subcollection not as a map

In some use cases, we need inner data as subcollections and not as a map.
Does this support it? If yes, What should be the structure of JSON?

Regards

How to import Firestore Collection's Documents with List of Map having Timestamp fields?

I'm trying this command obviously as guessing :
node import.js firestore-export.json date=FoodGrains[0].product_adding_timestamp
^What I'm trying above is I have a List of Product objects as Maps in one Firestore Document and all these objects have two Timestamp fields "product_adding_timestamp" and "product_updating_timestamp". Trying the above-guessed command giving me this error :

Here's the screenshot of FoodGrains collection :

Here's the screenshot of fields every List object contains :

^I want to import product_adding_timestamp field values as Timestamp object, not as Map of _nanoseconds and _seonds keys.
Sorry. Because of not being a native English speaker I could not make this question brief.

Sub Collection Cannot be retrieved

Hi,

Appreciate your work. However I am having some issues with getting the sub collections using both the npm package as well as the git hub repo.

My database is something like:

shops_collection
-document_1
-menu_collection
-document 2
-menu_collection

so i tried to input "shops_collection" then "menu_collection"

like node exportjs shops_collection menu_collection but it returns an empty list. But node exportjs shops_collection works.

Also if I do not input variables and just .backup([]), the root collections are returned but not the subcollections.

Maybe you can help me take a look with that? Thanks

auto-id?

Is there an option to auto-id documents as they are uploaded? thank you

Firestore warning message displays in console

Thank you for the helpful plugin!

Recently, I started getting this warning message when using this plugin:

The behavior for Date objects stored in Firestore is going to change
AND YOUR APP MAY BREAK.
To hide this warning and ensure your app does not break, you need to add the
following code to your app before calling any other Cloud Firestore methods:

  const firestore = new Firestore();
  const settings = {/* your settings... */ timestampsInSnapshots: true};
  firestore.settings(settings);
With this change, timestamps stored in Cloud Firestore will be read back asFirebase Timestamp objects instead of as system Date objects. So you will also
need to update code expecting a Date to instead expect a Timestamp. For example:
  // Old:
  const date = snapshot.get('created_at');
  // New:
  const timestamp = snapshot.get('created_at');
  const date = timestamp.toDate();

Please audit all existing usages of Date when you enable the new behavior. In a
future release, the behavior will change to the new behavior, so if you do not
follow these steps, YOUR APP MAY BREAK.

It's due to a recent update of Firestore, and not an issue with the plugin, but it would be nice if the warning message could be suppressed in the console, maybe via a code change of some sort? Thanks for considering.

invalid authentication credentials.

i am getting this error

{ Error: 16 UNAUTHENTICATED: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential.

Dates are not supported

Everything worked correctly for me except the export of dates.
They seem to be imported back into the database as a string.

ex:
2018-05-31T04:00:00.000Z

Any advice on getting around this?

Thanks,

sync like feature?

Is it possible to add a feature where I run firestore-import-export as a cronjob, So that it reads only what is changed since the last update and update the file with the same documents? Instead of reading the thousands of documents each time I run it.

Exported Json File on One Line

I have two test documents in a collection right now and all the information is exported, but it's all in one line. Is there anything I need to add to get it to format nicely like the example json file without having to manually do it by entering.

it doesn't work with deep level JSON

please test with this sample JSON:

{
	"glossary": {
		"title": "example glossary",
		"GlossDiv": {
			"title": "S",
			"GlossList": {
				"GlossEntry": {
					"ID": "SGML",
					"SortAs": "SGML",
					"GlossTerm": "Standard Generalized Markup Language",
					"Acronym": "SGML",
					"Abbrev": "ISO 8879:1986",
					"GlossDef": {
						"para": "A meta-markup language, used to create markup languages such as DocBook.",
						"GlossSeeAlso": ["GML", "XML"]
					},
					"GlossSee": "markup"
				}
			}
		}
	}
}

Not all the data-collections are saved recursively ?

Is there a way to export all the sub-collections and data structures at once rather than specifically defining each collection?

Import customers, workers and categories to firestore

Related to https://github.com/prolike/dillydally/issues/107

Use a value from key-value pair to be the doc id

For example, I would like my doc id to be the value from the key-value pair courseid

How to import to Firestore With An Auto ID

Hie Im still a newbie to Firestore. I have the import working when i enter a manual ID like :
{
"menu" : {

	"1" : {
		"menu_item" : "Menu 1",
	},
	"2" : {
		"menu_item" : "Menu 2",
	}

}
}

but i would like to let Firestore auto generate the ID
Please help. Thank you in advance

When working with big data, it gives an error. Deadline Exceeded

This project is running but when I want to download large amounts of data I get the following error. The data I want to download is about 4 million. I upgraded the Firebase account to the paid plan and I did not understand the reason for this problem. I would be glad if you can help with this.

The error I got:

Error: 4 DEADLINE_EXCEEDED: Deadline Exceeded
at Object.exports.createStatusError (C:\Users\ixast\Downloads\firestore-import-export-master\firestore-import-export-master\node_modules\grpc\src\common.js:91:15)
at ClientReadableStream._emitStatusIfDone (C:\Users\ixast\Downloads\firestore-import-export-master\firestore-import-export-master\node_modules\grpc\src\client.js:233:26)
at ClientReadableStream._receiveStatus (C:\Users\ixast\Downloads\firestore-import-export-master\firestore-import-export-master\node_modules\grpc\src\client.js:211:8)
at Object.onReceiveStatus (C:\Users\ixast\Downloads\firestore-import-export-master\firestore-import-export-master\node_modules\grpc\src\client_interceptors.js:1277:15)
at InterceptingListener._callNext (C:\Users\ixast\Downloads\firestore-import-export-master\firestore-import-export-master\node_modules\grpc\src\client_interceptors.js:568:42)
at InterceptingListener.onReceiveStatus (C:\Users\ixast\Downloads\firestore-import-export-master\firestore-import-export-master\node_modules\grpc\src\client_interceptors.js:618:8)
at C:\Users\ixast\Downloads\firestore-import-export-master\firestore-import-export-master\node_modules\grpc\src\client_interceptors.js:1033:24 {
code: 4,
metadata: Metadata { _internal_repr: {}, flags: 0 },
details: 'Deadline Exceeded'
}

Stackoverflow url:
https://stackoverflow.com/questions/62860410/cloud-firestore-big-data-error-deadline-exceeded

Dates

Hello,

What format can I use to ensure that my dates are imported as either timestamp or date. It is importing them as strings or objects.

typo in import.js line 45 (startUpading)

there is a typo

would be upDating?

the function will terminate if one of the subcollection is empty

I am trying to export the firestore database with subcollection,
but the program will terminate if one of the subcollection is empty.

For example, my database structure is like : users//userinfo

but some users didn't have a userinfo yet, and the function will end at this point

is there anyway to solve it?

Firestore GeoPoint Field Not Importing/Exporting Properly

The GeoPoint field is exported as a latitude and longitude, but when imported back into firestore, becomes a map with two numbers fields. Any fixes? Thanks in advance!

Getting error Error: Argument "data" is not a valid Document. Input is not a plain JavaScript object.

Not sure what is wrong...file is of the form...

{
"countries": {
"path": "/config/types/countries",
"values": [{
"id": "AF",
"name": "Afghanistan",
"dialingCode": "+93"
}, {
"id": "AL",
"name": "Albania",
"dialingCode": "+355"
}, {
"id": "DZ",
"name": "Algeria",
"dialingCode": "+213"
}, {

Is it possible? [InitializeApp] without 'databaseURL'

I just tried with importing serviceAccountKey.json for usage, and I didn't changed value of databaseURLat InitializeApp.

1. It works fine, it it possible?
2. In the case of me, is it possible to submit my datas to your databaseUrl: "https://ionic-firestore-dn.firebaseio.com"?

Can't import

Hi, Great work... This was just what I needed :-)

I get the following error when trying to import. Export works fine.
Hope you can help.

$ node import.js import-to-firestore.json
undefined:1
{
^

SyntaxError: Unexpected token  in JSON at position 0
at JSON.parse ()
at /Users/ralf/Documents/GitHub/firebase/import.js:40:22
at FSReqWrap.readFileAfterClose [as oncomplete] (internal/fs/read_file_context.js:53:3)

JSON is just for testing a looks like:

{
"songList": {
"FpVWamw6LKfxhsdBaNsd": {
"artistName": "Muse",
"albumName": "The Nightwatcher",
"id": "FpVWamw6LKfxhsdBaNsd",
"songName": "Sweet Home Alabama",
"songDescription": "Deep trouble"
},
"PFJJaHZtM53V71na6vM5": {
"songDescription": "Desc",
"artistName": "Queen",
"albumName": "Boehemiam Rapsody",
"id": "PFJJaHZtM53V71na6vM5",
"songName": "Radio Gaga"
},
"WqMHifsWETX3CpiyDjft": {
"songDescription": "Germany",
"artistName": "Madonna",
"albumName": "Summer",
"id": "WqMHifsWETX3CpiyDjft",
"songName": "Like a Prayer"
},
"g9Eja99SlX1EywEMOhOk": {
"songDescription": "About winning",
"artistName": "Queen",
"albumName": "Best of Queen",
"id": "g9Eja99SlX1EywEMOhOk",
"songName": "We are the Champions"
},
"mgNDi0p4FUKFaMJ7Oo7m": {
"songDescription": "Ghost Town - desc",
"artistName": "Adam Lampert",
"albumName": "Best of",
"id": "mgNDi0p4FUKFaMJ7Oo7m",
"songName": "Ghost Town"
},
"kklnILJMKMHLKMLKJNnk": {
"songDescription": "Norwegian band",
"artistName": "AHA",
"albumName": "Fjeldaberne",
"id": "123456789",
"songName": "Take on me"
}
}
}

Subcollections are created as objects

I have a simple database with games>users and when I get the exported I have a json like this

{
   "games":{
      "21hRLgKFxGDep7s7bDyY":{
         "Date":1536256816002,
         "State":"COMPLETED",
         "users":{
            "ElaQCEJxQlNmECwei1SFQ1nqs4y1":{
               "Debt":0,
               "Endavans":56000,
               "Id":"ElaQCEJxQlNmECwei1SFQ1nqs4y1",
               "LifeSavers":0,
               "Name":"kevin el original",
               "Active":false,
               "Admin":false
            },
            "EukvVjXCwOO6aSl5bkXjTZK2RJ52":{
               "LifeSavers":0,
               "Name":"cris",
               "Active":false,
               "Admin":false,
               "Debt":0,
               "Endavans":18000,
               "Id":"EukvVjXCwOO6aSl5bkXjTZK2RJ52"
            },
            "QZPw7rhpimQiptje2OamYixrUWy2":{
               "Endavans":0,
               "Id":"QZPw7rhpimQiptje2OamYixrUWy2",
               "LifeSavers":0,
               "Name":"Ger",
               "Active":false,
               "Admin":false,
               "Debt":0
            },
            "zwNfvrE6YYPAwcNtCD9GrXT6BHh1":{
               "Debt":0,
               "Endavans":58000,
               "Id":"zwNfvrE6YYPAwcNtCD9GrXT6BHh1",
               "LifeSavers":0,
               "Name":"L",
               "Token":"ckRLEEq6Fw8:APA91bG9djdyN1kNzta9hU0OYE8Zaaf2dV2fOmxjcOOh602yRV8bsoRrCuP0mULUJhvM85FEOjCThow6XBnANaypFHITXBOXfPH94uL9pOh5UcJjgwAZaXsFn8pO5_NhCh2EL8c_1DJC",
               "Active":false,
               "Admin":false
            }
         }
      }
   }
}

But once i import that i have this:

Instead of something like this:

Is there any way to accomplish that?

Btw nice project!

Thank you for your time!

Regards,

Germán.

Doesn't work with big collections: DEADLINE_EXCEEDED: Deadline Exceeded

When I want to export a big collection
Then I get this error: DEADLINE_EXCEEDED: Deadline Exceeded