dalenguyen / firestore-import-export Goto Github PK
View Code? Open in Web Editor NEWAn application that can help you to backup and restore from Cloud Firestore | Firebase
Home Page: https://www.npmjs.com/package/firestore-export-import
An application that can help you to backup and restore from Cloud Firestore | Firebase
Home Page: https://www.npmjs.com/package/firestore-export-import
Hello,
What format can I use to ensure that my dates are imported as either timestamp or date. It is importing them as strings or objects.
In some use cases, we need inner data as subcollections and not as a map.
Does this support it? If yes, What should be the structure of JSON?
Regards
I have two test documents in a collection right now and all the information is exported, but it's all in one line. Is there anything I need to add to get it to format nicely like the example json file without having to manually do it by entering.
Hi, Great work... This was just what I needed :-)
I get the following error when trying to import. Export works fine.
Hope you can help.
$ node import.js import-to-firestore.json
undefined:1
{
^
SyntaxError: Unexpected token in JSON at position 0
at JSON.parse ()
at /Users/ralf/Documents/GitHub/firebase/import.js:40:22
at FSReqWrap.readFileAfterClose [as oncomplete] (internal/fs/read_file_context.js:53:3)
JSON is just for testing a looks like:
{
"songList": {
"FpVWamw6LKfxhsdBaNsd": {
"artistName": "Muse",
"albumName": "The Nightwatcher",
"id": "FpVWamw6LKfxhsdBaNsd",
"songName": "Sweet Home Alabama",
"songDescription": "Deep trouble"
},
"PFJJaHZtM53V71na6vM5": {
"songDescription": "Desc",
"artistName": "Queen",
"albumName": "Boehemiam Rapsody",
"id": "PFJJaHZtM53V71na6vM5",
"songName": "Radio Gaga"
},
"WqMHifsWETX3CpiyDjft": {
"songDescription": "Germany",
"artistName": "Madonna",
"albumName": "Summer",
"id": "WqMHifsWETX3CpiyDjft",
"songName": "Like a Prayer"
},
"g9Eja99SlX1EywEMOhOk": {
"songDescription": "About winning",
"artistName": "Queen",
"albumName": "Best of Queen",
"id": "g9Eja99SlX1EywEMOhOk",
"songName": "We are the Champions"
},
"mgNDi0p4FUKFaMJ7Oo7m": {
"songDescription": "Ghost Town - desc",
"artistName": "Adam Lampert",
"albumName": "Best of",
"id": "mgNDi0p4FUKFaMJ7Oo7m",
"songName": "Ghost Town"
},
"kklnILJMKMHLKMLKJNnk": {
"songDescription": "Norwegian band",
"artistName": "AHA",
"albumName": "Fjeldaberne",
"id": "123456789",
"songName": "Take on me"
}
}
}
Is there an option to auto-id documents as they are uploaded? thank you
In an attempt to migrate data from Firestore to the RealtimeDB I was hoping the collections of my data would be placed inside of an object. It seems trivial to accomplish and maybe I'm the only one with this use case so I will be working on this. (First PR coming through so bear with me!)
@dalenguyen Thanks for contributing this library for data import. I managed to use it to upload my data.
{
"collectionName" : {
"ID00001" : {
"venue" : "Super Ring"
}
}
}
It would be to be able to automatic generate the key because I am uploading new data. At the moment, I am trying to generate the ID outside this import utility.
line 43, 48 "udpateCollection"
udpate -> update
:)
unable to solve this
When I want to export a big collection
Then I get this error: DEADLINE_EXCEEDED: Deadline Exceeded
Is it possible to add a feature where I run firestore-import-export
as a cronjob, So that it reads only what is changed since the last update and update the file with the same documents? Instead of reading the thousands of documents each time I run it.
I'm trying this command obviously as guessing :
node import.js firestore-export.json date=FoodGrains[0].product_adding_timestamp
^What I'm trying above is I have a List of Product objects as Maps in one Firestore Document and all these objects have two Timestamp fields "product_adding_timestamp" and "product_updating_timestamp". Trying the above-guessed command giving me this error :
Here's the screenshot of FoodGrains collection :
Here's the screenshot of fields every List object contains :
^I want to import product_adding_timestamp field values as Timestamp object, not as Map of _nanoseconds and _seonds keys.
Sorry. Because of not being a native English speaker I could not make this question brief.
This project is running but when I want to download large amounts of data I get the following error. The data I want to download is about 4 million. I upgraded the Firebase account to the paid plan and I did not understand the reason for this problem. I would be glad if you can help with this.
The error I got:
Error: 4 DEADLINE_EXCEEDED: Deadline Exceeded
at Object.exports.createStatusError (C:\Users\ixast\Downloads\firestore-import-export-master\firestore-import-export-master\node_modules\grpc\src\common.js:91:15)
at ClientReadableStream._emitStatusIfDone (C:\Users\ixast\Downloads\firestore-import-export-master\firestore-import-export-master\node_modules\grpc\src\client.js:233:26)
at ClientReadableStream._receiveStatus (C:\Users\ixast\Downloads\firestore-import-export-master\firestore-import-export-master\node_modules\grpc\src\client.js:211:8)
at Object.onReceiveStatus (C:\Users\ixast\Downloads\firestore-import-export-master\firestore-import-export-master\node_modules\grpc\src\client_interceptors.js:1277:15)
at InterceptingListener._callNext (C:\Users\ixast\Downloads\firestore-import-export-master\firestore-import-export-master\node_modules\grpc\src\client_interceptors.js:568:42)
at InterceptingListener.onReceiveStatus (C:\Users\ixast\Downloads\firestore-import-export-master\firestore-import-export-master\node_modules\grpc\src\client_interceptors.js:618:8)
at C:\Users\ixast\Downloads\firestore-import-export-master\firestore-import-export-master\node_modules\grpc\src\client_interceptors.js:1033:24 {
code: 4,
metadata: Metadata { _internal_repr: {}, flags: 0 },
details: 'Deadline Exceeded'
}
Stackoverflow url:
https://stackoverflow.com/questions/62860410/cloud-firestore-big-data-error-deadline-exceeded
please test with this sample JSON:
{
"glossary": {
"title": "example glossary",
"GlossDiv": {
"title": "S",
"GlossList": {
"GlossEntry": {
"ID": "SGML",
"SortAs": "SGML",
"GlossTerm": "Standard Generalized Markup Language",
"Acronym": "SGML",
"Abbrev": "ISO 8879:1986",
"GlossDef": {
"para": "A meta-markup language, used to create markup languages such as DocBook.",
"GlossSeeAlso": ["GML", "XML"]
},
"GlossSee": "markup"
}
}
}
}
}
My original database uses Firestore's timestamp type.
The export script creates:
"endDate": { "_seconds": 1577318400, "_nanoseconds": 0 }
Then if I import, the newly imported DB no longer has the timestamp field, instead it has a map with the above structure.
How do you convert the server timestamp to datetime while exporting from firestore
thanks for the help.
i am getting this error
{ Error: 16 UNAUTHENTICATED: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential.
I am trying to export the firestore database with subcollection,
but the program will terminate if one of the subcollection is empty.
For example, my database structure is like : users//userinfo
but some users didn't have a userinfo yet, and the function will end at this point
is there anyway to solve it?
I just tried with importing serviceAccountKey.json
for usage, and I didn't changed value of databaseURL
at InitializeApp
.
databaseUrl: "https://ionic-firestore-dn.firebaseio.com"
?The GeoPoint field is exported as a latitude and longitude, but when imported back into firestore, becomes a map with two numbers fields. Any fixes? Thanks in advance!
Is there a way to export all the sub-collections and data structures at once rather than specifically defining each collection?
Hie Im still a newbie to Firestore. I have the import working when i enter a manual ID like :
{
"menu" : {
"1" : {
"menu_item" : "Menu 1",
},
"2" : {
"menu_item" : "Menu 2",
}
}
}
but i would like to let Firestore auto generate the ID
Please help. Thank you in advance
Not sure what is wrong...file is of the form...
{
"countries": {
"path": "/config/types/countries",
"values": [{
"id": "AF",
"name": "Afghanistan",
"dialingCode": "+93"
}, {
"id": "AL",
"name": "Albania",
"dialingCode": "+355"
}, {
"id": "DZ",
"name": "Algeria",
"dialingCode": "+213"
}, {
I have tried providing date arguments twice in command lines for both fields. But it's only properly handling the first field "product_adding_timestamp"
The commands I tried :
node import.js firestore-export.json date=product_adding_timestamp, product_updating_timestamp
node import.js firestore-export.json date=product_adding_timestamp, product_updating_timestamp
Here's the snippet of JSON data for both fields :
"product_adding_timestamp": {
"_seconds": 1570476289,
"_nanoseconds": 16000000
}
"product_updating_timestamp": {
"_seconds": 1570476289,
"_nanoseconds": 16000000
}
Hi,
Appreciate your work. However I am having some issues with getting the sub collections using both the npm package as well as the git hub repo.
My database is something like:
shops_collection
-document_1
-menu_collection
-document 2
-menu_collection
so i tried to input "shops_collection" then "menu_collection"
like node exportjs shops_collection menu_collection but it returns an empty list. But node exportjs shops_collection works.
Also if I do not input variables and just .backup([]), the root collections are returned but not the subcollections.
Maybe you can help me take a look with that? Thanks
how can i get it out
I don't use this package directly, but it helped me a lot when I was building my own import export tool. I recently started adding update timestamps into my firestore data using the FieldValue.serverTimestamp() sentinel. The problem is, when I exported the data and reimported it, the field was no longer a Timestamp object, but rather some unusable json junk... I was able to fix this using some pretty basic replacer/reviver functions in my JSON.parse and JSON.stringify calls:
const replacer = (key, value) => {
if (typeof value === "object" && value instanceof admin.firestore.Timestamp) {
return value.toDate().toISOString();
}
return value;
}
const isoRegex = /^\d{4}-[01]\d-[0-3]\dT[0-2]\d:[0-5]\d:[0-5]\d\.\d+Z$/
const reviver = (key: any, value: any) => {
if(typeof value === 'string' && value.match(isoRegex)) {
return admin.firestore.Timestamp.fromDate(new Date(value));
}
return value;
}
Then you just need to add the second argument to your stringify and parse calls:
// during export
... JSON.stringify(data, replacer) ...
// during import
... JSON.parse(data, reviver) ...
It wouldn't fix timestamps previously exported, but you could always adjust the reviver to account for objects that look like {"_seconds":1223456,"_nanoseconds":38575}
-- that seems to be how the default stringify call was serializing the objects.
Everything worked correctly for me except the export of dates.
They seem to be imported back into the database as a string.
ex:
2018-05-31T04:00:00.000Z
Any advice on getting around this?
Thanks,
Thank you for the tool. Really useful!
Reading through the exported that, it seems to get everything right about the reference paths.
But when I try to import the same json, the references are just set as objects.
Is there any special way to arrange the data to be able to set that data as an actual reference?
Cheers
there is a typo
would be upDating?
I have a simple database with games>users and when I get the exported I have a json like this
{
"games":{
"21hRLgKFxGDep7s7bDyY":{
"Date":1536256816002,
"State":"COMPLETED",
"users":{
"ElaQCEJxQlNmECwei1SFQ1nqs4y1":{
"Debt":0,
"Endavans":56000,
"Id":"ElaQCEJxQlNmECwei1SFQ1nqs4y1",
"LifeSavers":0,
"Name":"kevin el original",
"Active":false,
"Admin":false
},
"EukvVjXCwOO6aSl5bkXjTZK2RJ52":{
"LifeSavers":0,
"Name":"cris",
"Active":false,
"Admin":false,
"Debt":0,
"Endavans":18000,
"Id":"EukvVjXCwOO6aSl5bkXjTZK2RJ52"
},
"QZPw7rhpimQiptje2OamYixrUWy2":{
"Endavans":0,
"Id":"QZPw7rhpimQiptje2OamYixrUWy2",
"LifeSavers":0,
"Name":"Ger",
"Active":false,
"Admin":false,
"Debt":0
},
"zwNfvrE6YYPAwcNtCD9GrXT6BHh1":{
"Debt":0,
"Endavans":58000,
"Id":"zwNfvrE6YYPAwcNtCD9GrXT6BHh1",
"LifeSavers":0,
"Name":"L",
"Token":"ckRLEEq6Fw8:APA91bG9djdyN1kNzta9hU0OYE8Zaaf2dV2fOmxjcOOh602yRV8bsoRrCuP0mULUJhvM85FEOjCThow6XBnANaypFHITXBOXfPH94uL9pOh5UcJjgwAZaXsFn8pO5_NhCh2EL8c_1DJC",
"Active":false,
"Admin":false
}
}
}
}
}
But once i import that i have this:
Instead of something like this:
Is there any way to accomplish that?
Btw nice project!
Thank you for your time!
Regards,
Germán.
Thank you for the helpful plugin!
Recently, I started getting this warning message when using this plugin:
The behavior for Date objects stored in Firestore is going to change
AND YOUR APP MAY BREAK.
To hide this warning and ensure your app does not break, you need to add the
following code to your app before calling any other Cloud Firestore methods:
const firestore = new Firestore();
const settings = {/* your settings... */ timestampsInSnapshots: true};
firestore.settings(settings);
With this change, timestamps stored in Cloud Firestore will be read back asFirebase Timestamp objects instead of as system Date objects. So you will also
need to update code expecting a Date to instead expect a Timestamp. For example:
// Old:
const date = snapshot.get('created_at');
// New:
const timestamp = snapshot.get('created_at');
const date = timestamp.toDate();
Please audit all existing usages of Date when you enable the new behavior. In a
future release, the behavior will change to the new behavior, so if you do not
follow these steps, YOUR APP MAY BREAK.
It's due to a recent update of Firestore, and not an issue with the plugin, but it would be nice if the warning message could be suppressed in the console, maybe via a code change of some sort? Thanks for considering.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.