Bug report

Describe the bug

Getting collection, but when trying to run node --trace-deprecation firestore2json.js firebase getting 0 records written to firebase.json

To Reproduce

Steps to reproduce the behavior, please provide code snippets or a repository:

1. Go to '…'
2. Click on '…'
3. Scroll down to '…'
4. See error

System information

• OS: Windows

I have users whose provider is custom token login in fireauth and phone.

The reason why I use custom token was that firebase didn't provide kakao social login, even though It was actually kakao login, but I used custom token at that time.
Its user json data looks like below.

 {
    "uid": "kakao:123",
    "emailVerified": false,
    "displayName": "OOO",
    "photoURL": "https://photourl.jpg",
    "disabled": false,
    "metadata": {
      "lastSignInTime": "Fri, 29 Dec 2023 15:27:29 GMT",
      "creationTime": "Fri, 29 Dec 2023 15:21:03 GMT"
    },
    "tokensValidAfterTime": "Fri, 29 Dec 2023 15:21:03 GMT",
    "providerData": []
  },

And I also have users whose provider is phone.

  {
    "uid": "123456",
    "emailVerified": false,
    "phoneNumber": "+82101111111",
    "disabled": false,
    "metadata": {
      "lastSignInTime": "Tue, 16 May 2023 06:58:35 GMT",
      "creationTime": "Tue, 16 May 2023 06:58:35 GMT"
    },
    "providerData": [
      {
        "uid": "+82101111111",
        "providerId": "phone",
        "phoneNumber": "+82101111111"
      }
    ]
  },

When I run node import_users.js users.josn, It only migrates users whose provider is email.
and import_users.js file shows each json file should have email and providerData property.

function createUser(user) {
  var sql =
    "(\n        '00000000-0000-0000-0000-000000000000', /* instance_id */\n        uuid_generate_v4(), /* id */\n        'authenticated', /* aud character varying(255),*/\n        'authenticated', /* role character varying(255),*/\n        '"
      .concat(
        user.email,
      )
      .concat(
        user.emailVerified ? "NOW()" : "null",      )
      .concat(
        formatDate(user.metadata.creationTime),
      )
      .concat(
        getProviderString(user.providerData),
        "', /* raw_app_meta_data jsonb,*/\n        '{\"fbuser\":"
      )
      .concat(
        JSON.stringify(user),
        "}', 
  return sql;
}

But user's json file who signed up with custom token doesn't have email and providerData property.

In this case how can I migrate users of fireauth to supabase authentication successfully?

Bug report

Per title, import_users.js generates invalid SQL.

To Reproduce

1. Export with firestoreusers2json.js (after fixing the batching bug that generates invalid data if you have more than 1 batch).
2. Attempt to import generated JSON node import_users.js ./users.json
3. Observe error

loading users from ./users.json
insertUsers: 100
runSQL error: error: syntax error at or near "s"
    at Parser.parseErrorMessage (/Users/imagio/dev/firebase-to-supabase/node_modules/pg-protocol/dist/parser.js:287:98)
    at Parser.handlePacket (/Users/imagio/dev/firebase-to-supabase/node_modules/pg-protocol/dist/parser.js:126:29)
    at Parser.parse (/Users/imagio/dev/firebase-to-supabase/node_modules/pg-protocol/dist/parser.js:39:38)
    at Socket.<anonymous> (/Users/imagio/dev/firebase-to-supabase/node_modules/pg-protocol/dist/index.js:11:42)
    at Socket.emit (node:events:513:28)
    at addChunk (node:internal/streams/readable:315:12)
    at readableAddChunk (node:internal/streams/readable:289:9)
    at Socket.Readable.push (node:internal/streams/readable:228:10)
    at TCP.onStreamRead (node:internal/stream_base_commons:190:23) {
  length: 94,
  severity: 'ERROR',
  code: '42601',
  detail: undefined,
  hint: undefined,
  position: '150346',
  internalPosition: undefined,
  internalQuery: undefined,
  where: undefined,
  schema: undefined,
  table: undefined,
  column: undefined,
  dataType: undefined,
  constraint: undefined,
  file: 'scan.l',
  line: '1188',
  routine: 'scanner_yyerror'
}
sql was:
......many thousands of lines of SQL

Expected behavior

Import works or at least fails with a useful error message

Screenshots

If applicable, add screenshots to help explain your problem.

System information

• OS: macos
• Browser: N/A
• Version of supabase-js: N/A
• Version of Node.js: 16.19.0

Additional context

Hello guys,

First of all, thanks a lot for this really helpful guide. Well, i manage to migrate the users to supabase but they failed to authenticate. How do i use this guide effectively? Thanks

I got the following error message when trying to run the upload.js file, but am I missing something obvious?

firebase-to-supabase % node storage/upload.js som sfda fdsa
/Users/tylershukert/firebase-to-supabase/storage/supabase-keys.js:15
export const keys = {
^^^^^^

SyntaxError: Unexpected token export
    at new Script (vm.js:80:7)
    at createScript (vm.js:274:10)
    at Object.runInThisContext (vm.js:326:10)
    at Module._compile (internal/modules/cjs/loader.js:664:28)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:712:10)
    at Module.load (internal/modules/cjs/loader.js:600:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:539:12)
    at Function.Module._load (internal/modules/cjs/loader.js:531:3)
    at Module.require (internal/modules/cjs/loader.js:637:17)
    at require (internal/modules/cjs/helpers.js:22:18)

Bug report

I am trying to migrate data from firestore database to supabase. I have successfully created the json files from firebase collections. The issue is when I am running node json2supabase.js agent_invites.json bigserial id, it generate multiple copies of one object in json in Postgres.

To Reproduce

Steps to reproduce the behavior, please provide code snippets or a repository:

1. Clone this repository
2. Download agent_invites.json
3. Place this file in firestore folder
4. run node json2supabase.js agent_invites.json bigserial id
5. It will generate event more records in supabase then there are lines in this json

Expected behavior

It should generate exact same number of records that are there in json

System information

• OS: Windows
• VS Code
• "@supabase/supabase-js": "^1.30.7",
• "firebase-admin": "^10.0.2",

Bug report

Describe the bug

While using the auth solution in this repo, if you export a list of users to a .json and then attempt to import those users, if any user has a display name with an ' in it (example: "O'Leary"), the SQL that's generated throws an error.

To Reproduce

1. Create a user in Firebase Auth with a displayName containing an apostrophe.
2. Export that user to a list of users.
3. Try to import that user with the import_users.js script
4. See error

Expected behavior

Import works for names with apostrophes.

After commend is ran, users.json file is created. When check for validation of json file in online tools it says exprected ','. Same resulte when running import_users.

Improve documentation

Link

Add a link to the page which needs improvement (if relevant)
https://github.com/supabase-community/firebase-to-supabase/tree/main/auth

Describe the problem

For JS beginners, it is not immediately obvious that to run the scripts, one needs to install the dependencies. So when running node firestoreusers2json.js, I got this error and was initially lost:

$ node firestoreusers2json.js
node:internal/modules/cjs/loader:927
  throw err;
  ^

Error: Cannot find module 'firebase-admin'
Require stack:
- /~/firebase-to-supabase/auth/firestoreusers2json.js
    at Function.Module._resolveFilename (node:internal/modules/cjs/loader:924:15)
    at Function.Module._load (node:internal/modules/cjs/loader:769:27)
    at Module.require (node:internal/modules/cjs/loader:996:19)
    at require (node:internal/modules/cjs/helpers:92:18)
    at Object.<anonymous> (/~/firebase-to-supabase/auth/firestoreusers2json.js:41:13)
    at Module._compile (node:internal/modules/cjs/loader:1092:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1121:10)
    at Module.load (node:internal/modules/cjs/loader:972:32)
    at Function.Module._load (node:internal/modules/cjs/loader:813:14)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:76:12) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '~/firebase-to-supabase/auth/firestoreusers2json.js'
  ]
}

I only got to fix this after running yarn install from the project root.

Describe the improvement

Add package installation steps to the setup instructions
Add an explicitly state in this section that the resulting firebase-service.json file also needs to be copied from your download location into the current auth folder, i.e.: firebase-to-supabase/auth/firebase-service.json

I couldn't find in the Supabase Migrate Documentation how to pull subcollections inside a collection in Firestore.

For example, under a workspace collection, there is a sub-collection of users in that workspace. Supabase codes only get the /workspaces collection, not the users collection.

Eg : /workspaces/{workspaceId}/users

Can you help with an example?
Thank you

Bug report

Describe the bug

We keep getting TCP IP disconnect on trying to upload and create a new table

A clear and concise description of what the bug is.

TCPConnectWrap.afterConnect
Error: connect ETIMEDOUT xx.xx.xxx.xx:xxxx
at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1161:16) {
errno: -4039,
code: 'ETIMEDOUT',
syscall: 'connect',
address: '23.22.253.25',
port: 5432
}

To Reproduce

Steps to reproduce the behavior, please provide code snippets or a repository:

1. Go to '…'
2. Click on '…'
3. Scroll down to '…'
4. See error

Expected behavior

A clear and concise description of what you expected to happen.
Would expect it to upload to the table correctly

Screenshots

If applicable, add screenshots to help explain your problem.

System information

• OS: [e.g. macOS, Windows]

• Browser (if applies) [e.g. chrome, safari]

• Version of supabase-js: [e.g. 6.0.2]

• Version of Node.js: [e.g. 10.10.0]

• OS: Windows (I know)

• Chrome

• Latest

• v16

Additional context

Trying to migrate from Supabase from Firebase

Add any other context about the problem here.
It's not working

I get the following error when attempting to run node import_users.js users.json. I've followed all the instructions on the README for Auth migration, running locally (not fly.io). My users.json file looks well-formed on instpection.

console.logging seems to indicate the error ocurrs on runSQL but not sure exactly what is wrong.

node:events:368
      throw er; // Unhandled 'error' event
      ^

Error: Parser cannot parse input: expected ','
    at Parser._processBuffer (/Users/gerardoascenciomorales/Documents/firebase-to-supabase/node_modules/stream-json/Parser.js:291:76)
    at Parser._transformBuffer (/Users/gerardoascenciomorales/Documents/firebase-to-supabase/node_modules/stream-json/utils/Utf8Stream.js:24:10)
    at Parser._transform (/Users/gerardoascenciomorales/Documents/firebase-to-supabase/node_modules/stream-json/utils/Utf8Stream.js:19:10)
    at Parser.Transform._write (node:internal/streams/transform:184:23)
    at writeOrBuffer (node:internal/streams/writable:389:12)
    at _write (node:internal/streams/writable:330:10)
    at Parser.Writable.write (node:internal/streams/writable:334:10)
    at Chain._write (/Users/gerardoascenciomorales/Documents/firebase-to-supabase/node_modules/stream-chain/index.js:152:18)
    at writeOrBuffer (node:internal/streams/writable:389:12)
    at _write (node:internal/streams/writable:330:10)
Emitted 'error' event on Chain instance at:
    at Chain.<anonymous> (/Users/gerardoascenciomorales/Documents/firebase-to-supabase/node_modules/stream-chain/index.js:146:71)
    at Chain.emit (node:events:402:35)
    at emitErrorNT (node:internal/streams/destroy:157:8)
    at emitErrorCloseNT (node:internal/streams/destroy:122:3)
    at processTicksAndRejections (node:internal/process/task_queues:83:21)

Bug report

Describe the bug

When importing from firebase uids are not preserved. They're instead ignored and regenerated as new uuids. This prevents using nearly any imported firestore data since most of the time non-public firestore documents are authenticated by matching on uid.

To Reproduce

1. Follow steps in docs to import users from firebase
2. Observe that the newly imported users don't have the same uid as they did in firebase making them impossible to relate to data imported from firestore

Expected behavior

Firebase uids should be reused or at least preserved somewhere to use with imported data

Screenshots

If applicable, add screenshots to help explain your problem.

System information

• OS: MacOS
• Version of supabase-js: N/A
• Version of Node.js: v16.19.0

Additional context

N/A

The README for the hooks feature of the firestore migration code states that hooks can be created in a js file matching the name of the collection which is being migrated. This is only true though as long as one is running the TS versions of the scripts. The JS version requires the hook file be called ${collectionName}_processDocument.js instead, that last part (_processDocument) being the missing key.

Since I'm not sure which version of the naming y'all would prefer I figured an issue would be best, rather than a PR. Happy to submit the fix if you let me know which naming convention you'd like to move forward with!

Bug report

Describe the bug

Firestore data migration tool: When I try to import 245 rows from the JSON file, a bit over 25,000 rows get uploaded.

I tested with JSON files of 5 entries or so and it works perfectly, but in case see below template:

{
  "age_range_min": 18,
  "age_range_max": 65,
  "distance_range_max": 42,
  "division": "All",
  "gender": 0,
  "heigth_range_min": 100,
  "heigth_range_max": 215,
  "looking_for_date": true,
  "looking_for_friends": true,
  "looking_for_short_term": false,
  "nationality": "All",
  "weight_range_min": 70,
  "weight_range_max": 320,
  "firestore_id": "GqMchob6ruTuUZs35CwkfUfpViQ2"
},
{
  "age_range_min": 18,
  "age_range_max": 65,
  "distance_range_max": 42,
  "division": "All",
  "gender": 0,
  "heigth_range_min": 100,
  "heigth_range_max": 215,
  "looking_for_date": true,
  "looking_for_friends": true,
  "looking_for_short_term": true,
  "nationality": "All",
  "weight_range_min": 70,
  "weight_range_max": 320,
  "firestore_id": "HGS7JysWZhfPPtbexH0d7OK0nts2"
},

To Reproduce

Steps to reproduce the behavior:

1. Go to 'Firestore directory'
2. Command line: 'node json2supabase.js users_filters.json serial'

No errors, but on supabase 25,000+ rows are added instead of 245

Expected behavior

A clear and concise description of what you expected to happen.

Screenshots

If applicable, add screenshots to help explain your problem.

System information

• OS: Windows
• Node.js: v16.13.2

"dependencies": {
    "@supabase/supabase-js": "^1.30.7",
    "firebase-admin": "^10.3.0",
    "firebase-scrypt": "^2.1.0",
    "moment": "^2.29.1",
    "pg": "^8.7.1",
    "stream-json": "^1.7.3"

Additional context

Seems like TS files were not compiled into the JS files (I've provided a few corrections as a push, I'll let you see), but it was not affecting this problem.

There are a few commented out lines in the code, it could possibly come from there.
ChatGPT says it has probably something to do with the fact that it runs asynchronously at some places.

When you run the command node firestore2json.js users.json you received a message 0 records written to users.json.json

But when you run node collections.js it does list the collections.

Any ideas?
Thank you.