kevlened / fireway Goto Github PK

It would be really great if fireway didn't throw when encountering files that didn't meet the migration filename spec. This would allow for configuration files and helpers relevant to migrations to co-exist in the migrations folder.
Maybe just return null like when the filename doesn't comply with semver.

Your package raises a lot of security alerts in your repo. Can you upgrade your dependencies to your latest version?

This means we need to clear the Fireway collection whenever we want to test the migrations. This is a huge hassle.

Hi there,

I'm currently running into an issue attempting to run migrations whilst specifying the environment variable FIREBASE_EMULATOR_HOST, my script is something along the lines of;

    "emulators": "firebase emulators:start",
    "firebase-migration": "yarn cross-env FIRESTORE_EMULATOR_HOST=localhost:8080 fireway --path=\"./scripts/migrations\" migrate --dryRun"

on starting the emulator, then running firebase-migration I get this error;

 FirebaseAppError: Failed to determine project ID: Error while making request: getaddrinfo ENOTFOUND metadata.google.internal. Error code: ENOTFOUND

which is quite strange considering the README file for the package explicitly states that once the FIRESTORE_EMULATOR_HOST env value is set, there should be no need to set a project ID, I also looked into how tests are ran in package, and I see it's very similar to the setup I shared above. I'd appreciate any insights you might have as why it's not working as one might expect.

My fireway version is 1.1.0

I'm trying to run migration script, but unfortunately it cannot read my credentials.

Currently using

set GOOGLE_APPLICATION_CREDENTIALS="%cd%\google-services.json"
fireway migrate

results in

ERROR: Failed to read credentials from file "C:\Users\[...]\google-services.json": Error: ENOENT: no 
such file or directory, open '"C:\Users\[...]\google-services.json"

Also when setting a relative path (they're in the same folder) set GOOGLE_APPLICATION_CREDENTIALS="google-services.json" does not fix it

I've just run my first migration and there was a lot of output. It looks like in cli.js that debug is set to true with no other possible value. This triggers all the logging behaviour.

Would you be OK with receiving a PR that adds a --debug flag to the CLI?

Gist here: https://gist.github.com/josheverett/df0a8662595f8d4a13053e5116b5e2b4

Using node v16.11.0 on an M1 mac. Searching for no template named 'remove_cv_t' in namespace 'std' led me to lots of similar issues where the combination of node 16 and Apple silicon was the culprit.

Sure enough, after downgrading to node 15 (which took forever), yarn / yarn setup / yarn test all work fine.

Hey @kevlened, I really love that you've solved a problem I (and I imagine many others) have found ourselves with. I was wondering if you had an open-source project to show how this actually looks in practice (for example: how to handle failed migration data rollbacks, formatting longer descriptions in the migration filename, etc).

More than happy to help set up a skeleton project if that would help, but I wanted to see if you had something open already to look at. Thanks

Hi! Thanks for the library. Unfortunately, I run into an issue when running fireway migrate

.nvm/versions/node/v10.22.0/lib/node_modules/fireway/src/index.js:31
		while (this._fireway_queue?.length) {
		                           ^

SyntaxError: Unexpected token .

Does it mean only Node.js versions that support optional chaining are supported?

The collection used for storing previous migration runs is hard coded as 'fireway'. This creates issues when you want to have multiple migrations for different services in the same project.

One option is to add a flag like:
--migrationsCollection service-fireway. This is similar to golang-migrate which has a config option of migrationsTable.

I'm getting Unexpected token '.' when running npx fireway. Version 0.4.2 is the latest working version for me.

Is there any way to use this library against a local instance of firestore running in the emulator? I'd love to test migration scripts against my local instance running at http://localhost:4000/firestore but when I do:
fireway migrate

I get:

Found 2 migration files
ERROR: 9 FAILED_PRECONDITION: The Cloud Firestore API is not available for Cloud Datastore projects.

I thought that maybe it needs the --projectId option but I wouldn't know which 'project' to use since the target database is on a [locally] emulated instance. Any ideas?

Hi,
Could you please let me know whether someone did any tools comparison with Fireway - https://github.com/kevlened/fireway and SchemaChange? If so, could you please share the details? Thanks!

I have a problem with forceWait. When I run migrations with forceWait fireway runs only the first migration and stops working after that without writing migration result to firestore. Without forceWait I works as expected but I get a lot of warnings. Any idea what's going on?

Example migration:

import { firestoreRoutes } from '@travel-together/core/routes';
import { MigrateOptions } from 'fireway';
import { createTraverser } from '@firecode/admin';

export const migrate = async ({ firestore, FieldValue }: MigrateOptions) => {
  const usersCollection = firestore
    .collection(firestoreRoutes.users)
    .where('active', '==', true)
    .where('deleted', '==', false);
  const storiesCollection = firestore.collection(firestoreRoutes.stories);
  const usersTraverser = createTraverser(usersCollection);

  const { docCount } = await usersTraverser.traverse(
    async (batchDocs, batchIndex) => {
      const batchSize = batchDocs.length;
      for (const doc of batchDocs) {
        const userData = doc.data();
        const storyData = {
          creatorId: doc.id,
          createdAt: userData.createdAt,
          lastActivityAt: userData.createdAt,
        };
        await storiesCollection.add(storyData);
        await doc.ref.update({
          numStories: FieldValue.increment(1),
        });
      }
      console.log(
        `Batch ${batchIndex} done! Migrated ${batchSize} users in this batch.`,
      );
    },
  );

  console.log(`Migrated ${docCount} users`);
};

Hello,
This is a great tool! So, thanks!

I ran into this problem when I was figuring out how this package works, and it made it really difficult to figure out what I needed to do!
I also don't have a solution to it, other than a lot of emulator reloads.

I tried running a migration against my local emulators, but it only ever returned 1 document.
I exported the data from my emulators, restarted them and re-imported the data, then the migration found the additional documents.
I think I had 1 document in the collection I was migrating when I first loaded the emulators and dry-ran the migration. I then added more documents for more testing, which it never picked up.
I had to export the new data from the emulators, restart the emulators, then reimport the data for it to be picked up by fireway.

Also, the saved migration details seemed to disappear, but still function for fireway.
So, I would run a v0.0.1 migration against the local emulators. It would do its thing.
No new collection called fireway would turn up in the emulators, however fireway would no longer run that migration version.
I looked through all my firebase projects in case it had leaked to a dev/prod project, however I didn't see it in any of them.

Once I confirmed my migrations (with lots of emulator restarts), everything ran fine on my dev/prod projects.

Based on the official docs, I have been trying to implement a migration that removes some deprecated fields in my documents. Here's the error I have consistently faced:

Update() requires either a single JavaScript object or an alternating list of field/value pairs that can be followed by an optional precondition. Value for argument "dataOrField" is not a valid Firestore document. Couldn't serialize object of type "DeleteTransform" (found in field "test1"). Firestore doesn't support JavaScript objects with custom prototypes (i.e. objects that were created via the "new" operator).

Here's the minimal code that reproduces the error (this same code works as expected when run as a script of its own, or when it's executed from a REST endpoint function):

// ./migrations/v0.0.1__test_field_deletion.js
const { admin } = require("firebase-admin");

const FieldValue = admin.firestore.FieldValue;

/**
 * remove deprecated fields were left behind by previous migrations
 */

module.exports.migrate = async ({ firestore }) => {
  try {
    // Get the `FieldValue` object
    const FieldValue = admin.firestore.FieldValue;

    // Grab the doc id.
    const docId = "1i9oM3IWWI2Q2VU5Pzjp";
    // Grab the collection name
    const collName = "consignments";
    // Create a document reference
    const docRef = firestore.collection(collName).doc(docId);
    let docItem = await docRef.get();

    const payload = {
      test1: FieldValue.delete()
    };

    console.log("payload ==> ", payload); // { test1: DeleteTransform {} }
    const res = await docRef.update(payload); // run the actual update op to delete the field
    console.log("deletion result: ", res);
  } catch (error) {
    console.log("error performing field delete migration: ", error.toString());
  }
};

Any ideas on how I can get it working with fireway?

My stack:

• macOS Catalina
• fireway --version = 0.4.0
• node --version = v14.3.0

Imagine you have this (Renamed FieldValue to fv to avoid confusion below).

function migrate({ firestore, FieldValue: fv }: MigrateOptions) {}

So in this case, if you do something like

const timestamp = fv.serverTimestamp()

Typescript will complain: Property serverTimestamp is a static member of type FieldValue.

It's because fv is typed as instance of class FieldValue, not the class FieldValue itself.

The solution would be to type it like this FieldValue: typeof FieldValue.

I haven't checked, but it might be the case for FieldPath and Timestamp as well.

For some reason, I keep getting warnings that "fireway detected open async" calls whenever I create a document in a migration.
All of these example migrations that do manage their async calls properly, are getting that warning.

module.exports.migrate = async ({ firestore: db }) => {
    await db.collection('test').add({ foo: 'bar' });
};
module.exports.migrate = async ({ firestore: db }) => {
    await db.collection('test').doc().set({ foo: 'bar' });
};
module.exports.migrate = ({ firestore: db }) => {
   db.collection('test').doc();
};

I'm not a very good with javascript/typescript, so maybe I'm talking nonsense.

It would be lovely to be able to write the migration scripts in typescript.
I've tried it, and got a syntax error when trying to import.

Is it possible to support typescript?

vi creates .*swp files while editing. So when I'm developing the migration script and I run fireway with --dryrun, I get the following error:

ERROR: Both v0.0.1__move-data-from-user-to-profile.js and .v0.0.1__move-data-from-user-to-profile.js.swp have the same version

My expectation is that files with a leading . would be ignored.

Thanks for this tool! I'm not sure how database migrations is completely ignored by the firebase CLI tool / firestore team.

The following migration script has an error in it. serverTimestamp is missing parens.

module.exports.migrate = ({firestore, FieldValue}) => {
  firestore.collection("users").get()
    .then((querySnapshot) => {
      querySnapshot.forEach(async (queryDocumentSnapshot) => {
        const id = queryDocumentSnapshot.id;
        const data = queryDocumentSnapshot.data();

        await firestore.collection("profiles").doc(id).set({
          createdAt: FieldValue.serverTimestamp,
          userId: id
        }); 
      });
    });
};

Dryrun does not catch this. It may be difficult for fireway to be able to detect such errors.

But what this issue is about is that the document stored by fireway sets success to true even though this error is thrown:

(node:19102) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 63)
(node:19102) UnhandledPromiseRejectionWarning: Error: Cannot encode value: serverTimestamp() {
        return ServerTimestampTransform.SERVER_TIMESTAMP_SENTINEL;
    }

My expectation is that the fireway document would set success to false.

This is low priority as my understanding is that fireway, whether success is true or false, will not process the migration again until I remove this document.

Thanks.

I noticed an inconsistency when running migration scripts with the --dryrun option.
I was testing out my code to copy data from one collection to a new one. Whenever a change to the script was made, I ran the migration script which worked as expected on most occasions but made changes to the db sometimes.
The new collection is created and the supplied data is added.

The expected behaviour is that the new collection should not be created when using --dryrun.

Command ran: fireway migrate --dryrun
Fireway [email protected]
Node: v14.16.0
OS: macOS Catalina

Is there a way to run a specific migration if I have multiple scripts living in my migrations/ folder? Lately, I have had a few use cases where I need to re-run a script whose entry is already in the fireway collection on Firestore. However, to actually make a script to be executed, I have found that I have to manually delete the entry corresponding to that script and others that came after it. For example, let's say I have:

migrations/v0.0.1__first_migration.js
migrations/v0.0.2__second_migration.js
migrations/v0.0.3__third_migration.js

If I need to run just v0.0.2__second_migration.js, I have to delete both the corresponding v2 and v3 entries in the fireway collection and then rerun fireway migrate. Wouldn't it be nice if I could do something like fireway migrate --script=v0.0.2?

Noticed this after doing a few migrations:

I would expect the install_rank to be a number and to be incremented however it looks be treated as string here hence the appending behaviour. Any ideas?

I need to do a migration and would love to use this tool but the README is not very helpful. Can you do a tutorial or add more documentation?

I installed yarn then tried yarn add -G fireway but it said error Missing list of packages to add to your project.

I want to configure my firestore settings for a specific migration. I need to set "ignoreUndefinedProperties: true". When I do this within the migrate function, I receive an error that Firestore has already been initialized. Is there a way to configure firestore settings?

export async function migrate({firestore} : MigrateOptions) {
  firestore.settings({ timestampsInSnapshots: true, ignoreUndefinedProperties: true });
  ...
}

Hey, I'm done searching for firestore migration, and just found this great tools. But seems only support for CommonJS.

Can I use ES6 module import ? because I faced this error when i'm using ES6 Module import.

v0.0.1__foods.js

import * as fs from 'fs';
import * as path from 'path';

async function migrate({ firestore, FieldValue }) {
  const foods = JSON.parse(fs.readFileSync(path.resolve('src/seeds/foods.json')));

  foods.forEach(async (item) => {
    await firestore.collection('foods').doc(item.id).set(item);
  });
}

export default migrate;

I'm attempted to setup a VSCode debugger launch.json to debug fireway migrations.
The migrations are Typescript so that adds an additional level of complexity.
Wondering if you can provide an example.

Hi,

Have you had any thoughts on how you might want to do reverse migrations? I'm likely to have some time to implement this and would want to do it in a way that lines up with your plans.

Is it even possible to access the admin object from the migration script?

I want to change some filed names inside customUserClaims, so I thought it would be nice to create a migration for that. But for doing that I would need to access admin.auth().listUsers() and then run setCustomUserClaims() for each individual user.

I've added const admin = require("firebase-admin"); before the script, which amazingly worked, but that feels like a workaround and not a proper solution.

I would suggest adding admin object to MigrateOptions

I got firebase token by using command firebase login:ci, and after deploy with flag --token

How to run migrations with this token?