onCreate fires two times about firebase-functions

[brianchirls]

@laurenzlong If this is expected behavior, it should be reflected in the Firebase Authentication triggers documentation. There is no mention of the possibility of multiple executions, nor of the need for idempotency.

In fact, the document referenced above specifically mentions sending a welcome email as a use case, which is decidedly not an idempotent operation. The Firebase samples repo even includes such an example, which does nothing to account for multiple executions.

This is happening on one of my apps in production, where users are receiving 3 welcome emails. At best, this makes us look bad. Worse, if we get flagged as spam, it could affect our ability to send emails in the future.

If this is not going to be fixed, could you (or someone) please:

1. Update the documentation
2. Update the example

Thanks

[schankam]

Am I the only one to find this behaviour completely ridiculous ? Does it mean that I need to implement an event system at the beginning of EACH of my functions if I don't want them to randomly be ran more than twice ??? This sounds a bit crazy to me tbh. How can we even use Cloud Function in production with such a weird behaviour.

Plus if I need to write to Firestore just for this specific use case, it's increasing my number of writes to the database, which will increase my bill at the end of the month. What the...

[kylecordes]

I find also, that onCreate triggers often happen 1, 2, or 3 times.

While it is no doubt possible to create some additional mechanism to attempt to only execute the intended logic once... that certainly seems like the sort of thing the platform should be able to do for us?

[radar155]

Any update from the backend team? The issue is still present and it's very annoyng. This should be fixed.

[kossnocorp]

@erperejildo I don't think this will be ever resolved. I believe that is the price we have to pay for the serverless.

I created a high-order function that runs a transaction on Firestore that checks if a record with given eventId is present in the DB and if so skip the callback, otherwise it creates the record and runs the callback. I wrap all the events with it, and it works fine. I only wish it could be builtin functionality.

[mikob]

what exactly is the benefit of this "intended behavior"? This seems to cause unnecessary problems for a lot devs using firebase. Can this be re-opened?

[laurenzlong]

Thanks for the report @notsonotso. We've made the backend team aware of this issue. I agree that it is annoying, and should be fixed. In the meanwhile, you can make use of event.eventId to handle the case where there are duplicate invocations, since the duplicate events will have the same eventId. In fact, I would recommend:
admin.database().ref('userqueue/' + event.eventId).set({...})
Closing this since it is not a SDK bug.

Thank you @laurenzlong

1. Where am I supposed to submit backend bugs?
2. Can this happen on all kinds of events, or just onCreate?

[thechenky]

To reiterate again, this is expected behavior. You would need to write your functions in an idempotent way so that they are able to handle cases when they are invoked more than once.

[ewindso]

Question, has anyone ever noticed this with an onCall or onRequest function? We're trying to make everything idempotent, but these don't appear to have an event ID.

[kossnocorp]

@neilpoulin you can ensure it with transactions, here's the code I use:

import { EventContext } from 'firebase-functions'
import { collection, ref, transaction } from 'typesaurus'

export interface EventClaim {
  eventType: string
  eventId: string
  time: Date
}

export const eventClaims = collection<EventClaim>('eventClaims')

export default function once<EventData>(
  fn: (data: EventData, context: EventContext) => Promise<any>
) {
  return async (data: EventData, context: EventContext) => {
    const { eventId, eventType } = context
    const claimRef = ref(eventClaims, eventId)

    await transaction(async ({ get, set }) => {
      const claimDoc = await get(claimRef)

      if (claimDoc) throw new Error('The claim is already exist')

      await set(claimRef, {
        eventType,
        eventId,
        time: new Date()
      })
    })

    return fn(data, context)
  }
}

[wooliet]

It never occurred to me to wrap the check in a transaction. Thanks @kossnocorp !

Well, "consistently". I just got three messages when adding a user.

[itaydr]

I'm also experiencing this occasionally with firestore().onCreate(...).

[laurenzlong]

Hi everyone, I was mistaken when I said that this is a bug. Cloud Functions aims to fire at least once, which means it could fire more than once. So you should write your functions in a idempotent way in order to handle the possibility of multiple triggering. I've filed an internal bug to better document this.

[Araknos]

https://stackoverflow.com/questions/54541177/firebase-cloud-functions-executed-several-times

[wooliet]

I would guess there were technical considerations, and it was decided to opt for triggering "at least once, maybe more" rather than "zero or once". You can't handle an event you never get, but you can deal with an event received multiple times.

[mikob]

Part of the reason of using a framework is to abstract away such technical considerations. There are other minutea that firebase abstracts away for us, why not this?

[erperejildo]

Still getting this issue after more than a year. Any updates?

[wooliet]

We have a couple dozen onCall and I have not run into this issue.

[neilpoulin]

@kossnocorp Since writes to firestore are async, how can you be sure that your first event wrote to firestore before your second (read: duplicate) event tries to read the event ID? While the approach seems like a great start, it doesn't seem like it will work 100% of the time (unless I'm misunderstanding how the transaction block works). The only way I can figure out how to handle this is to write the function to be idempotent.

[tsdexter]

@kossnocorp am I thinking about this correctly?

It seems that your once function will only process the event once, even if the processing fails for some reason. This is not ideal for aggregation. For example, if the event is onDelete and you want to decrement a total then if the decrement transaction fails for some reason, when it retries, the once function will stop it and the counter will never be decremented.

Am I correct in thinking this is a plausible scenario (it's hard for me to test as I can't find a way to trigger the same event again)?

Here is some code that I've implemented to ensure the decrement happens if, for some reason, the original events transaction fails.

utils/once.ts

import { EventContext } from 'firebase-functions'
import * as admin from "firebase-admin";

export default function once<EventData>(eventHandler: (data: EventData, context: EventContext) => Promise<any>) {
  return async (data: EventData, context: EventContext) => {
    await admin.firestore().runTransaction(async (transaction) => {
      const eventRef = admin.firestore().doc(`events/${context.eventId}`)
      const eventDoc = await transaction.get(eventRef);
      if (eventDoc.exists && eventDoc.data()?.success) throw new Error("Event already successfully processed");
      transaction.set(eventRef, {eventId: context.eventId, createdAt: admin.firestore.FieldValue.serverTimestamp()});
    });
    return eventHandler(data, context);
  }
}

export function setEventSuccess(transaction: admin.firestore.Transaction, context: EventContext): admin.firestore.Transaction {
  const eventRef = admin.firestore().doc(`events/${context.eventId}`);
  return transaction.set(eventRef, { success: true, successAt: admin.firestore.FieldValue.serverTimestamp() }, { merge: true });
}

index.ts

import * as functions from "firebase-functions";
import * as admin from "firebase-admin";
import once, { setEventSuccess } from "./utils/once";
admin.initializeApp();

export const customerDeletedAggregation = functions.firestore
  .document("customers/{id}")
  .onDelete(once(async (snap, context) => {
    // aggregate the deletion within stats/customers.total
    const statsRef = admin.firestore().doc('stats/customers');
    return admin.firestore().runTransaction(async (transaction) => {
      // if this transaction succeeds the events success property will be true
      // if it fails for any reason it will still be undefined so the event is still
      // processed when retried      
      const statsDoc = await transaction.get(statsRef);
      const data = statsDoc.data();
      const total = data?.total || 1; // this is a delete so even if total is not set for some reason, there is at least this one
      setEventSuccess(transaction, context)
        .set(statsRef, {...data, total: total - 1 }, { merge: true });
    });
  }));