[destination-bigquery] After an upgrade to 2.8.0 a connection to a postgres source started failing about airbyte

[killthekitten]

@edgao downgrade to 2.6.3 worked!

[edgao]

that sounds like an unrelated thing (platform logs out some record validation stuff, but afaik it's purely informational and doesn't actually affect sync success)

pinged the db sources team internally to take a look at this, stream statuses are a relatively new thing so it's possible that there's some rough edges in that region

[evantahler]

@killthekitten are you in our community slack? We could do the log transfer there

[PieterBos94]

I am having the same issue with LinkedIn, PostHog and Hubspot after the upgrade to 2.8.0. Downgrading to 2.6.3 solved the problem. I had the same error in the logs about the stream status. Let me know if I can supply any additional information :)

io.airbyte.workers.exception.WorkerException: A stream status has been detected for a stream not present in the catalog

[benmoriceau]

Thanks @killthekitten,
Is the error reported in the ticket from the 1 step?

Could you send me the result of the following query so I can see the failure reason of all the attempts and make sure that we are going to address the right error:

SELECT j.id, j.config, j."status", a."status", a.failure_summary, a."output" 
FROM public.jobs j
join attempts a on a.job_id = j.id
where j."scope" = '<YOUR_CONNECTION_ID>'
order by j.id desc
limit 50;

I am also on the community slack under the name of Benoit Moriceau (Airbyte)

[edgao]

@killthekitten for the thing about only having 54 records after two attempts - is that still happening? (if yes, can you post the logs? I'm on the airbyte slack, feel free to DM me)

[edgao]

submitted #41036 - this doesn't explain the A stream status (null.ad_campaign_analytics) has been detected for a stream not present in the catalog thing, but it does explain @killthekitten 's observation (where the second attempt of a sync doesn't write all the records correctly). I think this is an easy fix, will try and get a PR out today/tomorrow.

[killthekitten]

@benmoriceau I would be able to do it after 14th

[alexnikitchuk]

we observe the same issue with Snowflake destination 3.11.1 while syncing from Postgres source. Looks fixed by downgrading to 3.10.1

2024-07-11 04:00:20 platform > readFromSource: exception caught
io.airbyte.workers.exception.WorkerException: A stream status (public.primary_record_balance_segments) has been detected for a stream not present in the catalog
	at io.airbyte.workers.helper.StreamStatusCompletionTracker.track(StreamStatusCompletionTracker.kt:36) ~[io.airbyte-airbyte-commons-worker-0.63.4.jar:?]
	at io.airbyte.workers.general.BufferedReplicationWorker.readFromSource(BufferedReplicationWorker.java:361) ~[io.airbyte-airbyte-commons-worker-0.63.4.jar:?]
	at io.airbyte.workers.general.BufferedReplicationWorker.lambda$runAsyncWithHeartbeatCheck$3(BufferedReplicationWorker.java:242) ~[io.airbyte-airbyte-commons-worker-0.63.4.jar:?]
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]

[tturkenitz]

After upgrading to 0.63.8 and connector version of 2.8.4 the issue persists but with a different message.
All connections which sync to BigQuery continue to fail. The error message has changed from:

Warning from destination: Some streams were unsuccessful due to a source error: [io.airbyte.protocol.models.v0.StreamDescriptor@709f0202[name=HOSTED_DOMAINS,namespace=IpInfo,additionalProperties={}]

To:

2024-07-17 08:50:56 destination > ERROR main i.a.c.i.b.IntegrationRunner(runInternal):216 caught exception! io.airbyte.commons.exceptions.TransientErrorException: Some streams were unsuccessful due to a source error. See logs for details.
	at io.airbyte.cdk.integrations.destination.async.AsyncStreamConsumer.close(AsyncStreamConsumer.kt:210) ~[airbyte-cdk-core-0.41.4.jar:?]
	at kotlin.jdk7.AutoCloseableKt.closeFinally(AutoCloseableJVM.kt:46) ~[kotlin-stdlib-1.9.23.jar:1.9.23-release-779]
	at io.airbyte.cdk.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.kt:209) [airbyte-cdk-core-0.41.4.jar:?]
	at io.airbyte.cdk.integrations.base.IntegrationRunner.run(IntegrationRunner.kt:116) [airbyte-cdk-core-0.41.4.jar:?]
	at io.airbyte.integrations.destination.bigquery.BigQueryDestinationKt.main(BigQueryDestination.kt:386) [io.airbyte.airbyte-integrations.connectors-destination-bigquery.jar:?]

Stack Trace: io.airbyte.commons.exceptions.TransientErrorException: Some streams were unsuccessful due to a source error. See logs for details.
	at io.airbyte.cdk.integrations.destination.async.AsyncStreamConsumer.close(AsyncStreamConsumer.kt:210)
	at kotlin.jdk7.AutoCloseableKt.closeFinally(AutoCloseableJVM.kt:46)
	at io.airbyte.cdk.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.kt:209)
	at io.airbyte.cdk.integrations.base.IntegrationRunner.run(IntegrationRunner.kt:116)
	at io.airbyte.integrations.destination.bigquery.BigQueryDestinationKt.main(BigQueryDestination.kt:386)

2024-07-17 08:50:56 platform > readFromDestination: exception caught
io.airbyte.workers.internal.exception.DestinationException: Destination process exited with non-zero exit code 1
	at io.airbyte.workers.general.BufferedReplicationWorker.readFromDestination(BufferedReplicationWorker.java:493) ~[io.airbyte-airbyte-commons-worker-0.63.8.jar:?]
	at io.airbyte.workers.general.BufferedReplicationWorker.lambda$runAsync$2(BufferedReplicationWorker.java:235) ~[io.airbyte-airbyte-commons-worker-0.63.8.jar:?]
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
2024-07-17 08:50:56 platform > readFromDestination: done. (writeToDestFailed:false, dest.isFinished:true)

I will try out the image as well. Thanks!

[killthekitten]

@davinchia this is of course unfortunate, but I understand the reasoning and I'm incredibly grateful for the support the Airbyte team already provided with this issue.

We are out of ideas, so we decided to set up a fresh instance of Airbyte with the latest versions of connectors, and we are migrating all of our destinations to this instance. It is fully isolated: a new GCP project, a new airbyte_internal dataset, a new bucket and a new config DB. It's been running without any issues for more than a week already, and I hope it will continue this way.

The new setup doesn't use abctl but I've scheduled a call with the Airbyte team to migrate to it. Could you explain why you think abctl could help with this issue? I don't yet see how it might be relevant and would appreciate your insight.

And before we wrap up: @benmoriceau so far, the dev-02eae48501 worker hasn't been failing, but also could it be masking actual errors? Could you tell me what string should I search for in logs to make sure it's running smooth? Also, should I revert to the default version, or should I keep running on this?

[benmoriceau]

@killthekitten My suspicion is that there is something which is not properly done on docker. Using abctl will make airbyte to run on kuberbete instead of docker without requiring the installation of a local kube cluster.

The xustom image is changing the error into a warning and let the connection to continue: Here is the change

[killthekitten]

Downgrading to 2.7.1 results in the same error.

[marcosmarxm]

@airbytehq/destinations can someone take a look into this issue?
My guess during the upgrade running discover schema a table was deleted or lost permission but the state still exist. @killthekitten can you check the state message (Settings -> Advance) and check if all streams/tables are in there also match the ones in Replication tab?

[edgao]

tl;dr this sounds like a source bug, but try downgrading destination-bigquery to 2.6.3 as a workaround?

WorkerException: A stream status has been detected for a stream not present in the catalog

this sounds like the platform detecting that the source emitted an invalid stream status message (which is not the same as a state message 😓 ). (relevant platform code here)

And I think the reason this works on the pre-dv2 destination-bigquery version is that platform only triggers this validation when the destination supports refreshes (code.) Destination Bigquery added support for refreshes in 2.7.0, so downgrading to an older version will mask the bug.

so @marcosmarxm I think it's worth tagging either DB sources to investigate source-postgres, or platform move to check the platform code - not sure which one is more likely though :/

[killthekitten]

@marcosmarxm can't find the state information anywhere (looked under general settings, connection and connector settings). Could it be that the latest Airbyte doesn't show this info anymore? I've been looking for it earlier today already.

@edgao it could very well be a source issue. Not sure how relevant this is, but we've been seeing a lot of these errors in logs:

#17319

Is there anything else I could look up to help with investigation?

[rodireich]

The table in question is "public"."admin_notes".
The error is happening in the context of initial sync completing.
We need more logging, context in order to try to reconstruct what went wrong here.

This looks like user cursor incremental sync (i.e no CDC or xmin)

[rodireich]

I tried recreating a similar scenario with all the information we have so far.
Not seeing the problem

[killthekitten]

@rodireich admin_notes showing up in logs before the error could be a coincidence, as this stream is first in the list when you sort alphabetically.

Is there any secure channel where I can send you the full logs? I could send both the successful and the failing sync logs. Also, don't have the access to the link you shared so can't confirm whether the setup matches ours.

[killthekitten]

This looks like user cursor incremental sync (i.e no CDC or xmin)

Correct, we use the cursor mode. That said, we perform a full rewrite on all streams

[killthekitten]

@evantahler messaged you just now 👌

[evantahler]

Thank you!
Internal reference to slack thread for Airbyte folks looking for the log

[benmoriceau]

Hello @killthekitten and @PieterBos94,
We have created a new release which is changing the error being logged by adding which is the missing stream. Would it be possible to upgrade your platform and re-run with the version 2.8.0 of big-query?

Thanks,

[benmoriceau]

Would it be possible to send me the catalog associated with the connection?
The request to get it is SELECT "catalog" FROM public."connection" where id = <CONNECTION_ID>;

Could you also let me know if there is a stream prefix set. It can be find in the setting tab of the connection.

[killthekitten]

@benmoriceau

Would it be possible to upgrade your platform and re-run with the version 2.8.0 of big-query?

I won't be able to test the new release on our current instance, unfortunately, at least not this week (please ping me again if needed in the future).

That said, I took time to re-create our airbyte installation in a new GCP project and to my surprise, the sync didn't fail on 2.8.0. The only differences I can think off:

1. It's a fresh Airbyte DB with just one configured connection to the same PostgreSQL data source that we use in a real setup
2. It uses a clean GCS staging area

Would it be possible to send me the catalog associated with the connection?

Sure, I've exported the catalog data and passed it to @evantahler.

Could you also let me know if there is a stream prefix set. It can be find in the setting tab of the connection.

There's no prefix

[evantahler]

Internal link with @killthekitten's Catalog - thanks!

[benmoriceau]

Thank you @killthekitten,
Looking at the catalog, the schema looks ok. I would like to make sure that it is the catalog coming from the old deployment and not the new one.

What is happening is that the version 2.8.0 of big query is adding the a new functionality named refresh. This now require having a stream status being forwarded to the destination. When this functionality is activated, we are validating that the stream status are valid. We have deployed this to our internal deployment of Airbyte and we didn't run into any issue.

I am going to run a comparison of the format of the catalog that you send and an example of our internal catalog to see if I see anything that could explain the issue.

Thanks,

[killthekitten]

@benmoriceau confirmed, this is the catalog from the old deployment but downgraded to bigquery 2.6.3.

I can also make a dump of the fresh deployment's catalog (2.8.0) if that is helpful.

Could you briefly describe what is a status?

[benmoriceau]

@killthekitten, I believe that I am starting to understand the issue. It seems to be the same root cause than #39900. We have introduce a new metadata in the connector definition. In order to properly propagate the metadata, we need to upgrade the platform before we upgrade the connector so the platform can properly read the metadata.

In order to confirm this could you let me know the result of the following query

SELECT supports_refreshes 
FROM public.actor_definition_version
where docker_repository = 'airbyte/destination-bigquery' and docker_image_tag = '2.8.0';

I would like to confirm my assumption because I don't understand why it wasn't the same error than the other tickets.

I am thinking about a way to fix that for the old deployment which requires as few manual action as possible.

[killthekitten]

@benmoriceau here you go:

airbyte=> SELECT supports_refreshes 
FROM public.actor_definition_version
where docker_repository = 'airbyte/destination-bigquery' and docker_image_tag = '2.8.0';
 supports_refreshes 
--------------------
 t
(1 row)

In order to properly propagate the metadata, we need to upgrade the platform before we upgrade the connector so the platform can properly read the metadata.

FYI I saw the message in BigQuery changelog and made sure the platform was running on 0.63.0 before I updated to BigQuery 2.8.0.

[benmoriceau]

Thanks @killthekitten, It seems that the sequence of updates was right which means that the functionalities supported by the destination have been properly imported.

I am still looking for the root cause, I will make a fix to prevent failing the sync in the platform.

Could you confirm that only the attempt were failing, if a new job is created, is it still failing? I wonder if we were using the old image tag for the config but for a connector which has been updated. It was fixed here But this got release in a platform version you are currently using (release version was 0.63.1).

[killthekitten]

@benmoriceau sorry, not sure I understood the question! IIRC the moment we switched the connector version, all attempts started failing and/or crashing the instance, only one succeeded out of a dozen. Is that what you asked?

[benmoriceau]

@killthekitten I am wondering if the jobs start failing after the X attempts failed. It should then create a new job, I am wondering what is the status of this job.

[killthekitten]

@benmoriceau our logs are a mess, as a bit of panic was involved and I don't remember how many times I cancelled the jobs while switching the platform version etc, so can't say which of the jobs were created after using up all attempts. Here is how it looked like (from earliest to latest):

1. Sync failed, 260k records extracted, nothing loaded in any of the 5 attempts
2. Clear data initiated by me and succeeded
3. Sync partially succeeded, 2 attempts, various number of records extracted/loaded
4. Sync partially succeeded, 4 attempts, the first attempt looks like all of the data was synced but still failed
5. Another clear succeeded
6. A cancelled sync
7. A successful sync (reverted the version back)

[benmoriceau]

Hello @killthekitten,
Would you be able to upgrade the platform, bump the version of the big query destination to 2.8.0 or more and re-run the sync.

I am not seeing anything which would point me to a root cause in the query result. Upgrading to the latest version of the platform will add some stream name information in the failure message. It will help me to check if it is in the catalog of the job.

Thanks,

[killthekitten]

I could try tomorrow afternoon!

[benmoriceau]

Thanks!

[killthekitten]

On a side note we have a related issue: every time an airbyte instance goes down, any failed syncs are retried, but the retried attempt leaves the dataset in an inconsistent state. I believe this has started when I rolled back 2.8.0.

• Attempt 1, failed: 4,274,883 records extracted, 4,274,883 records loaded
• Attempt 2, succeeded: 54 records extracted, 54 records loaded
• The final sync job status: 4,274,937 records extracted, 4,274,937 records loaded, but the dataset only contains a handful of records (probably 54)

Could this be related to the refresh functionality that I reverted? Should I file a new issue?

[benmoriceau]

@killthekitten Yes it seems related to the refreshes functionality on the destination side. For some reason the destination only has the latest attempt data.
@stephane-airbyte FYI as destination OC.

[killthekitten]

@benmoriceau

Would you be able to upgrade the platform, bump the version of the big query destination to 2.8.0 or more and re-run the sync.

I upgraded the platform to 0.63.4 and the destination to 2.8.1, and the sync has been completed successfully. Here are the differences compared to the time it failed:

• I made sure to wait until ALL of the syncs finish before running the update, and additionally switched all of the syncs off before turning off the instance. The last time it wasn't the case, I believe I shut down the instance abruptly without cancelling the syncs. There could've been some temp state stored from the cancelled syncs.
• Last week, when the syncs failed, I've made sure to delete the sync folder in the staging GCS bucket. I did it at the same time as I rolled back to 2.6.3, I believe.
• Last time it was 2.8.0, this time it's 2.8.1
• Last time I went from 0.63.0 to Airbyte 0.63.2 then to 0.63.1, this time it was 0.63.3 -> 0.63.4

Could the issue have been caused by the fact that the sync was cancelled, and then 2.8.0 attempted to resume it?

[killthekitten]

@benmoriceau update:

• After I applied the 2.8.1 update, the syncs succeeded.
• Then I've paused and unpaused a few connectors, and all of them failed on the first attempt.
• After that, some of the connectors are failing, and some are successful.

[killthekitten]

A couple of syncs are not stable anymore (maybe a coincidence), so I'm reverting back to 2.6.3 once again. Would appreciate any hints about resolving the inconsistent state issues on 2.6.3.

If that is still relevant, I've found the "stream status has been detected" message in one of the logs:

io.airbyte.workers.exception.WorkerException: A stream status (public.transaction) has been detected for a stream not present in the catalog
        at io.airbyte.workers.helper.StreamStatusCompletionTracker.track(StreamStatusCompletionTracker.kt:36) ~[io.airbyte-airbyte-commons-worker-0.63.4.jar:?]
        at io.airbyte.workers.general.BufferedReplicationWorker.readFromSource(BufferedReplicationWorker.java:361) ~[io.airbyte-airbyte-commons-worker-0.63.4.jar:?]
        at io.airbyte.workers.general.BufferedReplicationWorker.lambda$runAsyncWithHeartbeatCheck$3(BufferedReplicationWorker.java:242) ~[io.airbyte-airbyte-commons-worker-0.63.4.jar:?]
        at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]

These are present only in some cases. The table naming is confusing in the logs: in some places I see public.{{stream_name}}, in others it is public_{{stream_name}} and also there's a bunch of public:{{stream_name}}. I'd assume this is just formatting issues.

[killthekitten]

Another failure of this kind but in Stripe 3.17.4 <> BigQuery 1.2.20 sync:

io.airbyte.workers.exception.WorkerException: A stream status (public.transaction) has been detected for a stream not present in the catalog
        at io.airbyte.workers.helper.StreamStatusCompletionTracker.track(StreamStatusCompletionTracker.kt:36) ~[io.airbyte-airbyte-commons-worker-0.63.4.jar:?]
        at io.airbyte.workers.general.BufferedReplicationWorker.readFromSource(BufferedReplicationWorker.java:361) ~[io.airbyte-airbyte-commons-worker-0.63.4.jar:?]
        at io.airbyte.workers.general.BufferedReplicationWorker.lambda$runAsyncWithHeartbeatCheck$3(BufferedReplicationWorker.java:242) ~[io.airbyte-airbyte-commons-worker-0.63.4.jar:?]
        at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]

[benmoriceau]

@killthekitten Thanks for the logs,
The way of writing the namespace and name is just a formating in the log.
For this connection, is the stream transaction selected? It will also be helpful to get the catalog use in the job which fails with the error A stream status (public.transaction) has been detected for a stream not present in the catalog.

select config
from jobs
where id = <JOB_ID>

[killthekitten]

The transaction stream is selected in this case, yes. What I've noticed, this could be any of the streams, I've seen different streams failing in different jobs. I've shared the config in DM 👍

[benmoriceau]

@edgao I think that i have a fix for the missing stream in catalog. I wonder if the missing data in the destination is a side effect of that because we are not sending the COMPLETED status to the destination since the orchestrator failed.

[killthekitten]

@edgao yes, it is still an issue! logs sent 👍

[benmoriceau]

@killthekitten Sorry if I missed that before but I would like to confirm that you are running with docker and without having the orchestrators activated.

[killthekitten]

@benmoriceau correct, we run with docker. What would be an orchestrator? Pretty sure we don't run anything like that.

[benmoriceau]

The orchestrator is a component which run on its own pod an is transferring data between the source and the destination. It is disabled by default on docker but enabled by default on kube. There might be some differences around how the catalog is loaded in the different modes.
That being said, I didn't managed to reproduce the issue on my local with the same configuration than you.

[killthekitten]

That's a pity. We did major version bumps and changed the stream configuration too many times in the last year, so it must be some broken state either in the config DB or airbyte_internal that is hard to reproduce

[PieterBos94]

I upgraded airbyte and BigQuery to the latest version and tried again and got different errors:

From posthog connector:
readFromDestination: exception caught java.lang.IllegalStateException: Destination process is still alive, cannot retrieve exit value. at com.google.common.base.Preconditions.checkState(Preconditions.java:515) ~[guava-33.2.0-jre.jar:?] at io.airbyte.workers.internal.DefaultAirbyteDestination.getExitValue(DefaultAirbyteDestination.java:210) ~[io.airbyte-airbyte-commons-worker-0.63.4.jar:?] at io.airbyte.workers.general.BufferedReplicationWorker.readFromDestination(BufferedReplicationWorker.java:492) ~[io.airbyte-airbyte-commons-worker-0.63.4.jar:?] at io.airbyte.workers.general.BufferedReplicationWorker.lambda$runAsync$2(BufferedReplicationWorker.java:235) ~[io.airbyte-airbyte-commons-worker-0.63.4.jar:?] at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]

2024-07-07 08:37:48 platform > failures: [ { "failureOrigin" : "destination", "failureType" : "transient_error", "internalMessage" : "io.airbyte.commons.exceptions.TransientErrorException: Some streams were unsuccessful due to a source error: [io.airbyte.protocol.models.v0.StreamDescriptor@649f25f3[name=annotations,namespace=posthog,additionalProperties={}], io.airbyte.protocol.models.v0.StreamDescriptor@2e62ead7[name=insights,namespace=posthog,additionalProperties={}], io.airbyte.protocol.models.v0.StreamDescriptor@3b98b809[name=projects,namespace=posthog,additionalProperties={}], io.airbyte.protocol.models.v0.StreamDescriptor@bf75b5c[name=feature_flags,namespace=posthog,additionalProperties={}], io.airbyte.protocol.models.v0.StreamDescriptor@16da476c[name=cohorts,namespace=posthog,additionalProperties={}], io.airbyte.protocol.models.v0.StreamDescriptor@2fd72332[name=persons,namespace=posthog,additionalProperties={}]]", "externalMessage" : "Some streams were unsuccessful due to a source error: [io.airbyte.protocol.models.v0.StreamDescriptor@649f25f3[name=annotations,namespace=posthog,additionalProperties={}], io.airbyte.protocol.models.v0.StreamDescriptor@2e62ead7[name=insights,namespace=posthog,additionalProperties={}], io.airbyte.protocol.models.v0.StreamDescriptor@3b98b809[name=projects,namespace=posthog,additionalProperties={}], io.airbyte.protocol.models.v0.StreamDescriptor@bf75b5c[name=feature_flags,namespace=posthog,additionalProperties={}], io.airbyte.protocol.models.v0.StreamDescriptor@16da476c[name=cohorts,namespace=posthog,additionalProperties={}], io.airbyte.protocol.models.v0.StreamDescriptor@2fd72332[name=persons,namespace=posthog,additionalProperties={}]]", "metadata" : { "attemptNumber" : 0, "jobId" : 10800, "from_trace_message" : true, "connector_command" : "write" }, "stacktrace" : "io.airbyte.commons.exceptions.TransientErrorException: Some streams were unsuccessful due to a source error: [io.airbyte.protocol.models.v0.StreamDescriptor@649f25f3[name=annotations,namespace=posthog,additionalProperties={}], io.airbyte.protocol.models.v0.StreamDescriptor@2e62ead7[name=insights,namespace=posthog,additionalProperties={}], io.airbyte.protocol.models.v0.StreamDescriptor@3b98b809[name=projects,namespace=posthog,additionalProperties={}], io.airbyte.protocol.models.v0.StreamDescriptor@bf75b5c[name=feature_flags,namespace=posthog,additionalProperties={}], io.airbyte.protocol.models.v0.StreamDescriptor@16da476c[name=cohorts,namespace=posthog,additionalProperties={}], io.airbyte.protocol.models.v0.StreamDescriptor@2fd72332[name=persons,namespace=posthog,additionalProperties={}]]\n\tat io.airbyte.cdk.integrations.destination.async.AsyncStreamConsumer.close(AsyncStreamConsumer.kt:199)\n\tat kotlin.jdk7.AutoCloseableKt.closeFinally(AutoCloseableJVM.kt:46)\n\tat io.airbyte.cdk.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.kt:209)\n\tat io.airbyte.cdk.integrations.base.IntegrationRunner.run(IntegrationRunner.kt:116)\n\tat io.airbyte.integrations.destination.bigquery.BigQueryDestinationKt.main(BigQueryDestination.kt:386)\n", "timestamp" : 1720341407780 }

[PieterBos94]

Found a bit clearer error in the linkedIn Ads connector:
83)\nCaused by: io.airbyte.workers.exception.WorkerException: A stream status (null.ad_campaign_analytics) has been detected for a stream not present in the catalog

[edgao]

@PieterBos94 just to check - how are you running airbyte? (helm chart, docker compose?)

[PieterBos94]

Docker compose on compute engine in GCP. Used this guide: https://docs.airbyte.com/deploying-airbyte/on-gcp-compute-engine

[benmoriceau]

Hello @killthekitten @PieterBos94,
Could deploy the worker with the image tag dev-02eae48501? This image changes the error to a warning and add a new log.
If you managed to deploy it and have a sync to run it would help us a lot to have the logs of it.

Thanks,

[timorkal]

These are my versions:
Airbyte: 0.63.6
Postgres: 3.4.24
BigQuery: 2.8.2

I get the same error:

2024-07-11 14:34:21 platform > readFromSource: exception caught io.airbyte.workers.exception.WorkerException: A stream status (public.transaction_analytics) has been detected for a stream not present in the catalog at io.airbyte.workers.helper.StreamStatusCompletionTracker.track(StreamStatusCompletionTracker.kt:36) ~[io.airbyte-airbyte-commons-worker-0.63.6.jar:?] at io.airbyte.workers.general.BufferedReplicationWorker.readFromSource(BufferedReplicationWorker.java:361) ~[io.airbyte-airbyte-commons-worker-0.63.6.jar:?] at io.airbyte.workers.general.BufferedReplicationWorker.lambda$runAsyncWithHeartbeatCheck$3(BufferedReplicationWorker.java:242) ~[io.airbyte-airbyte-commons-worker-0.63.6.jar:?] at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]

[benmoriceau]

@alexnikitchuk @timorkal Thank you for reporting the issue,
Did you try to use the image mentioned in #40555 (comment)?

Could you let me know if you are using docker to deploy airbyte? We are in the process to deprecate docker #40599 and since we haven't been able to reproduce the error internally I wonder if moving to kube through abctl will help.

[timorkal]

I am using docker and I've managed to deploy it.
For now it looks like it works and doesn't fail, I will monitor tomorrow too and update.

[PieterBos94]

@benmoriceau

Sorry I am not sure how to do this. I don't see a worker image in the https://docs.airbyte.com/operator-guides/configuring-airbyte

[killthekitten]

@PieterBos94 I haven't done this myself yet, but it should be enough to edit the docker compose yaml file:

• Find image: airbyte/worker:${VERSION}
• Replace with image: airbyte/worker:dev-02eae48501
• Restart compose

Make sure to backup your DB if you run it on compose too.

[whanata]

we observe the same issue with Snowflake destination 3.11.1 while syncing from Postgres source. Looks fixed by downgrading to 3.10.1

2024-07-11 04:00:20 platform > readFromSource: exception caught
io.airbyte.workers.exception.WorkerException: A stream status (public.primary_record_balance_segments) has been detected for a stream not present in the catalog
	at io.airbyte.workers.helper.StreamStatusCompletionTracker.track(StreamStatusCompletionTracker.kt:36) ~[io.airbyte-airbyte-commons-worker-0.63.4.jar:?]
	at io.airbyte.workers.general.BufferedReplicationWorker.readFromSource(BufferedReplicationWorker.java:361) ~[io.airbyte-airbyte-commons-worker-0.63.4.jar:?]
	at io.airbyte.workers.general.BufferedReplicationWorker.lambda$runAsyncWithHeartbeatCheck$3(BufferedReplicationWorker.java:242) ~[io.airbyte-airbyte-commons-worker-0.63.4.jar:?]
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]

Is there a seperate issue for Snowflake, I'm experiencing the same issue also here (although it works after it retries once).

We are using Mongo Connector to Snowflake Destination.
Currently I am on:

• Airbyte: 0.63.5
• Mongo Connector: 1.4.2
• Snowlfake Destination: 3.11.1

I believe it coincided with an upgrade on Snowflake Destination from 3.10 to 3.11 also when it started happening (we wanted the ability to refresh the stream in case we need to do a full refresh due to some missing data.

[benmoriceau]

Hello @whanata,
There is no specific issue for Snowflake as we believe that the root cause is the same. Are you also running on docker? If so did you try to deploy the custom platform version (link). It will help us to gather more logs to understand what is the issue.

Thanks,

[tturkenitz]

Hey, I want to mention that I'm experiencing the same issue. Regardless of the source I'm using, if the destination is BigQuery, I always get the following exception:

Warning from destination: Some streams were unsuccessful due to a source error: [io.airbyte.protocol.models.v0.StreamDescriptor@709f0202[name=HOSTED_DOMAINS,namespace=IpInfo,additionalProperties={}]

This error occurs for every stream in the source. I can reproduce this behavior with any source when the destination is BigQuery (tested with Snowflake, Google Analytics, and others).

My deployment runs on Docker as well. Happening on v062.3 and BQ connector version of 2.8.0.
Will upgrading to v0.63.8 resolve this issue? I can see that the CDK for BigQuery and Snowflake is bumped, but the PR #41041 is not part of the changelist.

I see that 41041 was released in 0.63.6. I'll upgrade to the latest version.

[killthekitten]

I switched to dev-02eae48501 and BigQuery 2.8.4 today. The first few syncs went without errors, and I shared the logs with @benmoriceau, I'll report again tomorrow.

A few observations since I last wrote here:

• We are experiencing #30691 again. Could it be relevant for this thread?
• A couple weeks ago we have reduced the frequency of syncs on all connectors, and the setup has been more stable since
• We have noticed stale data in the Stripe <> BigQuery 1.2.20 destination. The most recent data in the destination was more than three weeks ago, which is coincided with the date of the original issue that started the thread.

[davinchia]

Could it be relevant for this thread?
Not relevant. That's from hitting the out-of-the-box Temporal deployment Airbyte ships with.

@killthekitten we've been trying really hard to reproduce this internally without success. At this point we aren't going to devote more resources as this doesn't seem widespread. I understand this isn't always what users want to hear so I apologise for any inconvenience. I'll check in with Benoit with the logs just in case something does jump out.

To the rest of the folks in this issue, Docker support is going to be deprecated soon. Because of this, we've stopped investing in Docker support. I'd recommend giving upgrading to our new abctl tool a shot.

[killthekitten]

For the future record, we found these added warnings both in the fresh and old installations. I won't post them here, but as @benmoriceau summarized it, all namespaces are null instead of public in the stream extracted from the catalog, and it could have something to do with the docker setup.

I'm testing out an abctl installation and will report back in case I have any success with it.

[samcz1]

Seems like everyone on here has been experiencing this bug on Docker installs, but I want to flag I'm encountering the same error on a helm install on GKE. Airbyte 0.63.4, Bigquery destination 2.8.4. All my streams are syncing to the same BigQuery destination, but I'm only experiencing failures from some of them (HubSpot and LinkedIn Ads). HubSpot stream was working on smaller syncs but started to fail after some schema changes caused it to attempt a large backfill (~1M rows). Looking back on the logs it appears it had been having this issue the whole time, but previously had been able to restart and succeed on the second attempt (now it is doing 20 attempts and then failing). It isn't failing on the same stream each time, for a while it was on the contacts stream but yesterday I disabled the backfill setting and restored an older connection state and now contacts are loading successfully but it fails on the engagements stream.

Error logs:

2024-07-26 12:14:15 platform > readFromSource: exception caught
io.airbyte.workers.exception.WorkerException: A stream status (null.engagements) has been detected for a stream not present in the catalog
        at io.airbyte.workers.helper.StreamStatusCompletionTracker.track(StreamStatusCompletionTracker.kt:36) ~[io.airbyte-airbyte-commons-worker-0.63.4.jar:?]
        at io.airbyte.workers.general.BufferedReplicationWorker.readFromSource(BufferedReplicationWorker.java:361) ~[io.airbyte-airbyte-commons-worker-0.63.4.jar:?]
        at io.airbyte.workers.general.BufferedReplicationWorker.lambda$runAsyncWithHeartbeatCheck$3(BufferedReplicationWorker.java:242) ~[io.airbyte-airbyte-commons-worker-0.63.4.jar:?]
        at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
2024-07-26 12:14:15 platform > readFromSource: done. (source.isFinished:false, fromSource.isClosed:false)
2024-07-26 12:14:15 platform > processMessage: done. (fromSource.isDone:true, forDest.isClosed:false)
2024-07-26 12:14:15 platform > writeToDestination: exception caught
java.lang.IllegalStateException: Source process is still alive, cannot retrieve exit value.
        at com.google.common.base.Preconditions.checkState(Preconditions.java:515) ~[guava-33.2.0-jre.jar:?]
        at io.airbyte.workers.internal.DefaultAirbyteSource.getExitValue(DefaultAirbyteSource.java:136) ~[io.airbyte-airbyte-commons-worker-0.63.4.jar:?]
        at io.airbyte.workers.general.BufferedReplicationWorker.writeToDestination(BufferedReplicationWorker.java:454) ~[io.airbyte-airbyte-commons-worker-0.63.4.jar:?]
        at io.airbyte.workers.general.BufferedReplicationWorker.lambda$runAsyncWithTimeout$5(BufferedReplicationWorker.java:263) ~[io.airbyte-airbyte-commons-worker-0.63.4.jar:?]
        at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
2024-07-26 12:14:15 platform > writeToDestination: done. (forDest.isDone:true, isDestRunning:true)
2024-07-26 12:14:24 platform > readFromDestination: exception caught
java.lang.IllegalStateException: Destination process is still alive, cannot retrieve exit value.
        at com.google.common.base.Preconditions.checkState(Preconditions.java:515) ~[guava-33.2.0-jre.jar:?]
        at io.airbyte.workers.internal.DefaultAirbyteDestination.getExitValue(DefaultAirbyteDestination.java:210) ~[io.airbyte-airbyte-commons-worker-0.63.4.jar:?]
        at io.airbyte.workers.general.BufferedReplicationWorker.readFromDestination(BufferedReplicationWorker.java:492) ~[io.airbyte-airbyte-commons-worker-0.63.4.jar:?]
        at io.airbyte.workers.general.BufferedReplicationWorker.lambda$runAsync$2(BufferedReplicationWorker.java:235) ~[io.airbyte-airbyte-commons-worker-0.63.4.jar:?]
        at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
2024-07-26 12:14:24 platform > readFromDestination: done. (writeToDestFailed:true, dest.isFinished:false)

[killthekitten]

@samcz1 looks identical to our issue.

CC @davinchia @marcosmarxm @benmoriceau would this mean migrating to abctl won't solve the issue?