[MongoSource] Error reading from MongoDB: Circular reference detected about mage-ai

[wangxiaoyou1993]

@DevBey can you share the traceback or stacktrace?

[DevBey]

Hello @wangxiaoyou1993,

but this is the only log we get for this, apologies but please let me know if I am missing anything

INFO:cold_sea/2/20240418T113235:{"pipeline_run_id": 5, "pipeline_schedule_id": 2, "pipeline_uuid": "cold_sea", "hostname": "d89761f2784e", "level": "INFO", "message": "[MongoSource] Error reading from MongoDB: Circular reference detected", "timestamp": 1713463027.410202, "uuid": "cae3c7f1925a44a49a101c916d5f1a58"}

[wangxiaoyou1993]

@DevBey can you run the pipeline in the editor page and see there's stacktrace?

[DevBey]

apologies but how ?? @wangxiaoyou1993

[wangxiaoyou1993]

@DevBey you should have the button to execute the pipeline. You can cancel editing dependency and you'll see the button.

[DevBey]

yep @wangxiaoyou1993 we tried this same traceback, apart from this just logs of message received and message being inserted into postgres with the mongodb document.

then this error, no other trackback.

[DevBey]

Hello @wangxiaoyou1993,

could you please help with any update on this ??

[DevBey]

Hello @wangxiaoyou1993,

quick update on this, mongdb to postgres streaming does work in case we do some kind of transformation and then pass on it to postgres data exporter.

[wangxiaoyou1993]

Hello @wangxiaoyou1993,

quick update on this, mongdb to postgres streaming does work in case we do some kind of transformation and then pass on it to postgres data exporter.

What transformation did you add?

[DevBey]

nothing much just few custom transforms that we do for data type issues like dates etc.

we figured that mongodb issue would be eliminated if we instead of passing mongodb loader to postgres exporter directly just, do some kind of initial transform and then pass on the message to postgres exporter and it worked.

[wangxiaoyou1993]

nothing much just few custom transforms that we do for data type issues like dates etc.

we figured that mongodb issue would be eliminated if we instead of passing mongodb loader to postgres exporter directly just, do some kind of initial transform and then pass on the message to postgres exporter and it worked.

Thanks for the info! Will test with the dates type.

[DevBey]

apologies but my hunch is that I don't think it's related to any particular datatype. it's a generaly issue when fetching data from mongodb and dumping into Postgres. it works MongoDB to MongoDB