bug: public discovery.yml is throwing 404 about meltano

[edgarrmondragon]

@dsuarez692 Looking into it.

In the meantime, are there any blockers for you to migrate to Meltano v3¹?

Footnotes

1. https://docs.meltano.com/guide/v3-migration/ ↩

[edgarrmondragon]

@dsuarez692 in the meantime, can you try setting the environment variable MELTANO_DISCOVERY_URL='https://raw.githubusercontent.com/meltano/meltano/v2.19.0/src/meltano/core/bundle/discovery.yml'?

[dsuarez692]

I was able to solve it, but sadly I ended up updating the Meltano code 😞

Inside of .meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/tap_airbyte/tap.py when the code prepares the command to call Docker, the line 372 ("--",) is breaking the call. I removed it and it worked again

[dsuarez692]

@dsuarez692 in the meantime, can you try setting the environment variable MELTANO_DISCOVERY_URL='https://raw.githubusercontent.com/meltano/meltano/v2.19.0/src/meltano/core/bundle/discovery.yml'?

Same error message 😢

[edgarrmondragon]

@dsuarez692 in the meantime, can you try setting the environment variable MELTANO_DISCOVERY_URL='https://raw.githubusercontent.com/meltano/meltano/v2.19.0/src/meltano/core/bundle/discovery.yml'?

Same error message 😢

@dsuarez692 I don't know what your environment and workflow looks like so I need to ask: what command did you run and did you make sure that it ran with that environment variable set?

[dsuarez692]

@dsuarez692 in the meantime, can you try setting the environment variable MELTANO_DISCOVERY_URL='https://raw.githubusercontent.com/meltano/meltano/v2.19.0/src/meltano/core/bundle/discovery.yml'?

Same error message 😢

@dsuarez692 I don't know what your environment and workflow looks like so I need to ask: what command did you run and did you make sure that it ran with that environment variable set?

I was able to reproduce it locally. I've created a new Meltano project with tap-postgres-airbyte as the extractor and target-jsonl as the loader. I have a example database running on my local and when I execute

export MELTANO_DISCOVERY_URL='https://raw.githubusercontent.com/meltano/meltano/v2.19.0/src/meltano/core/bundle/discovery.yml'
meltano run tap-postgres target-jsonl

I get the same error

2024-04-05T21:21:32.208010Z [error    ] Block run completed.           block_type=ExtractLoadBlocks err=RunnerError('Cannot start plugin tap-postgres: Catalog discovery failed: command [\'/Users/dsuarez/Documents/Code/my-meltano-project/.meltano/extractors/tap-postgres/venv/bin/tap-airbyte\', \'--config\', \'/Users/dsuarez/Documents/Code/my-meltano-project/.meltano/run/tap-postgres/tap.ebc47f1c-53ff-476e-84ba-5e1370a1857e.config.json\', \'--discover\'] returned 1 with stderr:\n 2024-04-05 18:21:31,673 | INFO     | tap-airbyte          | Checking for docker on PATH.\n2024-04-05 18:21:31,673 | INFO     | tap-airbyte          | Found docker on PATH.\n2024-04-05 18:21:31,673 | INFO     | tap-airbyte          | Checking docker version.\n2024-04-05 18:21:31,756 | INFO     | tap-airbyte          | Successfully executed docker version.\nTraceback (most recent call last):\n  File "/Users/dsuarez/Documents/Code/my-meltano-project/.meltano/extractors/tap-postgres/venv/bin/tap-airbyte", line 8, in <module>\n    sys.exit(TapAirbyte.cli())\n  File "/Users/dsuarez/Documents/Code/my-meltano-project/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/click/core.py", line 1157, in __call__\n    return self.main(*args, **kwargs)\n  File "/Users/dsuarez/Documents/Code/my-meltano-project/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/click/core.py", line 1078, in main\n    rv = self.invoke(ctx)\n  File "/Users/dsuarez/Documents/Code/my-meltano-project/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/click/core.py", line 1434, in invoke\n    return ctx.invoke(self.callback, **ctx.params)\n  File "/Users/dsuarez/Documents/Code/my-meltano-project/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/click/core.py", line 783, in invoke\n    return __callback(*args, **kwargs)\n  File "/Users/dsuarez/Documents/Code/my-meltano-project/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/tap_airbyte/tap.py", line 268, in cli\n    tap: TapAirbyte = cls(  # type: ignore\n  File "/Users/dsuarez/Documents/Code/my-meltano-project/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 110, in __init__\n    self.setup_mapper()\n  File "/Users/dsuarez/Documents/Code/my-meltano-project/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 180, in setup_mapper\n    self.mapper.register_raw_streams_from_catalog(self.catalog)\n  File "/Users/dsuarez/Documents/Code/my-meltano-project/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 173, in catalog\n    self._catalog = self.input_catalog or self._singer_catalog\n  File "/Users/dsuarez/Documents/Code/my-meltano-project/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 326, in _singer_catalog\n    for stream in self.streams.values()\n  File "/Users/dsuarez/Documents/Code/my-meltano-project/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 135, in streams\n    for stream in self.load_streams():\n  File "/Users/dsuarez/Documents/Code/my-meltano-project/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 359, in load_streams\n    for stream in self.discover_streams():\n  File "/Users/dsuarez/Documents/Code/my-meltano-project/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/tap_airbyte/tap.py", line 769, in discover_streams\n    for stream in self.airbyte_catalog["streams"]:\n  File "/Users/dsuarez/Documents/Code/my-meltano-project/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/tap_airbyte/tap.py", line 654, in airbyte_catalog\n    raise AirbyteException(\ntap_airbyte.tap.AirbyteException: Discover failed with return code 1: Unknown option: discover\n\n') exit_codes={} set_number=0 success=False
Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps, or to
join our friendly Slack community.

Run invocation could not be completed as block failed: Cannot start plugin tap-postgres: Catalog discovery failed: command ['/Users/dsuarez/Documents/Code/my-meltano-project/.meltano/extractors/tap-postgres/venv/bin/tap-airbyte', '--config', '/Users/dsuarez/Documents/Code/my-meltano-project/.meltano/run/tap-postgres/tap.ebc47f1c-53ff-476e-84ba-5e1370a1857e.config.json', '--discover'] returned 1 with stderr:
 2024-04-05 18:21:31,673 | INFO     | tap-airbyte          | Checking for docker on PATH.
2024-04-05 18:21:31,673 | INFO     | tap-airbyte          | Found docker on PATH.
2024-04-05 18:21:31,673 | INFO     | tap-airbyte          | Checking docker version.
2024-04-05 18:21:31,756 | INFO     | tap-airbyte          | Successfully executed docker version.
Traceback (most recent call last):
  File "/Users/dsuarez/Documents/Code/my-meltano-project/.meltano/extractors/tap-postgres/venv/bin/tap-airbyte", line 8, in <module>
    sys.exit(TapAirbyte.cli())
  File "/Users/dsuarez/Documents/Code/my-meltano-project/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/Users/dsuarez/Documents/Code/my-meltano-project/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/Users/dsuarez/Documents/Code/my-meltano-project/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/dsuarez/Documents/Code/my-meltano-project/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/Users/dsuarez/Documents/Code/my-meltano-project/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/tap_airbyte/tap.py", line 268, in cli
    tap: TapAirbyte = cls(  # type: ignore
  File "/Users/dsuarez/Documents/Code/my-meltano-project/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 110, in __init__
    self.setup_mapper()
  File "/Users/dsuarez/Documents/Code/my-meltano-project/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 180, in setup_mapper
    self.mapper.register_raw_streams_from_catalog(self.catalog)
  File "/Users/dsuarez/Documents/Code/my-meltano-project/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 173, in catalog
    self._catalog = self.input_catalog or self._singer_catalog
  File "/Users/dsuarez/Documents/Code/my-meltano-project/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 326, in _singer_catalog
    for stream in self.streams.values()
  File "/Users/dsuarez/Documents/Code/my-meltano-project/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 135, in streams
    for stream in self.load_streams():
  File "/Users/dsuarez/Documents/Code/my-meltano-project/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 359, in load_streams
    for stream in self.discover_streams():
  File "/Users/dsuarez/Documents/Code/my-meltano-project/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/tap_airbyte/tap.py", line 769, in discover_streams
    for stream in self.airbyte_catalog["streams"]:
  File "/Users/dsuarez/Documents/Code/my-meltano-project/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/tap_airbyte/tap.py", line 654, in airbyte_catalog
    raise AirbyteException(
tap_airbyte.tap.AirbyteException: Discover failed with return code 1: Unknown option: discover

[edgarrmondragon]

Ah, that's a different error though! I have to run but I'd try tweaking the capabilities of the tap in meltano.yml. Maybe it doesn't support the --discover option?

[dsuarez692]

The documentation says that it supports the discover capability, and it was working until two weeks ago. If I remove the discover from the meltano.yml file I get this:

2024-04-05T21:36:35.707578Z [error    ] Cannot start plugin tap-postgres: Applying catalog rules failed: catalog file is missing.
2024-04-05T21:36:35.707861Z [error    ] Block run completed.           block_type=ExtractLoadBlocks err=RunnerError('Cannot start plugin tap-postgres: Applying catalog rules failed: catalog file is missing.') exit_codes={} set_number=0 success=False
Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps, or to
join our friendly Slack community.

Run invocation could not be completed as block failed: Cannot start plugin tap-postgres: Applying catalog rules failed: catalog file is missing.

[dsuarez692]

I dug into the code and I found that the line that fails is when the plugin starts a docker image airbyte/source-postgres:3.3.23, latest to execute the command discover. At that moment, that line throws the error Unknown option: discover
Did something change recently on that Docker image?

[edgarrmondragon]

I dug into the code and I found that the line that fails is when the plugin starts a docker image airbyte/source-postgres:3.3.23, latest to execute the command discover. At that moment, that line throws the error Unknown option: discover Did something change recently on that Docker image?

Can't see anything obvious in https://github.com/airbytehq/airbyte/commits/master/airbyte-integrations/connectors/source-postgres that would indicate that 😕

[WillDaSilva]

Thank you for this information @dsuarez692.

This commit seems responsible for adding "--" to the command: MeltanoLabs/tap-airbyte-wrapper@e1349aa

I don't understand why the addition of "--" would break the command in this way.

[edgarrmondragon]

Thank you for this information @dsuarez692.

This commit seems responsible for adding "--" to the command: MeltanoLabs/tap-airbyte-wrapper@e1349aa

I don't understand why the addition of "--" would break the command in this way.

Yeah, I wonder if it's a problem caused by having an older version of docker.