"Test DATASETTE_LOAD_PLUGINS" test shows errors but did not fail the CI run about datasette

[simonw]

Here's the failure in CI, which did not cause the workflow to fail even though it should have:

[simonw]

The script:

datasette/tests/test-datasette-load-plugins.sh

Lines 1 to 29 in b0d0a0e

	#!/bin/bash
	# This should only run in environemnts where both
	# datasette-init and datasette-json-html are installed
	PLUGINS=$(datasette plugins)
	echo "$PLUGINS" | jq 'any(.[]; .name == "datasette-json-html")' | \
	grep -q true || ( \
	echo "Test failed: datasette-json-html not found" && \
	exit 1 \
	)
	# With the DATASETTE_LOAD_PLUGINS we should not see that
	PLUGINS2=$(DATASETTE_LOAD_PLUGINS=datasette-init datasette plugins)
	echo "$PLUGINS2" | jq 'any(.[]; .name == "datasette-json-html")' | \
	grep -q false || ( \
	echo "Test failed: datasette-json-html should not have been loaded" && \
	exit 1 \
	)
	echo "$PLUGINS2" | jq 'any(.[]; .name == "datasette-init")' | \
	grep -q true || ( \
	echo "Test failed: datasette-init should have been loaded" && \
	exit 1 \
	)
	# With DATASETTE_LOAD_PLUGINS='' we should see no plugins
	PLUGINS3=$(DATASETTE_LOAD_PLUGINS='' datasette plugins)
	echo "$PLUGINS3"| \
	grep -q '\[\]' || ( \
	echo "Test failed: datasette plugins should have returned []" && \
	exit 1 \
	)

I'm not sure why those exit 1 lines did not cause a CI failure.

[simonw]

GPT-4 says:

In the script, you're using a subshell ( ... ) to group commands. If you exit 1 within the subshell, it will only exit the subshell and not the main script. This is why GitHub Actions does not see it as a failure.

It suggested doing this instead:

#!/bin/bash

PLUGINS=$(datasette plugins)
if ! echo "$PLUGINS" | jq 'any(.[]; .name == "datasette-json-html")' | grep -q true; then
  echo "Test failed: datasette-json-html not found"
  exit 1
fi

PLUGINS2=$(DATASETTE_LOAD_PLUGINS=datasette-init datasette plugins)
if ! echo "$PLUGINS2" | jq 'any(.[]; .name == "datasette-json-html")' | grep -q false; then
  echo "Test failed: datasette-json-html should not have been loaded"
  exit 1
fi

if ! echo "$PLUGINS2" | jq 'any(.[]; .name == "datasette-init")' | grep -q true; then
  echo "Test failed: datasette-init should have been loaded"
  exit 1
fi

PLUGINS3=$(DATASETTE_LOAD_PLUGINS='' datasette plugins)
if ! echo "$PLUGINS3" | grep -q '\[\]'; then
  echo "Test failed: datasette plugins should have returned []"
  exit 1
fi

[simonw]

Yes, the new script seems to work. On Python 3.11:

tests/test-datasette-load-plugins.sh 
echo $?
0

On Python 3.8:

tests/test-datasette-load-plugins.sh
Test failed: datasette-json-html not found
echo $?
1

[simonw]

https://github.com/simonw/datasette/actions/runs/6267146158/job/17019594849 failed on 3.9 this time.

    plugin_info["name"] = distinfo.name or distinfo.project_name
AttributeError: 'PathDistribution' object has no attribute 'name'
Test failed: datasette-json-html should not have been loaded

[simonw]

The test fails as expected now. Closing this issue, will solve the remaining problems in:

• #2057