Comments (12)
Once I have this working I should use it to ship a demo of a datasette.yml
file somewhere, so I can then add a link to that demo to the new documentation for:
It should go in this section: https://docs.datasette.io/en/latest/introspection.html#config
from datasette.
As soon as datasette publish cloudrun
has this I can re-enable this bit of the demo deploy:
datasette/.github/workflows/deploy-latest.yml
Lines 91 to 97 in 2da1a6a
Which should fix this broken demo from https://simonwillison.net/2022/Dec/2/datasette-write-api/
from datasette.
The @add_common_publish_arguments_and_options
decorator described here is bad. If I update it to support a new config
option all plugins that use it will break.
datasette/docs/plugin_hooks.rst
Lines 347 to 355 in f130c7c
I want to deprecate it and switch to a different, better design to address the same problem.
from datasette.
I think the actual design of this is pretty simple. Current help starts like this:
Usage: datasette publish cloudrun [OPTIONS] [FILES]...
Publish databases to Datasette running on Cloud Run
Options:
-m, --metadata FILENAME Path to JSON/YAML file containing metadata
to publish
--extra-options TEXT Extra options to pass to datasette serve
The -s
and -c
short options are not being used.
So I think -c/--config
can point to a JSON or YAML datasette.yaml
file, and -s/--setting key value
can mirror the new -s/--setting
option in datasette serve
itself (a shortcut for populating the config file directly from the CLI).
Here's the relevant help section from datasette serve
:
-m, --metadata FILENAME Path to JSON/YAML file containing
license/source metadata
-c, --config FILENAME Path to JSON/YAML Datasette configuration
file
-s, --setting SETTING... nested.key, value setting to use in
Datasette configuration
from datasette.
Here's the full help for Cloud Run at the moment:
datasette publish cloudrun --help
Usage: datasette publish cloudrun [OPTIONS] [FILES]...
Publish databases to Datasette running on Cloud Run
Options:
-m, --metadata FILENAME Path to JSON/YAML file containing metadata
to publish
--extra-options TEXT Extra options to pass to datasette serve
--branch TEXT Install datasette from a GitHub branch e.g.
main
--template-dir DIRECTORY Path to directory containing custom
templates
--plugins-dir DIRECTORY Path to directory containing custom plugins
--static MOUNT:DIRECTORY Serve static files from this directory at
/MOUNT/...
--install TEXT Additional packages (e.g. plugins) to
install
--plugin-secret <TEXT TEXT TEXT>...
Secrets to pass to plugins, e.g. --plugin-
secret datasette-auth-github client_id xxx
--version-note TEXT Additional note to show on /-/versions
--secret TEXT Secret used for signing secure values, such
as signed cookies
--title TEXT Title for metadata
--license TEXT License label for metadata
--license_url TEXT License URL for metadata
--source TEXT Source label for metadata
--source_url TEXT Source URL for metadata
--about TEXT About label for metadata
--about_url TEXT About URL for metadata
-n, --name TEXT Application name to use when building
--service TEXT Cloud Run service to deploy (or over-write)
--spatialite Enable SpatialLite extension
--show-files Output the generated Dockerfile and
metadata.json
--memory TEXT Memory to allocate in Cloud Run, e.g. 1Gi
--cpu [1|2|4] Number of vCPUs to allocate in Cloud Run
--timeout INTEGER Build timeout in seconds
--apt-get-install TEXT Additional packages to apt-get install
--max-instances INTEGER Maximum Cloud Run instances
--min-instances INTEGER Minimum Cloud Run instances
--help Show this message and exit.
from datasette.
I'd really like to remove --extra-options
. I think the new design makes that completely obsolete?
Maybe it doesn't. You still need --extra-options
for the --crossdb
option for example.
from datasette.
https://github.com/search?q=datasette+publish+extra-options+language%3AShell&type=code&l=Shell shows 17 matches, I'll copy in illustrative examples here:
--extra-options="--setting sql_time_limit_ms 5000"
--extra-options="--config default_cache_ttl:3600 --config hash_urls:1"
--extra-options "--setting sql_time_limit_ms 3500 --setting default_page_size 20 --setting trace_debug 1"
--extra-options="--config default_page_size:50 --config sql_time_limit_ms:30000 --config facet_time_limit_ms:10000"
--extra-options="--setting sql_time_limit_ms 5000"
--extra-options "--setting suggest_facets off --setting allow_download on --setting truncate_cells_html 0 --setting max_csv_mb 0 --setting sql_time_limit_ms 2000"
from datasette.
Found more when I searched for YAML.
Here's the most interesting: https://github.com/labordata/warehouse/blob/0029a72fc1ceae9091932da6566f891167179012/.github/workflows/build.yml#L59
--extra-options="--crossdb --setting sql_time_limit_ms 100000 --cors --setting facet_time_limit_ms 500 --setting allow_facet off --setting trace_debug 1"
Uses both --cors
and --crossdb
.
from datasette.
Maybe I should add --cors
and --crossdb
to datasette publish cloudrun
as well?
from datasette.
Worth noting that it already sets --cors
automatically without you needing to specify it:
datasette/datasette/utils/__init__.py
Line 374 in d97e82d
I wonder if that's actually surprising behaviour that we should change before 1.0.
from datasette.
Hi @simonw,
Awesome work with Datasette as always. We talked previously with a colleague about a project we were interested in hosting on Datasette Cloud. While waiting for DS Cloud to be ready, we decided to publish Datasette ourselves first. We were able to deploy the latest version (1.0a13) on Google Cloud Run but I did notice that datasette publish did not support an option for the datasette.yaml/json.
Because of this, our deployed instance lacked features that are available when ran locally. Specifically, the config/options (previously in metadata.yaml) that were moved to datasette.yaml that our instance is lacking are:
- extra_css_urls (i.e. the custom styles aren't being rendered)
- canned queries (i.e. the canned queries aren't being listed w/ the database)
As a test, we published the same database with Datasette stable (0.64.6) using the old format of metadata.yaml file (i.e. contains extra_css_urls and canned queries) and the custom styling and canned queries work as expected. From what I understand, we should be able to use the same metadata.yaml file with version 1.0a13. However, when we tested deploying Datasette latest using the same metadata.yaml file (using the old format), the custom CSS and canned queries did not work but everything else was working fine. We used the same parameters for datasette publish in both cases with the only difference being one used 0.64.6 while the other used 1.0a13.
I wanted to ask if there are any workarounds or solutions that would allow me to specify the aforementioned options (extra_css_urls, canned queries) during the deployment of 1.0a13 using datasette publish (similar to --extra-options perhaps?). Maybe I missed something in the documentation. Or do you think we'd need to deploy datasette manually for this to work?
Lastly, although using the stable version works as intended, we plan to use the 1.0 version because we want to include the datasette-write-ui plugin (w/c requires v 1.0a3) in the deployment. As such, we're also interested to know the status of datasette publish's support for using the datasette.yaml file.
Any help or pointers would be much appreciated.
Thanks!
from datasette.
Hi Simon,
Just an update: we managed to overcome the limitations mentioned above by deploying datasette 1.0a13 on a dedicated VPS (there were other requirements that Cloud Run wasn't able to address such as having mutable databases and using the upload-csvs plugin) but we're still very much interested in seeing and maybe even helping datasette publish eventually support settings in datasette.yaml
from datasette.
Related Issues (20)
- Proposal - store metadata inside `internal.db` tables HOT 2
- Broken link in documention: fivethirtyeight.datasettes.com
- Fix font size on filter inputs
- base_url getting appended twice in redirects when applying filters? HOT 3
- Accessibility: add a `lang` attribute to `html` HOT 1
- What minimal SQLite version should Datasette support? HOT 9
- Remove upserts in `set_XXX_metadata()` methods
- PyOdide test failure HOT 7
- Canned queries with named parameters fail with error against SQLite 3.46.0 HOT 11
- derive_named_parameters() method that works with latest SQLite HOT 5
- Flaky test_max_csv_mb test HOT 3
- Very weird flaky test_create_table_ignore_replace and test_upsert tests HOT 18
- multiple plugins extending the same base template?
- Consider using isolation_level="IMMEDIATE" for write connections HOT 5
- Database/Table/Row not found errors echo back text from URL HOT 7
- Proposal — Datasette JSON API changes for 1.0 HOT 4
- Block or rate limit based on User Agent? HOT 3
- Datasette 1.0 Canned Queries Broken HOT 3
- [Error] no such table: main.events HOT 1
- Cannot run datasette on 3.13 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from datasette.