Comments (7)
Some options for where this could go:
- Directly in the release notes? I'm not sure about that, those are getting pretty long already. I think the release notes should link to relevant upgrade guides.
- On a new page? We could have a "upgrade instructions" page in the documentation.
- At the bottom of the new https://docs.datasette.io/en/latest/configuration.html page
I'm leaning towards the third option at the moment.
But... we may also need to provide upgrade instructions for plugin authors. Those could live in a separate area of the documentation though, since issues affecting end-users who configure Datasette and issues affecting plugin authors are unlikely to overlap much.
from datasette.
Related idea: how about a datasette-upgrade
plugin which adds a datasette upgrade
command that can be used to automate this process?
Maybe something like this:
datasette install datasette-upgrade
datasette upgrade metadata-to-config metadata.json
This would output two new files: metadata.yaml
and datasette.yaml
. If files with those names existed already in the current directory they would be called metadata-new.yaml
and datasette-new.yaml
.
The command would tell you what it did:
Your metadata.json file has been rewritten as two files:
metadata-new.yaml
datasette.yaml
Start Datasette like this to try them out:
datasette -m metadata-new.yaml -c datasette.yaml
The command is datasette upgrade metadata-to-config
because metadata-to-config
is the name of the upgrade recipe. The first version of the plugin would only have that single recipe, but we could add more recipes in the future for other upgrades.
from datasette.
I dig it - I was thinking an Observable notebook where you paste your metadata.json
/metadata.yaml
and it would generate the new metadata + datasette.yaml files, but an extensible datasette upgrade
plugin would be nice for future plugins.
One thing to think about: If someone has comments in their original metadata.yaml
, could we preserve them in the new files? tbh maybe not too important bc if people cared that much they could just copy + paste, and it might be too distracting
from datasette.
I think I'm OK with not preserving comments, just because it adds a level of complexity to the tool which I don't think is worth the value it provides.
If people want to keep their comments I'm happy to leave them to copy those over by hand.
from datasette.
Started playing with this plugin idea, now tearing myself away to work on something more important:
from datasette import hookimpl
import click
import pathlib
@hookimpl
def register_commands(cli):
@cli.group()
def upgrade():
"""
Apply configuration upgrades to an existing Datasette instance
"""
pass
@upgrade.command()
@click.argument(
"metadata", type=click.Path(exists=True)
)
@click.option(
"new_metadata", "-m", "--new-metadata", help="Path to new metadata.yaml file", type=click.Path(exists=False)
)
@click.option(
"new_datasette", "-c", "--new-datasette", help="Path to new datasette.yaml file", type=click.Path(exists=False)
)
@click.option(
"output_dir", "-e", "--output-dir", help="Directory to write new files to", type=click.Path(), default="."
)
def metadata_to_config(metadata, new_metadata, new_datasette, output_dir):
"""
Upgrade an existing metadata.json/yaml file to the new metadata.yaml and
datasette.yaml split introduced prior to Datasette 1.0.
"""
print("Upgrading {} to new metadata.yaml format".format(metadata))
output_dir = pathlib.Path(output_dir)
if not new_metadata:
# Pick a filename for the new metadata.yaml file that does not yet exist
new_metadata = pick_filename("metadata", output_dir)
if not new_datasette:
new_datasette = pick_filename("datasette", output_dir)
print("New metadata.yaml file will be written to {}".format(new_metadata))
print("New datasette.yaml file will be written to {}".format(new_datasette))
def pick_filename(base, output_dir):
options = ["{}.yaml".format(base), "{}-new.yaml".format(base)]
i = 0
while True:
option = options.pop(0)
option_path = output_dir / option
if not option_path.exists():
return option_path
# If we ran out
if not options:
i += 1
options = ["{}-new-{}.yaml".format(base, i)]
from datasette.
Demo of that logic:
$ datasette upgrade metadata-to-config ../datasette/metadata.json
Upgrading ../datasette/metadata.json to new metadata.yaml format
New metadata.yaml file will be written to metadata-new-1.yaml
New datasette.yaml file will be written to datasette.yaml
$ touch metadata-new-1.yaml
$ datasette upgrade metadata-to-config ../datasette/metadata.json
Upgrading ../datasette/metadata.json to new metadata.yaml format
New metadata.yaml file will be written to metadata-new-2.yaml
New datasette.yaml file will be written to datasette.yaml
$ touch datasette.yaml
$ datasette upgrade metadata-to-config ../datasette/metadata.json
Upgrading ../datasette/metadata.json to new metadata.yaml format
New metadata.yaml file will be written to metadata-new-2.yaml
New datasette.yaml file will be written to datasette-new.yaml
from datasette.
Pushed that incomplete code here: https://github.com/datasette/datasette-upgrade
from datasette.
Related Issues (20)
- Mechanism for executing "isolated" database operations without prepared connection stuff HOT 4
- Cog is failing with round_trip_load_all() YAML error HOT 3
- datasette install datasette-comments break search-all
- jinja2_environment_from_request plugin hook HOT 8
- Support for duckdb HOT 1
- Datasette docker errors when db schema uses "strict"
- Tests and docs to guarantee extras used by datasette-search-all
- ?_extra=columns parameter causes all rows to return `"column-name": "column-name"` HOT 6
- black leaves extra whitespace in my rendered documentation HOT 2
- Show how to register a plugin in a test fixture HOT 1
- Empty list in plugin introspection demo
- After releasing 1.0a8 update projects that test against a commit hash
- Upgrade to most recent Black
- track_event() mechanism for analytics HOT 8
- 1.0a8 release notes and upgrade instructions HOT 7
- Upgrade Datasette Publish to work with 1.0a8 alpha HOT 3
- Fill out the TODO sections on the configuration docs page HOT 1
- Table configuration settings should work in both metadata.yml and datasette.yml HOT 12
- Treat "plugins" in metadata as if it was "plugins" in configuration on startup HOT 3
- Treat allow blocks in metadata as if they were in config HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from datasette.