metagov / gateway Goto Github PK
View Code? Open in Web Editor NEWAn API gateway for online communities
Home Page: https://docs.metagov.org/
License: MIT License
An API gateway for online communities
Home Page: https://docs.metagov.org/
License: MIT License
Hi @metagov, we're thinking of moving the metagov repo to an organization, and were curious if you would consider donating your Github handle (github.com/metagov) to the project. We're nice people running a nonprofit research project, and we'd really appreciate it!
Let me know if you have any questions :)
Thanks,
Josh
Each Plugin has a one-to-one relationship with a state
model, which is just a datastore that can be used to store any data for the life of the plugin. (Life of the plugin = from when it's activated for the community, until it's deactivated or the community is deleted). I originally added this because I wanted to store data that was fetched on initialization so that it can be accessed later on. I was thinking of state
as something that can be re-created on plugin init.
HOWEVER, I realized I could just use the state
to support the web monetization revshare use-case. Here's how: https://github.com/metagov/metagov-prototype/blob/2eb8931212ed705fff3de382d1f8943d700530f4/metagov/metagov/plugins/webmonetization/models.py
Instead of the action
s performing actions on an external platform, the action
s are updating the state of the plugin itself. Instead of the resource
functions getting data from an external platform, the resource
functions are returning data from the plugin state.
I think this is a fine approach to use for small structured documents. I don't think we need to go beyond this right now. If we see it being repeatedly used in a common way, maybe we'll want to add an abstraction specifically for creating/updating/deleting structured documents.
Q&A about using plugin state to store "documents":
1. Can the Driver govern changes to the document?
Yes. In the revshare example, the document is changed by invoking a metagov action (eg POST /api/internal/action/webmonetization.remove-pointer
). Only the Driver is able to invoke that request, so, naturally the Driver can "govern" that action however it wants to.
2. Can the Driver govern access to the document?
Same as (1) –– the Driver is the only thing that can access the /resource
endpoint, so yes. Unless...
3. Can we allow public access to the document?
This is really a separate issue. As mentioned on #15 we are considering making all /resource
endpoints totally public, or allowing plugin authors to declare whether resources are public (anyone on internet) or private (driver only).
Currently we destroy and re-create the plugin (and its state
) whenever the plugin config
changes (code). Instead of deleting and recreating plugin instances, we should have plugin authors implement apply_config
or some pre-save hook that lets them choose what to do when the config changes. Likewise, we should just mark the plugin instance as inactive
when a community deactivates it, instead of deleting the instance.
The prototype should integrate with an external PB tool. Evaluate the below options and decide which one to use.
https://github.com/decidim/decidim
https://github.com/openbudgets/participatory-budgeting
https://github.com/consul/consul
https://github.com/StanfordCDT/pb
https://github.com/socialappslab/appcivist
As Miriam wrote in policykit/policykit#452:
In order to change the repos you have access to, you currently need to delete and re-enable the integration in PolicyKit, which is not ideal.
We should add a method to the Github plugin to dynamically change which repos are covered, or some other approach to smooth out this experience.
Issue for adding (and restructuring) Metagov installation docs to include a section about Django drivers that use the gateway as django app.
There are 3 ways to use Metagov:
For (1), that section should include:
The identity system is working okay for linking, but our system of storing metadata about the links (quality, type) doesn't work as designed. Essentially, the issue is that we're storing data on individual linked accounts (we say, for instance, that a Github oauthed account is of type "oauth" and quality "high") but users will typically want to assess the quality of a link between two accounts ("only do X for a Slack user if we're sure they're the same as the corresponding Github user".)
This is low priority to fix until we have a clear use case, so feel free to share potential use cases (or other thoughts) in the comments.
The end-user should be able to use this widget to build a list of all the platforms and services that they use. In particular, it will list which platforms & services (1) have Metagov plugins and (2) are already integrated into Metagov. The list should be exportable.
Use Hypha's list as the target use-case: https://hackmd.io/@patcon/HyH0W9bAE
This issue refers to this code comment. The installation process requires the installer to be an admin, because PolicyKit requires it. We're only able to check if the installing user is an admin after they perform the install, at which point the app is already installed from Slack's point of view. As you can see in the linked code, we raise an error if the installing user is not an admin, or if we fail to get their user info. At that point we should also uninstall the app using apps.uninstall because we are failing the installation.
When the configuration of an active plugin changes, metagov-core deletes the plugin instance and creates a brand new one. This means that all the plugin state
is lost. This might not be desirable for some plugins that use state to persist data that is updated via actions (see the "revshare" plugin).
Instead of deleting and recreating plugins, expose a handler that lets the plugin author decide what should happen when the config is changed.
Related to discussion #16.
Relevant code in core:
https://github.com/metagov/metagov-prototype/blob/70e8b4752e82b1bfc11dd3274071acd5734ce94c/metagov/metagov/core/serializers.py#L30-L34
In combination with #19, we can support Drivers that want to communicate exclusively via hooks. Right now some GovernanceProcesses require the Driver to be able to poll Metagov for changes to GovernanceProcesses, which means the Driver needs to have some scheduler.
Plugin for https://github.com/StanfordCDT/pb
Stub for issue to decide whether / when / how we need to include an IFTTT or Zapier integration into Metagov.
Rough summary of current conversation: it seems like low-hanging fruit that could deliver a lot of functionality, but we would need to first spend time figuring out how it would interact with / be governed by other Metagov services.
Tl;dr: we should see if Solid or some other service in Web3 has implemented a minimal, "claims-based" identity metasystem, and evaluate whether it's possible to import that metasystem into Metagov along with some basic support for building plugins and policies using the units of that metasystem.
Parts of this post are
Problem
Linked identity is a clear prerequisite to a range of possible governance policies that communities may wish to adopt. Linked identity is also necessary for certain Metagov services to function. For example:
This is clear value in having linked or shared digital identities for users as well as for other entities such as organizations, roles, and resources.
Direct solutions (copied from @mashton's docs)
Linked identity can be accomplished directly through various forms of account linking; account linking is when user accounts from various identity providers are associated with the same user profile. In the above example, the record for OpenCollective user “joshua-tan” would be linked to the record for Discourse user “joshua.” There would be some core user profile record that lists all the accounts that belong to this user. That profile record could be curated
We got as far as the above before deciding that identity was out-of-scope for phase 1 of the Metagov prototype.
Proposal: an identity metasystem
I hypothesize that Metagov should not be in the business of providing a shared identity layer for its plugins, nor should it be in the business of determining the identity policies of particular platforms and communities.
Therefore, instead of an explicit identity layer, I propose that we implement (or import) a minimum viable identity metasystem (MVIM) for Metagov. Such a metasystem should expose a unified interface that allows services to "loosely couple" to a set of 3rd-party identity management systems. By implementing an identity metasystem as opposed to an identity system, we maintain Metagov's status as a backend service for governance authors.
Background and Definitions
This is clear value in having linked or shared digital identities for users as well as for other entities such as organizations, roles, and resources. Indeed, corporations spend billions on a generalization of this problem every year, called "[entity resolution[(https://www.sciencedirect.com/topics/computer-science/entity-resolution)". But the problem of providing a linked, much less unified, identity layer is extremely difficult. Kim Cameron has an excellent summary of why:
Why is it so hard to create an identity layer for the Internet? Mainly because there is little agreement on what it should be and how it should be run. This lack of agreement arises because digital identity is related to context, and the Internet, while being a single technical framework, is experienced through a thousand kinds of content in at least as many different contexts, all of which flourish on top of that underlying framework. The players involved in any one of these contexts want to control digital identity as it impacts them, in many cases wanting to prevent spillover from their context to any other.
Instead of an identity layer, Cameron argues that we need a identity metasystem:
The technology of “device drivers” enabled interchangeable hardware to be plugged in as required. [...] Digital identity requires a similar approach. We need a unifying identity metasystem that can protect applications from the internal complexities of specific implementations and allow digital identity to become loosely coupled. This metasystem is in effect a system of systems that exposes a unified interface much like a device driver or network socket does. That allows one-offs to evolve towards standardized technologies that work within a metasystem framework without requiring the whole world to agree a priori. [...]
She further argues that the concept of a "claim" is central to such an identity metasystem and "encompasses all the known digital identity systems and therefore allows us to begin to unify the rational elements of our patchwork conceptually. It allows us to define digital identity for a metasystem embracing multiple implementations and ways of doing things." To Cameron,
A claim is: “An assertion of the truth of something, typically one which is disputed or in doubt.” Some examples of claims in the digital realm will likely help:
- A claim could just convey an identifier: for example, that the subject's student number is 490-525, or that the subject's Windows name is REDMOND\kcameron. This is the way many existing identity systems work.
- Another claim might assert that a subject knows a given key; and should be able to demonstrate this fact.
- A set of claims might convey personally identifying information; name, address, date of birth and citizenship, for example.
- A claim might simply propose that a subject is part of a certain group; for example, that she has an age less than 16.
- And a claim might state that a subject has a certain capability; for example, to place orders up to a certain limit, or modify a given file.
Note that, contrary to the problem setting above described by Cameron, Metagov does not need an identity layer for the whole Internet. It just needs an "identity open set" that covers the services needed for a single community. (Indeed the problem seems intractable at the scale of the Internet: Cameron posed her idea of an identity metasystem in 2006.)
Next steps
It's my belief that some kind of claims-based identity metasystem has almost certainly been implemented somewhere, i.e. the claims-based approach sounds vaguely familiar to docs I've read for things like Solid, DID, and self-sovereign identity. So the next step is to do some research / ask a knowledgeable expert who can point us in the right direction. I'll update this issue as we collect more information.
Currently you need to update the whole Community config each time with PUT, which could lead to accidentally disabling other plugins (and requires the caller to have the configs-- incl API keys-- for all plugins). Add endpoint(s) for enabling/disabling individual plugins.
This issue is for making the Metagov webhook functionality more useful to the driver.
DRIVER_EVENT_RECEIVER_URL
in the metagov env, and ALL events go to that one Driver endpoint.some considerations:
Note: this issue has been closed and the remaining items moved to separate tickets:
#181
#183
#184
Tracking feature requests to the slack.emoji-vote
process:
channel
and users
are provided, start the vote in a public channel but only allow the specified uses ("eligible voters") to vote. When ineligible users attempt to cast votes, remove their emoji-react and show an ephemeral message explaining why. #122disallowed_participants: str[]
for list of voters that are not allowed to vote. The disallowed voters have their emoji-reactions removed. (Used by PK to exclude "proposer")allowed_participants: str[]
for list of voters that are allowed to vote. This could be performed as a group message or in a channel.poll_type: boolean|choice
and min: int
and max: int
to specify the min and max vote casts permitted for choice votes.private: bool
for performing private vote as direct messages to each participant.close
. Currently it edits the existing message to display the final vote count next to each option. Screenshot of current behavior:1000 USDC split between the top 10 projects (100 each)
Every DAO has a wallet and some form of voting, but most governance discussions and decisions still happen outside the DAO, on other applications and platforms. The Metagov Gateway is an open-source API gateway that makes it easy for developers to access multiple governance applications and build governance systems for communities that exist across multiple technology stacks. It also ships with a powerful governance authoring tool, PolicyKit, for authoring Web2 governance policies.
These are some of the applications and platforms currently integrated with the Metagov Gateway that you can use in a given DAO’s governance:
We want you to use the Metagov Gateway in your DAO or DAO tool.
The top 10 projects by these measures will win 100 USDC each.
Add endpoint(s) for Driver's to access the available actions, processes, and events for a given community. Expose the jsonschemas if available.
These can then be used by PolicyKit to add hints to the policy editor –– like listing available actions + processes, along with their input and output types. PolicyKit could also use the jsonschemas to generate forms for no-code policy authoring.
1000 USDC split between the top 10 projects (100 each)
Every DAO has a wallet and some form of voting, but most governance discussions and decisions still happen outside the DAO, on other applications and platforms. The Metagov Gateway is an open-source API gateway that makes it easy for developers to access multiple governance applications and build governance systems for communities that exist across multiple technology stacks. It also ships with a powerful governance authoring tool, PolicyKit, for authoring Web2 governance policies. In this bounty, we want to see how and whether it can be use in Web3 governance.
These are some of the applications and platforms currently integrated with the Metagov Gateway that you can use in a given DAO’s governance:
We want you to use the Metagov Gateway in your DAO or DAO tool.
The top 10 projects by these measures will win 100 USDC each.
Zoom has APIs that support creating polls and reading poll results, seeing who attending meetings, chat bots, webhooks, etc: https://marketplace.zoom.us/docs/api-reference/zoom-api
For instance, in the Slack plugin we just re-use Slack's name for the community identifier, team id
, but we want something standard across all plugins.
Currently in the identity pr #119 I use community_platform_id
but we could choose a better or less verbose name.
When installing a policy, adding a bot, or installing an application, it would be nice to have a staging or sandbox feature in order to preview the effects of that action without necessarily finalizing it to the community. Kind of like a "preview" button one sees in blog and forum software, with some sort of easy-to-use revert / rollback functionality. Or even some minimum security assurance.
Reasoning: users have a bunch of qualms with making governance changes, making them slow. We want to address as many of those qualms as possible, in order to make uptake faster. E.g. in the DADA Discord server, EdgarAllanPoe writes, "We have been playing with a MEE6 bot in out moderator server. Almost ready to try moving him over here but very scared to deploy him in case we mess up by missing a setting somewhere and people have access to commands they shouldn't while we're setting him up. We were thinking to maybe bring him in and immediately restrict his access to only a hidden channel so that nobody can see him. Maybe you could give us some advice on how bots are normally deployed in already active servers."
Not sure if something like this is already on PolicyKit's priority list. What do you think @amyxzhang?
Use cases should incorporate Loomio, Open Collective, Open Monetization, PolicyKit, and Participatory Budgeting. These use cases should be based on what is possible with the APIs/webhooks that are currently exposed by each tool. At the end of phase 1, all these use cases should be implemented.
Will require more research into some tools, especially Web Monetization/Coil and PB (#3).
Acceptance Criteria
The goals with this are:
If channel
and users
are provided, start the vote in a public channel but only allow the specified uses ("eligible voters") to vote. When ineligible users attempt to cast votes, remove their emoji-react and show an ephemeral message explaining why
The caller should be allowed to provide users
as a list of slack platform identifiers OR as list of metagov IDs. Pull this param out into a special param that is preprocessed by core, maybe.
So far, all platforms that we integrate with via plugins have the following structure:
This issue is for supporting plugins for platforms like Slack and Discord where you can create a single bot (or app) that is distributed across many organizations/servers/communities. At minimum, this requires supporting...
This goes hand-in-hand with #52
action
for sending an email. This is for the Agreement Engine https://github.com/metagov/agreements-prototype.This is a meta-issue for the Big™️ things that we've considered adding to the Metagov Prototype, but were deferred from the initial prototype phase (Jan-June 2021).
Ideas for how to make Metagov Prototype a more useful tool. Hypothesis is that these enhancements will increase chances of adoption. We're early in the learning phase!
These are split into two categories: enhancements that are driven by "Driver DX" and enhancements that are driven by "Plugin DX."
"Driver DX" is the developer experience of creating a Driver. It concerns the interface and functionality exposed to the Driver from Metagov (the leftmost arrow), also known as the "Metagov API."
"Plugin DX" is the developer experience of creating a Plugin. It concerns the interface and functionality exposed to the Plugin author from metagov-core.
The "Driver Developer" is creating a driver that uses Metagov, or updating an existing piece of governance software to use Metagov (ex PolicyKit, Kybern). This person is (1) hitting the Metagov API from the Driver code, (2) deploying Metagov, (3) deploying updates to Metagov (ie migrating to latest version of metagov-prototype which could include plugin changes, new plugins, or core changes).
/community
endpoints). Right now there is not enough information to create a robust community mgmt UI on the Driver side. Metagov should expose plugin config schemas.The "Plugin Developer" is authoring a Plugin to connect to some platform(s) or implement some governance process(es). They are running Metagov locally in order to develop their plugin in this repo under metagov-prototype/metagov/metagov/plugins
.
The "community admin" is the member of a community who sets up some part of the governance infrastructure. This UX will vary widely depending on which Driver is used. If PolicyKit is used, the "community admin" is going to policykit.org and installing PolicyKit to their community, and configuring Metagov through the PolicyKit settings screen.
Ideas for making Metagov "production-ready."
use discussion_id
https://www.loomio.org/help/api?api_key=xx
This post isn't an architecture question exactly, so it may belong on Discourse rather than GitHub issues. I'll post it here for now as a test-case of our new documenting-decisions-on-GitHub policy.
We've been planning since January to work with DADA.art as our initial use-case, to be deployed in May; you can read more in the DADA + Metagov project plan. However, the main use-case that we've obtained from DADA so far, "Activating a community through task applets", can be implemented through IFTTT workflows without using Metagov at all. Metagov may at some point want to add an IFTTT integration, but that will probably not happen by May.
My sense right now is that we need to work more closely with Bea and Abraham @ DADA to spec out a use-case that directly involves Metagov or an associated service. I think it would also be valuable to write a use-case involving DADA that explores how Metagov can/cannot engage with the normative+cultural facets of governance.
Right now (March 22), we are working full-time preparing Metagov for the Open Web hackathon in mid-April. If we decide that DADA is not the right use-case to explore after mid-April, then I need to adapt quickly and start finding / speccing out a new use-case.
Next step: it would be helpful to get Prima to comment on this issue, since she knows the most about DADA's governance.
Currently plugins need to define config as a JSON schema on the model, and configs are all community-specific.
This issue is for:
bot_token
in the Slack. These settings should not be exposed in the client, they are configured automatically using oauth flow.Create Python package that Drivers can use to interact with the Metagov APIs. Let's start with Python since that's what Kybern is written in. Should be in a separate repo.
Dockerize metagov to make install process easier.
Update install docs- https://docs.metagov.org/en/latest/installation.html
@mashton and I have been going back and forth on whether Metagov really needs a UI at this point. My guess is that, up to mid-April (the Open Web hackathon) we do not need and do not have the time to build a UI.
For the DADA use-case in May, which still needs to be refined, my guess is that we still do not need a UI beyond the existing config UI in Django. I.e. we need to own some sort of UI for an editor, but we do not need to own the UI for the day-to-day governance (see functional spec diagram below).
However, it has been helpful to build mockups, and we should continue doing so. Mockups have helped us visualize some of the functionalities we are speccing out and possible user flows.
Background: the target UI / UX has been (1) some sort of widget-based dashboard, (2) a Jupyter Notebook-like UX for authoring power-users, and (3) a simpler "match resource to gov-process" UX for regular users. We can discuss changes to this UI in another issue.
This issue is for expanding the use of VotingStandard
to other plugins. It could also include adding more properties to the VotingStandard available properties –– it will evolve as our "metagov voting standard" evolves, but currently it only has 4 properties (a subset of the current draft standard).
It is currently only used in DiscoursePoll (PR #112) here:
https://github.com/metagov/metagov-prototype/blob/c5e9c28ed840fe87e9768c32ca22042b03055977/metagov/metagov/plugins/discourse/models.py#L264-L270
Up to this point, we've been imagining Metagov being used by a Driver to govern activities on social platforms. The main Driver we've considered during architecture is PolicyKit, with some thought to how Kybern and Dada could function as a drivers as well.
The current architecture requires:
Requirement (1) lets us not have to deal with the situation where there are multiple drivers using a single instance of Metagov. In the code we assume that there is only one Driver, and that the Driver is governing 1 or more communities.
Requirement (2) lets us restrict (most of) the metagov endpoints to be accessibly to local traffic only. That's why https://prototype.metagov.org/api/internal/resource/sourcecred.cred?username=miriam is forbidden if you open it in a browser, but metagov instance of PolicyKit can access this URL because it's making the request from the same network.
Requirement (2) means that we don't have to deal with authenticating requests to those endpoints, because the Driver is fully trusted. The Driver is responsible for authentication and authorization–– ie making sure that people say who they say they are, and that people can only access Metagov data that they have access to (though the Driver) –– for example in PolicyKit, I as an author for community X should not be able to write a policy that performs an action in community Y.
Here's a list of things that aren't currently supported, because of the requirements listed above.
A. Read access from non-Driver system: An external system wants to make a read-only request to Metagov to read a resource. For example, a separately hosted instance of Discourse[1] requests a revenue share config from Metagov. The community already uses PolicyKit as the driver.
→ we could either let certain read-only endpoints be configured as completely public to the internet, OR implement auth so that the caller can authenticate against metagov.
B. Write access from non-Driver system: An external system wants to make a write request to Metagov. For example, some code somewhere wants to perform an action. The community already uses policykit as the driver.
→ the caller needs to authenticate against metagov somehow
C. Smart contract as Driver: A smart contract wants to act as a single-community Driver, with the ability to perform metagov actions, governance processes, and be notified of events occurring on other platforms. This isn't possible right now because of requirement (2)–– that the Driver is collocated with Metagov.
D. Smart contract as governed platform, PolicyKit as Driver. I think this is basically the same as (C). In both cases, Metagov needs a way to push data to the chain.
📌 Feedback requested: Which of these unsupported behaviors do we actually need to support? Do we have a concrete use-case for (B)?
My current feeling: we can get by with the naive solution for (A) if needed since everything we're using is public anyway, skip (B) for now though we'll eventually want it(?), and need to figure out (C) ASAP because we'll need it to support the upcoming NEAR challenge and possible peerkat collaboration.
Smart contracts can't just make API requests directly to services on the internet. Oracles are a special kind of smart contracts that allow a smart contract to look like it's making an API call, but really the oracle is making the API calls off-chain and posting the result on-chain for smart contracts to use. This tutorial how/why. Existing tools like Chainlink make it easier to perform API requests, receive responses, & retrieve data from the outside world. See Chainlink docs: request and receive data.
TODO- figure out exactly how Metagov could act as an oracle. Once I have an actual proposal for that, I'll do a separate issue.
[1] I'm specifically thinking about a Discourse instance that has a Discourse plugin (that we created) that invokes Metagov. Obviously there aren't any existing non-driver platforms that want to invoke Metagov right now.
Currently plugin authors are given a broad handle in processing incoming webhook requests. This requires the plugin author to understand a bunch of stuff about how metagov-core works, which might not be ideal. This issue suggests chunking up webhook handling into smaller pieces, and give the control back to the core. The core can then delegate each piece back to whichever handlers are defined by the plugin author.
Create some kind of middleware chain so plugin authors can hook in handlers for...
handle_incoming_webhook
handler, example)*Look at how this works when authoring a Zapier app..
Support having multiple plugin instances of the same type for a single community. For example community X should be able to have 2 enabled instance of the RevShare
Plugin, each with different configurations.
[updated]
Add a scheduling system in metagov-core. Enable Plugin authors to access the scheduler for these 2 purposes:
closing_at
date?N communities * M listener plugins * P governance processes
[old]
The current "listener" interface requires that the external platform already has support for web hooks, and that it emits web hooks for all the events that the governance author cares about.
Read about implementing a Listener in a Plugin here: https://docs.metagov.org/en/latest/plugin_tutorial.html#listener
Problem: not all platforms have support for webhooks!
One example is NEAR: as a plugin author, I want to support "listening" to transactions occurring on the community's smart contract, so that the governance Driver can react to events. There is a public NEAR indexer for explorer that exposes a public API for querying transactions (or DataHub API which is similarly a public indexer). There are also services like BlockNative that let you monitor blockchain transactions using webhooks, but none of these support NEAR.
Alternative to webhooks: the alternative to webhooks is some kind of "pull" API where we fetch recent events from an API at regular intervals. This is what PolicyKit does using Celery tasks (with RabbitMQ broker). How can we make it easy for plugin authors to implement something like that? What technology should we use?
Driver should be able to set the event receiver URL dynamically (eg with POST /api/internal/webhook
). The Driver may even want to use several event receiver URLs–– like separate endpoints per-community or per-plugin or per-plugin-per-community.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.