dxos / cli Goto Github PK

DXOS CLI and extensions.

License: GNU Affero General Public License v3.0

JavaScript 22.13% Shell 1.30% Dockerfile 0.28% TypeScript 76.25% HTML 0.04%

cli's Introduction

DXOS

We build Composer: an extensible app platform where developers can extend their intelligence. Open-source. Multiplayer. Local-first. Private.

The DXOS SDK takes care of peer-to-peer collaboration for local-first apps without central sync servers.

Learn more about the the mission.

SDK Demo

Shown below are two sessions of an application using ECHO to replicate state peer-to-peer in real-time.

Documentation

DXOS Resources

Contributions

Your ideas, issues, and code are most welcome. Please take a look at our community code of conduct, the issue guide, and the PR contribution guide. To learn about how to set up for development and contribution to DXOS, see the Repository Guide

cli's People

Stargazers

Watchers

Forkers

nytyn

cli's Issues

Browser to CLI Party invitations are not working

The other way (CLI to Browser) are working.

Reproduced in Tasks app in protocols and CLI, and in Chess repo.

Use Dot for app registering and serving

Publish RegistryApi
create protobuf schema for an app
dx app register - register on dot blockchain
dx app serve - reads data from dot

feat: suggest `dx COMMAND help`

If you enter only dx:

Type dx help for full list of commands.

If you enter dx app:

Type dx app help for full list of commands.

This should extend for all the extensions.

We should avoid having empty outputs 🤔

CLI: Make it more verbose

Overview

We would like to get more information about errors, warnings, or instructions when deploying/publishing an app.
Even if we have a great doc, sometimes humans make mistakes, and providing output right away on the console will make the adjustments faster 😄 .

Config Errors
Config Warnings
Publish Errors
Publish Warnings
Deploy Errors
Deploy Warnings
Once the deployment is already done we can provide a default URL so the app owner can take a look at the current deployment.
Check if app.yml is set, otherwise notify the developer that we can't find it.
create a apps deployed list command

CLI extensions not installing

13:20 ➜ dx extension install @dxos/cli-machine

✔ Installing @dxos/cli-machine

Error: Cannot find module 'got'Require stack:
- /Users/dmaretskyi/.config/yarn/global/node_modules/do-wrapper/dist/request-helper.js
- /Users/dmaretskyi/.config/yarn/global/node_modules/do-wrapper/dist/do-wrapper.js
- /Users/dmaretskyi/.config/yarn/global/node_modules/@dxos/cli-machine/dist/es/modules/machine.js
- /Users/dmaretskyi/.config/yarn/global/node_modules/@dxos/cli-machine/dist/es/index.js
- /Users/dmaretskyi/.config/yarn/global/node_modules/@dxos/cli/dist/es/pluggable.js
- /Users/dmaretskyi/.config/yarn/global/node_modules/@dxos/cli/dist/es/modules/pluggable.js
- /Users/dmaretskyi/.config/yarn/global/node_modules/@dxos/cli/dist/es/main.js
- /Users/dmaretskyi/.config/yarn/global/node_modules/@dxos/cli/bin/dx.js

ECHO cli

Unify config for runtime/tools

Single schema for all DXOS related configs: runtime and publish

Schema defintion is in dxos/protocols/packages/common/proto/src/proto/dxos/config.proto

Example of how it would look in yaml:

version: 1

module:
  name: dxos:app.tasks
  type: dxos:type.app
  version: 1.2.3 # Loaded from package.json

  author: DXOS.org
  license: GPL-3.0
  description: Braneframe
  repository: 'https://github.com/dxos/frame-app'

  record:
    app: # Same as dxos.type.App (from DXNS)
      contentType:
        - 'dxos:type.chess.board'

build:
  command: npm run build

runtime:
  services:
    signal:
      server: "wss://apollo3.kube.moon.dxos.network/dxos/signal"
    dxns:
      server: "https://apollo3.kube.moon.dxos.network/dxos/wns/api"
  
  client:
    storage:
      persistent: true

  bot:
    factory:
      id: 123
  
  props:
    title: "Tasks App"

Config loading

Both apps at runtime and cli tools use @dxos/config package to load the config.

In general case it has an array of "config sources": yaml files, remote kube config, cli profile to load config from.
Those are then merged into a single config.

Config files:

dx.yml Deploy configuration + runtime defaults
dx-dev.yml Dev overrides for runtime configuration

Tasks

Update config protos with new design
Use @dxos/config to load config in CLI extensions (app/bot publish)

Consolidate the way `dxns` commands are working

generating address should store it in halo
dx halo should print current profile
verbs
persistence mechanisms should work the same way - "default" files are different - one is a symlink
Remove "normal connection closure" from the log
- Also some similar logs which happen when the connection to Polkadot API is opened and closed too quickly
A single command to print all stuff - profiles, addresses, accounts
Migrate docs from dxos/dxos#860 and readmes to docs
Should be able to override address selection by command line parameter, just as account currently
- Account - DX_DXNS_ACCOUNT
- Address - DX_DXNS_ADDRESS
Rename dx ns list to dx ns info

[CLI] Include QR code from Keybase in Tutorials docs

Ideally, the QR code would be served by an unprotected route by the Kube, but as a first version, we will have the QR code in the docs.

We need to add it to the Deployment docs into the Tutorial and explain how should be used to be able to access the Console.

Upgrade `protocols` dependencies to newest version

Required for dxos/dxos#68

Party invitations in CLI not working

Main problem

Party invitations not working.

wire-local party create
[dx] party invite

(...)

// Use the invitation to claim it, e.g. in Wallet extension

The invitation times out.

Side problem (Fixed)

wire party create not working after upgrading to latest protocols

wire-local party create

AssertionError [ERR_ASSERTION]: Incorrect arguments format.
    at NetworkManager.joinProtocolSwarm (/Users/przemek/Projects/wireline/cli/node_modules/@dxos/network-manager/src/network-manager.ts:72:11)
    at PartyProtocol.start (/Users/przemek/Projects/wireline/cli/node_modules/@dxos/echo-db/src/parties/party-protocol.ts:69:33)
    at PartyInternal.open (/Users/przemek/Projects/wireline/cli/node_modules/@dxos/echo-db/src/parties/party-internal.ts:195:26)
    at warnAfterTimeout (/Users/przemek/Projects/wireline/cli/node_modules/@dxos/echo-db/node_modules/@dxos/debug/src/timeout-warning.ts:21:12)
    at Lock.executeSynchronized (/Users/przemek/Projects/wireline/cli/node_modules/@dxos/async/src/lock.ts:40:21)
    at PartyManager.open (/Users/przemek/Projects/wireline/cli/node_modules/@dxos/echo-db/src/parties/party-manager.ts:86:9)
    at warnAfterTimeout (/Users/przemek/Projects/wireline/cli/node_modules/@dxos/echo-db/node_modules/@dxos/debug/src/timeout-warning.ts:21:12)
    at Lock.executeSynchronized (/Users/przemek/Projects/wireline/cli/node_modules/@dxos/async/src/lock.ts:40:21)
    at ECHO.open (/Users/przemek/Projects/wireline/cli/node_modules/@dxos/echo-db/src/echo.ts:219:7)
    at Client.initialize (/Users/przemek/Projects/wireline/cli/node_modules/@dxos/client/src/client.ts:136:5) {
  generatedMessage: false,
  code: 'ERR_ASSERTION',
  actual: false,
  expected: true,
  operator: '=='
}

Error: Incorrect arguments format.

Update DXNS CLI and app publishing to adjust to new spec impl in the blockchain

Enable different kinds of links serving

/app/DXN
- Should grab the latest
/app/DXN/tag
/app/DXN/version
/app/CID

Add typings for `argv: any` across handlers

At the minimum, replace with Arguments<CoreOptions>

DXNS get resource that doesn't exist crashes

WNS Migration

TODO(burdon): Link to Ashwin's issue.

Rename

@wirelineio/registry-client => @dxos/registry-client
@dxos/cli-wns => @dxos/cli-registry
WNS => registry
wrn => dxn (variable)
xxxWrn => xxx (avoid suffix?)
wrnXxx => xxx (avoid prefix?)
WIRE_ => DX_ (variables, env)
~/.wire => ~/.dx (config)
wire => dx (CLI)
services.wns => services.registry (config)

Resource Names

wrn://auth/path/sub => dxn://auth:path/usb (const) (e.g., dxn:dxos.org:app/console@alpha:/service-worker.js)
- NOTE: last slash is required for relative paths to work.
Util class to validate (e.g., create, parse, restrict characters).

Misc text

Copyright 2020 DXOS.org
DXOS.org
[email protected]
AGPL-3.0

Misc clean-ups

Port numbers (doc/list)
- KUBE service range 87xx?
- List ports in console services dashboard
- https://github.com/dxos/kube/blob/9d7333aeff52a504de6bda29c1d6673c53ad28e5/docs/content/misc/stack.md
- https://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_numbers
Delete @wirelineio (and other defunct) npm packages

Registry

id vs name (with valid character check, global util)
record types (e.g., wrn:app)
WRN names should not include wrn:// prefix in registry (this is implicit)
namespace vs domain vs authority?
remove tags (legacy for namespace?) [see CLI]

CLI

Normalize deploy/publish/register commands (CLI)
Normalize config files
Normalize logging (errors, help, verbose, json, "ok", etc.)
Consistent use of namespace/tag

wire machine commands are run twice

As of 2.0.20, wire machine delete and wire machine create are run twice, for example:

> wire machine delete --name thomas --verbose
Removing machine...
Removing DNS record...
Removing machine...
Removing ipfs service record...
Removing DNS record...
Removing ipfs service record...
Removing bot-factory service record...
Removing bot-factory service record...
Removing kube service record...
Removing kube service record...

I worked around this for 2.0.21 by adding a 'running' flag as a kind of lock around those two operations.

Support deploying a validator wns node in wire machine

Revive cli-bot

Revive bot-factory install, start, setup commands
Revive bot build, publish, register commands
Revive bot spawn, invite commands
Add e2e tests testing the whole flow
- Build, publish, register bot
- Run bot-factory
- Spawn bot via cli
- Check that bot is running

Error: Not supported when deploying

The deployment (wire app deploy) does not work for me in Teamwork.

Wire version and profile

14:22 ➜ wire version
v1.0.1-beta.29

14:27 ➜ wire profile config
{
  "cli": {
    "dashboard": {
      "package": "@dxos/xbox-dashboard",
      "channel": "beta",
      "bin": "dxos-dashboard"
    },
    "signal": {
      "package": "@dxos/signal",
      "channel": "beta",
      "bin": "dxos-signal"
    },
    "chess": {
      "package": "chess-cli",
      "bin": "chess-cli"
    },
    "npmClient": "yarn",
    "peerId": "b105f3ec295c11a6783bc8f0231f5235e60677fc878de497fd56fa450821559f"
  },
  "services": {
    "wns": {
      "server": "https://demo.kube.dxos.network/dxos/wns/api",
      "userKey": "657af063bd13e7359904d3d935e9e9b8339fa4a5effc61690d28a79417eafe7f",
      "bondId": "bd43083744e7ff15380f38ec672be7715b343c7529e21128fb0ac12c23f015ea",
      "chainId": "devnet-2",
      "gas": "200000",
      "fees": "200000uwire"
    },
    "ipfs": {
      "server": "https://apollo1.kube.moon.dxos.network/dxos/ipfs/api",
      "gateway": "https://apollo1.kube.moon.dxos.network/dxos/ipfs/gateway/"
    },
    "signal": {
      "server": "wss://apollo1.kube.moon.dxos.network/dxos/signal"
    },
    "ice": [
      {
        "urls": "stun:apollo1.kube.moon.dxos.network:3478"
      },
      {
        "urls": "turn:apollo1.kube.moon.dxos.network:3478",
        "username": "dxos",
        "credential": "dxos"
      },
      {
        "urls": "stun:apollo2.kube.moon.dxos.network:3478"
      },
      {
        "urls": "turn:apollo2.kube.moon.dxos.network:3478",
        "username": "dxos",
        "credential": "dxos"
      }
    ],
    "faucet": {
      "server": "https://faucet.kube.moon.dxos.network/api"
    }
  },
  "system": {
    "certEndpoint": ""
  }
}

Steps

Go to Teamwork, apps/teamwork-app
wire app deploy

Result

14:22 ➜ wire app deploy
✔ Installing @dxos/[email protected]
Preparing to deploy...
Building rzadp-Teamwork...
Build Ok.
Publishing rzadp-Teamwork...
Published [email protected] with cid QmVpXjEU61op5ZMbVN9BuVC5YKcrhEeZqgmvxH8i9C39Gz
Registering [email protected]...

Error: Not supported

It does not say what exactly is not supported, and doesn't give me a clue on what to do to fix this.

Add a script for linking

Add a script aiding in developing CLI modules outside of this repo, much like dxos-deprecated/sdk#303

README.md - Clarify nvm dependency

The CLI has an nvm dependency. This needs to be mentioned clearly as a prerequisite.

`@dxos/cli-dxns` installed via `npm` is broken

Image that shows the point where the cli is breaking:

Add 'dx' alias to cli

Normalize publishing to IPFS/DXNS

Currently, publishing is handled in each cli extension separately, e.g. via dx app deploy or dx bot deploy;

We would like to normalize process of publishing of app/bot/model/etc under single cli extension (supposedly cli-dxns)
By doing so, we need to follow those rules:

unify .yml file, e.g. dx.yml
support link to the model from other entities like apps and bots (new field with dxn or cid)
common build/publish command for deploy in cli-dxns
- build command to read corresponding build section of module.yml to find a command to run a build
- create a build tool for bot, e.g. expose https://github.com/dxos/protocols/blob/main/packages/bot-deprecated/botkit-deprecated/src/tools/build.ts#L13 as binary

gaFeed store reader stalled

Happens when redeeming an inivitation in cli, browser is the inviter.

[wire]> gaFeed store reader stalled: no message candidates were accepted after 1000ms timeout.
Current candidates: [
  {
    key: <Buffer b0 b5 a3 b1 0f 79 ba 9d 4f 3c 5f 15 98 ac 73 36 3d 08 13 87 14 25 10 cd 78 d4 db 63 7a af a0 13>,
    seq: 0,
    data: { halo: [Object] },
    sync: true
  }
]

Party cli extension to be able to inspect party state

Rename cli-data to cli-party

remove jq exec requirement

CLI is not picking up ENV variables when installed globally via `npm`

CLI is picking up ENV variables through its env-map file:

https://github.com/dxos/cli/blob/main/packages/cli-core/src/env-map.json

The setup is working fine when the CLI is installed in a repository and run through yarn dx - example setup is working in console.

However it seems to not be working when CLI is installed globally.

Steps to reproduce:

npm install --global @dxos/[email protected]
npm install --global @dxos/[email protected]
npm install --global @dxos/[email protected]

export DX_DXNS_ENDPOINT="wss://blabla"

dx dxns resource list

Expected output

An error about not being able to connect to blabla.

Actual output

Resources listed from DXNS setup in my profile.

`.wire` directory is hardcoded across the codebase

It should be configurable or at least specified in one place.
Also it should probably be .dx or .dxos.

Add `example` domain to the seed command for the DXNS

Bots, CLI, frame demo

Fix cli init for case of local development

Currently init func looks for extensions that are installed in the system, instead of using all packages from cli monorepo

Add docs for cli-dxns

Secure key storage in the CLI

Potentially integrate polkadot keys in keyring.

Update CLI DXNS query and view capabilities

CLI test coverage

CLI should use persistent storage by default.

Persistent halo and feeds per profile.
Command to reset storage.
cli-party - state manager API to create client and open current party - might happen automatically.

First time run wire wns command throws error about missing "chalk"

I see this every time on multiple machines. Ubuntu 18.

david@blackcube2:~$ wire wns account create
✔ Installing @dxos/[email protected]
Error: Cannot find module 'chalk'
Require stack:
- /home/david/.config/yarn/global/node_modules/inquirer/lib/objects/separator.js
- /home/david/.config/yarn/global/node_modules/inquirer/lib/inquirer.js
- /home/david/.config/yarn/global/node_modules/@dxos/cli-wns/dist/es/modules/wns.js
- /home/david/.config/yarn/global/node_modules/@dxos/cli-wns/dist/es/index.js
- /home/david/.config/yarn/global/node_modules/@dxos/cli/dist/es/pluggable.js
- /home/david/.config/yarn/global/node_modules/@dxos/cli/dist/es/modules/pluggable.js
- /home/david/.config/yarn/global/node_modules/@dxos/cli/dist/es/main.js
- /home/david/.config/yarn/global/node_modules/@dxos/cli/bin/wire.js
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:965:15)
    at Function.Module._load (internal/modules/cjs/loader.js:841:27)
    at Module.require (internal/modules/cjs/loader.js:1025:19)
    at require (internal/modules/cjs/helpers.js:72:18)
    at Object.<anonymous> (/home/david/.config/yarn/global/node_modules/inquirer/lib/objects/separator.js:2:13)
    at Module._compile (internal/modules/cjs/loader.js:1137:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1157:10)
    at Module.load (internal/modules/cjs/loader.js:985:32)
    at Function.Module._load (internal/modules/cjs/loader.js:878:14)
    at Module.require (internal/modules/cjs/loader.js:1025:19) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/home/david/.config/yarn/global/node_modules/inquirer/lib/objects/separator.js',
    '/home/david/.config/yarn/global/node_modules/inquirer/lib/inquirer.js',
    '/home/david/.config/yarn/global/node_modules/@dxos/cli-wns/dist/es/modules/wns.js',
    '/home/david/.config/yarn/global/node_modules/@dxos/cli-wns/dist/es/index.js',
    '/home/david/.config/yarn/global/node_modules/@dxos/cli/dist/es/pluggable.js',
    '/home/david/.config/yarn/global/node_modules/@dxos/cli/dist/es/modules/pluggable.js',
    '/home/david/.config/yarn/global/node_modules/@dxos/cli/dist/es/main.js',
    '/home/david/.config/yarn/global/node_modules/@dxos/cli/bin/wire.js'
  ]
}

upgrade not working

13:17 ➜ dx upgrade --npm-client yarn --force --version "2.0.36-alpha.0"

Found extensions: @dxos/pad-cli, @dxos/ipfs-cli, @dxos/cli-bot, @dxos/cli-data, @dxos/cli-signal, @dxos/cli-app
✖ Uninstalling @dxos/pad-cli

Error: Command failed: yarn global remove @dxos/pad-clierror This module isn't specified in a package.json file.

CLI error on party create

❯ wire-dev party create
Error: party.processor.keyOrInfoAdded.debounce is not a function

Workaround: Use yarn linked version of dxos/async in dxos/cli.

CLI cleanup

Improve CLI stability

feat: create dx devtools download to get the latest zip file from protocols

We need a way to get the latest zip file from protocols.

For now we can just download a harcoded version, hosted in IPFS maybe?

Let's discuss it 😄

`wire-local app serve` is not working

DEBUG=* wire-local app serve
  pm2:paths pm2 home resolved to /Users/przemek/.pm2 /Users/przemek +0ms
  pm2:conf Using 2 parallelism (CONCURRENT_ACTIONS) +0ms
  pm2:client Using RPC file /Users/przemek/.pm2/rpc.sock +0ms
  pm2:client Using PUB file /Users/przemek/.pm2/pub.sock +0ms
  interactor:client [PING INTERACTOR] Trying to connect to Interactor daemon +0ms
  axon:sock connect attempt null:/Users/przemek/.pm2/interactor.sock +0ms
  axon:sock error ENOENT +79ms
  axon:sock ignored ENOENT +0ms
  dxos:cli Error: spawn wire-app-server ENOENT
  dxos:cli     at Process.ChildProcess._handle.onexit (internal/child_process.js:269:19)
  dxos:cli     at onErrorNT (internal/child_process.js:467:16)
  dxos:cli     at processTicksAndRejections (internal/process/task_queues.js:82:21) +0ms

Error: spawn wire-app-server ENOENT

The issue can be worked around by running yarn dx app serve in the cli repo root.

CLI Update Tracking Issue

CLI tools are all using latest SDK, ECHO, HALO as needed
Code coverage exists for all packages in monorepo #54
Extraneous dependencies are pruned (ex: no jq exec)
Documentation is updated for all packages in monorepo

Windows OS : README.md install note - add yarn global instructions

For users on Windows, if the yarn global bin is not in %path%, they will not be able to use the CLI.

Add two steps for windows users:

Discover the yarn global path with 'yarn global bin'
Add the path to their environment variable PATH

CLI changes to make BF use profile config, history support

Make bot-factory use CLI profile config (custom config loading code currently)
History support for interactive CLI
Don’t quit on bad command

Console/cli-app url/path names.

Review and update: #120
Move @wirelineio/registry-client to @dxos/registry-client.
Utility class to create/parse WRN (DXN) names (with validation).
Fix console path generation and cli-app serving (see patterns below).
Ensure service_worker.js is loaded correctly (for console app). (See dxos/console#67).

Valid WRN: wrn://dxos:app/console@alpha
Valid URL: https://apollo1.kube.moon.dxos.network/app/dxos:app/console@alpha