cardstack / cardstack

The mono-repo for the core Cardstack framework.

Cardstack App Suite

This is the main repo for Cardstack, an open source application architecture for building cohesive experiences, including payments, on open, decentralized infrastructure.

Join the discussion around developing on the Cardstack framework on Discord, and read our documentation on docs.cardstack.com. (Note: documentation refers to Cardstack v2. The main branch of this respository has in-progress work on v3 of Cardstack.)

Orientation

This is a monorepo. Each directory under packages and cards is distributed as a standalone NPM package under the @cardstack NPM namespace. More information is available in the README.md within each package.

Developing and Testing within this Repo

Local host names

Set up app.cardstack.test and app-assets.cardstack.test to resolve to localhost (127.0.0.1). There are a variety of ways to accomplish this, with the most direct being to edit your /etc/hosts file.

Hub environment variables

See the README in the hub package for environment variables that you'll need to setup.

Fetching waypoint config

To retrieve environment variables for waypoint in a more readable way as an alternative to waypoint config get, you can use the convenience script yarn waypoint-vars.

// prints a table of variables, with values truncated for readability
yarn waypoint-vars
// prints JSON of variables that contain any of the strings provided (case-insensitive), full values
yarn waypoint-vars VAR_NAME1 VAR_NAME2

Javascript dependencies & Typescript compilation

We use volta to manage our global javascript dependencies. In this case, specifically, we use it to manage node and yarn. To use it simply install it following the instructions here: https://docs.volta.sh/guide/getting-started

In development, we use lerna to manage the inter-dependencies of all the packages. To get started:

1. Install node and yarn via volta.
2. yarn global add lerna (use >= 2.0.0 for yarn workspaces integration)
3. lerna bootstrap
4. Launch the typescript compiler with yarn compile --watch
5. Start ember-cli and the hub node server with yarn start and/or run tests with yarn test

Editors

If you use Visual Studio Code and work on the boxel, web-client or ssr-web packages, we recommend the following extensions to make working with glint and gts files better:

• Glint language server integration for VS Code
• VSCode extension for glimmer

Note that you will see TypeScript errors shown twice if you have the built-in VS Code TypeScript extension running as well as the Glint extension. You may also see some spurious errors from the built-in extension, because it does not understand the template tag. It can be beneficial to turn off the built-in TS extension when working purely in a workspace that is configured for glint, since glint is performing a super set of TS validation. To find the built-in extension in the extensions list to disable it, you will need to search for "@builtin typescript".

Understanding the respositories under the Cardstack organization

The following summary offers an overview of where development is currently ongoing at Cardstack. (Note: any projects linked below that are not currently public will become public soon.)

cardstack/card-protocol-xdai

• The Layer 2 contracts for the card protocol live here including
  • PrepaidCardManager contract
  • RevenuePool contract
  • L2 Token contract
  • SPEND token contract
  • BridgeUtilities contract (facilitates token bridge contract)

cardstack/tokenbridge-contracts

• The home bridge and foreign bridge token contracts

https://github.com/cardstack/card-protocol-relay-service

• our gnosis relay service, forked to provide additional prepaid card manager API's that support gasless interactions with our PrepaidCardManager contract

cardstack/safe-transaction-service

• our gnosis transaction service, this was forked to provide transaction service for Sokol (xDai uses the gnosis hosted transaction service)

cardstack/cardstack

• this one! It is our monorepo that contains our CardPay Dapp (as well as eventually cardstack hub runtime). Work on the "card compiler" is also occurring in PRs of this repository. A proof-of-concept for the dapp was developed here: https://github.com/cardstack/card-pay/tree/update-UI-depot

cardstack/cardwallet

• our rainbow wallet fork that supplies our mobile client experience. Currently it is focused around interacting with Layer 1 contracts, eventually we see it as interacting with the Layer2 protocol as well. A proof-of-concept was developed here: https://github.com/cardstack/rainbow/branches

cardstack/infra

• Holds our terraform scripts to provision AWS and cloudflare (and eventually GCP) services for our infrastructure.

cardstack/boxel

• our web UI component library

cardstack/catalog-experiment

• our planned Javascript build tooling and CDN that eliminates the need for running npm/yarn and eliminates the need to maintain a node_modules folder in your web projects

cardstack/animations-experiment

• proof of concept for an animation library that works well with Ember and meets Boxel's motion needs

Project-wide Policy and Community Governance

Cardstack follows semantic versioning. As a young, pre-1.0 project, this means you can continue to expect breaking changes in minor releases. Each package should endeavor to include a CHANGELOG.md once it begins to have a non-trivial number of external users.

We intend to adopt a community RFC governance process for public API changes, based on the Rust and Ember projects. At this pre-1.0 stage, RFCs are optional and changes may still be fast and furious.

Cardstack follows the Ember Community Guidelines, both because we are a proper subset of the existing Ember community and because we wholeheartedly endorse the same values.

cardstack's Issues

Add workflow package

Queue View (QV): The first level of the workflow, with a list of counters for each tag
Cue Card List View (CCLV): 01 Notifications.jpg
Message Thread View (MTV): 02 Dispute Resolution.jpg

• Make a decent looking CCLV, with the text of each message as content (Approval Card View)
• Display more priorities in the QV
• Adjust acceptance test (now failing)
• Group messages also by date range (Today, This week, This month, etc.) and display latest date range on top of QV
• Fix clicking on a different tag in the QV after a message has been selected and thus MTV is loaded.
• Fix displaying messages that match the selected date range (on the CCLV)
• Remove is-handled attribute from Message. It should be derived from the status attribute (which should be added and can have the values: pending, approved, denied).
• Add active class to selected tag in QV (see "License action")
• Rename "Automatically processed" to "Processed"
• Create first version of MTV. Summary card should have 1) message title as summary 2) three buttons and a copy of the cue card from the Approval Card View (ACV).
• Create the message-list-card component in the addon and just reimport it from the app
• Make clicking a tag that has no messages in it open an empty approval card in CCLV.
• Make it so that handling the last card with a tag doesn't make that tag's group vanish from the UI
• For non-urgent priority groups, display the number of handled messages (not the unhandled ones)
• On the CCLV, display all cards that match the clicked tag or date range, not just the ones that need attention => They are now displayed by group, that takes into consideration their priority, tag and status, so that might not be necessary.
• Buttons should not be displayed in the Processed and FYI priorities
• Color counters based on their priority (red vs. black)
• Change the priority of a handled message to 'processed'
• Introduce new priority: Delegated and put in on top of QV
• Find out if message cards should be added to the addon folder (instead of app) and how to do that
• CSS adjustments (see TODOs in addon.css)
• Write integration test for workflow service (tried my hand but failed but I now see there's a good example of how to create records in the store in packages/models/tests/integration/models-test.js

Disabled plugins still leak their descendant deps into ember build

The @cardstack/plugin-utils/when-disabled feature is supposed to let addons disable their ember build output when they are marked as disabled in cardstack hub, which is useful when you have plugins that should only be enabled in certain environments.

But the strategy I used isn't catching the recursive dependencies -- their artifacts still end up in the build.

One way to fix this is to switch from wrapping treeForAddon, etc, and instead just wrap treeFor (because treeFor is what recurses into all the other addons). This does the right thing within the broccoli build, but it results in build failures because we would also need to intercept the included hook. Which is problematic, because included is synchronous and we need to ask the hub if we're activated asynchronously.

Internal server error for invalid Authorization header

With @cardstack/authentication installed and activated

🚀 ~$ curl -XPOST http://localhost:4200/cardstack -H "Authorization: Bearer asdf"
Internal Server Error

Server log:

  Error: Not a valid signed message
      at Encryptor.verifyAndDecrypt (/Users/aaron/dev/probably-right/node_modules/@cardstack/hub/encryptor.js:41:13)
      at Authentication._tokenToSession (/Users/aaron/dev/probably-right/node_modules/@cardstack/authentication/cardstack/middleware.js:65:57)
      at /Users/aaron/dev/probably-right/node_modules/@cardstack/authentication/cardstack/middleware.js:98:28
      at dispatch (/Users/aaron/dev/probably-right/node_modules/koa-compose/index.js:44:32)
      at /Users/aaron/dev/probably-right/node_modules/koa-compose/index.js:36:12
      at dispatch (/Users/aaron/dev/probably-right/node_modules/koa-compose/index.js:44:32)
      at /Users/aaron/dev/probably-right/node_modules/koa-compose/index.js:36:12
      at /Users/aaron/dev/probably-right/node_modules/@cardstack/hub/middleware-stack.js:20:13
      at process._tickCallback (internal/process/next_tick.js:109:7)

It looks like the check here doesn't catch the exception thrown here

Ethereum indexer does not handle functions with a return value that named outputs

One quirk of solidity is that a function can actually return named outputs, so that its possible to return many values from a function (similar to javascript returning an object that has mulitple properties). We need to be able to handle this type of function return.

https://github.com/cardstack/cardstack-web/issues/248

Refactor grants so they dont default to "all"

Right now when you create a grant, when the type or the field is not specified for the grant it is assumed to apply to all types or all fields. It makes it very easy to set a grant that casts too wide of a net if you omit information. It may be safer to take a more conservative approach such that the grant only applies to the types and fields that you specify. For convenience we could include an all types or all fields flag.

re-running `hub:seed` results duplicate key constraint errors

when hub:seed sees that a model already exists for an ID it should update the model instead of trying to create a new model with a duplicate key

Remove ethereum buffer

Now that we have queued indexing and are lessening our reliance on elastic search via the pgsearch plugin, let's remove the ethereum buffer.

Add linting for requiring unspecified dependencies

We should pull in eslint-plugin-node, and specifically the no-extraneous-require rule: https://github.com/mysticatea/eslint-plugin-node/blob/master/docs/rules/no-extraneous-require.md

I tried to do this, but ran into trouble getting eslint to load the plugin. I can try again when I have some time to focus on it.

Because we're using a monorepo with yarn workspaces, we're especially susceptible to this - our modules can all require each other, and only break once published individually to the registry.

Solidify cross-datasource schema usage

Once the hub is up and running, and the index is fully populated, any data source can reference data or schema from any other data source. But during the initial indexing, errors can pop up due to load order.

When models from one data source try to reference schema from a datasource that hasn't yet been indexed, readOtherIndexers for the schema model can return undefined, and we'll see some error like this:

cardstack/indexers Unexpected error in _updateLoop Error: field "user" refers to missing related type "mock-users"

Here's an example of a seed file that can break it: https://github.com/cardstack/cardstack/blob/ad1344592a77b91b8ed8d963bf5e8d8e8d584446/packages/notification/tests/dummy/cardstack/seeds/development/ephemeral.js

You can check out that commit (ad13445), and run ember serve inside the notification package to reproduce.

We should probably do some more lightweight discovery of all the data sources before trying to validate a complete schema.

Git seeds cause implicit dependency on authentication

The development seeds cause ember serve to fail when @cardstack/authentication is not installed.

https://github.com/cardstack/cardstack/blob/master/packages/git/blueprints/cardstack-git-seeds/files/cardstack/seeds/development.js#L25

1. ember new my-app --yarn
2. yarn add @cardstack/hub @cardstack/git
3. yarn add dag-map to work around #3
4. ember generate cardstack-git-seeds
5. ember serve results in:

🚀 my-app$ ember serve
Livereload server on http://localhost:49153
(node:58110) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 2): Error: Cannot find module '@cardstack/authentication/package.json'
Error: Cannot find module '@cardstack/authentication/package.json'
    at Function.Module._resolveFilename (module.js:470:15)
    at Function.resolve (internal/module.js:27:19)
    at loadPlugin (/Users/aaron/dev/my-app/node_modules/@cardstack/hub/plugins.js:124:41)
    [ snip ]

Deleting the plugin config for authentication from cardstack/seeds/development.json:

  {
    type: 'plugin-configs',
    id: 2,
    attributes: {
      module: '@cardstack/authentication'
    }
  },

lets the app run

enhancement: add eslint browser rule to check for extraneous imports

Right now there is an eslint rule that checks for extraneous require which is super helpful. It would be great to have a similar rule to check for extraneous import. Something like this: https://github.com/benmosher/eslint-plugin-import/blob/master/docs/rules/no-extraneous-dependencies.md

Have both `date` and `datetime` core types

Currently there is only a date core type, which actually behaves more like a datetime. It would be nice if we could have core data types for both date, where the time isn't important (ex. 2018-06-26) and datetime (ex. 2018-06-26T04:00:00.000Z).

Document how to create a schema + maybe even have a guide

The process of setting up a schema for a person that is new to the system is a little opaque. We should include documentation around how to create a schema, including different relationship types.

pgsearch refactor tracking issue

Opening this as a place to collect notes on where we are and where we're going with the pgsearch refactor. The major goals here are

• allow us to consistently offer features like server-side computed properties (which can in turn power arbitrarily complex permissions, relationships, etc). By "consistently" I mean you can rely on them being present (1) in the search index, (2) in responses that bypass the search index via custom searchers, (3) in responses echoed back from writers.

• avoid needing to always go to upstream data sources when recomputing inter-record dependencies (because they can be arbitrarily slow and some may be fragile and easy to overload).

• make all writes (via hub:writers) also do the correct targeted update in the search index, without relying on triggering a general hub:indexers update (no more "update with hints"). This lets us maintain a clear separation between full updates (which are effectively background processing that is never in a request critical path) and ongoing concurrent writes (which are never blocked by background indexing and yet are always immediately visible in the search index).

In these notes it's useful to keep in mind three different representations for any resource:

• the upstream format is a single json:api resource as given to us by an indexer or searcher plugin. It never has computed fields, and it can't have includes because it's only a resource, not a complete document.
• the pristine format is a complete json:api document that is augmented with computed fields and includes.
• the search format is a flattened representation that nests are included resources inline, flattens away the attributes and relationships POJOs, and allows field-type plugins to customize the storage format of their values for optimal searchability.

The steps here are:

• replace our built-in elasticsearch-backed index with a postgres-backed index. We need to consistently read our own writes, we don't actually depend internally on any of the things elasticsearch excels at (like deep text searches). This work is already far along on the pgsearch branch.

• refactor DocumentContext so it can run not just at indexing time, but also when assembling a response from a writer or searcher. Optimize its internals so that it can build both or either of the searchDoc and pristineDoc while caching the common work between them and not preemptively building parts it wasn't asked for.

• change hub:writers so that it uses DocumentContext to send a complete pristine document format in the response to the client, and also so that it stores the search document format directly in the search index (this allows plugins like jsonapi to drop their explicit calls to indexers.update()).

• change hub:searchers so it uses DocumentContext to turn any upstream documents provided by searcher plugins into pristine documents. It needs to be smart enough to not do wasted work when a searcher returns an already-pristine document (which is what happens when the request is handled by our search index). But it also needs to be able to customize responses by respecting a ?include= query parameter. Basically it should opportunistically use what it was already given, but augment it wherever the user is asking for more.

• we have skipped tests for computed fields that can get unskipped once we've done this work.

• the read implementation in BranchUpdate should change. Today it always goes to the upstream data source, but now we can switch to the upstream_doc from pgsearch. This will require our invalidations step to get smarter

• standardize all writer, searcher, and indexer APIs to represent the upstream doc as a full json:api response, not just a resource. This creates opportunities for us to better optimize when plugins happen to have extra data already available.

Augment indexing to allow to allow indexing to occur in data source specific queues

At boot time we should clear out any indexing queue jobs (active and pending) pgboss so that new containers' indexing jobs wont have to wait on abandoned jobs from older container

need to explicitly check session tokens on client app start

I'm running into a scenario where it looks like the client is not explicitly checking tokens with the server when the application starts. This is visible when you start the hub without setting CARDSTACK_SESSIONS_KEY env var. then login (use the mock-auth plugin's dummy app), then restart the server (this invalidates all the tokens), then reload the client. In this scenario the cardstack-session service's isAuthenticated property is true, but the token presented to the server does not resolve to a valid session.

cardstack-session.user errors if authentication fails

cardstack-session.js:20 calls pushPayload with whatever comes back from the authentication request. This will create a user model if it succeeds, but it throws an error for a failure response. The particular failure payload I'm getting is:

{
  "errors": [{
      "title": "OK",
      "detail": "incorrect_client_credentials",
      "code": 200
    }]
}

Add a `mayLogin` attribute to to grants

allow grants to include an additional permissions attribute mayLogin that can be used to determine if a user has authorization to login to the hub. This grant would be asserted after the authentication rewrite has occurred and the hub has found/created the user record based on the information from the authentication plugin.

Standardize on Ember's module imports API

This is a good new contributor issue.

There are many Ember addons inside the packages directory of this repo. We should standardize them to use the new Ember module imports:

https://github.com/emberjs/rfcs/blob/master/text/0176-javascript-module-api.md

Use cs-svg instead of cs-icon in Mobiledoc editor

Work on adding markup to Mobiledoc editor (#98) was started before cs-svg helpers were introduced, we should use those instead of cs-icon.

missing is-component helper

Using the demo app at https://github.com/ef4/emberconf-2017-demo and accessing the speakers/new route (http://localhost:4200/speakers/new), we get the error below.

Error: Compile Error: is-component is not a helper
at Helper.compile (http://localhost:4200/assets/vendor.js:69162:23)
at http://localhost:4200/assets/vendor.js:69395:26
at Array.map (native)
at PositionalArgs.compile (http://localhost:4200/assets/vendor.js:69394:104)
at Args.compile (http://localhost:4200/assets/vendor.js:69353:93)
at OpcodeBuilder.compile (http://localhost:4200/assets/vendor.js:63332:29)
at OpcodeBuilder.putArgs (http://localhost:4200/assets/vendor.js:63251:84)
at IfSyntax.compile (http://localhost:4200/assets/vendor.js:67978:17)
at InlineBlockCompiler.compileStatement (http://localhost:4200/assets/vendor.js:65928:61)
at InlineBlockCompiler.compile (http://localhost:4200/assets/vendor.js:66006:22)

Per @ef4, this component should be available via the ember-cli-is-component addon, which is a dependency of the tools package.

I'm able to eliminate this error by updating the tools' package.json:
original:

"ember-cli-is-component": "https://github.com/ef4/ember-cli-is-component#544bf7a87689015c4f54c6697093d9f2b0d7b1ea",

updated:

"ember-cli-if-component": "https://github.com/ef4/ember-cli-is-component#544bf7a87689015c4f54c6697093d9f2b0d7b1ea",

(Note the change from -is-component to -if-component).

This seems to be due to this addon having a different name in its package.json file.

Please confirm that this is just a typo in the tools package.json file, vs. an issue that needs to be addressed elsewhere.

Github Torii config mismatch

The github login component currently doesn't work, because it writes config to the torii provider github-oauth2 here, but attempts to use the unconfigured github here.

Create `buildQueryExpression` for boolean type

Right now when you use a search query that has a filter on a boolean type field, the query uses string semantics for the filtering. Specifically if { filter: { is-built-in: false } }, where the field is-built-in is a boolean type field, will only match content where there is an is-built-in field. Ideally this filter should also match on content where there is no is-built-in field as well.

Add transitionToContentPage to routing service

I've found myself writing the following snippet more than once (ok, twice :) ) to go to the page of a resource (a cardstack model):

import routingService from '@cardstack/routing/services/routing';
import { transitionTo } from '@cardstack/tools/private-api';
import { getOwner } from '@ember/application';
import { pluralize } from 'ember-inflector';

export default routingService.extend({
  transitionToContentPage(resource) {
    let routingId = resource.get('id');
    let routeName = pluralize(resource.constructor.modelName);
    let { name, params, queryParams } = this.routeFor(routeName, routingId);
    queryParams.routingId = routingId;
    return transitionTo(getOwner(this), name, params.map(p => p[1]), queryParams);
  }
});

I think it'd be valuable to add this method to the routing service as it's what users will most often want instead of calling service.routeFor, assembling all the params correctly and transitioning to the new route, none of which is obvious to do.

Index individual ethereum events

Currently for the ethereum plugin we index at the level of mappings (aka ledger entries). This means that we only see the current state of a smart contract. We should index one level deeper: specifically at the ethereum event level. This would provide insight into historical ethereum state. This means that we would create a document for each configured ethereum event for the contract. Client apps could then use server side computed's to aggregrate these events and perform more powerful analysis on the contract's history.

hub:start doesn't check exit status of hub:build

Within the hub:start command, if buildAppImage fails we still continue on and try to start the container.

If the container already exists, you may not even notice your build failed. If it doesn't, you may get a confusing error about trying to download it from a repo that doesn't exist.

Cardstack authorizer

I'd open a PR, but I've been having trouble testing with local versions due to yarn & monorepo shenanigans.
Can we add this authorizer to packages/authentication? Right now I've had to define this in my app.

import Ember from 'ember';
import Authorizer from 'ember-simple-auth/authorizers/base';

const { isEmpty } = Ember;

export default Authorizer.extend({
  authorize(data, block) {
    const accessToken = data.meta.token;

    if (!isEmpty(accessToken)) {
      block('Authorization', `Bearer ${accessToken}`);
    }
  }
});

Provide capability for ethereum plugin's indexer's Updater to perform a read for non-schema docs

Right now the ethereum plugin's indexer's Updater class can only perform a read() for schema. Update this to also work for non-schema docs: https://github.com/cardstack/cardstack/blob/master/packages/ethereum/cardstack/indexer.js#L210-L214

Missing hub dependency dag-map

1. ember new my-app --yarn
2. yarn add @cardstack/hub
3. ember serve results in:

🚀 my-app$ ember serve
Livereload server on http://localhost:49153
Cannot find module 'dag-map'
Error: Cannot find module 'dag-map'
    at Function.Module._resolveFilename (module.js:470:15)
    at Function.Module._load (module.js:418:25)
    at Module.require (module.js:498:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (/Users/aaron/dev/my-app/node_modules/@cardstack/hub/middleware-stack.js:2:16)
    at Module._compile (module.js:571:32)
    at Object.Module._extensions..js (module.js:580:10)
    at Module.load (module.js:488:32)
    at tryModuleLoad (module.js:447:12)
    at Function.Module._load (module.js:439:3)

Issue indexing content types that have differing fields from ethereum event triggers

some of the content types that we use for ethereum events don't have fields that exactly match the fields that are fired for an Ethereum event. https://github.com/cardstack/cardstack-web/issues/249

In this case, we should just lookup call the method of the contract for the content type instead of using the ethereum event to populate the document for the content type.

Should we yield raw model value in block form for cs-field?

cardstack/packages/tools/addon/templates/components/cs-field.hbs

Line 4 in efdffed

{{yield value }}

When I used the block form I was expecting to get the raw value passed along. What is your thinking of passing along {{yield (get content fieldName)}} instead of the value which could potentially include a placeholder?

Prevent postgresql plugin from indexing pgboss tables

The postgresql plugin is trying to index the pgboss tables that are used for the index queing. The simplest way to stop this would be to use a different DB for pgboss (as we do for pgsearch).

workflow: Don't use moment via global

We should remove this:

cardstack/packages/workflow/.eslintrc.js

Line 12 in 33f510f

moment: false,

ember-moment is already providing an importable module: https://github.com/stefanpenner/ember-moment#es6-moment

user loading broken in test-support-authenticator

This

cardstack/packages/test-support-authenticator/middleware.js

Line 21 in bb8e12d

(type, id) => self.searcher.get(Session.INTERNAL_PRIVILEGED, 'master', type, id)

is supposed to be stubbing the same behavior as this:

cardstack/packages/authentication/cardstack/middleware.js

Lines 37 to 46 in bb8e12d

	get userSearcher() {
	return {
	get: (type, userId) => {
	return this.searcher.get(Session.INTERNAL_PRIVILEGED, this.controllingBranch.name, type, userId);
	},
	search: (params) => {
	return this.searcher.search(Session.INTERNAL_PRIVILEGED, this.controllingBranch.name, params);
	}
	};
	}

But it has the wrong signature and so session.loadUser() doesn't work when using the test-support-authenticator.

@alexspeller this is the issue you found.

Field caption and editor not rendered for some field names

I had the following schema definition in a project:

  initial.addResource('content-types', 'songs')
    .withRelated('fields', [
      initial.addResource('fields', 'master-recording-title').withAttributes({
        fieldType: '@cardstack/core-types::string'
      }),
  (...)

To my surprise, when editing the song on its own content page, both the caption and the field editor were missing (not rendered) in the right sidebar. I then changed master-recording-title to master-title but it did not change anything. Finally, I changed it to master and then it worked fine:

  initial.addResource('content-types', 'songs')
    .withRelated('fields', [
      initial.addResource('fields', 'master').withAttributes({
        fieldType: '@cardstack/core-types::string'
      }),
  (...)

The same happened with an integer field called number-of-songs. I changed it to song-count which did not work and finally songcount, which did.

Add information on how to debug and step through hub/node code

I suspect interacting with @cardstack/hub will be many people's first time needing to step through the node portions of an ember-cli addon (myself included). It would be fantastic to include some information on how to do that, or link to a resource that walks through it.

Error when Contracts that have many events result in

I'm seeing that when a contract has many thousands of events the following error is emitted Frame size of 1403431 bytes exceeds maximum accepted frame size when we index the contract. We should do historic indexing in smaller chunks, so as not to run into this error.

Renaming project folder breaks hub

1. My app was up and running
2. I stopped the server
3. I renamed my project folder from probably-right to probably-stack
4. I started the server and saw the following error:

🚀 probably-stack$ yarn start
yarn start v0.23.4
$ ELASTICSEARCH='http://localhost:9200' ember serve
Livereload server on http://localhost:49153
(node:31081) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): Error: Failed to resolve path '/Users/aaron/dev/probably-right/cardstack/seeds/../..': No such file or directory
Error: Failed to resolve path '/Users/aaron/dev/probably-right/cardstack/seeds/../..': No such file or directory
[ ... this repeats about every second ... ]

5. I stopped the server, moved the folder back, and got it working again.

I'm not sure whether it was actually that the folder name changed, or whether it would fail if you ran a new project for the first time, but the folder had a different name than the ember project.

Removed data-sources are not cleared from search index

If you entirely remove a data-source, there is nothing to forcefully clear its content from the search index.

Editing relationships

Dump from @ef4 on Slack, before it moves into the nether.

I don’t think there’s much downside to representing the enum as a relationship. It may seem more verbose, but it allows us make it less special. You would want a clear way to query, filter, and sort the choices anyway, and if they’re real resources in their own right you already have all that for free.
We can design good UI to make it all feel natural from the user’s perspective.

the general rule right now is that any resource gets edited and saved by visiting its own canonical URL. Content that’s embedded from elsewhere isn’t edited directly. I think this is a good rule, and we can allow more contextual editing of embedded content by having something like a modal card view (imagine clicking on a piece of embedded content and the content animates to float in front of the original page, where you can edit & save it and then let it go back into place.)

Basically, there are two edit modes, change the relationship to another existing entity (or delete it altogether) or edit the current entity. The first one will require finding all valid entities of that type, which might not always be via a findAll. Both modes of editing need to be easy to understand and do from the UI side.

@cardstack/hub installation issue

I was following along with Edward's EmberConf talk and everything was going great until I tried to install 'hub'. (ember install @cardstack/hub). The install failed with the following error:

~>ember install @cardstack/hub
NPM: Installed @cardstack/hub
Unexpected identifier
C:\Users\me\Projects\anapp\node_modules\@cardstack\hub\index.js:15
  async serverMiddleware({ app, options }) {
        ^^^^^^^^^^^^^^^^
SyntaxError: Unexpected identifier
    at createScript (vm.js:56:10)
    at Object.runInThisContext (vm.js:97:10)
    at Module._compile (module.js:542:28)
    at Object.Module._extensions..js (module.js:579:10)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)
    at Module.require (module.js:497:17)
    at require (internal/module.js:20:19)
   ...

Btw, Edward uses ember add @cardstack/hub in the talk which is not an available command in my version of ember cli (2.12.2) so maybe this is the issue? Also, I'm on Win 7.

Improved data source management for test suites

Planning notes based a call with @habdelra:

The near-term plan:

1. Generic data source seeding
   • lets rename seeds to data-sources, because that's all that's really supposed to be in there.
   • then we can use seeds for a more generic version of the ephemeral store's initialModels. You would manually load seed models into your data sources (any data sources, not just ephemeral) by running a command line tool (ember hub:seed).
   • By convention, the ephemeral store would still load effectively do hub:seed automatically at boot, so it behaves the same as today.
   • no more separate dev/test/prod directories for seeds (or data sources). We're finding it better to manage the distinction via env vars.
2. Postgres-specific seeding & testing
   • add command line tools for conveniently creating and destroying databases
   • create includes migrations (though they would still also run automatically if a running indexer sees that they're needed, as they do now).
   • the seed command from above would necessarily let you seed a postgres database, as long as it already exists.
   • ember test will require the database to already exist, it will not make it for you.
   • the baseline state between tests is a database that exists and has schema (from the migrations) but it does not include any automatic use of seeds.
   • each test is responsible for its own setup and teardown to get back into the baseline state.
   • the baseline state does not include seed models automatically, but individual tests can choose to load them explicitly (and then clean them up to get back to baseline)
3. Test Support package improvements
   • we need to expose a built-in privileged user for the clientside test-support code to run as, when setting up and tearing down tests
   • the existing test-support/addon/fixtures.js API needs to be extended to support data sources that can't do arbitrary checkpointing. The ephemeral-checkpoints and ephemeral-restores data types would go away in favor a generic middleware provided by test-support that can create and restore checkpoints.
   • writers will get new hooks for creating and restoring checkpoints. Create is optional (not every data source can create arbitrary checkpoints). But you can implement restore without implementing create by supporting pre-defined checkpoints (how they are predefined is data-source specific configuration).
   • in the case of postgres, the "empty checkpoint" could be configured as a series of sql statement to run that get you into that state (mostly of the form "delete from some_table", but arbitrary sql is allowed in case your baseline state isn't truly empty).

     factory.addResource('data-sources', 'postgres').withAttributes({
       sourceType: '@cardstack/postgresql',
       params: {
         branches: {
           master: {
             migrationsDir: 'cardstack/migrations'
           }
         },
         testCheckpoints: {
           empty: [
             `delete from users`,
             `delete from batches`
           ]
         }
       }
     })
     
     
     

In the longer-term, we would like to make fully dynamic management of data-sources be a viable way of writing test suites. For example,POST /api/data-sources and DELETE /api/data-sources/1 would actually create and destroy underlying resources like postgres databases or git repositories. This would let you drive arbitrarily deep testing all from a client-side test suite. This gives you the ability to do deep end-to-end testing to prove that an assumption holds, and elsewhere do shallower/faster testing based on the assumption.

Refactor hub:indexers to no longer accept `hints`

Refactor hub:indexers to no longer accept hints. Additionally, update all plugin indexers that use hints to use the pgsearch's Batch.saveDocument() instead to index specific documents in place of using hints to as a means to identify specific documents for indexing.

Prevent editor view from appearing when there is no session

When the user logs out, the editor view (right edge) is still displayed. We should not display the card editing view when the session is not authenticated.

Add a mechanism to associate users into groups

As part of asserting grants, we should have a mechanism to associate users into groups that can be specified in grants. Probably need to handle the ability for users to be added to or removed from a group when user's attributes/relationships change.

Create a mechanism so that the hub can recognize a postgres column type as case-insensitive text

Postgres has a module for representing case insensitive text: CITEXT https://www.postgresql.org/docs/9.1/static/citext.html. We can map columns of these types to the hub's case-insensitive string type.

Assert ethereum contract model names are plural

Right now the ethereum contract model name inflections has no opinions. this should change to assert that all the content types that are derived from the contract have a plural inflection.

Postgresql should use sql defined in checkpoints for a restore, not sql defined in a restore

Right now the sql restore uses sql that are part of that restore model. Rather, the postgresql restore should look up a checkpoint, and execute the sql associated the checkpoint. The checkpoint with it's sql can be established in the config for the postgresql data source.

Allow primary interaction with fields in edit mode

When some markup is wrapped in a cs-field and we're in edit mode, an overlay is rendered above the markup. That's exactly what we want in most cases, but in some cases the underlying markup has event handlers that cannot be triggered because of this.

The example where this came up is the Table of Contents for a document being wrapped in a cs-field which prevents clicking on each chapter title to load the related chapter content on the UI.

To make this possible, we should mark such fields in some way (allowHandlers). When such a field is first interacted with in Edit mode, it should allow the interaction "to go through" (by setting pointer-events: none on the overlay. A subsequent 2nd click should trigger the current behavior, bringing up the editor for that field.

(See https://github.com/cardstack/deck/pull/91#issuecomment-420333736)

Docker image build doesn't use yarn.lock

This pattern is trying to optionally match yarn.lock, but it doesn't:

    let yarn = dir+'yarn.lock?';        // COPY errors out if a directly specified file is missing, but is ok with a pattern matching 0 files

Matching follows the rules in https://golang.org/pkg/path/filepath/#Match

A ? is not a modifier like in Javascript regex, it's a wildcard equivalent to . in Javascript regex.