michael / hub Goto Github PK

So the first thing we'd need, is a service that receives, stores and renders Substance Documents.

1. Store multiple versions (snapshots) of a document using the Substance App.

POST /documents

A post service that takes a substance document as the body. The substance doc includes an id field, so we can either create a first or subsequent version in the hub database.

This is how the document looks like:

{
  "id": "some-doc-id",
  "properties": {},
  "nodes": {
    "section:1": {
      "content": "Hello world!",
      "id": "section:1",
      "type": "section",
      "prev": null,
      "next": "text:2"
    },
    "text:2": {
      "content": "Hey there.",
      "id": "text:2",
      "type": "text",
      "prev": "section:1",
      "next": null
    }
  },
  "annotations": {
    "suggestion:1": {
      "id": "suggestion:1",
      "type": "suggestion",
      "node": "text:2",
      "pos": [
        0,
        3
      ]
    }
  },
  "comments": {
    "comment:a": {
      "id": "comment:a",
      "node": "text:2",
      "content": "A way of saying hello.",
      "annotation": "suggestion:1"
    }
  },
  "head": "section:1",
  "tail": "text:2"
}

1. Render latest version

GET /some-doc-id

HTML-rendered version of latest document version

1. Access JSON version

GET /some-doc-id.json

Technology:

• Node.js/Express - perfect
• I was considering PostgreSQL as a database backend for the hub, since we can easily ensure referential integrity (e.g. users, networks, network subscriptions etc) this client seems to work well and is actively maintained (https://github.com/brianc/node-postgres)
• We stopped using SASS/Less CSS-Precompilers for various reasons, but if want to keep using it, it's really no problem.

We need a better way to coordinate breaking changes on the API side. At least for me it is quite hard to keep up with a number of API changes that need to be reflected on the composer side.

My suggestion: Let's try to apply API changes in smaller increments, so we can nail them one by one.

Proposed workflow:

1. Suggest an API change using a GitHub Issue. E.g. POST /register -> POST /users
2. Coordinate implementation, so the change is reflected in both repos (hub, composer)
3. Close Issue

Thoughts?

I find myself disliking the Jade template language... ;) so I wonder if we shouldn't switch to EJS instead.

• we can mix with existing html (i don't like the step of converting html to prooper jade code)
• we use the same templating as on the client-side (Composer)
• I like being able to write javascript code right in the templates. Sure we need to take care we don't put logic into our templates but that's more of a matter of applying good practices, I think it doesn't justify for introducing a new language.

What's your opinion on that, @yuchi ?

We've got 4900 registered Substance.io users. I'd like to send them an email asking for help and point them to our campaign.

First off, good idea? 2nd, what should the email sound like?

Can you add to the README a quick tutorial how to run the tests?

I wonder if we should moderate /explore, and just show some documents that are marked as featured. I guess it's important to have some quality content that people see as "examples".

Do you guys know any good open commons text repositories? We could just grab some texts from such sources and publish using substance.

In house solution? @vectorsize pointed me to very interesting packages.

Let's start the discussion

We'll create .md files in a /blog directory. At startup the application will render the files and store the to serve them correctly.

We need to think about those things at some point.

https://tent.is/terms

It'd be nice to have a just small selection of books (not copyrighted) that we could publish on substance.io so people have something to dig into.

E.g.
http://www.gutenberg.org/browse/scores/top

Ideally those books include high quality imagery so we can show off Substance image support as well.

This is currently stubbed, so you can login with any username/password combination ;)

I've implemented login/signup for in the Substance Composer, so you can check the latest version from (build from master).

@yuchi Let me know if you have some time during the next two weeks to help out with securing/optimizing the authentication/registration API. I just put everything in place and it works. We are actually quite close to Alpha-1. :)

Check: http://substance-hub.heroku.com

You can specify your local Hub URL here:

The pused code seems to be out of sync, somehow a table documents in the db is missing and this version of setup.js that was supposed to create it is totally gone, we need to have a working codebase in one of the branches so that each of us can work on it.

@yuchi could you please take a look at this?

ATM token authentication only populates the user field. Could be cumbersome.

One of the most crucial features that is still missing is collaboration. It should be rather easy to add real collaboration to our current stack, so people can invite other to work on their document.

On the Hub:

• Create table collaborators which is just a mapping user to document
• Adapt API (/users/michael/documents) so users see their own documents plus those docs where they collaborate.

On the Composer:

• Autosync (we should sync automatically so it's less likely to run into conflicts when multiple users are editing the same document)
• Collaborators panel (so you can easily invite new people)
• Optimize dashboard (we need to visually separate your own docs from docs where you're a collaborator)

General thoughts:

• Old Substance supported invitation by email (using a temporary TAN code that needs to be confirmed by the invited user, which may not yet have a substance account), it was fairly complex to implment, so maybe we wanna start by inviting existing substance users.
• Notification system: We sort of need a notification system in the long run. There could be a message reading for instance: "Michael invited you to collaborate on his document 'Substance'".

I guess it's rather cheap so maybe this should be part of the 0.2.0 release. What do you think?

• Create views for administration login
• Create code structure
• Define tools for interface? (don't kill me, but what about bootstrap?) (see #34)
• Create main views
• Prepare actual sections: users, pubblications... This is for you @michael :)

Please @yuchi as we spoke earlier, I will need a working CURL command to see if the erro comes from the hub or the Composer. just leave it on a comment here.

Grazie mille!

Depends on #33

BTW I do love the difference between GitHub style and GitHub Enterprise Admin styles.

At this moment I cannot fully reproduce the flow of a pubblication (for example) and I cannot guarantee to @vectorsize that APIs are working right (related to #38).

For this reason I'm going to create tests. I need @michael to implement the flow tests, btw.

Probably I'll use visionmedia's supertest

Substance Unsubscribers:

Mick [email protected]
Arie Lakeman [email protected]
Justin Mazzi [email protected]

@yuchi We need a reset password functionality rather sooner than later.

API suggestion

POST /user/reset_password

POST data:

{
  "user": "michael", // or email address optionally
  "client_id": "...",
  "client_secret": "..."
}

This sends an email to the provided email adress containing a password reset link.

In this case, I guess it's okay to provide a link to a webpage where the user can pick a new password. Then he needs to relogin with his new password using the Composer.

• Remove unused views
• Switch structure
• Switch every view

@vectorsize, you're welcome!

Depends on #35

Now that I'm tackling images I started wrapping my head around how we'd handle cover images, which can be specified per doc (optional).

Cover images are intended to serve for the following purposes:

• Attention (eye) catcher (e.g. on the Hub's document listing, but also on the doc itself)
• High resolution (I guess we'd like them to be high resolution so we can consider them a book cover. We need to think about nice styles when we display such documents on the hub.
• Within the composer they could work like cover images on facebook (providing a "add cover" link in the document title/abstract section.

I'll add some design sketches here. The display part for document listings can become tricky, since we need to be able to display documents /w cover next to document those that don't have one.

I'd see this as a higher priority since it will take time until people adapt it. So if we could just agree on the size I could start implementing them.

Slightly too obtrusive (too many effects) but I kind of like how the document + story cover is layed out here: http://www.nytimes.com/projects/2012/snow-fall/#/?part=tunnel-creek

We should take the change and take some time to organise better our codebase for the 0.2.0 release, before we look into implementing new features.

Here's an attempt of most important features I see for the 0.2.0 release

Composer

• Improved build system (@oliver----)
• Plugin System (@michael, @vectorsize)
• New Content Types (images, links)
• Support Networks on the Composer side

Hub

• Code cleanup, can we dump legacy code (should we do the switch to EJS)
• How can we have a better data layer (still against using an ORM as writing SQL directly has a lot of advantages imho)
• DB Migration system (in house, declarative but not to far away from sql statements)
• Focus on the presentation layer (optimize how documents look like and make reading a doc on Substance.io a real pleasure)
• Networks (yeah this plays into the usability discussion)

Will update this issue as thoughts come up. Let's have the discussion below. In general.. let's polish before looking into new features? Do you guys agree?

Depends on #35

This is mostly a refactor to structure the next API features.

Just recorded a little screencast... http://ma.zive.at/Composer-Screencast.mov

@vectorsize can you use that to tweak our campaign video?

I'm leaving the master branch for the oauth branch, to implement all the infrastructure for oauth.

Once done we'll merge to the visual changes I see you (@michael) are doing. See you soon on master :)

Is it ok?

I was about to integrate the API changes to the composer.

Here's some troubles I just ran into.

curl http://localhost:3000/api/v1/users/michael/documents/statuses

this route was matched

/users/:username/documents/:document

instead of this

/users/:username/documents/statuses

I could fix that by changing the order. However we should find a better way to distinguish such routes, because what if someone used the id "statuses" for his document?

One of the reason why I don't like REST so much... ;) Any ideas how we could solve it without ruining the nice url scheme?

So the other day I spent some design hours sketching some design improvements for the hub. I will use this issue to paste the pictures here for ideas and discussions.

Please see fullscreen
…not sure about the colors yet

Home page

Blog post

Networks

@michael the first who want to tackle this will auto-assign to himself the issue.

Actually errors are manually wrapped to errors.Unauthorized in any case.

This is problematic because we lose the error context ("Authentication Failed", ok, but why?)

In my crusade for consistency and code organization, I did a huge refactor of the APIs. Now they do look good, but are completely different, have a look 638582d

Michael, you'll need to change the Composer calls... sorry!

• Intro Alpha-1
• Explaining features
• What's working already
• What's missing in the current app (caveats)
• What are the next steps?

maybe less technical.. but with references to interior.substance.io.

Additional fields:

• location (string)
• company (organization ?)
• website

@yuchi what's the data property for? I mean we could just store JSON in it, holding additional properties. But then we can't use SQL-Indexes.

Would allow us to implement fulltext search on the client \o/. However, not a priority yet, assigning to "Pipedreaming".

https://github.com/reyesr/fullproof

This is a big challenge.

The HTTP Link Header should be sent. Queries in data accessors must accept limits and offset (of course), db.many should take into account the limit and return specialized information.

// faked syncronous calls

function userDocuments (username, limit, offset) {
  if (limit == null) {
    limit = db.DEFAULT_LIMIT;
    offset = 0;
  } else if (offset == null) {
    offset = 0;
  }

  // attion to the `+ 1`!
  var results = db.query(USER_DOCUMENTS_SQL, limit + 1, offset);

  var count = results.length;

  var response = {
    results: results.slice(0, limit);
  };

  if (count > limit) {
    response.next = {
      limit: limit,
      offset: offset + limit
    };
  }

  if (offset > 0) {
    response.prev = {
      limit: limit,
      offset: Math.max(0, offset - limit)
    }
  }

  return response;
}

An augmented array could also act as an array for our internal use.

Sorry if I put it here on the /hub, probably should be on /composer.

This document:

http://substance-hub.herokuapp.com/documents/35802e700ee0a9294bfbe3b2c7f3bc85

One of the paragraphs is full of   instead of real spaces.

I'd like to implement a proper logout. However I'd need an API endpoint to destroy an existing authorization. Probably by just sending the token that should be invalidated?

DELETE /api/v1/authorizations/TOKEN

Add validations on the serverside.

• check username format, email format, etc.
• check if username exists
• return proper error object like {errors: [{field: "username", "message": "Username already exists"}]}

I have not looked into the passport stuff yet. Maybe this offers some functionality.

@yuchi do you have time to take care of this?

ATM every request will download the assets (the separator/hr image for example).

We must ASAP configure the app to send decent Cache Headers.

As with refs we should have a way to detect local or remote metadata changes and have them synced appropriately.

http://substance.io/michael

should list all published documents of user "michael".

Maybe even add a user profile. Then we need to extend our user model.

This is a response to the "next steps" discussion we had via email.

Backlog for the 0.1.0 Alpha release:

1. Substance API documentation (@michael, @yuchi)
2. Overhaul interior.substance.io as there's some legacy information we should get rid of (@michael, @vectorsize , @yuchi)
3. Substance Store documentation (@oliver----,@michael)

.. more to come.

Related to username/pwd prompt problem.

substance/substance-legacy#264 (comment)

High, high priority

Let's propose 3 possible start dates (Thursdays):

• April 18th
• April 25th
• May 2nd

Location:

Linz

We can use the quasioffice for working, also we're flexible regarding timing. Victor and Pier can crash our couches/guest rooms.

Rent a space somewhere

We could rent a lodge or something somewhere, so we can all stay at one place.

I'll post options as they come to my mind here. Regarding possible dates, I can do all three. :)

application/substance.v1+json does not work.