cozy / cozy-data-system Goto Github PK

Deprecated - Data Layer for Cozy V2 Platform

License: GNU Affero General Public License v3.0

CoffeeScript 99.97% Shell 0.03%

cozy-data-system's Introduction

Cozy Data System

Little API that act as a middleware between Cozy Platform applications and data sent to the database, to the indexer and to the file system. It manages application permissions and provide helpers to make Cozy app development easier.

Install

To set it up inside your cozy instance:

# Get cozy monitor
npm install cozy-monitor -g
cozy-monitor install data-system

Contribution

You can contribute to the Cozy Data System in many ways:

Pick up an issue and solve it.
Add bulk features.
Improve mass deletion.
Write new tests.

Hack

Install CouchDB (>= 1.2.0), NodeJS (>= 0.10.0) then:

git clone git://github.com/cozy/cozy-data-system.git
cd cozy-data-system

# Load dependencies
npm install

Once datasystem is installed, run it with:

npm start # performs a node build/server.js

Or you can start it in dev mode with:

INDEXES_PATH="../search-indexes" coffee server.coffee

Tests

Run tests with following commmand

cake tests

Before submitting a pull request

Make sure the tests pass
Make sure you've built your modification:

cake tests
cake check-build
cake build

You can also use the provided hook:

cp pre-push .git/hooks/

About code coverage

The cake coverage command requires a bit of explanation. To achieve a proper code coverage, all CouchDB requests must be written in full javascript, otherwise the tests cannot work (since the coffee source is compiled with a special version).

License

Cozy Data System is developed by Cozy Cloud and distributed under the AGPL v3 license.

What is Cozy?

Cozy is a platform that brings all your web services in the same private space. With it, your web apps and your devices can share data easily, providing you with a new experience. You can install Cozy on your own hardware where no one profiles you.

Community

You can reach the Cozy Community by:

Chatting with us on IRC #cozycloud on irc.freenode.net
Posting on our Forum
Posting issues on the Github repos
Mentioning us on Twitter

cozy-data-system's People

Contributors

Stargazers

Watchers

Forkers

bleuse aenario poupotte kloadut cylwin obigroup frankrousseau paultranvan nono pkdevbox felix-lambert bnjbvr kosssi flyingrub digideskio fengweijp lesion snyk-community anatolinicolae

cozy-data-system's Issues

Error sending message if user name contains more than one space

When sending messages from the user, we create the sender address form user name, but escape it wrongly, so if for example user name contains two spaces, the sender address format is wrong and the message bounce.

Handle thumbnail creation

Thumbnails should be created when a new file is stored in the data system.

What is required:

Generate a thumbnail on image file creation
Generate a new thumbnail on attachment modification
Check for missing thumbnails and create it
Handle error case and mark the file as not previewable

@poupotte @aenario @jsilvestre What do you think about it?
.

Delete tmp file after attachment

cf cozy/cozy-photos-v2#2

The DS should delete files from the /tmp folder after attachment.

Location :
https://github.com/mycozycloud/cozy-data-system/blob/master/app/controllers/attachment_controller.coffee#L36-L55

@frankrousseau Do you want me to handle this one ?

Rename a directory doesn't rename sub items

Using the PUT /ds-api/data/merge/:id method with { name: 'newname'} on files, it works fine and rename the file without any problem.

But when doing it on folders, the folders is renamed but all the sub-content is "lost" (not linked to any visible folders).
The function may be (one of):

Recursively rename the sub files and folders
Or Throw an error if the folder is not empty

There is an advice about it on the code:
https://github.com/cozy/cozy-data-system/blob/master/server/controllers/data.coffee#L89-L90

Error: Can't set headers after they are sent

Seen in the logs (no more informations on the context):

[Fri, 22 Jan 2016 14:08:45 GMT] - GET /replication/?_nonce=1453471724944 - 403 - 7.263 ms -  242
[Fri, 22 Jan 2016 14:08:49 GMT] - POST /request/folder/byfullpath/ - 401 - 1.540 ms -  44
[Fri, 22 Jan 2016 14:08:49 GMT] - GET /replication/_changes?descending=true&limit=1 - 403 - 6.746 ms -  14
ute.js:107:5)
    at /usr/local/cozy/apps/data-system/node_modules/americano/node_modules/express/lib/router/index.js:213:24
    at param (/usr/local/cozy/apps/data-system/node_modules/americano/node_modules/express/lib/router/index.js:303:14)
    at param (/usr/local/cozy/apps/data-system/node_modules/americano/node_modules/express/lib/router/index.js:319:14)
    at param (/usr/local/cozy/apps/data-system/node_modules/americano/node_modules/express/lib/router/index.js:319:14)
[2016-01-22 14:08:20:877] error - replication | {}
[2016-01-22 14:08:20:877] error - replication | {}

http.js:690
    throw new Error('Can\'t set headers after they are sent.');
          ^
Error: Can't set headers after they are sent.
    at ServerResponse.OutgoingMessage.setHeader (http.js:690:11)
    at ServerResponse.res.set.res.header (/usr/local/cozy/apps/data-system/node_modules/americano/node_modules/express/lib/response.js:565:10)
    at ServerResponse.res.send (/usr/local/cozy/apps/data-system/node_modules/americano/node_modules/express/lib/response.js:134:12)
    at ServerResponse.res.json (/usr/local/cozy/apps/data-system/node_modules/americano/node_modules/express/lib/response.js:212:15)
    at ServerResponse.res.send (/usr/local/cozy/apps/data-system/node_modules/americano/node_modules/express/lib/response.js:122:21)
    at IncomingMessage.<anonymous> (/usr/local/cozy/apps/data-system/build/server/controllers/replication.js:143:19)
    at IncomingMessage.emit (events.js:117:20)
    at IncomingMessage.<anonymous> (_stream_readable.js:765:14)
    at IncomingMessage.emit (events.js:92:17)
    at emitReadable_ (_stream_readable.js:427:10)
[2016-01-22 14:08:49:163] error - app:error | Application is not authenticated
[2016-01-22 14:08:49:170] error - app:error | Error: Application is not authenticated
    at /usr/local/cozy/apps/data-system/build/server/helpers/utils.js:29:13
    at module.exports.checkDocType (/usr/local/cozy/apps/data-system/build/server/lib/token.js:53:14)
    at module.exports.checkPermissions (/usr/local/cozy/apps/data-system/build/server/helpers/utils.js:27:10)
    at Object.module.exports.checkPermissionsByType [as handle] (/usr/local/cozy/apps/data-system/build/server/middlewares/utils.js:65:10)
    at next_layer (/usr/local/cozy/apps/data-system/node_modules/americano/node_modules/express/lib/router/route.js:103:13)
    at Route.dispatch (/usr/local/cozy/apps/data-system/node_modules/americano/node_modules/express/lib/router/route.js:107:5)
    at /usr/local/cozy/apps/data-system/node_modules/americano/node_modules/express/lib/router/index.js:213:24
    at param (/usr/local/cozy/apps/data-system/node_modules/americano/node_modules/express/lib/router/index.js:303:14)
    at param (/usr/local/cozy/apps/data-system/node_modules/americano/node_modules/express/lib/router/index.js:319:14)
    at param (/usr/local/cozy/apps/data-system/node_modules/americano/node_modules/express/lib/router/index.js:319:14)
[2016-01-22 14:08:49:258] error - replication | {}
[2016-01-22 14:08:49:259] error - replication | {}

http.js:690
    throw new Error('Can\'t set headers after they are sent.');
          ^
Error: Can't set headers after they are sent.
    at ServerResponse.OutgoingMessage.setHeader (http.js:690:11)
    at ServerResponse.res.set.res.header (/usr/local/cozy/apps/data-system/node_modules/americano/node_modules/express/lib/response.js:565:10)
    at ServerResponse.res.send (/usr/local/cozy/apps/data-system/node_modules/americano/node_modules/express/lib/response.js:134:12)
    at ServerResponse.res.json (/usr/local/cozy/apps/data-system/node_modules/americano/node_modules/express/lib/response.js:212:15)
    at ServerResponse.res.send (/usr/local/cozy/apps/data-system/node_modules/americano/node_modules/express/lib/response.js:122:21)
    at IncomingMessage.<anonymous> (/usr/local/cozy/apps/data-system/build/server/controllers/replication.js:143:19)
    at IncomingMessage.emit (events.js:117:20)
    at IncomingMessage.<anonymous> (_stream_readable.js:765:14)
    at IncomingMessage.emit (events.js:92:17)
    at emitReadable_ (_stream_readable.js:427:10)

Exception on _bulk_docs

See this thread

[2015-12-01 15:04:30:395] ^[[31merror^[[39m - app:error | Cannot read property 'length' of undefined
[2015-12-01 15:04:30:399] ^[[31merror^[[39m - app:error | TypeError: Cannot read property 'length' of undefined
    at Object.async.each (/usr/local/cozy/apps/data-system/node_modules/async/lib/async.js:106:17)
    at Object.module.exports.checkPermissionsPostReplication [as handle] (/usr/local/cozy/apps/data-system/build/server/middlewares/utils.js:75:18)
    at next_layer (/usr/local/cozy/apps/data-system/node_modules/americano/node_modules/express/lib/router/route.js:103:13)
    at Route.dispatch (/usr/local/cozy/apps/data-system/node_modules/americano/node_modules/express/lib/router/route.js:107:5)
    at /usr/local/cozy/apps/data-system/node_modules/americano/node_modules/express/lib/router/index.js:213:24
    at Function.proto.process_params (/usr/local/cozy/apps/data-system/node_modules/americano/node_modules/express/lib/router/index.js:284:12)
    at next (/usr/local/cozy/apps/data-system/node_modules/americano/node_modules/express/lib/router/index.js:207:19)
    at next (/usr/local/cozy/apps/data-system/node_modules/americano/node_modules/express/lib/router/index.js:182:38)
    at next (/usr/local/cozy/apps/data-system/node_modules/americano/node_modules/express/lib/router/index.js:182:38)
    at next (/usr/local/cozy/apps/data-system/node_modules/americano/node_modules/express/lib/router/index.js:182:38)

Bulk update

I see there is an end-point for bulk delete, but not for bulk update. However, the db_remove_helper suggests that it shouldn't be too different to do (as a deletion seems to be actually an update, iiuc). Is there any way we could have a bulk update end-point, please?

Here's an API i imagine:

PUT /request/:type/:req_name/update
Param:
  type: the doctype name
  req_name: the name of the request
  Body {
    key: only returns document for this key
    keys: [only returns document for this array of keys]
    limit: number of documents to return
    skip: number of documents to skip
    startKey: only returns document after this key
    endKey: only returns document before this key
    update: [[key, value], [key, value]] or [key, value]
  }
  The body is fully optional.

For instance, with

body = {
  update: ['date', '1234']
}

Then the 'date' field of all values should be updated to '1234'.

body = {
  update: [['date', '1234'], ['bankAccount', '4321']]
}

Then the 'date' field of all values is updated to 1234, and the 'bankAccount' field of all values is updated to 4321.

Not sure about the naming of update, maybe values or replaceBy would be more adequate.

Prevent the DS from crashing when an app abuse the requests

So, this was caused by emails, which in some instances would ask the DS for
messages/xxx?include_docs=true&keys=[10000keys]

We can all agree this is a bug within emails, which will be fixed.
However, the DS should never crash, even if an app is being so rude 😈
Note : the crash was sometime caused by Allocation Failed (node.js error), sometime by the OOM, so we need to proactively prevent this error and not just catch & react.

2 Options :
A - Use streaming JSON Parsing / Permission management / JSON Writing so the whole request's results doesn’t need to be held in memory
B - Choose some heuristic (maybe 1000 docs at once, maybe 100Mo of results) at which point the DS should just return a 413 to the app and abort its request to couchdb.

This change is not urgent, filing as Enhancement

Avoid file creation with empty name

When making a request to POST /ds-api/data/
with:
{
docType: 'folder'
path: '',
name: ''
}

It creates a folder with no file name (which should not be not allowed).

View 'binary/byDoc' don't look to be properly created

Here is the error I got when I delete a file with the File app.

[Mon, 01 Dec 2014 00:17:20 GMT] - DELETE /data/4F2E2825-9D49-10E1-8889-0F41D26CE02B/ - 204 - 25.769 ms -  -
{ error: 'not_found', reason: 'missing_named_view' }

Allocation failed - process out of memory

Application keeps crashing with this message in logs:

FATAL ERROR: CALL_AND_RETRY_0 Allocation failed - process out of memory

Can't bulk delete objects when too many documents are returned for a given request

Overhaul request creation management

This part of cozydb should actually be in the DS :
https://github.com/cozy/cozy-db/blob/c960d01633827c59e354718817d7cbe9b4277e51/src/cozymodel.coffee#L212-L226

We need to be careful with the transition and mismatching versions of cozydb vs data-system.

Warning email / Please click here with no link.

It is because my cozy_instance has a domain set to empty string.

cozy-data-system/server/lib/encryption.coffee

Lines 34 to 35 in 89645d6

    
           if domain? 
        
               body += "Click here to login #{domain}."

Manage photo orientation

See cozy/cozy-emails#641

The latest version doesn't seem to like passwords

Hi,

To keep it short, I have this create call. The console.log sure isn't really secure, but I was working on an issue where I couldn't retrieve a supposed saved password, so I wanted to know what cozydb saved.
I have this output:

{ username: 'baboli18',
  password: '***********',
  tgc: 
   { key: 'CASTGC',
     value: 'TGT-598-rkm2FKaZBHIDLYFAORuWljLn1b2h26CVyW5ka6qlTeI3tfjeZj-cas',
     domain: 'web.isen-bretagne.fr',
     path: '/cas',
     secure: true,
     hostOnly: true,
     creation: '2015-08-24T16:29:18.975Z',
     lastAccessed: '2015-08-24T16:29:18.977Z' },
  jsessionid: 
   { key: 'JSESSIONID',
     value: 'F62053FD6C6FA9BAD8B96B6564919888',
     domain: 'web.isen-bretagne.fr',
     path: '/',
     secure: true,
     hostOnly: true,
     creation: '2015-08-24T16:29:17.800Z',
     lastAccessed: '2015-08-24T16:29:18.977Z' },
  _id: 'b359c1912482968bb7e0d84c76012f35',
  _attachments: undefined,
  _rev: undefined,
  id: 'b359c1912482968bb7e0d84c76012f35',
  docType: 'caslogin',
  binaries: undefined }

The censored "password" field is my password, which match with the valid value.

After this, I go look in the DataBrowser, the password field isn't filled. The next step in the app include a check with the said password, which seems to be undefined.
Here's the log from my DataSystem.

I updated the data-system to version 2.0.7 today, and it seems that this latest version is causing the issue. I tried with another cozydb version and the issue still happen.

If it can help, I'm running the app on an ISEN Cozy.

npm start error on ubuntu [with & w/o sudo]

Log:

[email protected] start: `node build/server.js`
npm ERR! `sh "-c" "node build/server.js"` failed with 1
npm ERR! Failed at the [email protected] start script.
npm ERR! This is most likely a problem with the cozy-data-system package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node build/server.js

Details: build/server.js is just another regular Generated by CoffeeScript,changing host or port doesn't fix it.
PS: I am only beginning to browse here and It may be early to raise this issue,but this is different from [fixable] npm error thrown at sudo npm install cozy-monitor -g,where is clearly says : No compatible version found: shelljs@'^0.3.0' or 'No compatible version found: path-extra@'^0.2.1' with sudo npm install cozy-files -g .

Improve handling of errors when sending messages

For now, messages sent from the user often use a wrong send address, displayName-noreply@domain. If the domain is not fully qualified and reachable, lots of email server reject the message. Would it be a privacy concern to use user's email address as the From address ?

If you think there may be privacy concerns, could we add another field in the preferences of the instance, where user can select the from address of this messages, defaulting to displayName-noreply@domain ?

We should also use this address in the reply-to (or sender ? or error-to ? or return path ?) message header, if application doesn't set it, so the rejection message don't get lost in some /dev/null, and users get informed when some message can't be sent.

Try to connect to redis regularly if the connection failed

Actually if the DS failed to connect to the Redis instance, it never tries to reach it again. So you have to restart your DS to take advantage of modification pub/sub.
Most of the user don't do that, so it will be better if the DS try to reconnect regurarly to the DS. @aenario could you investigate this ?

Self-hosted / Warning email when DS restart is sent from "no-replay at cozycloud.cc"

@vaz-ar mentioned that it's weird since he's not hosted by us. See https://github.com/cozy/cozy-data-system/blob/master/server/lib/encryption.coffee#L65

"Vital" doctypes should be created by the DataSystem and not the Home

Doctypes like cozyinstance, user, stackapplication, etc.

Creating them with the DataSystem can lead to a problem if the Home is started before the DataSystem.

Erreurs dans les logs

Bonjour,

J'ai des erreurs dans les logs de mon app data-system cependant mon application fonctionne très bien... (Pour chaque actions d'ajouts (ajout d'un fichier ou dans le calendrier...) dans cozy cela rajoute un bloque de log).
Je ne comprend pas comment le data-system peut fonctionner alors que dans les erreurs il est indiqué :

[2015-04-10 22:01:33:748] error - app:error | Application is not authorized
[2015-04-10 22:01:33:748] error - app:error | Error: Application is not authorized
    at /usr/local/cozy/apps/data-system/build/server/helpers/utils.js:29:13
    at /usr/local/cozy/apps/data-system/build/server/lib/token.js:43:22
    at checkToken (/usr/local/cozy/apps/data-system/build/server/lib/token.js:22:14)
    at module.exports.checkDocType (/usr/local/cozy/apps/data-system/build/server/lib/token.js:33:12)
    at module.exports.checkPermissions (/usr/local/cozy/apps/data-system/build/server/helpers/utils.js:23:10)
    at Object.module.exports.checkPermissionsByType [as handle] (/usr/local/cozy/apps/data-system/build/server/middlewares/utils.js:61:10)
    at next_layer (/usr/local/cozy/apps/data-system/node_modules/americano/node_modules/express/lib/router/route.js:103:13)
    at Route.dispatch (/usr/local/cozy/apps/data-system/node_modules/americano/node_modules/express/lib/router/route.js:107:5)
    at /usr/local/cozy/apps/data-system/node_modules/americano/node_modules/express/lib/router/index.js:213:24
    at param (/usr/local/cozy/apps/data-system/node_modules/americano/node_modules/express/lib/router/index.js:303:14)
[2015-04-10 22:01:33:896] error - app:error | Application is not authenticated
[2015-04-10 22:01:33:897] error - app:error | Error: Application is not authenticated
    at /usr/local/cozy/apps/data-system/build/server/helpers/utils.js:25:13
    at /usr/local/cozy/apps/data-system/build/server/lib/token.js:49:18
    at checkToken (/usr/local/cozy/apps/data-system/build/server/lib/token.js:24:14)
    at module.exports.checkDocType (/usr/local/cozy/apps/data-system/build/server/lib/token.js:33:12)
    at module.exports.checkPermissions (/usr/local/cozy/apps/data-system/build/server/helpers/utils.js:23:10)
    at Object.module.exports.checkPermissionsByDoc [as handle] (/usr/local/cozy/apps/data-system/build/server/middlewares/utils.js:53:10)
    at next_layer (/usr/local/cozy/apps/data-system/node_modules/americano/node_modules/express/lib/router/route.js:103:13)
    at Object.callback (/usr/local/cozy/apps/data-system/build/server/middlewares/utils.js:38:14)
    at /usr/local/cozy/apps/data-system/node_modules/cradle/lib/cradle/database/documents.js:45:18
    at Request._onResponse [as _callback] (/usr/local/cozy/apps/data-system/node_modules/cradle/lib/cradle.js:235:9)
[2015-04-10 22:01:33:917] error - app:error | Application is not authenticated
[2015-04-10 22:01:33:918] error - app:error | Error: Application is not authenticated
    at /usr/local/cozy/apps/data-system/build/server/helpers/utils.js:25:13
    at /usr/local/cozy/apps/data-system/build/server/lib/token.js:49:18
    at checkToken (/usr/local/cozy/apps/data-system/build/server/lib/token.js:24:14)
    at module.exports.checkDocType (/usr/local/cozy/apps/data-system/build/server/lib/token.js:33:12)
    at module.exports.checkPermissions (/usr/local/cozy/apps/data-system/build/server/helpers/utils.js:23:10)
    at Object.module.exports.checkPermissionsByDoc [as handle] (/usr/local/cozy/apps/data-system/build/server/middlewares/utils.js:53:10)
    at next_layer (/usr/local/cozy/apps/data-system/node_modules/americano/node_modules/express/lib/router/route.js:103:13)
    at Object.callback (/usr/local/cozy/apps/data-system/build/server/middlewares/utils.js:38:14)
    at /usr/local/cozy/apps/data-system/node_modules/cradle/lib/cradle/database/documents.js:45:18
    at Request._onResponse [as _callback] (/usr/local/cozy/apps/data-system/node_modules/cradle/lib/cradle.js:235:9)

Merci d'avance de vos éclaircissements.

Server crash: stream has no method setHeader

Data system of a user's instance keep crashing, here are the logs:

/usr/local/cozy/apps/data-system/build/server/controllers/attachments.js:86
        stream.setHeader('range', req.headers['range']);
               ^
TypeError: Object #<IncomingMessage> has no method 'setHeader'
    at /usr/local/cozy/apps/data-system/build/server/controllers/attachments.js:86:16
    at callback (/usr/local/cozy/apps/data-system/build/server/lib/downloader.js:47:7)
    at ClientRequest.<anonymous> (/usr/local/cozy/apps/data-system/build/server/lib/downloader.js:81:20)
    at ClientRequest.g (events.js:180:16)
    at ClientRequest.emit (events.js:95:17)
    at HTTPParser.parserOnIncomingClient [as onIncoming] (http.js:1689:21)
    at HTTPParser.parserOnHeadersComplete [as onHeadersComplete] (http.js:122:23)
    at Socket.socketOnData [as ondata] (http.js:1584:20)
    at TCP.onread (net.js:528:27)

Avoid file creation with file already used

When making a request to POST /ds-api/data/
with:
{
docType: 'folder'
path: '',
name: 'my_folder'
}

It allows me to create a folder with the same name of a folder in my root directory (it shouldn't be allowed).

Do not alow duplicates on doctype TAG

Currenlty, we can create more than one tag with the same name. This would normally not happens, as long as tag name is its user primary identifier.

Reproduce
In Contacts app:

add a tag (e.g cozy) to a contact
switch to another contact
add a tag to this new contact, but do not use the autocomplete suggest (type cozy in plain in the input field)
=> CouchDB now contains 2 docs tag with the same name.

Expected
DS should detect the duplicate entry and fallback to the existing one (so name acts as a primary key).

Does every request declaration leads to a query reindexing?

When I restart my apps, sometimes I notice slowness. I was wondering if redeclaring requests doesn't provoke some useless operations.

Need a way to send an email from the user to the user

Bug: Mails sender should not be the registered mail

Mails from cozy are send with the user email.
It should be an address like [email protected]
Maybe norepy@ or anything else

Actually it may cause :
Spam false positive detection for this address. Even on email not send by the cozy.

What I propose may enhance :
Spam Score should be higher because the mail sender match the dns records

It should be a feature to add a reply-to the cozy owner email (But it send out of the cozy the owner email over the internet, with all well known emails leaks)

Generate thumbnails for video files

On IRC, a user was talking about the Files addon that display a gallery of photos inside a folder, and said it would be great if the gallery could also show thumbnails for videos.
I wonder if we could create such thumbnails client-side with HTML5 API, or if we need something like ffmpeg/avconv on server side.

Sending email with Cozy's domain

Cf @LucsT concerns about using a domain that is not owned by the Cozy to send an email.

See http://www.nodemailer.com/docs/messages to add the Cozy's email as "replyTo" option.

Enhance request API.

In the request API, the HTTP method PUT is used either to manage the CouchDB request code and to delete results of the CouchDB request.
I would have used PUT for code management and POST for results handling.
Here is what I suggest :

'Request code' handling :
PUT /request/:req_name/
DELETE /request/:req_name/

'Request results' handling :
POST /request/:req_name/
POST /request/:req_name/destroy

Streaming upload

We do stream from couch to the app on download (cf. https://github.com/mycozycloud/cozy-data-system/blob/master/server/controllers/binaries.coffee#L72)

We should also stream on upload. For that we will need to drop express.bodyParser(), which is deprecated and replace it depending on the routes with express.json() and multiparty.

Sort on "Number" values

In requests on the database, numbers are not returned in the right order.

Fine-grained permission control through id

(following @Gara64's issue #204)

In order to share files between cozy we would need to be able to define fine-grained permissions based on the id of a document instead of the, more generic, docType.

The reason behind this is that with the actual configuration if a user wants to share, for example, an event from his calendar the sharing process needs access to the "event" docType, thus to all events. We think this behavior is too permissive (for our purpose) and might lead to unwanted access to other data.

Improve error handling

The problem occurs when an error is twice passed to the error constructor.

for instance

funA = (callback) ->
     callback new Error('hello')
funB = (callback) ->
     funA (err) -> callback new Error(err)

The error's message become [Object object] and is not usable.

Best fix would be have a handleError function like :

errorHandler = (err) ->
    return err if err instanceof Error
    return new Error(err) if typeof err is 'string'
    # also handle object error with a reason attributes (couchdb style)

Then we should do everywhere :

 return callback errorHandler err if err

'Your Cozy has been restarted' email is cryptic and scary

I have received the message below several times and my perception as a user is not great.

1 - I have no idea what are the implications
2 - the wording makes me anxious: My stuff has been deactivated for security reasons? Why? Did I lose data in the process?

Suggestion:

1 - rephrase the email (I'm happy to help once I understand the message to convey)
2 - link to a FAQ topic in the forum that is longer and more user friendly and offers human interaction if needed).

--Tristan

Hello,

Your Cozy has been recently restarted.
For security reasons, a restart disables encryption and decryption.
Some features of your applications are therefore desactivated.
They will be reactivated automatically when you will log into your Cozy
instance.Click here to login tristan-2.cozycloud.cc.
Cozy Team.

P-S: If you have any question, let us know at [email protected] or in
our IRC channel #cozycloud on freenode.net.

Unable to send email if domain contains a port number

See this thread.
We use the instance domain as sending domain. So if it contains a port number, the sending domain will be refused by email servers.

crapy update then crapy re-install, crash on data-system step

I'm stuck on data-system install

test@cubietruck:~$ sudo cozy-monitor install data-system
info - Install started for data-system...
error - An error occured:
error - Install failed for data-system.
[Error: data-system CANT START]

Is there a workaround ? I cannot use prepared system image, 've got other
dependencies already installed and configured :(

What I did

backup my datas
try normal update using web interface which (fail)
try normal update using command line tool which (fail)
decide to remove everything an re-install cozy using fabric script (fail)
try to add package repo, but didn't get cubietruck arch package (fail)

additional informations :

platform : cubietruck
test@cubietruck:~$ lsb_release -c
Codename: trusty

Mass deletion triggers infinite "doctype.delete" events

In the Files application, I tried to use the mass deletion API (remove all the results of a view) but it triggers "file.delete" events infinitely. I'm not sure if it's due to the API itself or the way we handle delete events in the Data System.

When error "master key and slave key don't exist" is raised encrypted field not visible

I have tons of these errors in ths logs :

error - lib/encryption | [decrypt]: master key and slave key don't exist

And no more encrypted data (password) in my DB.

I cannot understand the reason of this error.

Restarting the data-system does not help.

Server crash when a document has no view

Seen on the server of @kosssi. Seems related to mobile replication. Don't know if it is a “normal” case, but I'll just add some defensive code to prevent this crash.

TypeError: Object.keys called on non-object
    at Function.keys (native)
    at /usr/local/cozy/apps/data-system/build/server/lib/request.js:245:41
    at /usr/local/cozy/apps/data-system/node_modules/async/lib/async.js:111:13
    at Array.forEach (native)
    at _each (/usr/local/cozy/apps/data-system/node_modules/async/lib/async.js:32:24)
    at Object.async.each (/usr/local/cozy/apps/data-system/node_modules/async/lib/async.js:110:9)
    at /usr/local/cozy/apps/data-system/build/server/lib/request.js:244:24
    at recoverDocs (/usr/local/cozy/apps/data-system/build/server/lib/request.js:85:12)
    at Object.callback (/usr/local/cozy/apps/data-system/build/server/lib/request.js:82:14)
    at /usr/local/cozy/apps/data-system/node_modules/cradle/lib/cradle/database/documents.js:52:18

TypeError: Object.keys called on non-object
    at Function.keys (native)
    at removeEmptyView (/usr/local/cozy/apps/data-system/build/server/lib/request.js:191:16)
    at /usr/local/cozy/apps/data-system/build/server/lib/request.js:253:22
    at Object.async.each (/usr/local/cozy/apps/data-system/node_modules/async/lib/async.js:107:20)
    at /usr/local/cozy/apps/data-system/build/server/lib/request.js:248:26
    at /usr/local/cozy/apps/data-system/node_modules/async/lib/async.js:111:13
    at Array.forEach (native)
    at _each (/usr/local/cozy/apps/data-system/node_modules/async/lib/async.js:32:24)
    at Object.async.each (/usr/local/cozy/apps/data-system/node_modules/async/lib/async.js:110:9)
    at /usr/local/cozy/apps/data-system/build/server/lib/request.js:244:24

Server error: TypeError: Cannot read property 'docType' of undefined

Seen in server logs:

[2016-01-04 22:35:20:433] info - helper/db_feed | Publishing usage.application mon-telephone
[Mon, 04 Jan 2016 22:35:20 GMT] - GET /replication/_local/0Asqf.FrJMMnLNIRgJptvw%3D%3D?&_nonce=1451946918585 - 200 - 5.339 ms -  522
[2016-01-04 22:35:20:480] info - helper/db_feed | Publishing usage.application mon-telephone
[2016-01-04 22:35:20:482] info - helper/db_feed | Publishing usage.application mon-telephone
[Mon, 04 Jan 2016 22:35:20 GMT] - POST /replication/_bulk_docs?_nonce=1451946918624 - 201 - 6.457 ms -  67
[2016-01-04 22:35:20:575] info - helper/db_feed | Publishing usage.application mon-telephone
[Mon, 04 Jan 2016 22:35:20 GMT] - GET /replication/_local/0Asqf.FrJMMnLNIRgJptvw%3D%3D?&_nonce=1451946918728 - 200 - 4.944 ms -  522
[2016-01-04 22:35:20:694] info - helper/db_feed | Publishing usage.application mon-telephone
[Mon, 04 Jan 2016 22:35:20 GMT] - POST /replication/_revs_diff?_nonce=1451946918846 - 200 - 7.105 ms -  92

TypeError: Cannot read property 'docType' of undefined
    at Object.callback (/usr/local/cozy/apps/data-system/build/server/middlewares/utils.js:80:43)
    at /usr/local/cozy/apps/data-system/node_modules/cradle/lib/cradle/database/documents.js:45:18
    at Request._onResponse [as _callback] (/usr/local/cozy/apps/data-system/node_modules/cradle/lib/cradle.js:232:20)
    at Request.self.callback (/usr/local/cozy/apps/data-system/node_modules/request/request.js:368:22)
    at Request.emit (events.js:98:17)
    at Request.<anonymous> (/usr/local/cozy/apps/data-system/node_modules/request/request.js:1219:14)
    at Request.emit (events.js:117:20)
    at IncomingMessage.<anonymous> (/usr/local/cozy/apps/data-system/node_modules/request/request.js:1167:12)
    at IncomingMessage.emit (events.js:117:20)
    at _stream_readable.js:944:16

Returns 400 error when no login is given in the device creation controller

Request Delete do not trigger Realtime events

We should send 'type.destroy', id near this line :

https://github.com/mycozycloud/cozy-data-system/blob/master/app/controllers/request_controller.coffee#L75

DB Creation

When I start data system, I have to run it twice until server starting works well. The first time it displays the following stacktrace:

Railway server listening on port 7000 within development environment
database does not exists.

cozy-data-system/node_modules/cradle/lib/cradle.js:201
    callback(null, self.options.raw ? body : new cradle.Response(body, res
    ^
TypeError: undefined is not a function
    at Request._callback (cozy-data-system/node_modules/cradle/lib/cradle.js:201:9)
    at Request.callback (cozy-data-system/node_modules/request/main.js:120:22)
    at Request.<anonymous> (native)
    at Request.emit (events.js:70:17)
    at Request.<anonymous> (cozy-data-system/node_modules/request/main.js:633:16)
    at Request.emit (events.js:67:17)
    at IncomingMessage.<anonymous> (cozy-data-system/node_modules/request/main.js:595:14)
    at IncomingMessage.emit (events.js:88:20)
    at HTTPParser.parserOnMessageComplete [as onMessageComplete] (http.js:130:23)
    at Socket.socketOnData [as ondata] (http.js:1288:20)

Creation of a sharing controller

In order to implement the cozy-to-cozy sharing, we need to create a sharing controller.
As a first iteration, this controller could be conceptually split in 3 parts :

Creation (sharing creator) : creates a Sharing document including the sharing metadata (recipient urls, description, docids, etc), and sends a sharing proposition for each recipient (could be considered as a Cozy notification).
Validation (sharing recipient) : called when a recipient accepts or denies a sharing proposition. If accepted, it creates a new Access document for this sharing. The answer and the credentials (if any) are sent to the sharing creator.
Answer (sharing creator) : depending on the recipient's answer, saves the credentials and triggers a CouchDB replication, or removes the recipient from the Sharing document.

Handle devices through the DS

There are a few problems caused by the fact that devices are not treated as apps :

devices are not subjected to permissions
devices's views are not subjected to the request-conflict system, which lead to subtle bug.

Server crash on `RangeError: length > kMaxLength`

The server crash a few minutes after startup with the following message:

buffer.js:194
      this.parent = new SlowBuffer(this.length);
                    ^
RangeError: length > kMaxLength
    at new Buffer (buffer.js:194:21)
    at Function.Buffer.concat (buffer.js:495:16)
    at BufferList.copy (/usr/local/cozy/apps/data-system/node_modules/request/node_modules/bl/bl.js:113:21)
    at BufferList.slice (/usr/local/cozy/apps/data-system/node_modules/request/node_modules/bl/bl.js:88:15)
    at BufferList.toString (/usr/local/cozy/apps/data-system/node_modules/request/node_modules/bl/bl.js:155:15)
    at Request.<anonymous> (/usr/local/cozy/apps/data-system/node_modules/request/request.js:1199:36)
    at Request.emit (events.js:117:20)
    at IncomingMessage.<anonymous> (/usr/local/cozy/apps/data-system/node_modules/request/request.js:1167:12)
    at IncomingMessage.emit (events.js:117:20)
    at _stream_readable.js:943:16

Binary cleaning

I think I have more binaries than files, photos and attachments stored. We should thing about a way to clean unused binaries or fix what already exists.