Bulk update about cozy-data-system

[aenario]

Having the update as a map would be easier I think :

PUT /request/file/all/byPath/update/
{
  startkey: "/some/path/"
  endkey: "/some/path/ZZZZ"
  update: {
    lastModification: 3554232,
    author: "John"
  }
}

Or, more powerful (and similar performance: 1 request app->DS, 2X request DS->Couch) a function

PUT /request/file/all/byPath/update/
{
  startkey: "/some/path/"
  endkey: "/some/path/ZZZZ"
  update: "function(doc){
    doc.tags = doc.tags.map(function(tag){
        if(tag === 'old-name') return 'new-name';
        else return tag;
    }
    return doc
 }"
}

I agree this would be convenient and might improve performance of some cozy apps.
Not sure when we will have the time to do it

[bnjbvr]

Yes, a map would indeed be better (I wonder how I could have missed this). I'll try to add this feature if I get some time soon, this could show useful for Kresus.

[jankeromnes]

👍 This would be useful to import larger time series (for example, 15 years of currency data for ~10 different currencies is north of 25000 values).

EDIT: Actually, bulk insert would be useful for that, which also shouldn't be very different from bulk delete / update.

[jankeromnes]

FYI, according to this post about couchdb performance:

• Inserting docs one-by-one happens at ~260 docs/sec,
• Inserting docs in bulk (groups of 10000 docs) happens at ~2700 docs/sec,
• With even more optimizations (e.g. bypass HTTP and JSON conversions) it can go higher than 10500 docs/sec.

[aenario]

@jankeromnes , One thing to keep in mind is that cozy has a lot of views in couchdb. So adding many documents slow down the whole cozy. We already had some troubles with emails when importing large accounts.

So for your use case of currency, (ie. never changing values). The performance-optimal solution would be to have huge documents with, say, all data for a month and then using views to produce the index you need

function(doc){
    if (doc.docType == 'MonthOfCurrencyExchange')
          for (date in doc.dates)
                for (currency in date.currencies)
                       emit(date, currency, value);

[jankeromnes]

Interesting idea, thanks @aenario! Actually it might even be faster to store all my values in one single document, since the number of documents seems to be the bottleneck. I'll try different options.