cloudant / cdtdatastore

Cloudant Sync iOS datastore library.

Cloudant

Please use https://github.ibm.com/cloud-docs/Cloudant for contributions. Pushes to https://github.com/ibm-cloud-docs/Cloudant are only allowed by members of documentation team.

Build & CI

Jenkins

Jenkins is used to publish docs to staging and to production. The setup is managed by IBM Cloud Documentation team.

https://wcp-ace-docs-jenkins.swg-devops.com/job/Docs-build/job/Docs-build-Cloudant/

Travis

Travis is used to keep github.com and github.ibm.com repositories in sync. The setup is managed by Cloudant infra team.

https://travis.ibm.com/Bluemix-Docs

Documentation

Please use https://github.ibm.com/cloud-docs/Cloudant for contributions to the documentation. Pushes to https://github.com/ibm-cloud-docs/Cloudant are only allowed by members of the documentation team. See Documentation Update Process for instructions.

If you are not an IBM employee and want to make a documentation contribution, go to the IBM Cloudant documentation and click Feedback on the page where you want to comment.

cdtdatastore's Issues

Creating a cordova plugin for CDTDatastore

I notice that you are using cocapods for dependency management. Is it possible to create a cordova plugin for CDTDatastore without using cocapods?

Still some non-CDT prefixed items in logging.md

There are still some methods/constants missing their prefix in logging.md.

CDTDatastoreChangeNotification mangles userInfo keys "winner" with "source"

in this commit, we see the following diff hunk:

+    if (nil != nUserInfo[@"source"]) {
+        userInfo[@"winner"] = nUserInfo[@"source"];
+    }

Obviously a typo, and should be:

+        userInfo[@"source"] = nUserInfo[@"source"];

CocoaPods is a requirement to build

Cocoa Pods seems really cool, but what about distributing your code the good ol fashioned way? I can't build this project without Cocoa Pods, which seems like a bad idea since many people may not want this software to dictate whether or not they can use your software.

My suggestions:
List required frameworks/libraries in the documentation.
Include non apple frameworks/libraries in the project.
Build binaries that can be simply dragged and dropped into a project (Couchbase Lite does this).

Sorry for all the whining, but I'm using Couchbase Lite right now and wanted to try out your framework but this CocoaPod thing was an obstacle rather than helpful as you intended.

Make Replication a one liner

The aim here is to go from:

CDTReplicator *replicator = [datastore pushReplicationTarget:remoteDatabaseURL 
                                                withDelegate:self
                                                       error:&error];
[replicator start];

to this:

[datastore pushReplicationTarget: remoteDatabaseURL completionBlock: ...];

The work here is therefore to work out how:

• to call replicator.start from the pushReplicationTarget:completionBlock: method.
• the arguments to the callback (at least some status and error, like CloudKit).
• to encapsulate a delegate which calls the callback.

Improve error when hitting device disk capacity

When the device disk is full the developer will ultimately see a generic "500 Internal DB Error", which is indistinguishable from other errors that can produce the same response. FMDB will print something to the log (NSLog) about it, which seems to be the only way that a developer will know it's a capacity limiting error (assuming that FMDB/sqlite3 reports this error).

Throughout our code we use the FMDatabase -executeUpdate method, which doesn't capture any errors. It only returns a YES/NO. We could migrate to use -executeUpdate:withErrorAndBindings in order to capture FMDB errors and propagate them appropriately.

FMDB/sqlite supports disk full error codes, it is 13 - SQLITE_FULL (http://sqlite.org/rescode.html#full), we should make use of it to provide an improved error.

Remove Legacy test cases

Legacy Test cases for MYUtilites should be removed, an example of the test cases to be removed can be found here.

Can't query embedded document property

I have a document looking like the following: http://pastie.org/private/kmdctsbhgcxx8g9m2msipg
I want to query the documents given a concrete type and uriList.source.

To be able to query this, I create the indexes for those properties as following:

[self.dataStore ensureIndexed:@[@"type"] withName:@"docType"];
[self.dataStore ensureIndexed:@[@"uriList.source", @"uriList.url"] 
                     withName:@"documentUriList"];

So, I create the query:

CDTQResultSet *result = [[self.couchDBProvider getDataStore] 
    find:@{@"type": type, @"uriList.source": source} 
    skip:0 
   limit:0 
  fields:nil 
    sort:nil];

But get no result at all.

Seems that the problem is that uriList is an array of jsons and this is not managed

'TD_Database:openWithEncryptionKeyProvider:': Rework this method so we can rescue failures during migration

Although we exit the method when there is an error while updating the version, on other cases we update the database but we do not check the result, for example:

            while ([attachmentKeys next]) {
                NSData *keyData = [attachmentKeys dataNoCopyForColumn:@"key"];

                [TD_Database generateAndInsertFilenameBasedOnKey:*(TDBlobKey *)keyData.bytes
                                intoBlobFilenamesTableInDatabase:db];
            }

No concurrency control for operations in database files: 'open', 'close' & 'delete'

Once we create (and open) a FMDatabase using the methods in TD_Database, all operations in the database are executed in a FMDatabaseQueue, i.e. concurrency is managed through this serial queue. However, when the methods in TD_Database to open, close and delete one of these databases have no concurrency control at all, so it is possible to delete the database from different threads with undefined results.

This bug was first detected and commented #112 (comment) and #112 (comment).

TDPusher only writes checkpoint document at completion, rather than as batches are written

The last sequence value isn't written to the remote checkpoint doc from time-to-time by the push replicator. It only writes the lastSequence value at the end of replication, when TDReplicator -stopped is called. This is okay as long as the replicator exits gracefully. However, if the code crashes completely, then all push replication progress will be lost.

The push replicator only calls TDReplicator -saveLastSequence once, at the end. It doesn’t even try during replications. whereas, the pull replicator does make attempts at times during replication.

Remove Exposure of TDStatus Errors

Exposure of TDStatus errors should be eliminated to do this the following steps should be followed:

1. Use a single error domain across the CDT parts of the library -- CDTDatastoreErrorDomain.
2. Devote a block of 1000 error codes to each subsystem
   (replication, attachments, CRUD, indexing).

This allows the developer to see, for example, 1007 and know it's a CRUD
error, and we can expand the error codes produced by a subsystem (up to
999 codes, anyway).

1. Audit each error site, and decide what the most useful error is, and
   whether it needs a unique code.
2. Wrap TD Errors.
3. Document all our error codes properly. Using a single domain and
   declaring them a single header.

Geospatial Query Support

Hi,

I've looked in the source but cannot see support yet for geospatial queries. I have an existing project that would be a good fit for CDTDatastore but need to perform a few queries based on the location of a user.

Essentially I need to perform two queries:

1. Get documents within a rectangle (north, east, south and west)
2. Get document were the coordinates are closest to the user

I wonder if anyone has any ideas on how I could achieve this with CDTDatastore. One idea I had was to build a separate Sqlite DB with the doc._id and coordinates but I'm not sure if thats my only option.

Many Thanks

Oliver

Replication Acceptance doesn't clean up after itself.

Replication Acceptance tests do not remove the databases from the remote server, this causes issues when another test expects a clean server instance.

Remove remaining instances of networking not using `NSURLSession`

We need to ensure there is no networking remaining that is not using NSURLSession (see this).

A cursory search found two instances of NSURLConnection:

CDTDatastore/CDTDatastore/touchdb/TDMultipartUploader.m

Line 54 in c407be8

- (NSInputStream *)connection:(NSURLConnection *)connection

CDTDatastore/CDTDatastore/touchdb/TDRemoteRequest.m

Line 249 in c407be8

- (void)connectionDidFinishLoading:(NSURLConnection *)connection

Make sure CDTDatastore skips documents with empty ids during replication

There was a bug in CouchDB that would allow empty doc ids to be generated, since this is no longer valid, CDTDatastore should skip empty document ids during replication if encountered.

-CDTReplicator:replicatorStopped:n should set error

When this method sets the current state to CDTReplicatorStateError it should also set the error object by re-wrapping the underlying TDReplicator error.

Storing NSNumber "inifinity" value causes entire body to be dropped

The following unit test fails (add to Tests/Tests/DatastoreCRUD.m to run):

-(void)testNumberInfinity
{
    NSError *error;
    CDTMutableDocumentRevision *doc = [CDTMutableDocumentRevision revision];
    doc.body = [@{
                  @"hello": @"world",
                  @"infinity": @(INFINITY)
                  } mutableCopy];
    CDTDocumentRevision *ob = [self.datastore createDocumentFromRevision:doc error:&error];

    STAssertNil(error, @"Error creating document");
    STAssertNotNil(ob, @"CDTDocumentRevision object was nil");

    CDTDocumentRevision *check = [self.datastore getDocumentWithId:ob.docId error:&error];
    STAssertNil(error, @"Error creating document");
    STAssertNotNil(check, @"CDTDocumentRevision object was nil");
    STAssertTrue([check.docId isEqualToString:ob.docId],
                 @"docId Mismatch: %@ != %@", ob.docId, check.docId);
    STAssertTrue([check.revId isEqualToString:ob.revId],
                 @"revId Mismatch: %@ != %@", ob.revId, check.revId);

    // things start failing here
    STAssertTrue([check.body count] > 0,
                 @"check.body is empty");
    STAssertTrue([ob.body[@"hello"] isEqual:check.body[@"hello"]],
                 @"hello did not match: %@ != %@", ob.body[@"hello"], check.body[@"hello"]);
    STAssertTrue([ob.body[@"infinity"] isEqual:check.body[@"infinity"]],
                 @"infinity did not match: %@ != %@", ob.body[@"infinity"], check.body[@"infinity"]);
}

Also note that ob reports that all is fine.

DB Not opening

I am getting pretty often in iOS app the following message when trying to open the db:
error opening!: 23
Could not create database queue for path /var/mobile/Containers/Data/Application/xxx/dbname.touchdb

Is there any reason why this happens so often? I am using the v0.16 version of CDTDatastore

Query: Implement `ensureIndexed:(NSArray *)fieldNames`

Most users don't care about index names and it should be straightforward enough to generate unique ones on the fly.

This is also relevant because it's easy enough for users to specify index names which will cause an error.

SecKeychainFindGenericPassword expects nonnull value for password length

SecKeychainFindGenericPassword function used in CDTEncryptionKeychainStorage, with Xcode 7 requires a password length to be a nonnull value. This generates a warning, I think where we currently pass NULL passing 0 should be fine.

Cannot build as a framework

(I am working from one week of total iOS, and XCode experience, so please bear with me!)

I think this project cannot build as a framework when used by a Swift project (at least with XCode 7 beta 6). The main problem is "Include of non-modular header inside framework module".

From my limited research, I gather that this issue is somewhat new (as apparently frameworks are not the only way to organize subprojects); but it is becoming more urgent, since Swift projects cannot work around it. (One can disable this compiler error; however, the disabling is not honored in Swift projects, whether by bug or design.) A clear description of the problem is here: http://inessential.com/2015/03/16/swift_blocker

In any case, it sounds like the situation XCode caught is indeed problematic and should be fixed. (My understanding is that, .h files should not import system-wide .h files--.m files should; something about a naming conflict as multiple frameworks compile together). Here is your dependency, CocoaLumberjack struggling with the same issue.

So, modulo my naïve understanding, to be complete (except of course as a NOFIX), CDTDatastore needs to fix its own issues and then also upgrade to all dependencies when they fix theirs. This issue only applies to this project's specific build failure.

AFAICT, there is only one problem in the codebase:

/Users/jhs/src/cloudant/CDTDatastore/Classes/common/Encryption/Attachments/CDTBlobEncryptedDataConstants.h:20:9: Include of non-modular header inside framework module 'CDTDatastore.CDTBlobEncryptedDataConstants'

"start" and "stop" removed from CDTReplicatorFactory

Demo app is broken :)

Expecta uses NS_DESIGNATED_INITIALIZER which is undefined for SDKs before OSX10.10

Using xctool, which just wraps for xcodebuild
When building for OSX10.9:

$ (cd Tests; pod install)
$ xctool -workspace CDTDatastore.xcworkspace -scheme "Tests OSX" -sdk macosx10.9 test
...
In file included from /Users/jimix/work/iOS/CoreData/CDTDatastore/Tests/Pods/Expecta/Expecta/Matchers/EXPMatchers+beCloseTo.m:1:
In file included from /Users/jimix/work/iOS/CoreData/CDTDatastore/Tests/Pods/Expecta/Expecta/Matchers/EXPMatchers+beCloseTo.h:1:
In file included from /Users/jimix/work/iOS/CoreData/CDTDatastore/Tests/Pods/Expecta/Expecta/Expecta.h:10:
In file included from /Users/jimix/work/iOS/CoreData/CDTDatastore/Tests/Pods/Headers/Private/Expecta/ExpectaSupport.h:1:
/Users/jimix/work/iOS/CoreData/CDTDatastore/Tests/Pods/Headers/Private/Expecta/EXPExpect.h:31:159: error: expected ':'
- (instancetype)initWithActualBlock:(id)actualBlock testCase:(id)testCase lineNumber:(int)lineNumber fileName:(const char *)fileName NS_DESIGNATED_INITIALIZER;
                                                                                                                                                              ^
                                                                                                                                                              :
/Users/jimix/work/iOS/CoreData/CDTDatastore/Tests/Pods/Headers/Private/Expecta/EXPExpect.h:43:110: error: expected ':'
- (instancetype)initWithExpectation:(EXPExpect *)expectation selector:(SEL)selector NS_DESIGNATED_INITIALIZER;

I'll follow up on any issues with pre iOS8.

Query: special characters in index names

In the various [CDTQIndexManager ensureIndexed] variants, the indexName argument is used to dynamically build a table name via string concatenation. This is done because SQL placeholders cannot be used to form table names.

As a consequence, characters which have special meaning to the SQL query parser (such as space or semicolon and a whole host of others) cannot be used for index names, because they cause the SQL parser to throw an error.

Potential fixes:

• At a minimum this restriction should be documented
• We should also consider mitigating strategies, like restricting the index name to a "safe" range of characters

delete document with old rev that was already deleted returns 404 and not 409

n Cloudant/Couch, when one tries to delete a document for the second time with an old 'rev' value, a 409 error is returned.

14:58:44 > acurl -X POST https://gadamc2.cloudant.com/test -d '{"_id":"tt1", "hellow":"world", "_deleted":true, "_rev":"3-428bcbf5b7ad1cb5f7c9b9e740c0f4ea" }' -H 'Content-Type: application/json'

{"ok":true,"id":"tt1","rev":"4-23650c9a0128ae6820131175b23bad8a"}

15:07:40 > acurl -X POST https://gadamc2.cloudant.com/test -d '{"_id":"tt1", "hellow":"world", "_deleted":true, "_rev":"3-428bcbf5b7ad1cb5f7c9b9e740c0f4ea" }' -H 'Content-Type: application/json' --verbose

• Adding handle: conn: 0x7fd69a004000
• Adding handle: send: 0
• Adding handle: recv: 0
• Curl_addHandleToPipeline: length: 1
• - Conn 0 (0x7fd69a004000) send_pipe: 1, recv_pipe: 0
• About to connect() to gadamc2.cloudant.com port 443 (#0)
• Trying 184.173.163.133...
• Connected to gadamc2.cloudant.com (184.173.163.133) port 443 (#0)
• TLS 1.0 connection using TLS_RSA_WITH_RC4_128_SHA
• Server certificate: *.cloudant.com
• Server certificate: DigiCert High Assurance CA-3
• Server certificate: DigiCert High Assurance EV Root CA

  POST /test HTTP/1.1
  User-Agent: curl/7.30.0
  Host: gadamc2.cloudant.com
  Accept: /
  Content-Type: application/json
  Content-Length: 94

• upload completely sent off: 94 out of 94 bytes
  < HTTP/1.1 409 Conflict
  < X-Couch-Request-ID: 0ae6cab0
• Server CouchDB/1.0.2 (Erlang OTP/R14B) is not blacklisted
  < Server: CouchDB/1.0.2 (Erlang OTP/R14B)
  < Date: Sat, 22 Feb 2014 23:08:31 GMT
  < Content-Type: text/plain;charset=utf-8
  < Content-Length: 58
  < Cache-Control: must-revalidate
  <
  {"error":"conflict","reason":"Document update conflict."}
• Connection #0 to host gadamc2.cloudant.com left intact

However, when performing the same operations in CDTDatastore results in a 404 rather than a 409

Conflict revisions contain no attachments

In conflicting document, CDTDocumentRevision objects contain an empty attachment dictionary.

Create category on CDTDatastore to manage replications

The aim here is to go from:

CDTReplicatorFactory *replicatorFactory = [[CDTReplicatorFactory alloc] init];
CDTDatastore *datastore = [manager datastoreNamed:@"my_datastore"];
CDTPushReplication *pushReplication = [CDTPushReplication replicationWithSource:datastore target:remoteDatabaseURL];
CDTReplicator *replicator = [replicatorFactory oneWay:pushReplication error:&error];
[replicator start];

to this:

CDTReplicator *replicator = [datastore pushReplicationTarget:remoteDatabaseURL
withDelegate:self
error:&error];
[replicator start];

That is, we manage the replicator factory within the datastore. We shouldn't change the logic other than this.

Mismatch between documentation and version 1.0.0

The replication documentation references a method startWithTaskGroup. This method does not exist inside 1.0.0. I had to point cocoapods directly at master to use the feature.

I'm guessing someone forgot to release a new pod version when the feature was merged in?

iOS - Subscribing to Events

I notice that subscribing to events is not currently available in iOS, is this being worked on? Can the branch be made known so that I can contribute?

Update example app to "fire and forget" the CDTReplicator instance

Update the sample app to no longer retain the CDTReplicator instance now that it supports "fire and forget"

Query Nullability Annoations

There may be a mistake in the nullability annotations for Query, this line: https://github.com/cloudant/CDTDatastore/blob/master/CDTDatastore/query/CDTQIndex.m#L100 generates a warning in xcode for passing nil to something that requires a nonnull value.

Set Authorization Header

With reference to this issue here in Android: cloudant/sync-android#36

It would be preferable if this could also be made available in iOS. At the moment, not being able to set authorization headers and having to send username and password via the URL is very insecure.

Avoid calling CDTDocumentRevision initWithDocId: twice in CDTDatastore getDocumentWithId:

After the mutable doc code was landed, [[CDTDocumentRevision alloc] initWithDocId: became a bunch more expensive because we copy the JSON body into a new dictionary -- Adam Cox did some work profiling this, and it was one of our first successes in finding a performance regression from the performance work. We should look to eradicate times where we call it when we don't need to.

For starters, lets see if we can clear the two calls here:

- (CDTDocumentRevision *)getDocumentWithId:(NSString *)docId
                                       rev:(NSString *)revId
                                     error:(NSError *__autoreleasing *)error
{
    if (![self ensureDatabaseOpen]) {
        *error = TDStatusToNSError(kTDStatusException, nil);
        return nil;
    }

    TDStatus status;
    TD_Revision *rev =
        [self.database getDocumentWithID:docId revisionID:revId options:0 status:&status];
    if (TDStatusIsError(status)) {
        if (error) {
            *error = TDStatusToNSError(status, nil);
        }
        return nil;
    }

    CDTDocumentRevision *revision = [[CDTDocumentRevision alloc] initWithDocId:rev.docID
                                                                    revisionId:rev.revID
                                                                          body:rev.body.properties
                                                                       deleted:rev.deleted
                                                                   attachments:@{}
                                                                      sequence:rev.sequence];
    NSArray *attachments = [self attachmentsForRev:revision error:error];

    NSMutableDictionary *attachmentsDict = [NSMutableDictionary dictionary];

    for (CDTAttachment *attachment in attachments) {
        [attachmentsDict setObject:attachment forKey:attachment.name];
    }

    revision = [[CDTDocumentRevision alloc] initWithDocId:rev.docID
                                               revisionId:rev.revID
                                                     body:rev.body.properties
                                                  deleted:rev.deleted
                                              attachments:attachmentsDict
                                                 sequence:rev.sequence];

    return revision;
}

ATS Enforcement EOY 2016

Apple announced at WWDC that ATS will be strictly enforced from the end of the year. We should make sure we don't need a ATS exception, and we make it easier for users to replicate using https with cloudant.

CDTReplicatorFactory initWithDatastoreManager:nil returns a non-nil object

If you call:

CDTReplicatorFactory *repFactory = [[CDTReplicatorFactory alloc] initWithDatastoreManager:nil];

The result is a non-nil object that can then be used to create a CDTReplicator that is non-nil but is essentially useless when you try to use it.

Incorrect documentation on CDTDatastore

The documentation on CDTDatastore mentions that conflict resolution is coming, however it has been present for a number of releases, the documentation here needs to be updated to mention the conflicts API.

HTTP interceptors should be able to indicate failure/stop processing

At the moment HTTP interceptors have no way to prevent the request going ahead in a safe manner.

For Objective-C, this should probably be:

• Returns nil.

Question: should we add an error param to the callback?

Documentation issue regarding deleted datastores

The documentation for CDTDatastoreManager -deleteDatastoreNamed:error: says

"it is the responsibility of the caller to ensure that extensions should be shutdown (and their underlying databases closed) before calling this method."

However, a CDTDatastore -close does not exist nor is there any documentation on what an extension is and how to shut it down.

SQLite temporary files are not deleted when the database is deleted

Journal mode for main SQLite database is WAL. In this case, at least one temporary file is created in the same folder adding '-wal' to the name of the database (and also another one adding '-shm'). However the method in TD_Database to delete a database: 'TD_Database:deleteDatabase:' only removes the file for the database.

Implement Continous Replication

The underlying TDReplicator class seems to already support continuous replication. What needs to be done in order to enable continuous replication in CDTReplicator?

CDTReplicatior state tests in ReplicationAcceptance.

Add state tests for the following scenarios to ReplicationAcceptance:

• local database is deleted
• remote database is deleted
• loses network connection temporarily
  • for > 60s but < 120s
  • for < 60s
    (the default retry waiting period is 60s)
• loses network connection permanently
• remote credentials are changed

Deprecate SavedHTTP attachment and -createRevisionFromJson:forDocument:error

CloudantToolkit is deprecated and as a result we should move to deprecate the integration points for toolkit and CDTDatastore.

any support for replication of named documentation?

according the documentation of cloudant about the named documentation replication :

Example replication of specific documents:

{
"source": "http://username:[email protected]/example-database",
"target": "http://username:[email protected]:5984/example-database",
"doc_ids": ["foo", "bar", "baz"]
}

the CDTDatastore has any function for that?

like
var pullReplication = CDTPullReplication(source: datastore, target: remoteDatabaseURL, ["doc1","doc2"] )

thanks

Query: use enum for `type` in public API

Using an untyped string for type in CDTQIndexManager is confusing and error-prone.

• Deprecate methods which take an NSString for index
• Add enum for IndexType with values json and text
• Replace deprecated methods with new ones taking enum

Update Contributing

After the merge of PR #215 contributing will become out of date. It needs to be updated to provide the correct instructions for setting up a development environment for CDTDatastore.

Update Homepage

Update podspec homepage to point at the github repo.

Confusing Swift Interface

The generated Swift interface for CDTDatatore is confusing and for some features unusable. For example the CDTDocumentRevision class has 4 init methods each with different arguments and nullability. The macros used to manipulate the logging levels are not imported into swift thus Applications written in swift cannot control the logging levels for our classes.

It needs to be investigated how this can be improved, either using new Objc Language features announced at WWDC, or by creating a Swift front end which simplifies the interface. The problem with changing the mapping will result in the breaking changes for the swift interface so this may be in conflict with semver and have to be pushed to a major release rather than a minor.

TouchDB Nullability annotations

Add nullability annotations to TouchDB classes.

Could not build Objective-C module 'CDTDatastore' - Swift

Hi,
i'm having a problem running my project with CDTDatastore, i'm using Cocoapods and Swift.
When i build the project all goes fine, but when i try to do "import CDTDatastore", Xcode give me a compilation error: "Could not build Objective-C module 'CDTDatastore'", there are others error that are all of the same type, like "Include of non-modular header inside framework module..."
I have some additional information:

• i use Xcode 7.2
• in the Podfile i have "use_frameworks!"
• if i remove "use_frameworks!" and i import "CloudantSync.h" in the bridging header all goes fine, but i can't remove "use_frameworks!"