A distributed database with a built in streaming data platform
License: Apache License 2.0
Store your feelings here.
Written by Bazaarvoice: see the credits page for more details.
EmoDB is a RESTful HTTP server for storing JSON objects and for watching for changes to those events.
It is designed to span multiple data centers, using eventual consistency (AP) and multi-master conflict resolution. It relies on Apache Cassandra for persistence and cross-data center replication.
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
There are numerous places throughout EmoDB that use a Ticker as a time substitute to enable better unit testing. In all of these cases the production code substitutes in Ticker.systemTicker() for the actual instance. Most of these use the ticker for relative time-spans, such as for expiring caches. However, there are two instances where tickers are being used to return the current time. This is problematic because the system ticker does not guarantee that the epoch is used as the fixed time reference.
The ticker is being used to incorrectly determine whether an event on the databus from a remote data center that is no present in the local data center (that is, has been replicated by Cassandra) is stale. Fortunately this one is non-harmful, since the system ticker will be exceptionally low, causing all missing deltas to calculate as recent. The effect here therefore is that if a delta never gets replicated the databus will be re-polled indefinitely until the event naturally expires.
The ticker is being used to get the current time for the purpose of scheduling Stash execution and naming the destination directory. This is causing the Stash to run at the incorrect time and use a date far in the past, such as 1/1/1970 on the initial execution.
The use of Ticker is good for unit testing, but especially in the instances where the returned value must correspond with the current time it should be replaced with an object which will do so.
High for Stash since destination directory naming will not reflect actual Stash times.
Low. Although the impact is high the actual solution is straightforward.
There is an apparent contradiction between Emo's never-null doc get api and conditional application.
Start Emo and run through this scenario:
I start out by deleting the doc so we know it doesn't exist...
$ http DELETE :8080/sor/1/review:testcustomer/testdoc000 audit=="comment:'blah'" X-BV-API-Key:local_admin Content-Type:application/x.json-delta
HTTP/1.1 200 OK
Content-Type: application/json
Date: Wed, 08 Feb 2017 15:07:46 GMT
Transfer-Encoding: chunked
{
"success": true
}
$ http GET :8080/sor/1/review:testcustomer/testdoc000 X-BV-API-Key:local_admin
HTTP/1.1 200 OK
Content-Type: application/json
Date: Wed, 08 Feb 2017 15:07:50 GMT
Transfer-Encoding: chunked
{
"client": "TestCustomer",
"type": "review",
"~deleted": true,
"~firstUpdateAt": "2017-02-08T15:06:10.121Z",
"~id": "testdoc000",
"~lastUpdateAt": "2017-02-08T15:07:46.685Z",
"~signature": "233b2c5fcc29921f9728a317d5398b07",
"~table": "review:testcustomer",
"~version": 2
}
$ http POST :8080/sor/1/review:testcustomer/testdoc000 audit=="comment:'blah'" X-BV-API-Key:local_admin Content-Type:application/x.json-delta <<< 'if {..,"nofield":~} then {"author":"Fred","title":"Best Ever!","rating":5} end'
HTTP/1.1 200 OK
Content-Type: application/json
Date: Wed, 08 Feb 2017 15:07:59 GMT
Transfer-Encoding: chunked
{
"success": true
}
$ http GET :8080/sor/1/review:testcustomer/testdoc000 X-BV-API-Key:local_admin
HTTP/1.1 200 OK
Content-Type: application/json
Date: Wed, 08 Feb 2017 15:08:04 GMT
Transfer-Encoding: chunked
{
"client": "TestCustomer",
"type": "review",
"~deleted": true,
"~firstUpdateAt": "2017-02-08T15:06:10.121Z",
"~id": "testdoc000",
"~lastUpdateAt": "2017-02-08T15:07:59.769Z",
"~signature": "618e69f4f5592b4b9a39df057d3dbf51",
"~table": "review:testcustomer",
"~version": 3
}
$ http POST :8080/sor/1/review:testcustomer/testdoc000 audit=="comment:'blah'" X-BV-API-Key:local_admin Content-Type:application/x.json-delta <<< 'if or(~, {..,"nofield":~}) then {"author":"Fred","title":"Best Ever!","rating":5} end'
HTTP/1.1 200 OK
Content-Type: application/json
Date: Wed, 08 Feb 2017 15:08:31 GMT
Transfer-Encoding: chunked
{
"success": true
}
$ http GET :8080/sor/1/review:testcustomer/testdoc000 X-BV-API-Key:local_admin
HTTP/1.1 200 OK
Content-Type: application/json
Date: Wed, 08 Feb 2017 15:08:33 GMT
Transfer-Encoding: chunked
{
"author": "Fred",
"client": "TestCustomer",
"rating": 5,
"title": "Best Ever!",
"type": "review",
"~deleted": false,
"~firstUpdateAt": "2017-02-08T15:06:10.121Z",
"~id": "testdoc000",
"~lastUpdateAt": "2017-02-08T15:08:31.106Z",
"~signature": "582736be3895778a4cc2e1286ba6ffb6",
"~table": "review:testcustomer",
"~version": 4
}
$ http POST :8080/sor/1/review:testcustomer/testdoc000 audit=="comment:'blah'" X-BV-API-Key:local_admin Content-Type:application/x.json-delta <<< 'if {..,"nofield":~} then {"author":"Kenny","title":"Best Ever!","rating":5} end'
HTTP/1.1 200 OK
Content-Type: application/json
Date: Wed, 08 Feb 2017 15:14:14 GMT
Transfer-Encoding: chunked
{
"success": true
}
$ http GET :8080/sor/1/review:testcustomer/testdoc000 X-BV-API-Key:local_admin
HTTP/1.1 200 OK
Content-Type: application/json
Date: Wed, 08 Feb 2017 15:14:16 GMT
Transfer-Encoding: chunked
{
"author": "Kenny",
"client": "TestCustomer",
"rating": 5,
"title": "Best Ever!",
"type": "review",
"~deleted": false,
"~firstUpdateAt": "2017-02-08T15:06:10.121Z",
"~id": "testdoc000",
"~lastUpdateAt": "2017-02-08T15:14:14.226Z",
"~signature": "01c5de13fdf7fe5dc27ca6852226ee68",
"~table": "review:testcustomer",
"~version": 5
}
Low: I can just always use the redundant conditional or(~, {..,"nofield":~}). New users will find this behavior confusing, though... 'cause it is ;)
Make sure to label the issue.
Well documented description of use-cases and bugs.
Currently EmoDB explicitly has the system table placement configured in three places:
systemOfRecord:
systemTablePlacement: app_global:sys
blobStore:
systemTablePlacement: app_global:sys
auth:
tablePlacement: app_global:sys
Additionally, Stash implicitly piggy-backs on the systemOfRecord.systemTablePlacement configuration when creating the following table:
scanner:
scanStatusTable: "__system_scan_upload"
Is there really an advantage to allowing each of these to be configured independently? As a counter-argument, allowing these to be configured independently increases the opportunities for the administrator to set one or more of these incorrectly. Additionally, storing system data in multiple placements increases the possibility that some system table's placement is not configured with sufficient security to safeguard EmoDB internals.
We should at least consider unifying all of these into a single top-level attribute that would then become inject-able into all of the dependent modules (e.g.; DataStoreModule):
systemTablePlacement: app_global:sys
Not really testable, just the way EmoDB is configured.
This is a low-risk change since all of the current configurations are typically set to the same value anyway. This change only affects any EmoDB installs which pathologically chose to create multiple distinct system placements for each module where configurable.
Low
Currently when an API key is deleted it is fully removed from the system via a record deletion. There are several reasons why this is not desirable:
The proposed solution is to instead have a state on an API key with the following possible values:
There are no errors caused by this issue. The only way to very is to query the authorization tables directly and verify that a key has been deleted after the API key administration task to delete the key has been called.
Risk is fairly low. So long as the existing API key realm only authenticates or authorizes keys in the active state then the rest of the system should be unaffected.
Medium. While the changes are localized there are some nuances in ensuring authentication behaves as expected, especially concerning key migration.
Emo currently stores all rules in a flat namespace. This has worked well under the assumption that a single user, the administrator, is responsible for managing and assigning roles. However, the current move is toward a delegated administration system, where the administrator can create trusted API keys which themselves can create roles with limited permissions and assign those to API keys (see #63). To support this each API key must have a safe sandbox for creating and managing roles; it would be dangerous to have a flat all-or-nothing system where any user with permission to update roles could update any role in the system.
The issue being documented here is a perquisite to the permissions aspect. There should be the ability to group related roles by a common namespace. With this in place it would be possible to grant an API key permission such as "manage roles in namespace X" or "assign roles in namespace X". This way the API key would have a safe sandbox for role administration without permission to manage or assign roles outside of that sandbox.
By itself the ability to group roles by namespace is low. The riskiest aspects of this change are:
Medium
Make sure to label the issue.
Well documented description of use-cases and bugs.
./start-local.shLow
Currently, there is a rare bug that can present itself when a high update row is being compacted simultaneously in two data centers. Consider the following scenario:
| Sequence | DC-1 | DC-2 | Remarks |
|---|---|---|---|
| 1 | D1, D2, FCT, D3, D4, C1(D1,D2) | D1, D2, FCT, D3, D4 | DC-1 compacts, but C1 isn't replicated to DC-2 |
| 2 | D1, D2, D3, D4, C1(D1,D2), FCT | D1, D2, D3, D4, C1(D1,D2), FCT, | Compaction C1 and D3 are behind FCT in both data-centers |
| 3 | D3, D4, FCT C2(C1,D3) | D3, D4, FCT, C3(D3, D4) | D1,D2 and C1 gets deleted, while creating a new compaction, C2. This is where the problem is. Deletes got replicated to DC-2, and before C2 could make it to DC-2, DC-2 created another compaction C3, based on only D3 and D4. And thus, C3 becomes the last effective compaction which does not include C2 owned deltas |
| 4 | D3, D4, FCT, C2(C1,D3), C3(D3, D4) | D3, FCT, C3(D3, D4) | DC-1 replicated the corrupted compaction C3, and uses it as the best compaction |
To resolve the above, the delete of the base compaction should be deferred until the owner compaction is behind FCT. In the above scenario, C1 should not be deleted in Step 3.
MultiDCCompactionTest provided in the PR for this issue failsMedium. Although there is nothing destructive about this PR, it does deal with compactions that is a core functionality. Running it in QA and stress testing is recommended.
Medium
The cql-toggle task can be used to configure whether to use the Astyanax or Cql driver on a particular EmoDB server. There are several problems with this approach with regards to Stash:
cql-toggle task is not available when Emo is running in Stash mode.start.sh, then Stash using start-stash-role.shMedium. Once the Cql driver has been fully proven for Stash this is a moot point. However, until that time if a critical issue is found in the Cql implementation of scan queries used by Stash there is no way to change back without creating and deploying a new release.
Medium. There is already a settings module in place which is persistent. It should be possible to utilize that to control use of the Cql driver.
There exist intrinsics for ~firstUpdateAt and ~lastUpdateAt. While the latter is useful it does not provide sufficient context for some use cases. This is because ~lastUpdateAt is always set to the timestamp of the most recent delta, even if that delta did not mutate the content in any way. Consider the use case of an index which wants to be able to return all records that have changed since a given date. The last-update time can return false negatives for these cases where the last update was non-mutative.
Here are three possible solutions:
~lastMutateAt, which contains the timestamp of the most recent delta which actually mutated the content.~lastUpdateAt~lastUpdateAt to be the timestamp of the last mutative deltaJust as with the last mutation time there are also use cases to be made using the last update time so I don't advocate the second or third option. The new ~lastMutateAt intrinsic would follow the same rules as ~lastUpdateAt in terms of availability, such as being absent for deleted records.
For clarity, this is distinct from the existing ~signature intrinsic. Checking the signature only informs the client if two versions of a record contain the same delta history. Updating a record with a non-mutative delta results in a modified signature, making it insufficient for this use case.
Check out the project
Create a table, such as "mutation:demo"
Repeat this sequence
$ curl -s -XPUT -H "Content-Type: application/json" "http://localhost:8080/sor/1/mutation:demo/demo1?audit=comment:'initial+submission'" --data-binary '{"value": true}'
{"success":true}
$ curl localhost:8080/sor/1/mutation:demo/demo1 | jq .
{
"~lastUpdateAt": "2016-09-12T18:15:43.899Z",
"~firstUpdateAt": "2016-09-12T18:15:43.899Z",
"value": true,
"~id": "demo1",
"~table": "mutation:demo",
"~version": 1,
"~signature": "aff0bc9ee63bdb23b19bb92801636158",
"~deleted": false
}
$ curl -s -H "Content-Type: application/x.json-delta" "http://localhost:8080/sor/1/mutation:demo/demo1?audit=comment:'unchanged'" --data-binary '{..,"value":true}'
{"success":true}
$ curl localhost:8080/sor/1/mutation:demo/demo1 | jq .
{
"~lastUpdateAt": "2016-09-12T18:16:38.225Z",
"~firstUpdateAt": "2016-09-12T18:15:43.899Z",
"value": true,
"~id": "demo1",
"~table": "mutation:demo",
"~version": 2,
"~signature": "9d3d1af5e17275b714025343c8c648f5",
"~deleted": false
}
Note the ~lastUpdateAt, ~signature, and ~version changed even though the resolved record is unchanged. This is as expected, but it demonstrates the metadata for the last time the record content changed, "2016-09-12T18:15:43.899Z", is lost.
Medium. The resolution code already tracks the last mutation time, so the change is mostly just exposing the value as an intrinsic. However, any change in the delta resolution path carries at least a medium amount of risk.
Currently the databus supports a "ignoreSuppressedEvents" attribute which, when true,
automatically adds a "and not tags contains 're-etl'" condition to each databus subscription (default is true). This allows us to signify a set of databus events as ignorable by most subscriptions to prevent flooding their subscriptions with bulk updates which may not be of interest to them.
The issue is that the condition is hard-coded and therefore cannot be adapted as necessary. The actual deployment may choose to use a different value or dynamically set the condition based on current status, such as a one-time bulk update by a client which uses their own event tagging scheme.
The proposed solution is to keep the functionality to but to make the "suppressed event" condition configurable. Options include:
The former solution is less work but requires an ops update and re-deploy to change.
Create a databus subscription is "ignoreSuppressedEvents" set to true. The only events that will be suppressed are ones tagged as "re-etl".
Low. The functionality to suppress events is already in place; this only changes the source for the suppressed event condition.
Medium
For the databus, CachingSubscriptionDAO works by caching all subscriptions in a single map of subscription name to subscription object. Every time a subscription is inserted, updated/renewed, or deleted this invalidates the entire cache and causes it to be reloaded.
Before owned subscriptions were introduced this behavior was acceptable. However, with the introduction of owned subscriptions every API call for a subscription – peek, poll, ack, etc. – first requires reading the subscription from the lower layers to verify the owner. In our production system this leads to problems because 1) according to stats 60+ subscriptions per minute is the norm, 2) each subscription request invalidates the entire cache, and 3) there are over 800 subscriptions. This means many databus API calls must synchronously reload all subscriptions before returning a response. This can be seen in our dashboards with regular subscribe, poll, and ack request latencies of over 500ms, sometimes greater than 1 second.
CachingSubscriptionDAO should be reworked to be more efficient for retrieving a single subscription. At the very least, an invalidation of one subscription should not cause a full reload of all subscriptions from source.
Medium. The long response times already a risk since many clients will timeout as a result. The greatest risks are:
Medium
Make sure to label the issue.
Well documented description of use-cases and bugs.
Purging a table is a long process and should be asynchronous, using the same job framework used for databus replay.
Run a timed purge command.
Low; purge is generally only run by administrators and is infrequent at that.
Low.
In one of the Bazaarvoice environments, "anon", we started seeing the following errors in the logs:
ERROR [2016-11-03 15:25:17,004] com.bazaarvoice.emodb.web.scanner.rangescan.LocalRangeScanUploader: Scanning placement failed for task id=2, app_global:default: ScanRange[8000000000000000000000006bccac6e-955555555555555555555555c12201c3]
! com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.100.47.52:9042 (com.datastax.driver.core.exceptions.ConnectionException: [/10.100.47.52] Error while setting keyspace), /10.100.45.100:9042 (com.datastax.driver.core.exceptions.ConnectionException: [/10.100.45.100] Error while setting keyspace), /10.100.36.5:9042 (com.datastax.driver.core.exceptions.DriverException: Timeout while trying to acquire available connection (you may want to increase the driver number of per-host connections)), /10.100.36.111:9042, /10.100.46.26:9042, /10.100.43.43:9042, /10.100.38.91:9042, /10.100.36.88:9042, /10.100.42.136:9042, /10.100.42.249:9042 [only showing errors of first 3 hosts, use getErrors() for more details])
! at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:207) ~[emodb-web-5.4.15.jar:5.4.16]
! at com.datastax.driver.core.RequestHandler.access$1000(RequestHandler.java:43) ~[emodb-web-5.4.15.jar:5.4.16]
! at com.datastax.driver.core.RequestHandler$SpeculativeExecution.sendRequest(RequestHandler.java:273) ~[emodb-web-5.4.15.jar:5.4.16]
! at com.datastax.driver.core.RequestHandler$SpeculativeExecution$1.run(RequestHandler.java:396) ~[emodb-web-5.4.15.jar:5.4.16]
! ... 3 common frames omitted
! Causing: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.100.47.52:9042 (com.datastax.driver.core.exceptions.ConnectionException: [/10.100.47.52] Error while setting keyspace), /10.100.45.100:9042 (com.datastax.driver.core.exceptions.ConnectionException: [/10.100.45.100] Error while setting keyspace), /10.100.36.5:9042 (com.datastax.driver.core.exceptions.DriverException: Timeout while trying to acquire available connection (you may want to increase the driver number of per-host connections)), /10.100.36.111:9042, /10.100.46.26:9042, /10.100.43.43:9042, /10.100.38.91:9042, /10.100.36.88:9042, /10.100.42.136:9042, /10.100.42.249:9042 [only showing errors of first 3 hosts, use getErrors() for more details])
! at com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:84) ~[emodb-web-5.4.15.jar:5.4.16]
! at com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:37) ~[emodb-web-5.4.15.jar:5.4.16]
! at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37) ~[emodb-web-5.4.15.jar:5.4.16]
! at com.datastax.driver.core.ArrayBackedResultSet$MultiPage.prepareNextRow(ArrayBackedResultSet.java:312) ~[emodb-web-5.4.15.jar:5.4.16]
! at com.datastax.driver.core.ArrayBackedResultSet$MultiPage.one(ArrayBackedResultSet.java:275) ~[emodb-web-5.4.15.jar:5.4.16]
! at com.bazaarvoice.emodb.sor.db.cql.RowGroupResultSetIterator.nextRow(RowGroupResultSetIterator.java:123) ~[emodb-web-5.4.15.jar:5.4.16]
! at com.bazaarvoice.emodb.sor.db.cql.RowGroupResultSetIterator.access$300(RowGroupResultSetIterator.java:15) ~[emodb-web-5.4.15.jar:5.4.16]
! at com.bazaarvoice.emodb.sor.db.cql.RowGroupResultSetIterator$RowGroupImpl.computeNext(RowGroupResultSetIterator.java:97) ~[emodb-web-5.4.15.jar:5.4.16]
! at com.bazaarvoice.emodb.sor.db.cql.RowGroupResultSetIterator$RowGroupImpl.computeNext(RowGroupResultSetIterator.java:86) ~[emodb-web-5.4.15.jar:5.4.16]
Once this happens ALL Datastax driver connections return errors, including those on clusters unrelated to the one that caused this issue. Increasing the maximum number of connections did not have any effect. Root cause is under investigation.
Under investigation.
High
Unknown until root cause is determined.
Make sure to label the issue.
Well documented description of use-cases and bugs.
Low
The EmoDB Cassandra health check only verifies the Astyanax connection. It is possible that the CQL driver could lose connectivity independently of Astyanax. The health check should be updated to verify both connections.
This is difficult to verify since the circumstances where it happens are unusual, but a review of the Cassandra health check verifies this behavior.
Low due to its unlikeliness. However, we have seen issues where loss of Cassandra seed servers affects the Astyanax and Cassandra drivers differently, so a comprehensive health check would be beneficial.
Low
I'd like to consider the notion of "private" attributes. These are defined as attributes not visible by default via the sor api, databus, or stash. API keys with write permission would be able to assert a query parameter that they wish to see private fields.
Private attributes have three main use cases that I'm aware of right now:
Medium
We are introducing a new concept to Emo. This is something we should do very, very carefully. Try to erase my use case from your mind and think about what a new user will make of this feature. Does the documentation make the intended usage obvious? Is the feature sufficiently intuitive?
Make sure to label the issue.
Well documented description of use-cases and bugs.
To reproduce:
Perform a request such as:
curl -s -v "localhost:8080/sor/1/table/%99"
* Trying ::1...
* Connected to localhost (::1) port 8080 (#0)
> GET /sor/1/table/%99 HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 400 Bad Request
< Date: Mon, 11 Jul 2016 18:17:56 GMT
< X-BV-Exception: java.lang.IllegalArgumentException
< Content-Type: text/plain
< Transfer-Encoding: chunked
<
* Connection #0 to host localhost left intact
Invalid path string "/adhoc-throttles/GET_sor~1~table~�" caused by invalid charater @33
The issue is that the adhoc throttle is checking ZooKeeper to determine if that URL has any throttle applied. The character 0x99 is not a valid character for a ZK path, so the following exception is thrown:
! java.lang.IllegalArgumentException: Invalid path string "/adhoc-throttles/GET_sor~1~table~�" caused by invalid charater @33
! at org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:99) ~[emodb-web-4.32-SNAPSHOT.jar:4.32-SNAPSHOT]
! at com.bazaarvoice.emodb.common.zookeeper.store.ZkMapStore.toPath(ZkMapStore.java:92) ~[emodb-web-4.32-SNAPSHOT.jar:4.32-SNAPSHOT]
! at com.bazaarvoice.emodb.common.zookeeper.store.ZkMapStore.get(ZkMapStore.java:123) ~[emodb-web-4.32-SNAPSHOT.jar:4.32-SNAPSHOT]
! at com.bazaarvoice.emodb.web.throttling.AdHocThrottleManager.getThrottle(AdHocThrottleManager.java:67) ~[emodb-web-4.32-SNAPSHOT.jar:4.32-SNAPSHOT]
! at com.bazaarvoice.emodb.web.throttling.AdHocConcurrentRequestRegulatorSupplier.forRequest(AdHocConcurrentRequestRegulatorSupplier.java:43) [emodb-web-4.32-SNAPSHOT.jar:4.32-SNAPSHOT]
! at com.bazaarvoice.emodb.web.throttling.AdHocConcurrentRequestRegulatorSupplier.forRequest(AdHocConcurrentRequestRegulatorSupplier.java:34) [emodb-web-4.32-SNAPSHOT.jar:4.32-SNAPSHOT]
! at com.bazaarvoice.emodb.web.throttling.ConcurrentRequestsThrottlingFilter.filter(ConcurrentRequestsThrottlingFilter.java:28) [emodb-web-4.32-SNAPSHOT.jar:4.32-SNAPSHOT]
Recommended solution:
The ZK path for adhoc throttles already has a substitution filter to replace "/" with "~". That filter should be expanded to replace invalid ZK path characters with hex representations.
Medium. In our use case keys are overwhelmingly in the standard ASCII range, but there may be other use cases where the full unicode character set is more widely used for record keys, especially if the keys are computed from natural sources.
Low. The portion of code required for this update is centralized and does not directly impact any of the more complex core Emo functionality.
The following issue impacts both Stash and the getSplits() API call.
Emo uses the Astyanax call describeSplitsEx() to get splits for token ranges. Frequently this returns reasonable values, but sometimes it returns splits which are demonstrably incorrect. Typically the returned split is equivalent to the requests token range and the returned CfSplit.getRowCount() always returns 128. This negatively impacts Stash because Stash is serialized on these excessively large splits until it detects the issue and re-splits. This also negatively affects table splits because a returned split may contain far more records than the caller requested.
Note that there is no equivalent native CQL call and according to Cassandra tickets there is no similar support until C* 2.1.5. (see CASSANDRA-7688).
Note: To reproduce you must be using a Cassandra ring with at least 4 nodes. It'll become clear why later.
Although the system works correctly in the event of over-sized splits it noticeably slows Stash down. We've also seen some evidence that extremely large result sets can lead to corruption on the web-client, although this is still under investigation.
Medium. Regardless of the effort to resolve this the changes would be localized to a small part of the system. That said, that small part directly impacts splits and Stash, so a regression would have significant impact.
This issue is in the larger context of creating a system where Emo can have delegated API key management. Today Emo is best administered by a single cabal of administrators who create roles and API keys at the client's request. As Emo expands there will be a need to delegate role and API key management. That is, if the admin has a trusted client who should be able to create API keys and give a discrete set of permissions to those keys there should be a safely delegated way to do so. For example:
project_x:app placement: sor|*|if(intrinsic("~placement":"project_x:app"))sor|*|if(and(intrinsic("~placement":"project_x:app")),{..,"project":"runway"}))For this to work admin needs to be sure that xadmin cannot create roles outside of the scope of his authority. For example, xadmin should not be able to create a role with permissions such as sor|*|if(intrinsic("~placement":"restricted:app")) because xadmin himself does not have that permission.
Note that this is note a complete solution in itself as it still leaves open many holes such as restricting which roles a delegated API key can update and which API keys the delegate can modify, but it is a crucial piece of the overall puzzle for distributed API key management.
sor|read|* and including system|manage_api_keys and system|manage_roles and assign it to a new API key.sor|*With the current Emo this role creation is allowed. With the update in place the role creation should fail with a message that the user does not have permission to grant sor|*.
The risk is mitigated so long as all API keys and roles are maintained by admins only. However, the point of this issue is undoing that restriction. For that reason, the risk is low if operations remain at status quo, but high as soon as delegation is put into practice.
High. The level of changes to ensure one unbounded permission, such as sor|if(in("read","update"))|if(intrinsic("~table":not(like("*:sys")))) is wholly contained within a users other permissions is a non-trivial exercise.
Make sure to label the issue.
Well documented description of use-cases and bugs.
Requests to the databus are partitioned by subscription. In the application cluster a single server is responsible for each subscription as determined by a hash function and Ostrich partitioning. Any requests to Emo using Ostrich use the same hash function and so databus requests are routed to the correct server. If a request in to the wrong server, such as through a load balancer, Emo is supposed to forward that request to the correct server for the subscription. However, this functionality has regressed and is currently not working.
It looks the the problem was introduced here:
45b1ae0#diff-2e81d793f3b18dae728f5140e0e538ebR385
45b1ae0#diff-9f186017d63374d1f96db11c087edd46R397
45b1ae0#diff-9960abdfcc869098ddf67001c1443f11R32
The partitioning function was removed from the service pool configuration and the partition key is one call removed from the service factory so the partition key, the subscription name, is not provided at the point necessary for proper routing.
As a result of this defect pollers using a load balancer will regularly not poll their subscription correctly. As a result the subscription will appear empty even though it still has pending events.
Medium. The fix isn't too complicated; DatabusClientSubjectProxy needs to be replaced with an interface which directly contains subscription names and therefore can be correctly forwarded. However, it changes much of the logic in DatabusResource1 and the corresponding injections.
Make sure to label the issue.
Well documented description of use-cases and bugs.
There is a simple bash recipe for recursively traversing symlinks to get to the real installed directory of a script. Then, you can pushd to it in order to run the script in context. If you do this, I won't have to actually be in my emodb directory to start it.
I'll send a PR.
Low
Currently, stash is not a "snapshot" of EmoDB. It is a long scanning process that lasts for several hours and produces an inconsistent snapshot. The issue is that there is no way in this dataset to deterministically arrive at some subset of data such that we are assured that all records as of that time are present. Consider the following situation. Let's say we start the Stash scanning process at 7 p.m. It is not enough to say that any record with the "lastUpdateAt" later than 7 p.m. should be discarded from the dataset. If we do that, then we will take out records that should be present in the dataset, since lastUpdated after 7 p.m., doesn't mean the record didn't exist before 7 p.m., and if so, we don't have the state in which the record existed at 7 p.m.
We need to take a global snapshot such that we record exactly the state of records as they existed at the start time of the scan.
To do this, the easiest way in EmoDB seems to be:
At the time of scan start time, halt any deletion of deltas. Note that compactions can still take place, but no deltas will be deleted
When resolving deltas, any delta later than scan start time should be ignored. This can be done either at the Resolver layer, or at the DAO layer that only iterates over deltas less than scan start time.
Using the two steps above, we can guarantee that a global snapshot of EmoDB is provided as of the scan start time.
Make sure to label the issue.
Well documented description of use-cases and bugs.
Tracking issue for:
One of the changes introduced in #19 is that each databus subscription is now associated with a single API key owner. With this change the unique identifier for a subscription effectively changed from (name) to (name, API key). However, the system still requires that each subscription name is unique system-wide.
Emo should be changed to permit different API keys to have subscriptions with the same name. The largest barrier to this is that the underlying event channels names are the subscription names. This would have to be changed to include the owner's internal ID as part of the event channel name. There would also have to be a way to grandfather in events from the existing databus event channels.
Recommend changes:
This is a fairly high risk change because it fundamentally changes one of the central pieces of the Emo architecture. As previously noted there needs to be a smooth migration from old to new event channel names with no downtime and without losing any preexisting events in the old channels.
High based on the areas touched by this change, even though the effort and scale of implementing the change may be lower.
Databus subscriptions currently do not honor table read permissions. For example, assume subscription "sub1" has condition alwaysTrue() and is polled by API key "apikey1". Assume apikey1 has the necessary permissions for databus access but the only SoR-related permission it has is sor|read|if(intrinsic("~table","only_accessible_table")). If anyone updates a record in table "inaccessible_table" the update notification and record contents will be available to apkey1 when it polls sub1.
A proper remedy to this requires that each subscription is associated with an API key. This way the fanout process can determine whether the record is accessible not only by the subscription but also by the subscriber before putting the record onto the subscription. Of course, the downside to this is that each subscription must now be polled by only a single user, but realistically this is the only observed use case to-date anyway. Additionally, after much discussion with the team we could neither come up with a use-case nor a satisfactory solution for sharing a databus subscription with multiple API keys with potentially different read permissions.
Follow the example from the issue description: create an API key with limited read permissions, create a databus subscription, and perform updates on records in tables which the API key cannot read. The API key will not be able to read those records directly from the sor module but it will be able to see the records by polling the subscription.
High, since this impacts multiple systems. Namely,
High
Some time ago there was some evidence of a potential memory leak in Emo. Because of the size of Emo and the number of requests and background processes running at any given time we started looking for correlations between periods of high GC memory utilization and heap uses and the activity going on at those times. Through this process we were able to narrow the problem to being related to the databus, but without deeper profiling we couldn't get any more specific than this.
By good fortune, before we had the chance to start profiling a client starting using the databus in such a way that the memory leak happened much faster than in the past. Most clients create a databus subscription and renew it on a long loop, such as hourly or daily. This client renewed their databus subscription in a loop with only a 100ms delay between subscription calls. Using this new information we focused the investigation to the databus subscription renewal. From here we were able to confirm a memory leak.
This issue lies in LocalDataCenterEndPointProvider. Every time a subscription is renewed an invalidation message is sent to all other Emo servers in the local data center so they will invalidate their copy of the subscription and reload from the backend on the next use. To ensure that no servers miss this important notification a fresh ZooKeeperHostDiscovery is created on each invalidation and closed after use. Aside from this being somewhat expensive we also found that the ZooKeeperHostDiscovery instance leaks memory on each use, even after being closed and dereferenced.
To verify we created a test environment with two Emo instances, each with 2.5G heap memory available. On each instance three separate bash scripts were started which subscribe to a unique databus subscription with a simple {..,"type":"review"} condition on an infinite loop with 100ms between calls, for a total of 6 subscriptions being constantly renewed. With this configuration it took less than 7 hours for the instances' heap to be fully utilized. One instance became non-responsive with an out-of-memory error in an admin thread. The other became slow and spent at least 34% of its CPU cycles in garbage collection, never recovering to less than 95% heap utilization. This continued even after all 6 subscriber scripts were stopped.
We then modified Emo such that only a single ZooKeeperHostDiscovery was created and maintained for the life if the LocalDataCenterEndPointProvider. With this configuration there was no evidence of a memory leak. Even after 2 days GC was able to recover heap to 5% utilization in the same circumstances.
#!/bin/bash
NAME=$1
echo $NAME
while true;
do
curl -s -XPUT "localhost:8080/bus/1/${NAME}?APIKey=XXX&ttl=604800&eventTtl=86400" -d '{..,"type":"review"}' -H "Content-Type: application/x.json-condition" > /dev/null
curl -s "localhost:8080/bus/1/${NAME}/poll?ttl=60&limit=100&APIKey=XXX" > /dev/null
sleep 0.1
donecurl -s localhost:8081/metrics | jq .gauges | jq '. | {"jvm.memory.heap.used", "jvm.memory.heap.usage", "jvm.memory.heap.max"}'
Over time usable heap will shrink, even after forcing GCs using the /tasks/gc task.
Medium
Make sure to label the issue.
Well documented description of use-cases and bugs.
Splits and Stash scans are still using Astyanax. They should be converted to use the CQL driver as part of the migration process to CQL.
Not applicable. The end result should be indistinguishable from current behavior, only the underlying query mechanism will have changed.
High, since the C* driver is at the heart of proper performance of SoR data access.
Medium. Although the impact of this change is high the actual code impact is localized to the data reader DAOs.
Currently API keys and roles are administered using DropWizard tasks, ApiKeyAdminTask and RoleAdminTask respectively. It would be more beneficial if API keys are roles were administered using standard REST endpoints for several reasons:
As part of this refactoring the administration API should also be changed to expose each API key's internal ID. Administration API methods to modify the API key (add roles, delete, etc.) should be changed to use the internal ID and not the key itself. There are several reasons for this change:
sor interfaces instead of using the task. This then requires restarting all EmoDB instances in all environments to force them to flush their API key caches.Not really anything to test or verify, other than that the current administration is indeed performed through tasks and requires use the of an API key to modify the key in any way.
This is a low risk change since it is largely just refactoring existing capabilities. Emo already has the ability to restrict REST calls based on permissions, so using that system to protect API key and role administration endpoints from users without the necessary permissions should be straightforward. The largest risk is the switch to using internal IDs for administration. However, this should be safe because:
__auth:keys or __auth:internal_ids tables, so there is no way for them to back-door lookup information about an API key from their internal ID.Medium
While validating data we found several events which existed in EmoDB which had not triggered an associated databus event. While the missing events were from different times the common thread was that all events were from updates within 200ms of when the table was created. We replayed all events into a new subscription and verified the events did not appear in the replay. This indicates that the events were never fanned out at all, as opposed to an issue with the subscription itself, such as an unlogged event ack.
This should not be possible since the create table command invalidates the table caches on all instances in the cluster and won't return until all caches were successfully invalidated. However, we did identify several use cases where this could happen. If the table is cached as unknown on one server and the table is created on a separate server the invalidation event may not be delivered if:
If fanout is running on the first server then the table would be unknown at fanout time and therefore discarded from fanout.
It's also possible some undiscovered bug in table caching and invalidation also exists. However, given a) there are already 2 identified possible use cases which explain the observed behavior and b) the scope of the bug must be within table caching the fix is the same. Fanout should be updated not to discard updates to unknown tables immediately but should delay their deletion until there is confidence that the table is unknown because it is dropped and not because of a dirty cache.
This is a difficult circumstance to intentionally reproduce. In this case the solution is based on the evidence and a review of the code rather than building a reliable reproduction.
This is a fairly low risk change. The worst side effect would be if the fanout queue were legitimately backed up with millions of events from a dropped table. This is unlikely, however, since those events would all have to have been generated before the table was actually dropped AND before the events were fanned out on the master queue. Additionally, the backlog caused by this would be quickly cleared once the minimum time to deletion has elapsed.
Medium
Make sure to label the issue.
Well documented description of use-cases and bugs.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.