line / centraldogma Goto Github PK

There are various situations where an operation can fail:

• A read operation may fail when a client attempts to access the revision in a replica with replication delay. In this case, the failure can be easily recovered by retrying after a small delay.
• A replica may go offline due to some reason (e.g. machine dead, process killed, ...) and thus an operation may fail unexpectedly. If the operation is a read operation, we can just retry against another replica.

Currently, the client library exposes the situations mentioned above to our users as they are, forcing them to implement the recovery and retry logic by themselves. centraldogma-client should take care of them so that they don't have to.

We may also want to insert an artificial delay into watchRepository() or watchFile() operation so that the successive read operations have less chance of getting RevisionNotFoundException, as well as implementing the forementioned automatic retry mechanism.

.. because:

• libthrift isn't stable enough yet, e.g. 0.10.0 introduced breaking changes since 0.9.x. We do not want to ship two client library JARs - one for Thrift 0.9 and the other for 0.10.0.
• A client library can have fewer dependencies, especially without libthrift and Armeria.

To make this come true, we need to:

• #64 Revise our RESTful API & implementation so that it covers all features provided by the current Thrift service.
• Rewrite the client library so it uses the RESTful API.

In an emergency situation, a user may want to revert a repository into a certain revision, which is basically pushing a revert commit. There's currently no simple and quick way to do that; a user has to set up Git-to-CD sync, to send a PR with a revert commit and to merge it. It would be done much quicker in emergency if CD web UI has a dedicated UI.

.. because a change pushed directly to a mirrored repository will be reverted on next sync.

However, do not block a user from doing so because he may want to make a temporary change (e.g. GHE is down and need to make an immediate change.)

e.g. Permanently delete repositories/projects which were removed 7+ days ago.

.. as well as on-demand purging of course.

Although only logged at DEBUG level, it's always better avoid this sort of failure because exception instantiation is expensive.

Looking from the exception messages, this can be fixed by one of the following:

• setting the SecurityManager in a subject context or
• binding and unbinding the SecurityManager to/from the thread local context using ThreadLocalContext.bind(SecurityManager) and ThreadLocalContext.unbindSecurityManager(). (not preferred, because thread local variable is involved)

21:38:35.949 [armeria-common-blocking-tasks-3-1] DEBUG org.apache.shiro.subject.support.DefaultSubjectContext - No SecurityManager available in subject context map.  Falling back to SecurityUtils.getSecurityManager() lookup.
21:38:35.950 [armeria-common-blocking-tasks-3-1] DEBUG org.apache.shiro.subject.support.DefaultSubjectContext - No SecurityManager available via SecurityUtils.  Heuristics exhausted.
org.apache.shiro.UnavailableSecurityManagerException: No SecurityManager accessible to the calling code, either bound to the org.apache.shiro.util.ThreadContext or as a vm static singleton.  This is an invalid application configuration.
	at org.apache.shiro.SecurityUtils.getSecurityManager(SecurityUtils.java:123)
	at org.apache.shiro.subject.support.DefaultSubjectContext.resolveSecurityManager(DefaultSubjectContext.java:106)
	at org.apache.shiro.mgt.DefaultSecurityManager.ensureSecurityManager(DefaultSecurityManager.java:411)
	at org.apache.shiro.mgt.DefaultSecurityManager.createSubject(DefaultSecurityManager.java:333)
	at org.apache.shiro.mgt.DefaultSecurityManager.createSubject(DefaultSecurityManager.java:183)
	at org.apache.shiro.mgt.DefaultSecurityManager.login(DefaultSecurityManager.java:283)
	at org.apache.shiro.subject.support.DelegatingSubject.login(DelegatingSubject.java:256)
	at com.linecorp.centraldogma.server.internal.admin.authentication.LoginService.lambda$null$0(LoginService.java:69)
	at com.linecorp.armeria.common.AbstractRequestContext.lambda$makeContextAware$1(AbstractRequestContext.java:72)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
	at java.lang.Thread.run(Thread.java:748)

/cc @hyangtack

ZooKeeperCommandExecutor currently serializes a PushCommand as-is. This can cause unexpectedly large log entry to be stored in ZooKeeper, which has limited capacity. We should:

• Calculate the diff between the changeset provided in a PushCommand and the HEAD,
• Perform `Repository.commit() with the calculated diff and
• Store the calculated diff into ZooKeeper.

Such an optimization could reduce the size of a serialized log entry dramatically when a user issues multiple UPSERTs on existing files.

We currently support only Git-to-CD mirroring. Having CD-to-CD mirroring will allow us run satelite replicas which are mirrored from another CD cluster in a different data center.

Currently, removed projects and repositories are not removed, but just renamed with having the suffix of .removed.
We need to provide a way to remove them permanently when N(configurable, can be 0) days has passed.

Currently, there's no automated way to set up a new replica or to replace a broken replica. A user has to perform rsync from an existing healthy replica's data directory using command-line interface, which is obviously error-prone.

We should implement internal remote API:

• restricted to inter-replica communication
• dedicated to replica setup and recovery

.. so that configuring a new replica is as simple as running a single command. For example:

$ cd centraldogma/bin
$ ./setup-replica 192.168.0.100:36462 # the IP address and port of an existing healthy replica
... retrieves conf/* and lib/ext/* ...
... generates UUID and writes to conf/dogma.json ...
... retrieves all repository data ...
$ ./startup
... a new replica joins the cluster ...

In case of recovery, you could simply wipe out the existing data and conf directory and run setup-replica.

Like git clone so that a user can:

• Retrieve the content of a CD repository into a file system, so that he or she can feed it into other third party components such as a shell script.
• Push a commit that contains multiple changes.

Although client profile loading is currently being tested during Spring Boot integration tests, it would be better testing client profile loading independently.

We did not open-source our Spring Boot integration yet, but we wish to soon.

Some customers want to store binary files in CD. I think it's fine as long as it's not too large.

Related issue: #35 because binary files tend to be larger than our typical text files.

There are so many noises from gradle build output that are emitted from assertThatThrownBy(....);
We need to remove those expected noises so a user can focus on real problems.

On each login or logout, Shiro SecurityManager ends up creating four commits into our session repository, which can increase the number of commits in the session repository very quickly. Quick investigation of login process shows that:

• Subject.Builder.buildSubject() generates three commits
  1. Initialization commit by createSession(context) in AbstractNativeSessionManager.start(context)
  2. Redundant commit of (1) by applyGlobalSessionTimeout(session) in AbstractNativeSessionManager.start(context).
  3. Updating the current principal
• Subject.login() generates two more comments:
  1. Updating the current principal again, because currentPrincipals.equals(existingPrincipals) returns false at DefaultSubjectDao.java:218 due to mismatching realm name.
     • It seems like we don't need to specify a principal when creating a Subject at all. Removing the Subject.Builder.principal() call, which was specified right before buildSubject(), gets rid of the third commit in buildSubject().
  2. Updating the boolean flag that tells if the session is authenticated or not.

I expect something similar occurs for logout, making a commit for each attribute updates before the session removal, which is totally inefficient.

Give this observation, it seems like Shiro has no notion of 'transaction' or 'delayed update' to reduce the number of SessionDao.update() or SessionDao.create() calls, leading CentralDogmaSessionDAO produce unnecessary commits. We need to work aroud this or to work with the upstream to fix this. Or, did the upstream fix this issue already in a newer release? We are using a little bit old version of Shiro.

/cc @hyangtack

CentralDogma.forProfile() and CentralDogmaBuilder.profile() currently creates an endpoint group whose name is centraldogma-<integer counter>. Instead, we could name it as: centraldogma-profile-<profile name>

The response we sent is sometimes fairly large. Sometimes it's greater than 300KiB. We could save quite a bit of bandwidth by compressing it.

• Recognize accept-encoding HTTP header and compress the response on the fly.
  • Support gzip, deflate and brotli
• Cache the compressed result if possible.

This will require the work on the Armeria side first.

Expose stats about:

• JVM and OS
• Netty (needs Armeria-side work)
• HTTP requests
• Which replica is a leader?
• Whether a replica is runing in read-only mode

Angular.js is showing its age and thus we should migrate to something with brighter future. During the migration, we also may want to migrate from Bootstrap to Material design

When the Java client library fails, internal.thrift.CentralDogmaException is raised. This exposes an internal class as well as our implementation detail (Thrift). We should convert it into a protocol-agnostic generic exception type.

... so that it covers all features provided by the current Thrift service, and eventually remove Thrift endpoint, as described in #59

Creating and editing mirror settings require manual JSON editing which is time-consuming and error-prone. We should provide a dedicated UI for a meta repository so that a user cannot make a mistake when configuring a mirror.

Also, for some reason, when a bad mirror setting leaked into a meta repository, the UI should provide an easy way to figure out what's wrong with his/her setting.

We currently do not enforce any limit of the size of a file, which can cause troubles such as OOME when a user attempts to push a very large file.

We currently enforce 10 MiB hard limit on request content, but a user may want to increase the maximum allowed content length while limiting the size of an individual file. Think about a commit that contains multiple files whose total size is greater than 10 MiB while their each size is less than 1 MiB.

It will be very convenient if a user can query Central Dogma using a GraphQL query.

/cc @imasahiro

A Central Dogma project (or its repositories) is updated automatically by a machine. e.g. Git-to-CD mirroring. When such a task fails, there's no way to know the reason of the failure other than digging into the log files, which is very inconvenient.

We should provide a generic API that allows such tasks to send a notification/log message so that a user can browse them via web admin UI or watchable API call.

When you create a bean instance using CentralDogmaBeanFactory, the Watcher is made and there's no way to close it actively. It is, currently, closed with an exception when the watching server is down and therefore, it pollutes the output.

.. which may be useful for integration with third party applications.

We may want to generalize this service and ship it with Armeria.

We currently support only Git-to-CD mirroring. We should support the opposite direction as well.

/cc @tokuhirom

Some users want to apply more complicated inclusion/exclusion rules for a mirror. We could support rsync-like exclusion settings.

GitRepository currently uses a tag to maintain the mappings between revisions and commitIds. It has the following problems:

• When using the default refstorage implementation (refdir), it creates a symbolic link for each revision, which is slow and which creates too many files in the long term.
• When using the 'reftree' refstorage implementation, the disk usage of the repository increases way too much (more than 2x). Also, it becomes impossible to browse the repository using vanilla git command because reftree is a jGit-only extension.

I'd like to experiment with storing the revision-commitId mappings in a dedicated store like B-tree.

Related: https://github.com/lmdbjava/lmdbjava

We need to implement per-app/user and per-organization/group access control, such as who or which group can read or write which path patterns. We may need to store which user belongs to which group in Central Dogma's internal storage.

/cc @tokuhirom

No pagination has been implemented in the history view of the administrative UI. This can cause a problem as the number of commits increases.

A user want to use CentralDogma with Netty 4.0 (e.g. Apache Flink uses 4.0) but because Armeria uses methods that is newly added in Netty 4.1, it causes NoSuchMethod exception.

Currently, he/she can relocate Netty 4.1 by himself/herself.

To solve this, we can 1) make Armeria shade Netty or 2) refactor the client library so it has Armeria-based provider and URLConnection-based provider.

Related: line/armeria#705

Currently, a diff of a JSON file will always be represented in JSON patch format, which may not be a desired behavior when a user expects unified format. For example, some languages might not have proper JSON patch implementation.

We need to generalize this feature so a user can potentially ask for an alternative diff format for any file formats.

From #91 (comment)

Using a decorator...? Or, other way?

Currently, there's no easy way to know if the CD instance he or she is accessing belongs to BETA or RELEASE zone. We currently show the host name of the instance, but it's not clear enough. We could add a simple text value option whose value is appended to the web page title. e.g. 'Central Dogma (BETA, {hostname})'

Related: line/armeria#852

Provide BOM for Central Dogma artifacts to manage dependencies at downstreams.

Here is sample BOM that a project provides:

• http://platform.spring.io/platform/ , io.spring.platform:platform-bom:Brussels-SR5
• https://repo1.maven.org/maven2/io/spring/platform/platform-bom/Brussels-SR5/

so that the administrator does not have to check it visiting all mirrors to see if the modification is propagated properly.

Currently, there's no way to find out who removes the projects and repositories and when it happens.

When a user browses a repository, it would be useful if the administrative console displays the content of the README file in the current directory, just like GitHub does with the README.* files.

Also, a user may be interested in what a project is for. We need to think about a way to supply a README for a project because we cannot put files into a project. Probably into the meta repository?

A user has to use a command-line client, but it would be nice to have a web UI for this.

.. which would simplify the automated retrieval and verification of a tarball. For example, a user may want to build the Central Dogma distribution derived from the official tarball, like adding some extra jars to lib/ext and overriding some configuration files such as dogma.json and logback.xml.

.. so that:

• client applications do not refuse to start due to the inability to retrieve the configuration
• clients send fewer number of requests potentially

Note that a user should be allowed to disable this feature just in case he or she wants to make sure the latest revision is always retrieved.