eventsourcing / es4j Goto Github PK
View Code? Open in Web Editor NEWEvent capture and querying framework for Java
Home Page: http://eventsourcing.com
License: Mozilla Public License 2.0
Event capture and querying framework for Java
Home Page: http://eventsourcing.com
License: Mozilla Public License 2.0
Wondering if you would be willing to also make this available under ASL 2.0?
At this moment, Min
/Max
queries are limited to finding the smallest or largest value of a particular attribute globally across an entity. Which means you can tell "what's the latest order" but not "what's the latest order for this person". IsLatestEntity
/LatestAssociatedEntryQuery
are currently being used to address this type of need, but they are very slow and cumbersome. Min
/Max
has been already optimized for PostgreSQL storage.
It would have been great if it was possible to fit this type query within Min
/Max
as an option:
// Find the last order for user with id=<id>
max(Order.TIMESTAMP, equal(Order.USER_ID, id))
// Find all last orders for all users
max(Order.TIMESTAMP, Order.USER_ID)
(Or something similar)
ReflectASM is a very small Java library that provides high performance reflection by using code generation. An access class is generated to set/get fields, call methods, or create a new instance. The access class uses bytecode rather than Java's reflection, so it is much faster. It can also access primitive fields via bytecode to avoid boxing.
In Kotlin, as
is a reserved keyword, and leads to this:
@JvmField var X = SimpleIndex.`as` { o: TestEntity -> o.x }
as opposed to
@JvmField var X = SimpleIndex.as { o: TestEntity -> o.x }
which is far from great
Querying for things like:
is quite expensive as it involved a lot of scanning (as can be seen in https://github.com/eventsourcing/es4j/blob/master/eventsourcing-queries/src/main/java/com/eventsourcing/queries/IsLatestEntity.java and https://github.com/eventsourcing/es4j/blob/master/eventsourcing-queries/src/main/java/com/eventsourcing/queries/LatestAssociatedEntryQuery.java)
Proposed solution: designate and formalize an aggregate indexing system.
Despite publish = true
in https://github.com/eventsourcing/es4j/blob/master/build.gradle#L74 stopped auto-publishing maven packages and they have to be published manually for now.
PostgreSQL-backed journal & indices will create necessary tables and types without consideration for others — meaning there might be a conflict and two nodes might attempt to do the same thing at the same time.
Proposed solution: ensure these storages are locking these initialization routines. PostgreSQL advisory locks can be used to easily accomplish this.
(Only if possible, first indications is that it is not)
In any system the schema of commands and events will naturally evolve. While this is typically not a problem when using MemoryJournal
(the data will not survive through restarts), it is an issue with persistent journals (MVStoreJournal
and others).
Every persistent journal will need to print out the discrepancies found between entities available in the journal and in runtime. Using layout's hash we can easily detect which entities in the journal are no longer covered by the code at hand.
The report should look like this:
Found 1 unrecognized entity:
d395f099078879b005797edf7ef37f83a673bb3582056490655984d3d5b18cf8 (234 records)
This reporter should not try to attempt facilitating schema migrations in any way, it should be just a reporter.
It would be great if some meta information can be stored every time the journal is used. This way, the report can be much more readable:
Found 1 unrecognized entity:
com.foo.bar.events.UserLoggedIn (234 records) # d395f099078879b005797edf7ef37f83a673bb3582056490655984d3d5b18cf8
Even better if the layout can be saved, too:
Found 1 unrecognized entity:
com.foo.bar.events.UserLoggedIn (234 records) # d395f099078879b005797edf7ef37f83a673bb3582056490655984d3d5b18cf8
login: String
And we're using the same name for es4j for naming maven artifacts and package names, which might make it more difficult to assign proper naming to other solutions.
Proposed solution 1: rename com.eventsourcing:eventsourcing-X
maven artifacts to com.eventsourcing:es4j-X
and packages com.eventsourcing.*
to com.eventsourcing.es4j.*
Proposed solution 2: rename com.eventsourcing:eventsourcing-X
maven artifacts to org.es4j:es4j-X
and packages com.eventsourcing.*
to org.es4j.*
Proposed solution 3: rename com.eventsourcing:eventsourcing-X
maven artifacts to com.eventsourcing:es4j-X
and packages com.eventsourcing.*
to org.es4j.*
Proposed solution 4: find a completely different name for es4j
The failure to connect to a local NTP server is hard to detect (nothing is happening, repository is not starting). Since NTP is a UDP protocol, we can't reliably tell if there's an NTP server listening. The only way to tell if we're not getting a response.
Proposed solution: throw an exception after some reasonably high delay (1s, 5s?)
I can't use the dependency "com.eventsourcing:eventsourcing-inmem:0.3.1"
because it is missing on the Bintray Maven repository [1].
Work-around: Deploy the dependency to a local maven repository.
[1] http://dl.bintray.com/eventsourcing/maven/com/eventsourcing/
However, it should have a corresponding CommandThrownException
event, and the original exception should still be thrown as it should be now.
This is most likely happening due to npgall/cqengine#74 — that said, we do have a workaround for this — if the iterator was actually finished, it should close.
According to a report by @bsrk, and subsequent verification, Scala does not support static fields in classes. It does have companion objects, however, since Scala does not consider it a class and it is therefore impossible to specify it in the @Indices(Array(MyEventIndices))
annotation.
Proposed solution: implement a ScalaObjectIndexLoader
that will use Scala's own reflection, but other ideas are welcome.
Temporary workaround (for those who need one): write your index definitions in separate Java classes and refer to them in @Indices(...)
annotations.
It is, however, still very desirable to detect parameter names. @ParameterName
is one possible workaround, but it is a bit verbose.
Proposed solution: In some cases, it might be possible to use @java.beans.ConstructorProperties
. Nice thing is that (for example) Lombok generates it automatically.
Writing tests for es4j-powered apps is always a little bit of a hassle. To list a few issues:
Proposed solution: implement an eventsourcing-test module that will cover this.
I would like to propose to use http://docs.sonarqube.org/display/HOME/SonarQube+Platform for code analysis.
For IntelliJ - Sonar Lint http://www.sonarlint.org/intellij/.
It can be done as part of CI pipeline and will improve quality of the code.
These fingerprints (1-2 bytes most of the time) are used for calculating Layout hash, and they don't seem to follow one predefined numbering approach right.
It is probably preferable to keep them as short as possible IF the schema layout needs to be transferred at scale (although, how big of a deal is this?).
It might make sense to rename the term altogether. Magic number?
Allow Layout to produce a non-readonly instance for classes without matching setters, but a matching constructor
Hey, dont know if you've seen the projects already but there is some good efforts put into eventsourced and eventuate when it comes to eventsourcing and persistance and distribution.
It's just too slow because it'll keep re-opening connections and re-scanning types.
Proposed solution 1: force pool usage. Not sure how yet — different pools are available.
Proposed solution 2: do away with dataSource.getConnection()
and do some kind of connection pooling/sharing internally.
Proposed solution 3: enforce pgjdbc-ng and do make use of HikariCP underneath. Comes with a tradeoff of needing to configure that pool, but can be done with HikariConfig
. Ideally needs to share the pool with indices.
In certain aspects, we are starting to outgrow cqengine, the indexing engine behind es4j and one day we'll likely need our own indexing engine. cqengine was a great way to bootstrap and still is a good way to do what we want, but the time to upgrade will likely come. The telling signs so far are, in no particular order:
Query<EntityHandle<..>>
or ResultSet<EntityHandle<..>>
versus Query<...>
or ResultSet<...>
getIndexedCollection(Car.class).retrieve(query)
returns ResultSet<Car>
sun.reflect
API); not sure if we can improve this significantly within Java's constraints (can we do this in Java 9?), but worth look at, at the very least.Proposed solution: implement our own indexing engine.
We can certainly learn from the experience with cqengine and take it to the next level. This issue is an open-ended "proposal collector" to figure out a longer term plan for a potential transition, or any ideas for any other way to address the problem.
(It is, of course, a significant undertaking)
When using something like a remote SQL database for indexing, we often rely on the fact that we might not be consuming the entire ResultSet to achieve a better performance profile (or not consuming it at all).
However, this only works for truly lazy queries when nothing is extra done before the ResultSet is consumed, and the overhead of consuming the next result is low. In the SQL database scenario, even if the query itself is lazy, the moment the first result is consumed, the entire query will be fired off, and if the size of the index is substantial, there will be a delay.
Proposed solution: implement a query option for sized (limited) queries (Limit(N)
). It can be ignored by truly lazy indices, but SQL-based indices can use it to optimize their queries.
If Command#events
takes long time to complete (for example, making external API calls), the entire queue of commands will be blocked, because command processing is a single-thread process (originally, to avoid the complexities associated with multithread access to journal, indexing, etc.). This also holds true for massive event streams.
Proposed solutions:
CompletableFuture<EventStream<S>>
version of #events()
that will call EventStream<S>
version of #events()
in a) the same thread or b) in a worker thread` and wrap its result. Publishing process, therefore, should only send the command down the pipeline when the result is ready. There is still a concern of using blocking operations during stream generation.(Since the timestamp is not really comparable as is, it needs to be transparently re-encoded to a comparable value in order to work)
eventsourcing=# \d+ index_v1_9c22ea3e2733ba23f1e23c8577698827575a4457_navigable;
Table "public.index_v1_9c22ea3e2733ba23f1e23c8577698827575a4457_navigable"
Column | Type | Modifiers | Storage | Stats target | Description
--------+----------------------------------------------------+-----------+----------+--------------+-------------
key | layout_v1_c47d416193cba63554e5a69ca701973bc3e44172 | | extended | |
object | uuid | | plain | |
Indexes:
"index_v1_9c22ea3e2733ba23f1e23c8577698827575a4457_navigable_key" btree (key)
"index_v1_9c22ea3e2733ba23f1e23c8577698827575a4457_navigable_obj" btree (object)
eventsourcing=# \d+ layout_v1_c47d416193cba63554e5a69ca701973bc3e44172
Composite type "public.layout_v1_c47d416193cba63554e5a69ca701973bc3e44172"
Column | Type | Modifiers | Storage | Description
----------------+--------+-----------+---------+-------------
logicalCounter | bigint | | plain |
logicalTime | bigint | | plain |
See RFC1/ELF and RFC2/BES (added in eventsourcing/rfc@3ccf130)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.