slashdotdash / conduit

RealWorld example backend implementing the CQRS/ES pattern in Elixir and Phoenix

Conduit

Discover why functional languages, such as Elixir, are ideally suited to building applications following the command query responsibility segregation and event sourcing (CQRS/ES) pattern.

Conduit is a blogging platform, an exemplary Medium.com clone, built as a Phoenix web application.

This is the full source code to accompany the "Building Conduit" eBook.

This book is for anyone who has an interest in CQRS/ES and Elixir. It demonstrates step-by-step how to build an Elixir application implementing the CQRS/ES pattern using the Commanded open source library.

---

MIT License

---

Getting started

Conduit is an Elixir application using Phoenix 1.4 and PostgreSQL for persistence.

Prerequisites

You must install the following dependencies before starting:

• Git.
• Elixir (v1.6 or later).
• PostgreSQL database (v9.5 or later).

Configuring Conduit

1. Clone the Git repo from GitHub:

   $ git clone https://github.com/slashdotdash/conduit.git

2. Install mix dependencies:

   $ cd conduit
   $ mix deps.get

3. Create the event store database:

   $ mix do event_store.create, event_store.init

4. Create the read model store database:

   $ mix do ecto.create, ecto.migrate

5. Run the Phoenix server:

   $ mix phx.server

This will start the web server on localhost, port 4000: http://0.0.0.0:4000

This application only includes the API back-end, serving JSON requests.

You need to choose a front-end from those listed in the RealWorld repo. Follow the installation instructions for the front-end you select. The most popular implementations are listed below.

• React / Redux
• Elm
• Angular 4+
• Angular 1.5+
• React / MobX

Any of these front-ends should integrate with the Conduit back-end due to their common API.

Running the tests

MIX_ENV=test mix event_store.create
MIX_ENV=test mix event_store.init
MIX_ENV=test mix ecto.create
MIX_ENV=test mix ecto.migrate
mix test

Need help?

Please submit an issue if you encounter a problem, or need support.

conduit's Issues

Naming convension (plural or singular?)

Hi, I am new to commanded and I come from the rails.
I found there are two big contexts or aggregates. one is called blog (lib/conduit/blog), another is called accounts(lib/conduit/accounts).

I am not sure if we need to keep some kind of consistency for these two naming ?

which is better or how to choose , as in rails, we normally have convention about naming.

Query composition

Hi!

I happened to come across this project while studying different implementations on CQRS. This is not really an issue but more of a question as I happened to notice that query composition is not used and I wondered if there is a reason for this? As an example this code.

defmodule Conduit.Blog.Queries.ArticleComments do
  import Ecto.Query

  alias Conduit.Blog.Projections.Comment

  def new(article_uuid) do
    from c in Comment,
    where: c.article_uuid == ^article_uuid,
    order_by: [desc: c.commented_at]
  end
end

With query composition this could be written like:

defmodule Conduit.Blog.Queries.ArticleComments do
  import Ecto.Query

  alias Conduit.Blog.Projections.Comment

  def new(article_uuid) do
    Comment
    |> Comment.with_article_uuid(article_uuid)
    |> Comment.ordered_by_commented_at()
  end
end

defmodule Conduit.Blog.Projections.Comment do
  use Ecto.Schema

  import Ecto.Query

  #....

  def with_article_uuid(query, article_uuid) do
    query
    |> where([c], c.article_uuid == ^article_uuid)
  end

  def ordered_by_commented_at(query) do
    query
    |> order_by([c], desc: c.commented_at)
  end
end

Some of the queries do not return queryable but a tuple. By a quick look it seems that all of these also have pagination logic in it which I guess is the reason for the tuple. Instead of this these could return a queryable too in which case the pagination logic could be removed and replaced with a 3rd party one like this: https://github.com/drewolson/scrivener_ecto

Any thoughts on this?

Question: How to write into metadata field?

Hi,
this might be a noobish question, but I could not figure out how to write into events.metadata.
As I see it, maybe correct me if I am wrong, this would be the right place to save for example (phoenix) connection information like user IP etc.

Thanks fro advance.

Dispatch multiple commands per workflow (Commanded.Event.Handler) file?

Hi, we have been using this awesome example for a big project that we are building.

We are creating some mermaid diagrams to have a lower cognitive load in our developer's team. But during this process, we notice that the Workflows files that use Commanded.Event.Handler only can execute one command per file.

It's possible to execute multiple commands per workflow or it is a bad practice?

Thank you so much!

Add password validation to RegisterUser command

RegisterUser only validates username and email rules. It is possible to register user with

POST /api/users
{
	"user": {
		"username": "homer",
		"email": "[email protected]"
	}
}

mix test fails

18:03:43.094 [error] GenServer EventStore.Writer terminating
** (FunctionClauseError) no function clause matching in EventStore.Storage.QueryLatestEventId.handle_response/1
...

There were lots of warnings in the compile step.
Any insights?

Gratitude and questions

Hello Ben, I just wanna thank you for putting this repo up. As a beginner in Elixir and ES, it has been tremendously useful for me. I will soon start a new project based on what I learned from this repo and can't wait to buy your book when it's done!

Questions

1. Do you think that it's a good idea to have modules folder for aggregates, etc and schema folder for Ecto schema?

/conduit
  /modules      <----------- new
    /accounts
      /aggregates
      /commands
      /events
      /projectors
      /queries
      /validators
      /schema     <----------- new
      accounts.ex
      notifications.ex
      supervisor.ex
    /auth
    /blog
  /validation

2. How would you do cross context validation? For example, I have Users and Players contexts. I want to make sure that User exists when inserting a new Player. Based on my readings, I should create a workflow that listens to events from Users context and calls register_player in Players context. But what if I need to call register_player independently without listening to events? One thing that I thought of doing was to create a global Vex validator to validate the existence of a user. However, that doesn't feel right as Players context is now depending on Users context.

3. Would you recommend mixing non-ES stuffs (such as CRUD only APIs, etc.) within context?

4. Would it be possible to an example of using Process Manager?

Once again, thank you very much for this repo!

Update dependencies and improve local env setup

Hello there!

Thanks for this repo and the associated book.

While going through the online version of the book, I tried to run this repository locally, but struggled overall. For example, it was difficult to get the older version of Elixir, I was not sure about the appropriate OTP version, etc. It was a bit tricky to set this up with all appropriate system/application dependencies and getting it to compile/run.

After a while, I did however get a minimal version of this running on my own fork/ branch here. I updated some dependencies (not all), made it compile and also made the tests pass.

I wonder if there's any interest in getting some/any of these changes into master here, via a PR ? (Link to diff.)

Password hashing is unconditional (possible Denial of Service attack)

Hi.

I've been following along with the book and really like it so far. However, from what I understand, there might be a Denial of Service problem with the password hashing.

Since the password is hashed unconditionally and is not restricted in length, someone could upload a password that is a couple of megabytes in size, and the server would be quite busy hashing that.

The simplest solution I could come up with was to restrict the password length using a byte_size(password) <= 64 guard clause on hash_password/1, but I'd love to hear about a more elegant way of doing this, especially if it lets you enforce a minimum password length as well.

Custom mix task interacting with ES subsystem

Hello,

Is it possible to add an example custom mix task to demonstrate how to send a command from within a mix task? which applications should be started and how that is going to work in clustered mode?

Thanks,
Ilgar

function EventStore.MonitoredServer.child_spec/1 is undefined or private

I'm following the Conduit book, and I get this error.

** (Mix) Could not start application eventstore: EventStore.Application.start(:normal, []) returned an error: an exception was raised:
conduit_1 | ** (UndefinedFunctionError) function EventStore.MonitoredServer.child_spec/1 is undefined or private

My mix.exs

defmodule Conduit.Mixfile do
  use Mix.Project

  def project do
    [
      app: :conduit,
      version: "0.0.1",
      elixir: "~> 1.4",
      elixirc_paths: elixirc_paths(Mix.env),
      compilers: [:phoenix, :gettext] ++ Mix.compilers,
      start_permanent: Mix.env == :prod,
      aliases: aliases(),
      deps: deps()
    ]
  end

  # Configuration for the OTP application.
  #
  # Type `mix help compile.app` for more information.
  def application do
    [
      mod: {Conduit.Application, []},
      extra_applications: [:logger, :runtime_tools, :eventstore]
    ]
  end

  # Specifies which paths to compile per environment.
  defp elixirc_paths(:test), do: ["lib", "test/support"]
  defp elixirc_paths(_),     do: ["lib"]

  # Specifies your project dependencies.
  #
  # Type `mix help deps` for examples and options.
  defp deps do
    [ 
      {:commanded, "~> 0.16.0", override: true},
      {:commanded_eventstore_adapter, "~> 0.4.0"},
      {:eventstore, "~> 0.15.1"},
      {:phoenix, "~> 1.3.4"},
      {:phoenix_pubsub, "~> 1.0"},
      {:phoenix_ecto, "~> 3.2"},
      {:postgrex, ">= 0.0.0"},
      {:gettext, "~> 0.11"},
      {:cowboy, "~> 1.0"}
    ]
  end

  # Aliases are shortcuts or tasks specific to the current project.
  # For example, to create, migrate and run the seeds file at once:
  #
  #     $ mix ecto.setup
  #
  # See the documentation for `Mix` for more info on aliases.
  defp aliases do
    [
      "event_store.reset": ["event_store.drop", "event_store.create", "event_store.init"],
      "ecto.setup": ["ecto.create", "ecto.migrate", "run priv/repo/seeds.exs"],
      "ecto.reset": ["ecto.drop", "ecto.setup"],
      "test": ["ecto.create --quiet", "ecto.migrate", "test"]
    ]
  end
en

I can't find this error anywhere in the web. Any insights?

UDPDATE
Comparing with this' repo mix.exs, I have removed the {:eventstore, "~> 0.15.1"}, dependency, and still the same error appears.

updating username does not update authors table

How can one authorise queries rather than commands?

I've learned command authorisation by reading your article "Building a CQRS/ES web application in Elixir using Phoenix". However, I want to perform query authorisation in a bounded context. Could you give me some hints about how to do that?

Cheers,
Zhen Zhang

Release claimed unique values when command dispatch fails

The Conduit.Support.Middleware.Uniqueness middleware should use the after_failure/1 callback to release the claimed unique value when the command fails.

Uniqueness on account update

I may be wrong, but I believe this accounts implementation allows for multiple accounts to claim the same email/username by first creating accounts separately, then changing the emails/usernames of the different accounts to be the same. This is an issue I've faced in my own work, and found I had to solve it via a separate aggregate + a process manager.

My solution uses a map aggregate where each instance is a different map. All account emails are tracked in an account_emails map aggregate instance where the keys are emails, and values are account ids. The process manager starts in response to an account email change requested event, then reserves the email it plans to use in the account_emails map aggregate instance, then updates the account in the account aggregate, then releases the old email in the account_emails map aggregate instance. If the account aggregate update fails, it releases the new email in the account_emails map aggregate instance then fails.

Do you have a simpler solution?

Duplication of fields throughout the code

When we create a CRUD system usually we just need to define the fields in the migrations and schemas and that's it, but when we do CQRS (at least in this example) we need to define the same fields in:

• View
• Aggregates
• Commands
• Events
• Projections (Ecto Schemas)
• Projectors
• Migrations

I don't know if do in this way is a good approach because when we need to make a change in a single field we need to modify all those parts of the code, which sometimes is very complicated.

I know, CQRS has a lot of benefits comparing with CRUD, but I was wondering if this is the best way to do that.

Could you tell me if is this a good approach or we can do something to reduce the number of repeated fields definitions around the code?

"Building Conduit" book suggestions

Edits/suggestions for the book, Building Conduit. Have an edit or suggestion? Leave a comment here!

Edits

• in "Writing our first unit test" aggregates/user_test.exs example defmodule Conduit.Accounts.UserTest should be Conduit.Accounts.Aggregates.UserTest
• in "Writing our first read model projection" command validation example module names are wrong Conduit.Validations.Validators.String — the namespacing for validators changes over the course of a couple commits, and the examples in the book do not match
• The Unique validator module name does not match the module name added to the application supervisor

Suggestions

• i was confused about the phoenix-generated context code and tests until i looked at a commit where you removed all of that — it would have been nice to have that direction and some reassurance that “it’s going to be ok” in the book — it’s a brave new world!
• the refactoring of the readstore database setup to Conduit.Storage.reset! for tests is not mentioned in the book, but relied upon in the examples
• the guardian configuration is out of date. guardian 1.0.0 has been released and the setup/usage is a little different now
• generated phoenix test moved away from fixtures like fixture(:user) in favor of user_fixture(). This dealt with issues users were having once tests expanded beyond the most simple case.

Rename Project

Hey, I just noticed this project and the book you're writing are using the same name as my elixir libraries that I've been building for the past 2 years.

https://github.com/conduitframework
https://hex.pm/packages?_utf8=%E2%9C%93&search=conduit&sort=recent_downloads

I know it's a pain, but would you mind picking a different name?

Thanks and sorry for the inconvenience.

Remove `presence` from UpdateUser Command

Hey Ben, first off, great work!

I think these two lines can be removed in the command UpdateUser so they won't be mandatory when updating an existing user

https://github.com/slashdotdash/conduit/blob/master/lib/conduit/accounts/commands/update_user.ex#L21

https://github.com/slashdotdash/conduit/blob/master/lib/conduit/accounts/commands/update_user.ex#L27

(Phoenix.Router.NoRouteError) no route found for GET / (ConduitWeb.Router)

I follow the installation Readme.md, however I can't make it work, when I run mix phx.server I get the following:

➜  conduit git:(master) mix phx.server
warning: found quoted keyword "test" but the quotes are not required. Note that keywords are always atoms, even when quoted, and quotes should only be used to introduce keywords with foreign characters in them
  mix.exs:59

[debug] Attempting to start Postgrex
[debug] Successfully started Postgrex (#PID<0.258.0>)
[debug] Attempting to start Postgrex.Notifications
[debug] Successfully started Postgrex.Notifications (#PID<0.267.0>)
[debug] Attempting to start Postgrex
[debug] Successfully started Postgrex (#PID<0.269.0>)
[info] Running ConduitWeb.Endpoint with Cowboy using http://0.0.0.0:4000
[debug] Subscription "Accounts.Projectors.User"@"$all" subscribe to stream
[debug] Subscription "Blog.Projectors.Article"@"$all" subscribe to stream
[debug] Subscription "Blog.Projectors.Tag"@"$all" subscribe to stream
[debug] Subscription "Blog.Workflows.CreateAuthorFromUser"@"$all" subscribe to stream
[debug] Subscription "Accounts.Projectors.User"@"$all" subscribing to events
[debug] Subscription "Blog.Projectors.Article"@"$all" subscribing to events
[debug] Subscription "Blog.Workflows.CreateAuthorFromUser"@"$all" subscribing to events
[debug] Subscription "Accounts.Projectors.User"@"$all" requesting catch-up
[debug] Subscription "Blog.Projectors.Article"@"$all" requesting catch-up
[debug] Subscription "Blog.Projectors.Tag"@"$all" subscribing to events
[debug] Subscription "Blog.Workflows.CreateAuthorFromUser"@"$all" requesting catch-up
[debug] Subscription "Blog.Projectors.Tag"@"$all" requesting catch-up
[debug] Conduit.Accounts.Projectors.User has successfully subscribed to event store
[debug] Conduit.Blog.Projectors.Article has successfully subscribed to event store
[debug] Conduit.Blog.Workflows.CreateAuthorFromUser has successfully subscribed to event store
[debug] Conduit.Blog.Projectors.Tag has successfully subscribed to event store

And when I open http://localhost:4000, I get the next error:

[debug] ** (Phoenix.Router.NoRouteError) no route found for GET / (ConduitWeb.Router)
    (conduit) lib/conduit_web/router.ex:1: ConduitWeb.Router.__match_route__/4
    (conduit) lib/phoenix/router.ex:303: ConduitWeb.Router.call/2
    (conduit) lib/conduit_web/endpoint.ex:1: ConduitWeb.Endpoint.plug_builder_call/2
    (conduit) lib/plug/debugger.ex:102: ConduitWeb.Endpoint."call (overridable 3)"/2
    (conduit) lib/conduit_web/endpoint.ex:1: ConduitWeb.Endpoint.call/2
    (plug) lib/plug/adapters/cowboy/handler.ex:16: Plug.Adapters.Cowboy.Handler.upgrade/4
    (cowboy) /Users/yamildiazaguirre/Documents/grvty/conduit/deps/cowboy/src/cowboy_protocol.erl:442: :cowboy_protocol.execute/4

Uniqueness middleware issue after event removed

Support.Middleware.Uniqueness module works fine if the only thing that you need to validate is if the command hasn't been dispatched before with a common field name.

But what happens if you dispatch a command to remove something and later you try to create it again with the same value in the unique field? You would get an error, even if the record doesn't exist anymore.

To avoid that it would be great to add a timestamp comparing the dispatched commands in order to avoid comparing commands in long periods of time. At the end of the day, we just need this uniqueness feature to avoid duplicated event creation.

I'm going to try to make the changes and submit a pull request.

How can I populating the database with seeds

I'm looking for a way to publish the database but maintaining the consistency eventually.

%Example.TestContext.Aggregates.Industry{}
  |> execute(%Example.TestContext.Commands.CreateIndustry{
    uuid: UUID.uuid4(),
    name: "Agriculture"
  })

But it seems that the execute function is not loaded and I get this error:

** (CompileError) priv/repo/seeds.exs:27: undefined function execute/2
    (elixir) expanding macro: Kernel.|>/2
    priv/repo/seeds.exs:27: (file)

Integration with Absinthe GraphQL

I'm really interested to apply the CQRS approach in some of our future projects.
I've watching this awesome repo and I really like so much. But I'd wish have an example of how can we implement this using something like absinthe to get all the power of GraphQL with the CQRS approach.

I really want to know the vision from the experts about combine this technology with the CQRS pattern.

Thank you!

CORS

CORS is not enabled, I am having an issue sending requests using the react fronted you recommended in your README.md

Remove `username` field from `AuthorCreated` event

The author aggregate doesn't really need to know the username. Since it can be changed by the user aggregate we don't want to have to synchronise the value between two aggregates.