omgnetwork / elixir-omg

OMG-Network repository of Watcher and Watcher Info

The elixir-omg repository contains OMG Network's Elixir implementation of Plasma and forms the basis for the OMG Network.

IMPORTANT NOTICE: Heavily WIP, expect anything

Table of Contents

• Getting Started
  • Service start up using Docker Compose
    • Troubleshooting Docker
  • Install on a Linux host
• Installing Plasma contract snapshots
• Testing & development
• Working with API Spec's

Getting Started

A public testnet for the OMG Network is coming soon. However, if you are brave and want to test being a Plasma chain operator, read on!

Service start up using Docker Compose

This is the recommended method of starting the blockchain services, with the auxiliary services automatically provisioned through Docker.

Before attempting the start up please ensure that you are not running any services that are listening on the following TCP ports: 9656, 7434, 7534, 5000, 8545, 5432, 5433. All commands should be run from the root of the repo.

To bring the entire system up you will first need to bring in the compatible Geth snapshot of plasma contracts:

make init_test

It creates a file ./localchain_contract_addresses.env. It is required to have this file in current directory for running any docker-compose command.

docker-compose up

To bring only specific services up (eg: the childchain service, geth, etc...):

docker-compose up childchain geth ...

(Note: This will also bring up any services childchain depends on.)

To run a Watcher only, first make sure you sent an ENV variable called with INFURA_API_KEY with your api key and then run:

docker-compose -f docker-compose-watcher.yml up

Troubleshooting Docker

You can view the running containers via docker ps

If service start up is unsuccessful, containers can be left hanging which impacts the start of services on the future attempts of docker-compose up. You can stop all running containers via docker kill $(docker ps -q).

If the blockchain services are not already present on the host, docker-compose will attempt to pull the latest build coming from master. If you want Docker to use the latest commit from elixir-omg you can trigger a fresh build by building all three services with make docker-childchain, make docker-watcher and make docker-watcher_info.

Install on a Linux host

Follow the guide to install the Child Chain server, Watcher and Watcher Info.

Installing Plasma contract snapshots

To pull in the compatible snapshot for Geth:

make init_test

Testing & development

Docker building of source code and dependencies used to directly use common mix folders like _build and deps. To support workflows that switch between bare metal and Docker containers we've introduced _build_docker and deps_docker folders:

sudo rm -rf _build_docker
sudo rm -rf deps_docker

mkdir _build_docker && chmod 777 _build_docker
mkdir deps_docker && chmod 777 deps_docker

Pull in the compatible Plasma contracts snapshot:

make init_test

You can setup the docker environment to run testing and development tasks:

docker-compose -f docker-compose.yml -f docker-compose.dev.yml -f docker-compose.datadog.yml run --rm --entrypoint bash elixir-omg

Once the shell has loaded, you can continue and run additional tasks.

Get the necessary dependencies for building:

cd app && mix deps.get

Quick test (no integration tests):

mix test

Longer-running integration tests (requires compiling contracts):

mix test   --only integration

For other kinds of checks, refer to the CI/CD pipeline (https://app.circleci.com/pipelines/github/omgnetwork/elixir-omg) or build steps (https://github.com/omgnetwork/elixir-omg/blob/master/.circleci/config.yml).

To run a development iex REPL with all code loaded:

MIX_ENV=test iex -S mix run --no-start

Running integration cabbage tests

Integration tests are written using the cabbage library and they are located in a separated repo - specs. This repo is added to elixir-omg as a git submodule. So to fetch them run:

git submodule init
git submodule update --remote

Create a directory for geth:

mkdir data && chmod 777 data

Make services:

make docker-watcher
make docker-watcher_info

Start geth and postgres:

cd priv/cabbage
make start_daemon_services-2

If the above command fails with the message similar to:

Creating network "omisego_chain_net" with driver "bridge"
ERROR: Pool overlaps with other one on this address space

try the following remedy and retry:

make stop_daemon_services
rm -rf ../../data/*
docker network prune

Build the integration tests project and run tests:

cd priv/cabbage
make install
make generate_api_code
mix deps.get
mix test

Running reorg cabbage tests

Reorg tests test different assumptions against chain reorgs. They also use the same submodule as regular integration cabbage tests.

Fetch submodule:

git submodule init
git submodule update --remote

Create a directory for geth nodes:

mkdir data1 && chmod 777 data1 && mkdir data2 && chmod 777 data2 && mkdir data && chmod 777 data

Make services:

make docker-watcher
make docker-watcher_info

Start geth nodes and postgres:

cd priv/cabbage
make start_daemon_services_reorg-2

Build the integration tests project and run reorg tests:

cd priv/cabbage
make install
make generate_api_code
mix deps.get
REORG=true mix test --only reorg

Working with API Spec's

This repo contains gh-pages branch intended to host Swagger-based API specification. Branch gh-pages is totally diseparated from other development branches and contains just Slate generated page's files.

See gh-pages README for more details.

More details about the design and architecture

Details about the repository, code, architecture and design decisions are available here.

elixir-omg's Issues

Watcher API responses redefinition

I'd like to start a discussion to redefine data WatcherAPI returns to make it align better for new database structure.

Proposed changes

• Added Plasma's Block definition
• Changed Transaction definition for 4in/4out and reuse other definitions
• New definition of TransactionItem - list of transaction items with less data
• Changed Utxo definition - added owner and removed txbytes

You can take a look at swagger definition changes in commit or review it in SwaggerUI

MacOS Mojave setup instruction (w/o docker)

Environment

MacOS Mojave 10.14.3

Elixir

> elixir --version
Erlang/OTP 21 [erts-10.2.3] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe] [dtrace]

Elixir 1.8.1 (compiled with Erlang/OTP 21)

Solidity

> solc --version
solc, the solidity compiler commandline interface
Version: 0.4.25+commit.59dbf8f1.Darwin.appleclang

Elixir-omg compilation

Clone latest master into directory of you choice.
Fetch dependencies with mix deps.get

Compilation of dependencies should go well, probably you would need to install some requirements (e.g. openssl, automake...) your system is missing, but installing latest version should do the trick.

Only exception to above is eleveldb which is not maintained for some time now.
For this you need to add commandline tools (c++ includes) ❓
Replace also all occurence of 10.8 to 10.7 in <repo_root>\deps/eleveldb/c_src/leveldb/build_config.mk
Then try to compile eleveldb with

> CXXFLAGS=-stdlib=libc++ CPATH=/Library/Developer/CommandLineTools/CommandLineTools/usr/include/c++/v1 mix deps.compile eleveldb

Deposit to child chain not working

I have recently started exploring this project. and gone through setup and its working fine for me.
i have used docker, docker-compose for starting server on local machine.

i am using latest code from master branch. (i hope that Master branch is stable, if it is not stable then which branch is stable?)

i am executing following example https://github.com/omisego/elixir-omg/blob/master/docs/demo_01.md

but when i execute following line

deposit_blknum = DepositHelper.deposit_to_child_chain(alice.addr, 10)

it is giving me following error.

** (MatchError) no match of right hand side value: []
    lib/root_chain.ex:724: OMG.Eth.RootChain.deposit_blknum_from_receipt/1

this is happening because in this file apps/omg_eth/lib/root_chain.ex the function
deposit_blknum_from_receipt, the params/map %{logs => logs} is an empty list [].

and this logs coming as empty list from this function

file: apps/omg_api/test/support/integration/deposit_helper.ex

 def deposit_to_child_chain(to, value, @eth) do

if i do IO.inspect receipt in above function

%{
  "blockHash" => "0xbef4f10a52509aed801709f5679281bb8ad75288c099bdfd0217bbac5872c4d4",
  "blockNumber" => 1729,
  "contractAddress" => nil,
  "cumulativeGasUsed" => "0x6658",
  "from" => "0xe62c3d91689e3ca9b34431063bd1c566c3b799e0",
  "gasUsed" => "0x6658",
  "logs" => [],
  "logsBloom" => "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
  "status" => "0x1",
  "to" => "0x5506dc9e672633af38c4fd07c2dc02b4e9bc268d",
  "transactionHash" => "0xa56b4ba0bb2c592420fd2299371dc1217cec727f3b5ad1363a7af5fc799baea4",
  "transactionIndex" => "0x0"

again logs is empty list. :(

status.get endpoint might timeout and crash if Watcher is very busy

For example, when a large block is being processed, this can happen:

    March 25th 2019, 12:35:28.390        ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started⋅

    March 25th 2019, 12:35:28.390    2019-03-25 11:35:28.389 [error] ⋅#PID<0.8864.707> running OMG.Watcher.Web.Endpoint (cowboy_protocol) terminated

    March 25th 2019, 12:35:28.390    Server: watcher.ari.omg.network:80 (http)

    March 25th 2019, 12:35:28.390    ** (exit) exited in: GenServer.call(Sentry.TaskSupervisor, {:start_task, [{:nonode@nohost, #PID<0.8864.707>, #PID<0.8864.707>}, [#PID<0.8864.707>], :monitor, {:erlang, :apply, [#Function<3.26443835/0 in Sentry.Client.do_send_event/3>, []]}], :temporary, nil}, :infinity)

    March 25th 2019, 12:35:28.389    2019-03-25 11:35:28.388 [error] module=OMG.Watcher.Web.View.ErrorView function=template_....................ErrorView, view_template: "500.json"}⋅
    March 25th 2019, 12:35:28.387    2019-03-25 11:35:28.386 [info] module=Phoenix.Logger function=phoenix_error_render/3 request_id=2m7j1013m44465ffnkehhqr1 ⋅Converted exit {:timeout, {GenServer, :call, [OMG.Watcher.BlockGetter, :get_events, 5000]}} to 500 response⋅

See in particular
{:timeout, {GenServer, :call, [OMG.Watcher.BlockGetter, :get_events, 5000]}}.

Probably we should push data that status.get shows into some other process on a cadence decoupled from the API and fetch the status.get response from that new process?

This is the going to be same story with ExitProcessor - the detection of byzantine stuff shouldn't be done on every status.get as it can get quite involved.

Config override issue with omg_watcher

As part of the ~/config_watcher.exs generation and setup for the omg_watcher, I wanted to customize the port that the watcher listens on.

When I specified the following in my ~/config_watcher.exs:

use Mix.Config
config :omg_eth,
  contract_addr: "0x21c4127004b4ae732fc407dffe65d28b1d0b3594",
  txhash_contract: "0xbb2a364ded8099a9c5d11c9388d8179790fe1c497fc76796f57e159062738b95",
  authority_addr: "0xabe7a779d6b6f9cf254b62075605ef5e7129f998"

config :omg_db,
  leveldb_path: Path.join([System.get_env("HOME"), ".omg/data_watcher"])

config :omg_watcher, OMG.Watcher.Web.Endpoint,
  http: [port: 34727],
  url: [host: "localhost", port: 34727],
  debug_errors: true,
  code_reloader: true,
  check_origin: false,
  watchers: [],
  server: true

The following error occurred when attempting to POST to the Watcher:

2019-02-18 14:47:16.149 [error] ⋅GenServer OMG.Watcher.Eventer terminating
** (ArgumentError) no :pubsub server configured at, please setup :pubsub in your config.

By default this looks like:

    config :my_app, MyApp.PubSub,
      ...,
      pubsub: [name: MyApp.PubSub,
      adapter: Phoenix.PubSub.PG2]
(phoenix) lib/phoenix/endpoint.ex:510: Phoenix.Endpoint.__pubsub_server__!/1
    (omg_watcher) lib/web/endpoint.ex:16: OMG.Watcher.Web.Endpoint.broadcast!/3
    (omg_watcher) lib/eventer.ex:47: anonymous fn/1 in OMG.Watcher.Eventer.handle_cast/2
(elixir) lib/enum.ex:737: Enum."-each/2-lists^foreach/1-0-"/2
    (elixir) lib/enum.ex:737: Enum.each/2
    (omg_watcher) lib/eventer.ex:45: OMG.Watcher.Eventer.handle_cast/2
    (stdlib) gen_server.erl:616: :gen_server.try_dispatch/4
(stdlib) gen_server.erl:686: :gen_server.handle_msg/6
(stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3

To fix this I had to add the following to my ~/config_watcher.exs config:

pubsub: [name: OMG.Watcher.PubSub, adapter: Phoenix.PubSub.PG2]

This is already defined in the watchers config located here:

apps/omg_watcher/config/config.exs

It would appear that this config file is getting ignored/overridden by my updated config_watcher.exs file.

Is this the expected behaviour or should the original settings within apps/omg_watcher/config/config.exs be preserved if no overrides have been set in a custom config?

Integration tests failure on Mac OS

My environment:

• Mac OS High Sierra 10.13.3
• Erlang/OTP 20
• Elixir 1.6.6 (compiled with OTP 19)

How to reproduce:
mix test --include integration

Output from test run

1. omg_eth

==> omg_eth
Including tags: [:integration, :wrappers]
Excluding tags: [:test, {:wrappers, true}]



  1) test gets events with various fields and topics (OMG.EthTest)
     test/eth_test.exs:76
     ** (MatchError) no match of right hand side value: {:exit_status, 9}
     stacktrace:
       test/support/dev_geth.ex:45: OMG.Eth.DevGeth.stop/1
       (ex_unit) lib/ex_unit/on_exit_handler.ex:140: ExUnit.OnExitHandler.exec_callback/1
       (ex_unit) lib/ex_unit/on_exit_handler.ex:126: ExUnit.OnExitHandler.on_exit_runner_loop/0

<more tests failing>

Finished in 114.4 seconds
6 tests, 6 failures

2. Happy Path (omg-api) & omg-perftest

2018-09-21 12:54:50.657 [error] ⋅GenServer #PID<0.991.0> terminating
** (stop) exited in: GenServer.call(OMG.API.RootChainCoordinator, {:check_in, 14, :depositer}, 5000)
    ** (EXIT) time out
    (elixir) lib/gen_server.ex:836: GenServer.call/3
    (omg_api) lib/ethereum_event_listener.ex:87: OMG.API.EthereumEventListener.sync_height/2
    (omg_api) lib/ethereum_event_listener.ex:76: OMG.API.EthereumEventListener.handle_info/2
    (stdlib) gen_server.erl:616: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:686: :gen_server.handle_msg/6
    (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Last message: :sync⋅
2018-09-21 12:54:50.660 [error] ⋅GenServer #PID<0.992.0> terminating
** (stop) exited in: GenServer.call(OMG.API.RootChainCoordinator, {:check_in, 14, :exiter}, 5000)
    ** (EXIT) time out
    (elixir) lib/gen_server.ex:836: GenServer.call/3
    (omg_api) lib/ethereum_event_listener.ex:87: OMG.API.EthereumEventListener.sync_height/2
    (omg_api) lib/ethereum_event_listener.ex:76: OMG.API.EthereumEventListener.handle_info/2
    (stdlib) gen_server.erl:616: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:686: :gen_server.handle_msg/6
    (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Last message: :sync⋅
2018-09-21 12:54:53.836 [error] ⋅GenServer OMG.API.RootChainCoordinator terminating
** (MatchError) no match of right hand side value: :invalid_synced_height_update
    (omg_api) lib/root_chain_coordinator/core.ex:64: OMG.API.RootChainCoordinator.Core.check_in/4
    (omg_api) lib/root_chain_coordinator.ex:58: OMG.API.RootChainCoordinator.handle_call/3
    (stdlib) gen_server.erl:636: :gen_server.try_handle_call/4
    (stdlib) gen_server.erl:665: :gen_server.handle_msg/6
    (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Last message (from #PID<0.993.0>): {:check_in, 12, :depositer}⋅


  1) test deposit, spend, exit, restart etc works fine (OMG.API.Integration.HappyPathTest)
     test/integration/happy_path_test.exs:33
     ** (ExUnit.TimeoutError) test timed out after 60000ms. You can change the timeout:
     
       1. per test by setting "@tag timeout: x"
       2. per case by setting "@moduletag timeout: x"
       3. globally via "ExUnit.start(timeout: x)" configuration
       4. or set it to infinity per run by calling "mix test --trace"
          (useful when using IEx.pry)
     
     Timeouts are given as integers in milliseconds.
     
     stacktrace:
       (elixir) lib/task.ex:491: Task.await/2
       (omg_api) test/support/integration/deposit_helper.ex:56: OMG.API.Integration.DepositHelper.wait_deposit_recognized/1
       (omg_api) test/support/integration/deposit_helper.ex:31: OMG.API.Integration.DepositHelper.deposit_to_child_chain/3
       /Users/pawel/Projects/omisego/apps/omg_api/test/integration/fixtures.exs:60: OMG.API.Integration.Fixtures.fixture_create_alice_deposits/2
       (ex_unit_fixtures) lib/ex_unit_fixtures/imp.ex:154: ExUnitFixtures.Imp.create_fixture/2
       (elixir) lib/enum.ex:1899: Enum."-reduce/3-lists^foldl/2-0-"/3
       (ex_unit_fixtures) lib/ex_unit_fixtures/imp.ex:86: ExUnitFixtures.Imp.create_fixtures/4
       (ex_unit_fixtures) lib/ex_unit_fixtures/imp.ex:57: ExUnitFixtures.Imp.test_scoped_fixtures/2
       test/integration/happy_path_test.exs:15: OMG.API.Integration.HappyPathTest.__ex_unit_setup_0/1
       test/integration/happy_path_test.exs:15: OMG.API.Integration.HappyPathTest.__ex_unit__/2
       (ex_unit) lib/ex_unit/runner.ex:299: ExUnit.Runner.exec_test_setup/2
       (ex_unit) lib/ex_unit/runner.ex:246: anonymous fn/2 in ExUnit.Runner.spawn_test/3
       (stdlib) timer.erl:166: :timer.tc/1
       (ex_unit) lib/ex_unit/runner.ex:245: anonymous fn/4 in ExUnit.Runner.spawn_test/3

Deploying Rootchain contract to local geth in dev mode fails with “exceeds block gas limit” error

Setting up a local env, running geth --dev --dev.period 1.
OMG.Eth.DevHelpers.prepare_env fails with:

** (FunctionClauseError) no function clause matching in OMG.Eth.DevHelpers.deploy_sync!/1

    The following arguments were given to OMG.Eth.DevHelpers.deploy_sync!/1:

        # 1
        {:error, %{"code" => -32000, "message" => "exceeds block gas limit"}}

    Attempted function clauses (showing 1 out of 1):

        def deploy_sync!({:ok, txhash} = transaction_submission_result)

    test/support/dev_helpers.ex:116: OMG.Eth.DevHelpers.deploy_sync!/1
    test/support/dev_helpers.ex:51: OMG.Eth.DevHelpers.prepare_env!/2
    (stdlib) erl_eval.erl:680: :erl_eval.do_apply/6
...

The cause is that geth --dev lowers the block gas limit with each block and can quickly get down to under 6180000, which is what DevHelpers uses when it deploys the RootChain contract.

Two possible workarounds:

1. Restart geth with --dev.period 5. This means that the block gas limit will decrease more slowly, giving you time to deploy the contract.
2. Lower the value for @ gas_contract_rootchain in apps/omg_eth/test/support/deployer.ex to around 5500000. This should still be enough gas to deploy the contract (although it may not...).

N.B. Starting geth with --targetgaslimit 8000000 looks like it should solve the problem, but currently --targetgaslimit doesn't appear to have any effect

Using {:system, "DATABASE_URL"} for your :url configuration is deprecated.

watcher-0 watcher warning: Using {:system, "DATABASE_URL"} for your :url configuration is deprecated.
watcher-0 watcher
watcher-0 watcher Instead define an init/2 callback in your repository that sets
watcher-0 watcher the URL accordingly from your system environment:
watcher-0 watcher
watcher-0 watcher     def init(_type, config) do
watcher-0 watcher       {:ok, Keyword.put(config, :url, System.get_env("DATABASE_URL"))}
watcher-0 watcher     end
watcher-0 watcher
watcher-0 watcher   lib/ecto/repo/supervisor.ex:55: Ecto.Repo.Supervisor.compile_config/2
watcher-0 watcher   lib/db/repo.ex:16: (module)
watcher-0 watcher   (elixir) src/elixir_compiler.erl:79: :elixir_compiler.dispatch/4
watcher-0 watcher   (elixir) src/elixir_compiler.erl:63: :elixir_compiler.compile/3
watcher-0 watcher   (elixir) src/elixir_module.erl:309: :elixir_module.eval_form/6

`:unchallenged_exits` are unchallengable

:unchallenged_exit byzantine chain condition arises, when an invalid exit doesn't get challenged within some timeframe.

Unluckily, if an invalid exit gets such a "label" the moment the invalidating transaction gets processed (eg. during syncing), such exit becomes permanently unchallengable.

This is because as a general rule, block getting stops at any byzantine chain condition, and hence, the invalidating transaction doesn't get fully persisted/acknowledged into the state. Then as a consequence the exit is perceived as valid and one cannot get challenge data.

To see the error, try syncing to the staging chain and

exiting_utxopos = 40002000000000
exiting_utxopos = 41000000000000
%{"data" => challenge} =
  ~c(echo '{"utxo_pos": #{exiting_utxopos}}' | http POST localhost:7435/utxo.get_challenge_data) |>  :os.cmd() |> Poison.decode!()

the Watcher will sync until block 41000, whereas invaliding txs are in 42000 and 43000. The query will return as if the exit was valid

spotted on 0e61e3137b6a074e3dd1e843e50b9048360cd979 but eb0e191823983c303adf489e5eeb7e2e9965b8c7 on master should act equivalent

invalid json requests grill the http-rpc APIs

Run this query to a watcher instance:

~c(echo '{"address": yolo}' | http POST localhost:7434/account.get_utxos) |>  :os.cmd()

returns:

'<!DOCTYPE html>\n<html>\n<head>\n    <meta charset="utf-8">\n    <title>Plug.Parsers.ParseError at POST /account.get_utxos</title>\n    <meta name="viewport" content="width=device-............

same with the child chain api

commit 815c500

Cannot create an authority address

Getting error when attempting to create an authority address.

mix run --no-start -e '
contents = OMG.Eth.DevHelpers.prepare_env!() |> OMG.Eth.DevHelpers.create_conf_file()
"~/config.exs" |> Path.expand() |> File.write!(contents)
'

** (MatchError) no match of right hand side value: {:ok, %{"blockHash" => "0x9c6fddc0789f756fa796bc3119fe411e49150410e1557d4da16e4eccf4597a82", "blockNumber" => "0xcbb", "contractAddress" => "0xf13d901d39c263cac2a63dc5c2d0c0c9cd6f93f1", "cumulativeGasUsed" => "0x3fc820", "from" => "0x43f9925f9e7775e974524953cbc28df7c6876b9c", "gasUsed" => "0x3fc820", "logs" => [], "logsBloom" => "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000", "status" => "0x0", "to" => nil, "transactionHash" => "0xf6751a93fed5c4ae6a85786df5d7f68f52424bb45cc23226f01e877d07bc9f95", "transactionIndex" => "0x0"}}
test/support/dev_helpers.ex:98: OMG.Eth.DevHelpers.transact_sync!/1
test/support/dev_helpers.ex:107: OMG.Eth.DevHelpers.deploy_sync!/1
test/support/dev_helpers.ex:49: OMG.Eth.DevHelpers.prepare_env!/1
(stdlib) erl_eval.erl:680: :erl_eval.do_apply/6
(stdlib) erl_eval.erl:888: :erl_eval.expr_list/6
(stdlib) erl_eval.erl:411: :erl_eval.expr/5
(stdlib) erl_eval.erl:449: :erl_eval.expr/5
(stdlib) erl_eval.erl:126: :erl_eval.exprs/5
(elixir) lib/code.ex:232: Code.eval_string/3
(elixir) lib/enum.ex:765: Enum."-each/2-lists^foreach/1-0-"/2
(elixir) lib/enum.ex:765: Enum.each/2
(mix) lib/mix/tasks/run.ex:142: Mix.Tasks.Run.run/5
(mix) lib/mix/tasks/run.ex:85: Mix.Tasks.Run.run/1
(mix) lib/mix/task.ex:316: Mix.Task.run_task/3
(mix) lib/mix/cli.ex:79: Mix.CLI.run_task/2

I also had to change the tree id for the plasma contract branch, I used the tree id: f2f0b4793a1ee07a5a2aeb86fe4d8f1df83a86aa by updating elixir-omg/mix.lock with the new id. If this is causing my issue then which id should i use? The existing one was failing the build.

Setry causing exception during error handling

Exception thrown in Watcher after responding to invalid request

2019-02-12 10:34:37.670 [error] ⋅#PID<0.583.0> running OMG.Watcher.Web.Endpoint (cowboy_protocol) terminated
Server: localhost:7434 (http)
Request: POST /account.get_balance
** (exit) an exception was raised:
    ** (KeyError) key :peer not found in: %Plug.Conn{adapter: {Plug.Adapters.Cowboy.Conn, :...}, assigns: %{}, before_send: [], body_params: %Plug.Conn.Unfetched{aspect: :body_params}, cookies: %{}, halted: false, host: "localhost", method: "POST", owner: #PID<0.583.0>, params: %{}, path_info: ["account.get_balance"], path_params: %{}, port: 7434, private: %{}, query_params: %{}, query_string: "", remote_ip: {127, 0, 0, 1}, req_cookies: %{}, req_headers: [{"content-type", "application/json"}, {"cache-control", "no-cache"}, {"postman-token", "982e2847-c6c3-4ce6-976b-90f1c4ca348d"}, {"user-agent", "PostmanRuntime/7.1.1"}, {"accept", "*/*"}, {"host", "localhost:7434"}, {"accept-encoding", "gzip, deflate"}, {"content-length", "20"}, {"connection", "keep-alive"}], request_path: "/account.get_balance", resp_body: nil, resp_cookies: %{}, resp_headers: [{"cache-control", "max-age=0, private, must-revalidate"}], scheme: :http, script_name: [], secret_key_base: nil, state: :unset, status: nil}
        lib/sentry/plug.ex:174: Sentry.Plug.build_request_interface_data/2
        (omg_watcher) lib/web/endpoint.ex:15: OMG.Watcher.Web.Endpoint.call/2
        (plug_cowboy) lib/plug/cowboy/handler.ex:18: Plug.Adapters.Cowboy.Handler.upgrade/4
        (cowboy) /home/pnowosie/Proj/omisego/deps/cowboy/src/cowboy_protocol.erl:442: :cowboy_protocol.execute/4⋅

Here is Sentry repo issue. PR was merged, we're probably running older version still.

Watcher returns 500 Internal Server Error from /utxos if given an invalid address

If I pass an invalid address format to the watcher e.g. http://localhost:4000/utxos?address=0 the error response is 500 Internal Server Error.

I would have expected 400 Bad Request

I went to check if swagger validated the address param, but I can't find the swagger.json file. The omg_watcher/config.exs has

  swagger_files: %{
    "priv/static/swagger.json" => [router: OMG.Watcher.Web.Router]
  }

that file doesn't seem to exist in the repo, but calling http://localhost:4000/api/swagger/swagger.json does return the swagger doc, so I'm confused 😕

Fix the annoying non-failing error messages in integration tests

[error] module=DBConnection.Connection function=disconnect/2 ⋅Postgrex.Protocol (#PID<0.1727.0>) disconnected: ** (DBConnection.ConnectionError) owner #PID<0.1753.0> timed out because it owned the connection for longer than 15000ms⋅

`/transaction.submit` change proposition: remove the need to RLP encode signatures

We wanted to change Child chain and corresponding Watcher's /transaction.submit endpoint to receive:

• transaction - RLP & HEX encoded bytes of (raw) transaction
• signatures - array of HEX encoded bytes of signatures

❗ Endpoint version which accepts just transaction will be removed ❗

Currently these endpoints receive transaction with signatures together, RLP-encoded.

Reason for change:
We've currently introduced Informational Watcher helper /transaction.create which can help clients to compose transaction for a provided payment order. Helper will respond with RLP-encoded transaction bytes which is ready to sign by the UTXO owner(s).
If the client could just produce signatures (2 standard crypto operations) and not be forced to bind the transaction & signatures together, it would simplify the tech stack on client side.

More versatile clients who needs more control over UTxOs being used in transaction have to still produce correct transaction (which includes RLP encoding). However they could save to not RLP encode transaction & signatures together.

Rollout plan
We would introduce parameter signatures to ☝️ endpoints, so there will be:

1. possibility to send full RLP-encoded transaction with signatures using just transaction request field,
2. send raw transaction and signatures separately using transaction and signatures fields in the request body.

As we believe the 2nd version is better and is still the time to move, we will opt for obsolete and remove 1st version for the v1.0.

I looking forward for any feedback. Thanks ❗

Watcher not showing available UTXOs for deposits to Plasma contract

After following the steps within demo_01 to depositing ETH on Plasma, UTXOs are not showing as available upon queries on the watcher until the initial transaction is made on the Plasma chain

Elixir 1.8 Exceptions in test

https://circleci.com/gh/omisego/elixir-omg/665

In stage Integration Test & Coveralls there are several exceptions. Please determine the cause and verify if it's safe to move to Elixir 1.8.

Watcher does not run in MIX_ENV=prod mode

Watcher runs in dev mode, but the following exception is encountered when running in prod mode. Build 42c549a20c75a8d90db8a071bc177d6248cd2852.

watcher-0 watcher Last message: :sync⋅
watcher-0 watcher 2019-02-18 04:52:56.804 [error] module=gen_server function=error_info/7 ⋅GenServer :exit_challenger terminating
watcher-0 watcher ** (stop) exited in: GenServer.call(OMG.Watcher.ExitProcessor, {:challenge_exits, []}, 5000)
watcher-0 watcher     ** (EXIT) an exception was raised:
watcher-0 watcher         ** (Ecto.ConstraintError) constraint error when attempting to insert struct:
watcher-0 watcher
watcher-0 watcher     * unique: ethevents_pkey
watcher-0 watcher
watcher-0 watcher If you would like to convert this constraint into an error, please
watcher-0 watcher call unique_constraint/3 in your changeset and define the proper
watcher-0 watcher constraint name. The changeset has not defined any constraint.
watcher-0 watcher
watcher-0 watcher             (ecto) lib/ecto/repo/schema.ex:574: anonymous fn/4 in Ecto.Repo.Schema.constraints_to_errors/3
watcher-0 watcher             (elixir) lib/enum.ex:1327: Enum."-map/2-lists^map/1-0-"/2
watcher-0 watcher             (ecto) lib/ecto/repo/schema.ex:559: Ecto.Repo.Schema.constraints_to_errors/3
watcher-0 watcher             (ecto) lib/ecto/repo/schema.ex:222: anonymous fn/14 in Ecto.Repo.Schema.do_insert/4
watcher-0 watcher             (ecto) lib/ecto/repo/schema.ex:774: anonymous fn/3 in Ecto.Repo.Schema.wrap_in_transaction/6
watcher-0 watcher             (ecto) lib/ecto/adapters/sql.ex:576: anonymous fn/3 in Ecto.Adapters.SQL.do_transaction/3
watcher-0 watcher             (db_connection) lib/db_connection.ex:1283: DBConnection.transaction_run/4
watcher-0 watcher             (db_connection) lib/db_connection.ex:1207: DBConnection.run_begin/3
watcher-0 watcher     (elixir) lib/gen_server.ex:989: GenServer.call/3
watcher-0 watcher     lib/ethereum_event_listener.ex:101: OMG.API.EthereumEventListener.sync_height/2
watcher-0 watcher     lib/ethereum_event_listener.ex:84: OMG.API.EthereumEventListener.handle_info/2
watcher-0 watcher     (stdlib) gen_server.erl:637: :gen_server.try_dispatch/4
watcher-0 watcher     (stdlib) gen_server.erl:711: :gen_server.handle_msg/6
watcher-0 watcher     (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
watcher-0 watcher Last message: :sync⋅
watcher-0 watcher 2019-02-18 04:52:56.807 [info] module=OMG.API.EthereumEventListener function=init/1 ⋅Starting EthereumEventListener for in_flight_exit_processor⋅
watcher-0 watcher 2019-02-18 04:52:56.809 [info] module=OMG.API.EthereumEventListener function=init/1 ⋅Starting EthereumEventListener for exit_processor⋅
watcher-0 watcher 2019-02-18 04:52:56.812 [info] module=application_controller function=info_exited/3 ⋅Application omg_watcher exited: shutdown⋅
watcher-0 watcher 2019-02-18 04:52:56.874 [error] module=gen_event function=report_error/5 ⋅:gen_event handler Sentry.Logger installed in :error_logger terminating
watcher-0 watcher ** (stop) exited in: GenServer.call(Sentry.TaskSupervisor, {:start_task, [{:nonode@nohost, :error_logger, #PID<0.356.0>}, [#PID<0.356.0>], :monitor, {:erlang, :apply, [#Function<3.89165438/0 in Sentry.Client.do_send_event/3>, []]}], :temporary, nil}, :infinity)
watcher-0 watcher     ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
watcher-0 watcher Last message: {:error_report, #PID<0.393.0>, {#PID<0.419.0>, :crash_report, [[initial_call: {OMG.Watcher.ExitProcessor, :init, [:Argument__1]}, pid: #PID<0.419.0>, registered_name: OMG.Watcher.ExitProcessor, error_info: {:error, %Ecto.ConstraintError{constraint: "ethevents_pkey", message: "constraint error when attempting to insert struct:\n\n    * unique: ethevents_pkey\n\nIf you would like to convert this constraint into an error, please\ncall unique_constraint/3 in your changeset and define the proper\nconstraint name. The changeset has not defined any constraint.\n", type: :unique}, [{Ecto.Repo.Schema, :"-constraints_to_errors/3-fun-1-", 4, [file: 'lib/ecto/repo/schema.ex', line: 574]}, {Enum, :"-map/2-lists^map/1-0-", 2, [file: 'lib/enum.ex', line: 1327]}, {Ecto.Repo.Schema, :constraints_to_errors, 3, [file: 'lib/ecto/repo/schema.ex', line: 559]}, {Ecto.Repo.Schema, :"-do_insert/4-fun-1-", 14, [file: 'lib/ecto/repo/schema.ex', line: 222]}, {Ecto.Repo.Schema, :"-wrap_in_transaction/6-fun-0-", 3, [file: 'lib/ecto/repo/schema.ex', line: 774]}, {Ecto.Adapters.SQL, :"-do_transaction/3-fun-1-", 3, [file: 'lib/ecto/adapters/sql.ex', line: 576]}, {DBConnection, :transaction_run, 4, [file: 'lib/db_connection.ex', line: 1283]}, {DBConnection, :run_begin, 3, [file: 'lib/db_connection.ex', line: 1207]}]}, ancestors: [OMG.Watcher.Supervisor, OMG.Watcher.RootSupervisor, #PID<0.394.0>], message_queue_len: 5, messages: [{:"$gen_call", {#PID<0.424.0>, #Reference<0.210763670.3553624067.91748>}, {:piggyback_exits, []}}, {:"$gen_call", {#PID<0.423.0>, #Reference<0.210763670.3553624065.97256>}, {:new_in_flight_exits, []}}, {:"$gen_call", {#PID<0.422.0>, #Reference<0.210763670.3553624066.91237>}, {:challenge_exits, []}}, {:"$gen_call", {#PID<0.566.0>, #Reference<0.210763670.3553624066.91528>}, :check_validity}, {:"$gen_call", {#PID<0.557.0>, #Reference<0.210763670.3553624065.97451>}, :check_validity}], links: [#PID<0.396.0>], dictionary: [], trap_exit: false, status: :running, heap_size: 6772, stack_size: 27, reductions: 66248], []]}}⋅

Race condition in service startup prevents continuous deployment from working correctly

watcher-0 watcher 2019-02-27 03:43:25,762 INFO    :Starting Launcher
watcher-0 watcher 2019-02-27 03:43:25,769 INFO    :Service type to launch is Elixir Watcher
watcher-0 watcher 2019-02-27 03:43:25,769 INFO    :Starting launch process for build 9de723cf21ca81223f09d188e85c050a770c53db
watcher-0 watcher 2019-02-27 03:43:25,805 INFO    :Connected to the Ethereum client
watcher-0 watcher 2019-02-27 03:43:25,806 INFO    :Ethereum client is b'{"jsonrpc":"2.0","id":67,"result":"Geth/v1.8.22-stable/linux-amd64/go1.11.5"}\n'
watcher-0 watcher 2019-02-27 03:43:27,581 INFO    :Elixir mix compile successful
watcher-0 watcher 2019-02-27 03:43:27,581 INFO    :Using pre-deployed contract on network RINKEBY
watcher-0 watcher 2019-02-27 03:43:27,581 INFO    :Writing config_watcher.exs
watcher-0 watcher 2019-02-27 03:43:27,582 INFO    :Chain data found
watcher-0 watcher 2019-02-27 03:43:27,582 INFO    :Launcher process complete
watcher-0 watcher 2019-02-27 03:43:29.668 [info] module=OMG.Watcher.Application function=start/2 ⋅Starting OMG.Watcher.Application⋅
watcher-0 watcher 2019-02-27 03:43:29.670 [info] module=OMG.Watcher.Supervisor function=init/1 ⋅Starting OMG.Watcher.Supervisor⋅
watcher-0 watcher 2019-02-27 03:43:30.154 [info] module=OMG.API.EthereumEventListener function=init/1 ⋅Starting EthereumEventListener for depositor⋅
watcher-0 watcher 2019-02-27 03:43:30.167 [info] module=OMG.API.EthereumEventListener function=init/1 ⋅Starting EthereumEventListener for convenience_deposit_processor⋅
watcher-0 watcher 2019-02-27 03:43:30.173 [info] module=OMG.DB function=exit_infos/1 ⋅Reading exits' info, this might take a while. Allowing 60000 ms⋅
watcher-0 watcher 2019-02-27 03:43:30.532 [info] module=OMG.DB function=in_flight_exits_info/1 ⋅Reading in flight exits' info, this might take a while. Allowing 60000 ms⋅
watcher-0 watcher 2019-02-27 03:43:30.819 [info] module=OMG.DB function=competitors_info/1 ⋅Reading competitors' info, this might take a while. Allowing 60000 ms⋅
watcher-0 watcher 2019-02-27 03:43:31.138 [info] module=OMG.Watcher.ExitProcessor function=init/1 ⋅Initializing with: {:ok, %OMG.Watcher.ExitProcessor.Core{competitors: %{<<5, 25, 246, 202, 22, 10, 22, 70, 48, 85, 161, 205, 64, 232, 27, 213, 15, 100, 114, 178, 178, 199, 135, 141, 109, 176, 223, 160, 38, 20, 207, 27>> => %OMG.Watcher.ExitProcessor.CompetitorInfo{competing_input_index: 0, competing_input_signature: <<95, 47, 135, 253, 150, 132, 113, 83, 52, 114, 235, 166, 88, 44, 182, 27, 128, 6, 206, 198, 178, 176, 226, 146, 100, 6, 252, 204, 154, 191, 105, 2, 126, 54, 70, 229, 203, 172, 107, 25, 47, 37, 53, 217, ...>>, tx: %OMG.API.State.Transaction.Signed{raw_tx: %OMG.API.State.Transaction{inputs: [%{blknum: 92000, oindex: 0, txindex: 0}, %{blknum: 0, oindex: 0, txindex: 0}, %{blknum: 0, oindex: 0, txindex: 0}, %{blknum: 0, oindex: 0, txindex: 0}], outputs: [%{amount: 5, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, owner: <<211, 135, 183, 91, 96, 210, 71, 152, 137, 144, 134, 120, 136, 127, 42, 235, 50, 221, 215, 8>>}, %{amount: 0, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, owner: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>}, %{amount: 0, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, owner: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>}, %{amount: 0, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, owner: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>}]}, signed_tx_bytes: nil, sigs: [<<95, 47, 135, 253, 150, 132, 113, 83, 52, 114, 235, 166, 88, 44, 182, 27, 128, 6, 206, 198, 178, 176, 226, 146, 100, 6, 252, 204, 154, 191, 105, 2, 126, 54, 70, 229, 203, 172, 107, ...>>]}}, <<111, 131, 146, 125, 71, 86, 162, 126, 71, 74, 144, 199, 8, 208, 138, 113, 246, 168, 26, 245, 50, 231, 80, 211, 236, 69, 201, 157, 182, 149, 211, 238>> => %OMG.Watcher.ExitProcessor.CompetitorInfo{competing_input_index: 0, competing_input_signature: <<208, 138, 238, 15, 28, 71, 112, 128, 135, 30, 209, 68, 79, 99, 26, 237, 118, 221, 179, 32, 253, 66, 115, 3, 130, 6, 36, 231, 102, 247, 12, 80, 95, 91, 7, 239, 13, 21, 41, 2, 192, 76, 195, ...>>, tx: %OMG.API.State.Transaction.Signed{raw_tx: %OMG.API.State.Transaction{inputs: [%{blknum: 88002, oindex: 0, txindex: 0}, %{blknum: 0, oindex: 0, txindex: 0}, %{blknum: 0, oindex: 0, txindex: 0}, %{blknum: 0, oindex: 0, txindex: 0}], outputs: [%{amount: 9, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, owner: <<131, 22, 38, 25, 106, 35, 178, 202, 36, 239, 138, 121, 90, 96, 132, 55, 44, 185, 161, 193>>}, %{amount: 1, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, owner: <<211, 135, 183, 91, 96, 210, 71, 152, 137, 144, 134, 120, 136, 127, 42, 235, 50, 221, 215, 8>>}, %{amount: 0, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, owner: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>}, %{amount: 0, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, owner: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>}]}, signed_tx_bytes: nil, sigs: [<<208, 138, 238, 15, 28, 71, 112, 128, 135, 30, 209, 68, 79, 99, 26, 237, 118, 221, 179, 32, 253, 66, 115, 3, 130, 6, 36, 231, 102, 247, 12, 80, 95, 91, 7, 239, 13, 21, ...>>]}}, <<128, 179, 44, 122, 103, 36, 204, 127, 151, 184, 247, 6, 15, 128, 78, 110, 113, 109, 201, 125, 19, 52, 186, 194, 48, 235, 103, 160, 89, 22, 197, 201>> => %OMG.Watcher.ExitProcessor.CompetitorInfo{competing_input_index: 0, competing_input_signature: <<81, 133, 116, 4, 172, 120, 101, 166, 144, 71, 153, 108, 136, 4, 69, 49, 57, 16, 254, 2, 223, 44, 112, 129, 187, 250, 182, 84, 207, 79, 70, 8, 13, 81, 153, 214, 83, 103, 175, 119, 233, 28, ...>>, tx: %OMG.API.State.Transaction.Signed{raw_tx: %OMG.API.State.Transaction{inputs: [%{blknum: 93000, oindex: 0, txindex: 0}, %{blknum: 0, oindex: 0, txindex: 0}, %{blknum: 0, oindex: 0, txindex: 0}, %{blknum: 0, oindex: 0, txindex: 0}], outputs: [%{amount: 4, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, owner: <<211, 135, 183, 91, 96, 210, 71, 152, 137, 144, 134, 120, 136, 127, 42, 235, 50, 221, 215, 8>>}, %{amount: 0, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, owner: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>}, %{amount: 0, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, owner: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>}, %{amount: 0, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, owner: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>}]}, signed_tx_bytes: nil, sigs: [<<81, 133, 116, 4, 172, 120, 101, 166, 144, 71, 153, 108, 136, 4, 69, 49, 57, 16, 254, 2, 223, 44, 112, 129, 187, 250, 182, 84, 207, 79, 70, 8, 13, 81, 153, 214, 83, ...>>]}}}, exits: %{{:utxo_position, 41000, 0, 1} => %OMG.Watcher.ExitProcessor.ExitInfo{amount: 211200000, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, eth_height: 3909229, is_active: true, owner: <<248, 107, 91, 28, 44, 141, 225, 234, 77, 199, 55, 200, 73, 39, 35, 64, 250, 53, 97, 197>>}, {:utxo_position, 88000, 0, 2} => %OMG.Watcher.ExitProcessor.ExitInfo{amount: 1, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, eth_height: 3914646, is_active: true, owner: <<211, 135, 183, 91, 96, 210, 71, 152, 137, 144, 134, 120, 136, 127, 42, 235, 50, 221, 215, 8>>}}, in_flight_exits: %{<<5, 25, 246, 202, 22, 10, 22, 70, 48, 85, 161, 205, 64, 232, 27, 213, 15, 100, 114, 178, 178, 199, 135, 141, 109, 176, 223, 160, 38, 20, 207, 27>> => %{__struct__: OMG.Watcher.ExitProcessor.InFlightExitInfo, contract_id: 2370316990351448335720246544088407191758179935141667380790, eth_height: 3914978, exit_map: %{0 => %{is_finalized: false, is_piggybacked: false}, 1 => %{is_finalized: false, is_piggybacked: false}, 2 => %{is_finalized: false, is_piggybacked: false}, 3 => %{is_finalized: false, is_piggybacked: false}, 4 => %{is_finalized: false, is_piggybacked: false}, 5 => %{is_finalized: false, is_piggybacked: false}, 6 => %{is_finalized: false, is_piggybacked: false}, 7 => %{is_finalized: false, is_piggybacked: false}}, is_active: true, is_canonical: true, oldest_competitor: nil, timestamp: 1550853739, tx: %OMG.API.State.Transaction.Signed{raw_tx: %OMG.API.State.Transaction{inputs: [%{blknum: 92000, oindex: 0, txindex: 0}, %{blknum: 0, oindex: 0, txindex: 0}, %{blknum: 0, oindex: 0, txindex: 0}, %{blknum: 0, oindex: 0, txindex: 0}], outputs: [%{amount: 5, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, owner: <<211, 135, 183, 91, 96, 210, 71, 152, 137, 144, 134, 120, 136, 127, 42, 235, 50, 221, 215, 8>>}, %{amount: 0, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, owner: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>}, %{amount: 0, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, owner: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>}, %{amount: 0, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, owner: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>}]}, signed_tx_bytes: nil, sigs: [<<95, 47, 135, 253, 150, 132, 113, 83, 52, 114, 235, 166, 88, 44, 182, 27, 128, 6, 206, 198, 178, 176, 226, 146, 100, 6, 252, 204, 154, 191, 105, ...>>]}, tx_pos: nil}, <<31, 74, 173, 77, 68, 34, 166, 4, 205, 30, 146, 119, 237, 111, 157, 147, 234, 58, 235, 240, 130, 196, 193, 237, 144, 112, 108, 148, 183, 87, 253, 240>> => %{__struct__: OMG.Watcher.ExitProcessor.InFlightExitInfo, contract_id: 3781925471690620519351709730178971532336882711452340714464, eth_height: 3909406, exit_map: %{0 => %{is_finalized: false, is_piggybacked: false}, 1 => %{is_finalized: false, is_piggybacked: false}, 2 => %{is_finalized: false, is_piggybacked: false}, 3 => %{is_finalized: false, is_piggybacked: false}, 4 => %{is_finalized: false, is_piggybacked: false}, 5 => %{is_finalized: false, is_piggybacked: false}, 6 => %{is_finalized: false, is_piggybacked: false}, 7 => %{is_finalize (truncated)⋅
watcher-0 watcher 2019-02-27 03:43:31.138 [info] module=OMG.API.EthereumEventListener function=init/1 ⋅Starting EthereumEventListener for exit_processor⋅
watcher-0 watcher 2019-02-27 03:43:31.141 [info] module=OMG.API.EthereumEventListener function=init/1 ⋅Starting EthereumEventListener for convenience_exit_processor⋅
watcher-0 watcher 2019-02-27 03:43:31.144 [info] module=OMG.API.EthereumEventListener function=init/1 ⋅Starting EthereumEventListener for exit_finalizer⋅
watcher-0 watcher 2019-02-27 03:43:31.146 [info] module=OMG.API.EthereumEventListener function=init/1 ⋅Starting EthereumEventListener for exit_challenger⋅
watcher-0 watcher 2019-02-27 03:43:31.148 [info] module=OMG.API.EthereumEventListener function=init/1 ⋅Starting EthereumEventListener for in_flight_exit_processor⋅
watcher-0 watcher 2019-02-27 03:43:31.151 [info] module=OMG.API.EthereumEventListener function=init/1 ⋅Starting EthereumEventListener for piggyback_processor⋅
watcher-0 watcher 2019-02-27 03:43:31.154 [info] module=OMG.API.EthereumEventListener function=init/1 ⋅Starting EthereumEventListener for competitor_processor⋅
watcher-0 watcher 2019-02-27 03:43:31.156 [info] module=OMG.API.EthereumEventListener function=init/1 ⋅Starting EthereumEventListener for challenges_responds_processor⋅
watcher-0 watcher 2019-02-27 03:43:31.158 [info] module=OMG.API.EthereumEventListener function=init/1 ⋅Starting EthereumEventListener for piggyback_challenges_processor⋅
watcher-0 watcher 2019-02-27 03:43:31.161 [info] module=OMG.API.EthereumEventListener function=init/1 ⋅Starting EthereumEventListener for ife_exit_finalizer⋅
watcher-0 watcher 2019-02-27 03:43:31.222 [info] module=Phoenix.Endpoint.CowboyAdapter function=start_link/3 ⋅Running OMG.Watcher.Web.Endpoint with cowboy 1.1.2 at http://localhost:7434⋅
watcher-0 watcher 2019-02-27 03:43:31.223 [info] module=OMG.Watcher.BlockGetter.Supervisor function=init/1 ⋅Starting OMG.Watcher.BlockGetter.Supervisor⋅
watcher-0 watcher 2019-02-27 03:43:31.224 [info] module=OMG.DB function=utxos/1 ⋅Reading UTXO set, this might take a while. Allowing 600000 ms⋅
watcher-0 watcher 2019-02-27 03:43:31.708 [info] module=OMG.API.State function=init/1 ⋅Started State, height: 99000, deposit height: 93010⋅
watcher-0 watcher 2019-02-27 03:43:31.755 [error] module=gen_server function=error_info/7 ⋅GenServer OMG.Watcher.ExitProcessor terminating
watcher-0 watcher ** (KeyError) key :tx_seen_in_blocks_at not found in: %{__struct__: OMG.Watcher.ExitProcessor.InFlightExitInfo, contract_id: 2370316990351448335720246544088407191758179935141667380790, eth_height: 3914978, exit_map: %{0 => %{is_finalized: false, is_piggybacked: false}, 1 => %{is_finalized: false, is_piggybacked: false}, 2 => %{is_finalized: false, is_piggybacked: false}, 3 => %{is_finalized: false, is_piggybacked: false}, 4 => %{is_finalized: false, is_piggybacked: false}, 5 => %{is_finalized: false, is_piggybacked: false}, 6 => %{is_finalized: false, is_piggybacked: false}, 7 => %{is_finalized: false, is_piggybacked: false}}, is_active: true, is_canonical: true, oldest_competitor: nil, timestamp: 1550853739, tx: %OMG.API.State.Transaction.Signed{raw_tx: %OMG.API.State.Transaction{inputs: [%{blknum: 92000, oindex: 0, txindex: 0}, %{blknum: 0, oindex: 0, txindex: 0}, %{blknum: 0, oindex: 0, txindex: 0}, %{blknum: 0, oindex: 0, txindex: 0}], outputs: [%{amount: 5, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, owner: <<211, 135, 183, 91, 96, 210, 71, 152, 137, 144, 134, 120, 136, 127, 42, 235, 50, 221, 215, 8>>}, %{amount: 0, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, owner: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>}, %{amount: 0, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, owner: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>}, %{amount: 0, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, owner: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>}]}, signed_tx_bytes: nil, sigs: [<<95, 47, 135, 253, 150, 132, 113, 83, 52, 114, 235, 166, 88, 44, 182, 27, 128, 6, 206, 198, 178, 176, 226, 146, 100, 6, 252, 204, 154, 191, 105, 2, 126, 54, 70, 229, 203, ...>>]}, tx_pos: nil}
watcher-0 watcher     (omg_watcher) lib/exit_processor/core.ex:916: anonymous fn/1 in OMG.Watcher.ExitProcessor.Core.find_ifes_in_blocks/2
watcher-0 watcher     (elixir) lib/enum.ex:907: anonymous fn/3 in Enum.filter/2
watcher-0 watcher     (stdlib) maps.erl:257: :maps.fold_1/3
watcher-0 watcher     (elixir) lib/enum.ex:1956: Enum.filter/2
watcher-0 watcher     (omg_watcher) lib/exit_processor/core.ex:916: OMG.Watcher.ExitProcessor.Core.find_ifes_in_blocks/2
watcher-0 watcher     (omg_watcher) lib/exit_processor.ex:336: OMG.Watcher.ExitProcessor.prepare_validity_check/1
watcher-0 watcher     (omg_watcher) lib/exit_processor.ex:255: OMG.Watcher.ExitProcessor.handle_call/3
watcher-0 watcher     (stdlib) gen_server.erl:661: :gen_server.try_handle_call/4
watcher-0 watcher     (stdlib) gen_server.erl:690: :gen_server.handle_msg/6
watcher-0 watcher     (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
watcher-0 watcher Last message (from OMG.Watcher.BlockGetter): :check_validity⋅
watcher-0 watcher 2019-02-27 03:43:31.759 [info] module=OMG.DB function=exit_infos/1 ⋅Reading exits' info, this might take a while. Allowing 60000 ms⋅
watcher-0 watcher 2019-02-27 03:43:31.811 [error] module=gen_event function=report_error/5 ⋅:gen_event handler Sentry.Logger installed in :error_logger terminating
watcher-0 watcher ** (stop) exited in: GenServer.call(Sentry.TaskSupervisor, {:start_task, [{:nonode@nohost, :error_logger, #PID<0.371.0>}, [#PID<0.371.0>], :monitor, {:erlang, :apply, [#Function<3.89165438/0 in Sentry.Client.do_send_event/3>, []]}], :temporary, nil}, :infinity)
watcher-0 watcher     ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
watcher-0 watcher Last message: {:error_report, #PID<0.408.0>, {#PID<0.435.0>, :crash_report, [[initial_call: {OMG.Watcher.ExitProcessor, :init, [:Argument__1]}, pid: #PID<0.435.0>, registered_name: OMG.Watcher.ExitProcessor, error_info: {:error, {:badkey, :tx_seen_in_blocks_at, %{__struct__: OMG.Watcher.ExitProcessor.InFlightExitInfo, contract_id: 2370316990351448335720246544088407191758179935141667380790, eth_height: 3914978, exit_map: %{0 => %{is_finalized: false, is_piggybacked: false}, 1 => %{is_finalized: false, is_piggybacked: false}, 2 => %{is_finalized: false, is_piggybacked: false}, 3 => %{is_finalized: false, is_piggybacked: false}, 4 => %{is_finalized: false, is_piggybacked: false}, 5 => %{is_finalized: false, is_piggybacked: false}, 6 => %{is_finalized: false, is_piggybacked: false}, 7 => %{is_finalized: false, is_piggybacked: false}}, is_active: true, is_canonical: true, oldest_competitor: nil, timestamp: 1550853739, tx: %OMG.API.State.Transaction.Signed{raw_tx: %OMG.API.State.Transaction{inputs: [%{blknum: 92000, oindex: 0, txindex: 0}, %{blknum: 0, oindex: 0, txindex: 0}, %{blknum: 0, oindex: 0, txindex: 0}, %{blknum: 0, oindex: 0, txindex: 0}], outputs: [%{amount: 5, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...>>, owner: <<211, 135, 183, 91, 96, 210, 71, 152, 137, 144, 134, 120, 136, 127, 42, 235, 50, 221, ...>>}, %{amount: 0, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...>>, owner: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...>>}, %{amount: 0, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...>>, owner: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...>>}, %{amount: 0, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...>>, owner: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...>>}]}, signed_tx_bytes: nil, sigs: [<<95, 47, 135, 253, 150, 132, 113, 83, 52, 114, 235, 166, 88, 44, 182, 27, 128, 6, 206, 198, 178, ...>>]}, tx_pos: nil}}, [{OMG.Watcher.ExitProcessor.Core, :"-find_ifes_in_blocks/2-fun-2-", 1, [file: 'lib/exit_processor/core.ex', line: 916]}, {Enum, :"-filter/2-fun-0-", 3, [file: 'lib/enum.ex', line: 907]}, {:maps, :fold_1, 3, [file: 'maps.erl', line: 257]}, {Enum, :filter, 2, [file: 'lib/enum.ex', line: 1956]}, {OMG.Watcher.ExitProcessor.Core, :find_ifes_in_blocks, 2, [file: 'lib/exit_processor/core.ex', line: 916]}, {OMG.Watcher.ExitProcessor, :prepare_validity_check, 1, [file: 'lib/exit_processor.ex', line: 336]}, {OMG.Watcher.ExitProcessor, :handle_call, 3, [file: 'lib/exit_processor.ex', line: 255]}, {:gen_server, :try_handle_call, 4, [file: 'gen_server.erl', line: 661]}, {:gen_server, :handle_msg, 6, [file: 'gen_server.erl', line: 690]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 249]}]}, ancestors: [OMG.Watcher.Supervisor, OMG.Watcher.RootSupervisor, #PID<0.409.0>], message_queue_len: 0, messages: [], links: [#PID<0.411.0>], dictionary: [], trap_exit: false, status: :running, heap_size: 28690, stack_size: 27, reductions: 325946], []]}}⋅
watcher-0 watcher 2019-02-27 03:43:32.180 [info] module=OMG.DB function=in_flight_exits_info/1 ⋅Reading in flight exits' info, this might take a while. Allowing 60000 ms⋅
watcher-0 watcher 2019-02-27 03:43:32.619 [info] module=OMG.DB function=competitors_info/1 ⋅Reading competitors' info, this might take a while. Allowing 60000 ms⋅
watcher-0 watcher 2019-02-27 03:43:33.081 [info] module=OMG.Watcher.ExitProcessor function=init/1 ⋅Initializing with: {:ok, %OMG.Watcher.ExitProcessor.Core{competitors: %{<<5, 25, 246, 202, 22, 10, 22, 70, 48, 85, 161, 205, 64, 232, 27, 213, 15, 100, 114, 178, 178, 199, 135, 141, 109, 176, 223, 160, 38, 20, 207, 27>> => %OMG.Watcher.ExitProcessor.CompetitorInfo{competing_input_index: 0, competing_input_signature: <<95, 47, 135, 253, 150, 132, 113, 83, 52, 114, 235, 166, 88, 44, 182, 27, 128, 6, 206, 198, 178, 176, 226, 146, 100, 6, 252, 204, 154, 191, 105, 2, 126, 54, 70, 229, 203, 172, 107, 25, 47, 37, 53, 217, ...>>, tx: %OMG.API.State.Transaction.Signed{raw_tx: %OMG.API.State.Transaction{inputs: [%{blknum: 92000, oindex: 0, txindex: 0}, %{blknum: 0, oindex: 0, txindex: 0}, %{blknum: 0, oindex: 0, txindex: 0}, %{blknum: 0, oindex: 0, txindex: 0}], outputs: [%{amount: 5, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, owner: <<211, 135, 183, 91, 96, 210, 71, 152, 137, 144, 134, 120, 136, 127, 42, 235, 50, 221, 215, 8>>}, %{amount: 0, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, owner: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>}, %{amount: 0, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, owner: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>}, %{amount: 0, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, owner: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>}]}, signed_tx_bytes: nil, sigs: [<<95, 47, 135, 253, 150, 132, 113, 83, 52, 114, 235, 166, 88, 44, 182, 27, 128, 6, 206, 198, 178, 176, 226, 146, 100, 6, 252, 204, 154, 191, 105, 2, 126, 54, 70, 229, 203, 172, 107, ...>>]}}, <<111, 131, 146, 125, 71, 86, 162, 126, 71, 74, 144, 199, 8, 208, 138, 113, 246, 168, 26, 245, 50, 231, 80, 211, 236, 69, 201, 157, 182, 149, 211, 238>> => %OMG.Watcher.ExitProcessor.CompetitorInfo{competing_input_index: 0, competing_input_signature: <<208, 138, 238, 15, 28, 71, 112, 128, 135, 30, 209, 68, 79, 99, 26, 237, 118, 221, 179, 32, 253, 66, 115, 3, 130, 6, 36, 231, 102, 247, 12, 80, 95, 91, 7, 239, 13, 21, 41, 2, 192, 76, 195, ...>>, tx: %OMG.API.State.Transaction.Signed{raw_tx: %OMG.API.State.Transaction{inputs: [%{blknum: 88002, oindex: 0, txindex: 0}, %{blknum: 0, oindex: 0, txindex: 0}, %{blknum: 0, oindex: 0, txindex: 0}, %{blknum: 0, oindex: 0, txindex: 0}], outputs: [%{amount: 9, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, owner: <<131, 22, 38, 25, 106, 35, 178, 202, 36, 239, 138, 121, 90, 96, 132, 55, 44, 185, 161, 193>>}, %{amount: 1, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, owner: <<211, 135, 183, 91, 96, 210, 71, 152, 137, 144, 134, 120, 136, 127, 42, 235, 50, 221, 215, 8>>}, %{amount: 0, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, owner: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>}, %{amount: 0, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, owner: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>}]}, signed_tx_bytes: nil, sigs: [<<208, 138, 238, 15, 28, 71, 112, 128, 135, 30, 209, 68, 79, 99, 26, 237, 118, 221, 179, 32, 253, 66, 115, 3, 130, 6, 36, 231, 102, 247, 12, 80, 95, 91, 7, 239, 13, 21, ...>>]}}, <<128, 179, 44, 122, 103, 36, 204, 127, 151, 184, 247, 6, 15, 128, 78, 110, 113, 109, 201, 125, 19, 52, 186, 194, 48, 235, 103, 160, 89, 22, 197, 201>> => %OMG.Watcher.ExitProcessor.CompetitorInfo{competing_input_index: 0, competing_input_signature: <<81, 133, 116, 4, 172, 120, 101, 166, 144, 71, 153, 108, 136, 4, 69, 49, 57, 16, 254, 2, 223, 44, 112, 129, 187, 250, 182, 84, 207, 79, 70, 8, 13, 81, 153, 214, 83, 103, 175, 119, 233, 28, ...>>, tx: %OMG.API.State.Transaction.Signed{raw_tx: %OMG.API.State.Transaction{inputs: [%{blknum: 93000, oindex: 0, txindex: 0}, %{blknum: 0, oindex: 0, txindex: 0}, %{blknum: 0, oindex: 0, txindex: 0}, %{blknum: 0, oindex: 0, txindex: 0}], outputs: [%{amount: 4, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, owner: <<211, 135, 183, 91, 96, 210, 71, 152, 137, 144, 134, 120, 136, 127, 42, 235, 50, 221, 215, 8>>}, %{amount: 0, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, owner: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>}, %{amount: 0, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, owner: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>}, %{amount: 0, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, owner: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>}]}, signed_tx_bytes: nil, sigs: [<<81, 133, 116, 4, 172, 120, 101, 166, 144, 71, 153, 108, 136, 4, 69, 49, 57, 16, 254, 2, 223, 44, 112, 129, 187, 250, 182, 84, 207, 79, 70, 8, 13, 81, 153, 214, 83, ...>>]}}}, exits: %{{:utxo_position, 41000, 0, 1} => %OMG.Watcher.ExitProcessor.ExitInfo{amount: 211200000, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, eth_height: 3909229, is_active: true, owner: <<248, 107, 91, 28, 44, 141, 225, 234, 77, 199, 55, 200, 73, 39, 35, 64, 250, 53, 97, 197>>}, {:utxo_position, 88000, 0, 2} => %OMG.Watcher.ExitProcessor.ExitInfo{amount: 1, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, eth_height: 3914646, is_active: true, owner: <<211, 135, 183, 91, 96, 210, 71, 152, 137, 144, 134, 120, 136, 127, 42, 235, 50, 221, 215, 8>>}}, in_flight_exits: %{<<5, 25, 246, 202, 22, 10, 22, 70, 48, 85, 161, 205, 64, 232, 27, 213, 15, 100, 114, 178, 178, 199, 135, 141, 109, 176, 223, 160, 38, 20, 207, 27>> => %{__struct__: OMG.Watcher.ExitProcessor.InFlightExitInfo, contract_id: 2370316990351448335720246544088407191758179935141667380790, eth_height: 3914978, exit_map: %{0 => %{is_finalized: false, is_piggybacked: false}, 1 => %{is_finalized: false, is_piggybacked: false}, 2 => %{is_finalized: false, is_piggybacked: false}, 3 => %{is_finalized: false, is_piggybacked: false}, 4 => %{is_finalized: false, is_piggybacked: false}, 5 => %{is_finalized: false, is_piggybacked: false}, 6 => %{is_finalized: false, is_piggybacked: false}, 7 => %{is_finalized: false, is_piggybacked: false}}, is_active: true, is_canonical: true, oldest_competitor: nil, timestamp: 1550853739, tx: %OMG.API.State.Transaction.Signed{raw_tx: %OMG.API.State.Transaction{inputs: [%{blknum: 92000, oindex: 0, txindex: 0}, %{blknum: 0, oindex: 0, txindex: 0}, %{blknum: 0, oindex: 0, txindex: 0}, %{blknum: 0, oindex: 0, txindex: 0}], outputs: [%{amount: 5, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, owner: <<211, 135, 183, 91, 96, 210, 71, 152, 137, 144, 134, 120, 136, 127, 42, 235, 50, 221, 215, 8>>}, %{amount: 0, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, owner: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>}, %{amount: 0, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, owner: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>}, %{amount: 0, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, owner: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>}]}, signed_tx_bytes: nil, sigs: [<<95, 47, 135, 253, 150, 132, 113, 83, 52, 114, 235, 166, 88, 44, 182, 27, 128, 6, 206, 198, 178, 176, 226, 146, 100, 6, 252, 204, 154, 191, 105, ...>>]}, tx_pos: nil}, <<31, 74, 173, 77, 68, 34, 166, 4, 205, 30, 146, 119, 237, 111, 157, 147, 234, 58, 235, 240, 130, 196, 193, 237, 144, 112, 108, 148, 183, 87, 253, 240>> => %{__struct__: OMG.Watcher.ExitProcessor.InFlightExitInfo, contract_id: 3781925471690620519351709730178971532336882711452340714464, eth_height: 3909406, exit_map: %{0 => %{is_finalized: false, is_piggybacked: false}, 1 => %{is_finalized: false, is_piggybacked: false}, 2 => %{is_finalized: false, is_piggybacked: false}, 3 => %{is_finalized: false, is_piggybacked: false}, 4 => %{is_finalized: false, is_piggybacked: false}, 5 => %{is_finalized: false, is_piggybacked: false}, 6 => %{is_finalized: false, is_piggybacked: false}, 7 => %{is_finalize (truncated)⋅
watcher-0 watcher 2019-02-27 03:43:33.086 [info] module=application_controller function=info_exited/3 ⋅Application omg_watcher exited: OMG.Watcher.Application.start(:normal, []) returned an error: shutdown: failed to start child: OMG.Watcher.BlockGetter.Supervisor
watcher-0 watcher     ** (EXIT) shutdown: failed to start child: OMG.Watcher.BlockGetter
watcher-0 watcher         ** (EXIT) exited in: GenServer.call(OMG.Watcher.ExitProcessor, :check_validity, 5000)
watcher-0 watcher ** (MatchError) no match of right hand side value: {:error, {:omg_watcher, {{:shutdown, {:failed_to_start_child, OMG.Watcher.BlockGetter.Supervisor, {:shutdown, {:failed_to_start_child, OMG.Watcher.BlockGetter, {{{:badkey, :tx_seen_in_blocks_at, %{__struct__: OMG.Watcher.ExitProcessor.InFlightExitInfo, contract_id: 2370316990351448335720246544088407191758179935141667380790, eth_height: 3914978, exit_map: %{0 => %{is_finalized: false, is_piggybacked: false}, 1 => %{is_finalized: false, is_piggybacked: false}, 2 => %{is_finalized: false, is_piggybacked: false}, 3 => %{is_finalized: false, is_piggybacked: false}, 4 => %{is_finalized: false, is_piggybacked: false}, 5 => %{is_finalized: false, is_piggybacked: false}, 6 => %{is_finalized: false, is_piggybacked: false}, 7 => %{is_finalized: false, is_piggybacked: false}}, is_active: true, is_canonical: true, oldest_competitor: nil, timestamp: 1550853739, tx: %OMG.API.State.Transaction.Signed{raw_tx: %OMG.API.State.Transaction{inputs: [%{blknum: 92000, oindex: 0, txindex: 0}, %{blknum: 0, oindex: 0, txindex: 0}, %{blknum: 0, oindex: 0, txindex: 0}, %{blknum: 0, oindex: 0, txindex: 0}], outputs: [%{amount: 5, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...>>, owner: <<211, 135, 183, 91, 96, 210, 71, 152, 137, 144, 134, 120, 136, 127, ...>>}, %{amount: 0, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...>>, owner: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...>>}, %{amount: 0, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...>>, owner: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...>>}, %{amount: 0, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...>>, owner: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...>>}]}, signed_tx_bytes: nil, sigs: [<<95, 47, 135, 253, 150, 132, 113, 83, 52, 114, 235, 166, 88, 44, 182, 27, 128, ...>>]}, tx_pos: nil}}, [{OMG.Watcher.ExitProcessor.Core, :"-find_ifes_in_blocks/2-fun-2-", 1, [file: 'lib/exit_processor/core.ex', line: 916]}, {Enum, :"-filter/2-fun-0-", 3, [file: 'lib/enum.ex', line: 907]}, {:maps, :fold_1, 3, [file: 'maps.erl', line: 257]}, {Enum, :filter, 2, [file: 'lib/enum.ex', line: 1956]}, {OMG.Watcher.ExitProcessor.Core, :find_ifes_in_blocks, 2, [file: 'lib/exit_processor/core.ex', line: 916]}, {OMG.Watcher.ExitProcessor, :prepare_validity_check, 1, [file: 'lib/exit_processor.ex', line: 336]}, {OMG.Watcher.ExitProcessor, :handle_call, 3, [file: 'lib/exit_processor.ex', line: 255]}, {:gen_server, :try_handle_call, 4, [file: 'gen_server.erl', line: 661]}, {:gen_server, :handle_msg, 6, [file: 'gen_server.erl', line: 690]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 249]}]}, {GenServer, :call, [OMG.Watcher.ExitProcessor, :check_validity, 5000]}}}}}}, {OMG.Watcher.Application, :start, [:normal, []]}}}}
watcher-0 watcher             ** (EXIT) an exception was raised:
watcher-0 watcher                 ** (KeyError) key :tx_seen_in_blocks_at not found in: %{__struct__: OMG.Watcher.ExitProcessor.InFlightExitInfo, contract_id: 2370316990351448335720246544088407191758179935141667380790, eth_height: 3914978, exit_map: %{0 => %{is_finalized: false, is_piggybacked: false}, 1 => %{is_finalized: false, is_piggybacked: false}, 2 => %{is_finalized: false, is_piggybacked: false}, 3 => %{is_finalized: false, is_piggybacked: false}, 4 => %{is_finalized: false, is_piggybacked: false}, 5 => %{is_finalized: false, is_piggybacked: false}, 6 => %{is_finalized: false, is_piggybacked: false}, 7 => %{is_finalized: false, is_piggybacked: false}}, is_active: true, is_canonical: true, oldest_competitor: nil, timestamp: 1550853739, tx: %OMG.API.State.Transaction.Signed{raw_tx: %OMG.API.State.Transaction{inputs: [%{blknum: 92000, oindex: 0, txindex: 0}, %{blknum: 0, oindex: 0, txindex: 0}, %{blknum: 0, oindex: 0, txindex: 0}, %{blknum: 0, oindex: 0, txindex: 0}], outputs: [%{amount: 5, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, owner: <<211, 135, 183, 91, 96, 210, 71, 152, 137, 144, 134, 120, 136, 127, 42, 235, 50, 221, 215, 8>>}, %{amount: 0, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, owner: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>}, %{amount: 0, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, owner: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>}, %{amount: 0, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, owner: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>}]}, signed_tx_bytes: nil, sigs: [<<95, 47, 135, 253, 150, 132, 113, 83, 52, 114, 235, 166, 88, 44, 182, 27, 128, 6, 206, 198, 178, 176, 226, 146, 100, 6, 252, 204, 154, 191, 105, 2, 126, 54, 70, 229, 203, ...>>]}, tx_pos: nil}
watcher-0 watcher                     (omg_watcher) lib/exit_processor/core.ex:916: anonymous fn/1 in OMG.Watcher.ExitProcessor.Core.find_ifes_in_blocks/2
watcher-0 watcher                     (elixir) lib/enum.ex:907: anonymous fn/3 in Enum.filter/2
watcher-0 watcher                     (stdlib) maps.erl:257: :maps.fold_1/3
watcher-0 watcher                     (elixir) lib/enum.ex:1956: Enum.filter/2
watcher-0 watcher                     (omg_watcher) lib/exit_processor/core.ex:916: OMG.Watcher.ExitProcessor.Core.find_ifes_in_blocks/2
watcher-0 watcher                     (omg_watcher) lib/exit_processor.ex:336: OMG.Watcher.ExitProcessor.prepare_validity_check/1
watcher-0 watcher                     (omg_watcher) lib/exit_processor.ex:255: OMG.Watcher.ExitProcessor.handle_call/3
watcher-0 watcher                     (stdlib) gen_server.erl:661: :gen_server.try_handle_call/4
watcher-0 watcher                     (stdlib) gen_server.erl:690: :gen_server.handle_msg/6
watcher-0 watcher                     (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3⋅
watcher-0 watcher 2019-02-27 03:43:33.087 [info] module=application_controller function=info_exited/3 ⋅Application phoenix_ecto exited: :stopped⋅
watcher-0 watcher 2019-02-27 03:43:33.087 [info] module=application_controller function=info_exited/3 ⋅Application ecto exited: :stopped⋅
watcher-0 watcher 2019-02-27 03:43:33.088 [info] module=application_controller function=info_exited/3 ⋅Application cors_plug exited: :stopped⋅
watcher-0 watcher 2019-02-27 03:43:33.088 [info] module=application_controller function=info_exited/3 ⋅Application omg_eth exited: :stopped⋅
watcher-0 watcher 2019-02-27 03:43:33.088 [info] module=application_controller function=info_exited/3 ⋅Application exexec exited: :stopped⋅
watcher-0 watcher 2019-02-27 03:43:33.088 [info] module=application_controller function=info_exited/3 ⋅Application ethereumex exited: :stopped⋅
watcher-0 watcher 2019-02-27 03:43:33.088 [info] module=application_controller function=info_exited/3 ⋅Application httpoison exited: :stopped⋅
watcher-0 watcher 2019-02-27 03:43:33.088 [info] module=application_controller function=info_exited/3 ⋅Application poolboy exited: :stopped⋅
watcher-0 watcher 2019-02-27 03:43:33.088 [info] module=application_controller function=info_exited/3 ⋅Application ex_abi exited: :stopped⋅
watcher-0 watcher 2019-02-27 03:43:33.088 [info] module=application_controller function=info_exited/3 ⋅Application exth_crypto exited: :stopped⋅
watcher-0 watcher 2019-02-27 03:43:33.088 [info] module=application_controller function=info_exited/3 ⋅Application keccakf1600 exited: :stopped⋅
watcher-0 watcher 2019-02-27 03:43:33.088 [info] module=application_controller function=info_exited/3 ⋅Application binary exited: :stopped⋅
watcher-0 watcher 2019-02-27 03:43:33.088 [info] module=application_controller function=info_exited/3 ⋅Application libsecp256k1 exited: :stopped⋅
watcher-0 watcher 2019-02-27 03:43:33.096 [info] module=application_controller function=info_exited/3 ⋅Application omg_db exited: :stopped⋅
watcher-0 watcher 2019-02-27 03:43:33.096 [info] module=application_controller function=info_exited/3 ⋅Application exleveldb exited: :stopped⋅
watcher-0 watcher     lib/utils.ex:25: XomgTasks.Utils.generic_run/2
watcher-0 watcher     (mix) lib/mix/task.ex:331: Mix.Task.run_task/3
watcher-0 watcher     (mix) lib/mix/cli.ex:79: Mix.CLI.run_task/2
watcher-0 watcher     (elixir) lib/code.ex:767: Code.require_file/2
watcher-0 watcher 2019-02-27 03:43:33.096 [info] module=application_controller function=info_exited/3 ⋅Application eleveldb exited: :stopped⋅
watcher-0 watcher 2019-02-27 03:43:33.097 [info] module=application_controller function=info_exited/3 ⋅Application appsignal exited: :stopped⋅
watcher-0 watcher 2019-02-27 03:43:33.097 [info] module=application_controller function=info_exited/3 ⋅Application poison exited: :stopped⋅
watcher-0 watcher 2019-02-27 03:43:33.098 [info] module=application_controller function=info_exited/3 ⋅Application hackney exited: :stopped⋅
watcher-0 watcher 2019-02-27 03:43:33.098 [info] module=application_controller function=info_exited/3 ⋅Application metrics exited: :stopped⋅
watcher-0 watcher 2019-02-27 03:43:33.098 [info] module=application_controller function=info_exited/3 ⋅Application ssl_verify_fun exited: :stopped⋅
watcher-0 watcher 2019-02-27 03:43:33.098 [info] module=application_controller function=info_exited/3 ⋅Application certifi exited: :stopped⋅
watcher-0 watcher 2019-02-27 03:43:33.098 [info] module=application_controller function=info_exited/3 ⋅Application mimerl exited: :stopped⋅
watcher-0 watcher 2019-02-27 03:43:33.098 [info] module=application_controller function=info_exited/3 ⋅Application idna exited: :stopped⋅
watcher-0 watcher 2019-02-27 03:43:33.098 [info] module=application_controller function=info_exited/3 ⋅Application unicode_util_compat exited: :stopped⋅
watcher-0 watcher 2019-02-27 03:43:33.098 [info] module=application_controller function=info_exited/3 ⋅Application decorator exited: :stopped⋅
watcher-0 watcher 2019-02-27 03:43:33.099 [info] module=application_controller function=info_exited/3 ⋅Application postgrex exited: :stopped⋅
watcher-0 watcher 2019-02-27 03:43:33.100 [info] module=application_controller function=info_exited/3 ⋅Application db_connection exited: :stopped⋅
watcher-0 watcher 2019-02-27 03:43:33.100 [info] module=application_controller function=info_exited/3 ⋅Application connection exited: :stopped⋅
watcher-0 watcher 2019-02-27 03:43:33.100 [info] module=application_controller function=info_exited/3 ⋅Application fake_server exited: :stopped⋅
watcher-0 watcher 2019-02-27 03:43:33.101 [info] module=application_controller function=info_exited/3 ⋅Application briefly exited: :stopped⋅
watcher-0 watcher 2019-02-27 03:43:33.101 [info] module=application_controller function=info_exited/3 ⋅Application deferred_config exited: :stopped⋅
watcher-0 watcher 2019-02-27 03:43:33.101 [info] module=application_controller function=info_exited/3 ⋅Application socket exited: :stopped⋅
watcher-0 watcher 2019-02-27 03:43:33.102 [info] module=application_controller function=info_exited/3 ⋅Application runtime_tools exited: :stopped⋅

Investigate why postgres db was overloaded with moderate traffic

This incident happen in devcon4, while the blockexplorer polling the /getTransactions endpoint with 1 second interval on client, it overloaded the postgres instance and finally killed it cc. @JBunCE .

From the information we have, the amount of load shouldn't be enough to break the postgres / watcher, @JBunCE even upgraded the instance but still the polling managed to push the CPU up to close to ~100%

Usability of IFE-related endpoints in the Watcher

List of recognized issues:

• When calling status.get available piggybacks are reported as byzantine events
• IFE's that are never piggybacked, don't have a notion of being finalized. Will always show up in in_flight_exits and available_piggybacks

Rename omisego/omisego repository

The repository should be renamed prior to making it public because omisego is the name of the organization building the OMG Network.

Some initial suggestions (no particular order): omg-network, omg-poa, omg-child-chain-server, omg-poa-servers.

Let's decide on something sensible for now, as this will need to be propagated throughout thedocumentation.

Exception on Watcher startup

2019-01-25 13:59:51.856 [error] ⋅GenServer OMG.API.RootChainCoordinator terminating
** (MatchError) no match of right hand side value: :invalid_synced_height_update
    lib/root_chain_coordinator/core.ex:65: OMG.API.RootChainCoordinator.Core.check_in/4
    lib/root_chain_coordinator.ex:75: OMG.API.RootChainCoordinator.handle_call/3
    (stdlib) gen_server.erl:636: :gen_server.try_handle_call/4
    (stdlib) gen_server.erl:665: :gen_server.handle_msg/6
    (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Last message (from :depositor): {:check_in, 3753010, :depositor}⋅
2019-01-25 13:59:51.932 [error] ⋅GenServer :depositor terminating
** (stop) exited in: GenServer.call(OMG.API.RootChainCoordinator, {:check_in, 3753010, :depositor}, 5000)
    ** (EXIT) an exception was raised:
        ** (MatchError) no match of right hand side value: :invalid_synced_height_update
            lib/root_chain_coordinator/core.ex:65: OMG.API.RootChainCoordinator.Core.check_in/4
            lib/root_chain_coordinator.ex:75: OMG.API.RootChainCoordinator.handle_call/3
            (stdlib) gen_server.erl:636: :gen_server.try_handle_call/4
            (stdlib) gen_server.erl:665: :gen_server.handle_msg/6
            (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
    (elixir) lib/gen_server.ex:836: GenServer.call/3
    lib/ethereum_event_listener.ex:103: OMG.API.EthereumEventListener.sync_height/2
    lib/ethereum_event_listener.ex:84: OMG.API.EthereumEventListener.handle_info/2
    (stdlib) gen_server.erl:616: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:686: :gen_server.handle_msg/6
    (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Last message: :sync⋅
2019-01-25 13:59:51.975 [error] ⋅GenEvent handler Sentry.Logger installed in :error_logger terminating
** (stop) exited in: GenServer.call(Sentry.TaskSupervisor, {:start_task, [#PID<0.32.0>, :monitor, {:nonode@nohost, :error_logger}, {:erlang, :apply, [#Function<3.81924908/0 in Sentry.Client.do_send_event/3>, []]}], :temporary, nil}, :infinity)
    ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
Last message: {:error_report, #PID<0.704.0>, {#PID<0.722.0>, :crash_report, [[initial_call: {OMG.API.RootChainCoordinator, :init, [:Argument__1]}, pid: #PID<0.722.0>, registered_name: OMG.API.RootChainCoordinator, error_info: {:error, {:badmatch, :invalid_synced_height_update}, [{OMG.API.RootChainCoordinator.Core, :check_in, 4, [file: 'lib/root_chain_coordinator/core.ex', line: 65]}, {OMG.API.RootChainCoordinator, :handle_call, 3, [file: 'lib/root_chain_coordinator.ex', line: 75]}, {:gen_server, :try_handle_call, 4, [file: 'gen_server.erl', line: 636]}, {:gen_server, :handle_msg, 6, [file: 'gen_server.erl', line: 665]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 247]}]}, ancestors: [OMG.Watcher.Supervisor, OMG.Watcher.RootSupervisor, #PID<0.705.0>], message_queue_len: 0, messages: [], links: [#PID<0.707.0>, #PID<0.725.0>], dictionary: [], trap_exit: false, status: :running, heap_size: 1598, stack_size: 27, reductions: 6416], []]}}⋅

After restarting the service it starts:

2019-01-25 14:24:02,108 INFO    :Starting Launcher
2019-01-25 14:24:02,116 INFO    :Service type to launch is Elixir Watcher
2019-01-25 14:24:02,116 INFO    :Starting launch process for build fb3ca10f523cc9add9da516021a59e5e45d67bf7
2019-01-25 14:24:02,126 INFO    :Connected to the Ethereum client
2019-01-25 14:24:02,126 INFO    :Ethereum client is b'{"jsonrpc":"2.0","id":67,"result":"Geth/v1.8.21-stable/linux-amd64/go1.11.4"}\n'
omg_rpc: generated priv/static/swagger.json
omg_watcher: generated priv/static/swagger.json
2019-01-25 14:24:04,994 INFO    :Elixir mix compile successful
2019-01-25 14:24:04,994 INFO    :Using pre-deployed contract on network RINKEBY
2019-01-25 14:24:04,994 INFO    :Writing config_watcher.exs
2019-01-25 14:24:05,005 INFO    :Deleted Watcher LevelDB data
2019-01-25 14:24:09,267 INFO    :Initialised Watcher chain database
2019-01-25 14:24:14,794 INFO    :Watcher Postgres instance initialised
2019-01-25 14:24:14,795 INFO    :Launcher process complete
2019-01-25 14:24:17.898 [info] module=OMG.Watcher.Application function=start_watcher_supervisor/0 ⋅Started application OMG.Watcher.Application⋅
2019-01-25 14:24:18.137 [info] module=OMG.API.EthereumEventListener function=init/1 ⋅Starting EthereumEventListener for depositor⋅
2019-01-25 14:24:18.158 [info] module=OMG.DB function=exit_infos/1 ⋅Reading exits' info, this might take a while. Allowing 60000 ms⋅
2019-01-25 14:24:18.165 [info] module=OMG.DB function=in_flight_exits_info/1 ⋅Reading in flight exits' info, this might take a while. Allowing 60000 ms⋅
2019-01-25 14:24:18.165 [info] module=OMG.DB function=competitors_info/1 ⋅Reading competitors' info, this might take a while. Allowing 60000 ms⋅
2019-01-25 14:24:18.168 [info] module=OMG.API.EthereumEventListener function=init/1 ⋅Starting EthereumEventListener for exit_processor⋅
2019-01-25 14:24:18.170 [info] module=OMG.API.EthereumEventListener function=init/1 ⋅Starting EthereumEventListener for exit_finalizer⋅
2019-01-25 14:24:18.171 [info] module=OMG.API.EthereumEventListener function=init/1 ⋅Starting EthereumEventListener for exit_challenger⋅
2019-01-25 14:24:18.172 [info] module=OMG.API.EthereumEventListener function=init/1 ⋅Starting EthereumEventListener for in_flight_exit_processor⋅
2019-01-25 14:24:18.174 [info] module=OMG.API.EthereumEventListener function=init/1 ⋅Starting EthereumEventListener for piggyback_processor⋅
2019-01-25 14:24:18.175 [info] module=OMG.API.EthereumEventListener function=init/1 ⋅Starting EthereumEventListener for competitor_processor⋅
2019-01-25 14:24:18.177 [info] module=OMG.API.EthereumEventListener function=init/1 ⋅Starting EthereumEventListener for challenges_responds_processor⋅
2019-01-25 14:24:18.178 [info] module=OMG.API.EthereumEventListener function=init/1 ⋅Starting EthereumEventListener for piggyback_challenges_processor⋅
2019-01-25 14:24:18.180 [info] module=OMG.API.EthereumEventListener function=init/1 ⋅Starting EthereumEventListener for ife_exit_finalizer⋅
2019-01-25 14:24:18.267 [info] module=Phoenix.Endpoint.CowboyHandler function=start_link/3 ⋅Running OMG.Watcher.Web.Endpoint with Cowboy using http://0.0.0.0:7434⋅
2019-01-25 14:24:18.278 [info] module=OMG.DB function=utxos/1 ⋅Reading UTXO set, this might take a while. Allowing 600000 ms⋅
2019-01-25 14:24:18.281 [info] module=OMG.API.State function=init/1 ⋅Started State, height: 0, deposit height: 0⋅

• Verified geth not syncing

[elixir-omg] Watcher fails with unchallenged_exit

Tests fail: https://circleci.com/gh/omisego/elixir-omg/272

Output from deployed environment:

2019-01-07 05:20:16,105 INFO    :Starting Launcher
2019-01-07 05:20:16,120 INFO    :Service type to launch is Elixir Watcher
2019-01-07 05:20:16,120 INFO    :Starting launch process for build 67fe9a321141b277d509361700edcf99a246bff8
2019-01-07 05:20:16,129 INFO    :Connected to the Ethereum client
2019-01-07 05:20:16,129 INFO    :Ethereum client is b'{"jsonrpc":"2.0","id":67,"result":"Geth/v1.8.20-stable/linux-amd64/go1.11.2"}\n'
omg_rpc: generated priv/static/swagger.json
omg_watcher: generated priv/static/swagger.json
2019-01-07 05:20:19,129 INFO    :Elixir mix compile successful
2019-01-07 05:20:19,129 INFO    :Using pre-deployed contract on network RINKEBY
2019-01-07 05:20:19,129 INFO    :Writing config_watcher.exs
2019-01-07 05:20:19,161 INFO    :Deleted Watcher LevelDB data
2019-01-07 05:20:22,163 INFO    :Initialised Watcher chain database
2019-01-07 05:20:26,137 INFO    :Watcher Postgres instance initialised
2019-01-07 05:20:26,138 INFO    :Launcher process complete
2019-01-07 05:20:29.514 [info] module=OMG.Watcher.Application function=start_watcher_supervisor/0 ⋅Started application OMG.Watcher.Application⋅
2019-01-07 05:20:29.858 [info] module=OMG.API.EthereumEventListener function=init/1 ⋅Starting EthereumEventListener for depositor⋅
2019-01-07 05:20:29.883 [info] module=OMG.DB function=exit_infos/1 ⋅Reading exits' info, this might take a while. Allowing 60000 ms⋅
2019-01-07 05:20:29.894 [info] module=OMG.API.EthereumEventListener function=init/1 ⋅Starting EthereumEventListener for exit_processor⋅
2019-01-07 05:20:29.896 [info] module=OMG.API.EthereumEventListener function=init/1 ⋅Starting EthereumEventListener for exit_finalizer⋅
2019-01-07 05:20:29.898 [info] module=OMG.API.EthereumEventListener function=init/1 ⋅Starting EthereumEventListener for exit_challenger⋅
2019-01-07 05:20:30.014 [info] module=Phoenix.Endpoint.CowboyHandler function=start_link/3 ⋅Running OMG.Watcher.Web.Endpoint with Cowboy using http://0.0.0.0:7434⋅
2019-01-07 05:20:30.029 [info] module=OMG.DB function=utxos/1 ⋅Reading UTXO set, this might take a while. Allowing 600000 ms⋅
2019-01-07 05:20:30.031 [info] module=OMG.API.State function=init/1 ⋅Started State, height: 0, deposit height: 0⋅
2019-01-07 05:20:30.220 [info] module=OMG.Watcher.BlockGetter.Core function=log_downloading_blocks/2 ⋅Child chain seen at block #43000. Downloading blocks [1000, 2000, 3000, 4000, 5000, 6000, 7000, 8000, 9000, 10000, 11000, 12000, 13000, 14000, 15000, 16000, 17000, 18000, 19000, 20000, 21000, 22000, 23000, 24000, 25000, 26000, 27000, 28000, 29000, 30000, 31000]⋅
2019-01-07 05:20:30.265 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #1000 DF42265CFCB6ECC9... with 1 txs⋅
2019-01-07 05:20:30.265 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #2000 8BF8E6655129953F... with 1 txs⋅
2019-01-07 05:20:30.266 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #14000 8BA1E699CC91F808... with 6 txs⋅
2019-01-07 05:20:30.267 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #9000 D327F79BC066B820... with 15 txs⋅
2019-01-07 05:20:30.267 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #15000 7C6417D1D45AA00C... with 1 txs⋅
2019-01-07 05:20:30.268 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #5000 6583443CFD3B2D76... with 8 txs⋅
2019-01-07 05:20:30.268 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #4000 72EC5A216BD44C22... with 1 txs⋅
2019-01-07 05:20:30.268 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #12000 3EF9162217A3FDB5... with 4 txs⋅
2019-01-07 05:20:30.268 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #6000 C5C8893F26054CE2... with 1 txs⋅
2019-01-07 05:20:30.268 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #11000 B58DABD92E6F2D0A... with 1 txs⋅
2019-01-07 05:20:30.268 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #13000 259A1A06F6DFE798... with 1 txs⋅
2019-01-07 05:20:30.268 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #3000 521782ECC3DF9134... with 1 txs⋅
2019-01-07 05:20:30.268 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #7000 E97DB557085CB97A... with 7 txs⋅
2019-01-07 05:20:30.268 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #8000 A87578F5E28AE308... with 3 txs⋅
2019-01-07 05:20:30.268 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #10000 E432AF1978DC4676... with 1 txs⋅
2019-01-07 05:20:30.268 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #16000 386033F1A748DD81... with 1 txs⋅
2019-01-07 05:20:30.270 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #18000 1657DC22CF3A4C0E... with 4 txs⋅
2019-01-07 05:20:30.270 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #19000 4B6F1610300317FF... with 1 txs⋅
2019-01-07 05:20:30.271 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #20000 5AAA44DEF02F4F78... with 1 txs⋅
2019-01-07 05:20:30.275 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #22000 F25536AFB66B72C7... with 1 txs⋅
2019-01-07 05:20:30.276 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #21000 DFCF897780ABBFC1... with 10 txs⋅
2019-01-07 05:20:30.276 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #23000 AD136BD2EC628D9B... with 1 txs⋅
2019-01-07 05:20:30.281 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #24000 BE62EA890FC54B98... with 2 txs⋅
2019-01-07 05:20:30.282 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #25000 8DB77B9BF45AD948... with 1 txs⋅
2019-01-07 05:20:30.287 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #29000 EBC0F76A1B6E81FE... with 10 txs⋅
2019-01-07 05:20:30.288 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #26000 9057D0418C12EF53... with 4 txs⋅
2019-01-07 05:20:30.291 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #28000 C0D0EDDEC018890C... with 38 txs⋅
2019-01-07 05:20:30.297 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #17000 9700424909057951... with 1 txs⋅
2019-01-07 05:20:30.302 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #27000 002E4423D9B90F21... with 158 txs⋅
2019-01-07 05:20:30.410 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #30000 22F288B7AD90A9A3... with 759 txs⋅
2019-01-07 05:20:30.467 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #31000 60B267339C4EE52E... with 1020 txs⋅
2019-01-07 05:20:30.476 [info] module=OMG.Watcher.BlockGetter.Core function=log_downloading_blocks/2 ⋅Child chain seen at block #43000. Downloading blocks [32000]⋅
2019-01-07 05:20:30.477 [info] module=OMG.Watcher.BlockGetter.Core function=log_downloading_blocks/2 ⋅Child chain seen at block #43000. Downloading blocks [33000]⋅
2019-01-07 05:20:30.480 [info] module=OMG.Watcher.BlockGetter.Core function=log_downloading_blocks/2 ⋅Child chain seen at block #43000. Downloading blocks [34000]⋅
2019-01-07 05:20:30.483 [info] module=OMG.Watcher.BlockGetter.Core function=log_downloading_blocks/2 ⋅Child chain seen at block #43000. Downloading blocks [35000]⋅
2019-01-07 05:20:30.487 [info] module=OMG.Watcher.BlockGetter.Core function=log_downloading_blocks/2 ⋅Child chain seen at block #43000. Downloading blocks [36000]⋅
2019-01-07 05:20:30.488 [info] module=OMG.Watcher.BlockGetter.Core function=log_downloading_blocks/2 ⋅Child chain seen at block #43000. Downloading blocks [37000]⋅
2019-01-07 05:20:30.492 [info] module=OMG.Watcher.BlockGetter.Core function=log_downloading_blocks/2 ⋅Child chain seen at block #43000. Downloading blocks [38000]⋅
2019-01-07 05:20:30.494 [info] module=OMG.Watcher.BlockGetter.Core function=log_downloading_blocks/2 ⋅Child chain seen at block #43000. Downloading blocks [39000]⋅
2019-01-07 05:20:30.499 [info] module=OMG.Watcher.BlockGetter.Core function=log_downloading_blocks/2 ⋅Child chain seen at block #43000. Downloading blocks [40000]⋅
2019-01-07 05:20:30.502 [info] module=OMG.Watcher.BlockGetter.Core function=log_downloading_blocks/2 ⋅Child chain seen at block #43000. Downloading blocks [41000]⋅
2019-01-07 05:20:30.505 [info] module=OMG.Watcher.BlockGetter.Core function=log_downloading_blocks/2 ⋅Child chain seen at block #43000. Downloading blocks [42000]⋅
2019-01-07 05:20:30.515 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #41000 FDC5E5FD081232E9... with 1 txs⋅
2019-01-07 05:20:30.520 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #42000 1EB45065B380B18C... with 1 txs⋅
2019-01-07 05:20:30.576 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #40000 087C34F40B8AEE44... with 660 txs⋅
2019-01-07 05:20:30.576 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #33000 3DCCFC14576D1FDE... with 1003 txs⋅
2019-01-07 05:20:30.578 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #32000 F333E62BF035058A... with 876 txs⋅
2019-01-07 05:20:30.599 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #38000 05F7F569F751354F... with 926 txs⋅
2019-01-07 05:20:30.624 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #35000 236006C0C62543B2... with 908 txs⋅
2019-01-07 05:20:30.624 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #39000 6E554087C96CEEA7... with 960 txs⋅
2019-01-07 05:20:30.642 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #34000 6B8D2B788B5737E6... with 960 txs⋅
2019-01-07 05:20:30.646 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #37000 F71234F1543860C4... with 909 txs⋅
2019-01-07 05:20:30.650 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #36000 289C40A4BF72C5EC... with 1009 txs⋅
2019-01-07 05:21:23.793 [info] module=OMG.API.State.Core function=deposit/2 ⋅Recognized deposits [%{amount: 10, blknum: 1, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, eth_height: 3550612, owner: <<75, 107, 231, 155, 226, 180, 118, 223, 207, 172, 170, 104, 148, 137, 210, 32, 201, 209, 236, 66>>}]⋅
2019-01-07 05:21:23.848 [error] module=OMG.Watcher.BlockGetter function=handle_cast/2 ⋅Stopping OMG.Watcher.BlockGetter because of :unchallenged_exit⋅

EIP-712 signing support

ERC-712 support would allow metamask to sign our child chain transactions from a web3 application.

Open question: do we want to support the current signing method in addition to ERC-712? Or do we want to migrate all signing to ERC-712?

can't challenge an invalid exit from deposit UTXO

a regression has been made here

this line returns nil for deposit utxos (naturally).

Also this change spills reliance on WatcherDB (postgres) back into the challenger.

Discovery of the exit_id (txhash) must be done differently, e.g. by using the UTXO store in OMG.DB which I think now holds hashes of utxo-creating transactions (4th field of the utxo structure).

(NOTE this code is in ExitProcessor now, as Challenger has been refactored out)

Endpoints of the child chain server start too early

found by @JBunCE. This happens when child chain endpoints are being hit right after the child chain app start is invoked:

2019-01-10 17:19:03,855 INFO    :Starting Launcher
2019-01-10 17:19:03,867 INFO    :Service type to launch is Elixir Childchain
2019-01-10 17:19:03,867 INFO    :Starting launch process for build e19fc136400d2f56191d113ca777b0ebb3bd4406
2019-01-10 17:19:03,867 INFO    :Childchain data found
2019-01-10 17:19:03,876 INFO    :Connected to the Ethereum client
2019-01-10 17:19:03,876 INFO    :Ethereum client is b'{"jsonrpc":"2.0","id":67,"result":"Geth/v1.8.20-stable/linux-amd64/go1.11.2"}\n'
omg_rpc: generated priv/static/swagger.json
omg_watcher: generated priv/static/swagger.json
2019-01-10 17:19:07,486 INFO    :Elixir mix compile successful
2019-01-10 17:19:07,486 INFO    :Using pre-deployed contract on network RINKEBY
2019-01-10 17:19:07,486 INFO    :Writing config.exs
2019-01-10 17:19:12,205 INFO    :Childchain database initialised
2019-01-10 17:19:12,205 INFO    :Launcher process complete
2019-01-10 17:19:16.320 [info] module=OMG.RPC.Application function=start/2 ⋅Started application OMG.RPC.Application⋅
2019-01-10 17:19:16.478 [info] module=Phoenix.Endpoint.CowboyHandler function=start_link/3 ⋅Running OMG.RPC.Web.Endpoint with Cowboy using http://0.0.0.0:9656⋅
2019-01-10 17:19:16.520 [info] module=OMG.API.Application function=start/2 ⋅Started application OMG.API.Application⋅
2019-01-10 17:19:16.584 [info] module=OMG.DB function=utxos/1 ⋅Reading UTXO set, this might take a while. Allowing 600000 ms⋅
2019-01-10 17:19:16.824 [error] ⋅#PID<0.770.0> running OMG.RPC.Web.Endpoint terminated
Server: childchain:9656 (http)
Request: POST /block.get
** (exit) exited in: GenServer.call(OMG.API.FreshBlocks, {:get, <<96, 178, 103, 51, 156, 78, 229, 46, 15, 65, 35, 190, 100, 236, 165, 96, 176, 158, 77, 135, 138, 103, 16, 217, 175, 148, 246, 165, 167, 197, 8, 169>>}, 5000)
    ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started⋅
2019-01-10 17:19:16.824 [error] ⋅#PID<0.763.0> running OMG.RPC.Web.Endpoint terminated
etc....

After the entire child chain app is up, the problem disappears.

This is because omg_rpc (and endpoints) stand up as a dependency and as such - before omg_api's supervisor which sports FreshBlocks and the rest of stuff required to handle blocks.

Watcher sees unchallenged_exit exits

From 7a2375bb06f5c0ae721f1693f3fc0414764e378c on Ari:

2019-02-08 17:13:21.036 [warn] module=OMG.Watcher.BlockGetter function=handle_info/2 ⋅Chain invalid when trying to download blocks, because of {:error, []}, won't try again⋅

2019-02-08 17:13:19.424 [warn] module=OMG.Watcher.BlockGetter.Core function=consider_exits/2 ⋅Chain invalid when taking exits into account, because of {:error, :unchallenged_exit}⋅

Possible cause from @pdobacz:

deposits are traded in before exits, because exits have clearance to be pulled with no coordination (to enable validation and challenges regardless of chain being byzaintine). So for a brief moment Watcher sees them as invalid and worse yet - unchallenged

Piggybacked outputs are reported as available when calling status.get endpoint in Watcher

After in-flight exit output is piggybacked, status.get endpoint still reports it as available for piggybacking.

To reproduce that follow docs/demo04 and after starting an in-flight exit:

1. piggyback onto one of the outputs
2. wait until watcher recognizes the piggyback
3. call status.get endpoint and look for event that reports piggybacked output as still available

status.get breaks if geth is syncing

If geth is syncing it returns an object as below, not true

Message

  no case clause matching: {:ok, %{"currentBlock" => "0x3bb704", "highestBlock" => "0x3bb705", "knownStates" => "0x43588f7", "pulledStates" => "0x43588f7", "startingBlock" => "0x3bb704"}}

Backtrace (last 10 lines)

(omg_eth) lib/geth.ex:22: OMG.Eth.Geth.node_ready/0
(omg_eth) lib/geth.ex:36: OMG.Eth.Geth.syncing?/0
(omg_watcher) lib/api/status.ex:54: OMG.Watcher.API.Status.get_status/0
(omg_watcher) lib/web/controllers/status.ex:29: OMG.Watcher.Web.Controller.Status.get_status/2
(omg_watcher) lib/web/controllers/status.ex:15: OMG.Watcher.Web.Controller.Status.action/2
(omg_watcher) lib/web/controllers/status.ex:15: OMG.Watcher.Web.Controller.Status.phoenix_controller_pipeline/2
(omg_watcher) lib/web/endpoint.ex:15: OMG.Watcher.Web.Endpoint.instrument/4
(phoenix) lib/phoenix/router.ex:275: Phoenix.Router.__call__/1
(omg_watcher) lib/web/endpoint.ex:15: OMG.Watcher.Web.Endpoint.plug_builder_call/2
(omg_watcher) lib/web/endpoint.ex:17: OMG.Watcher.Web.Endpoint."call (overridable 2)"/2

node_ready is broken. It should allow syncing? to just get true/false and only break if an explicit error is thrown.

Probably use of node_ready in BlockQueue and API.Status can be cleaned to make more sense.

Writes of deposits/exits to WatcherDB not idempotent

There's idempotency handling done for inserts to WatcherDB related to txs here.

There's not similar thing for deposits and exits, see here and here.

In the case when these insert requests are executed, but app goes down before they are acknoledged in the markers held in OMG.DB, we end up in corrupt state resulting in:

2019-02-21 12:06:45.387 [error] module=gen_server function=error_info/7 ⋅GenServer :convenience_exit_processor terminating
** (Ecto.ConstraintError) constraint error when attempting to insert struct:

    * unique: ethevents_pkey

To reproduce locally, insert a line like:

if not Enum.empty?(exits_or_deposits), do: raise ArgumentError.message("stop)")

just after the insert invocations above. Start a fresh sync to a non-empty child chain (e.g. staging as of today). When it borks on this raise, comment the raise out and attempt to continue syncing.

The easiest and consistent way to handle this would be to do a similar thing to what is done for txs in BlockGetter and have an idempotency mechanism in place.

personal_unlockAccount JSON Request encoding breaks Parity compatibility

Parity:

2018-11-16 10:59:00 UTC  TRACE rpc  Request: {"params":["ThisIsATestnetPassphrase"],"method":"personal_newAccount","jsonrpc":"2.0","id":0}.
2018-11-16 10:59:00 UTC  DEBUG rpc  [Some(Num(0))] Took 10ms
2018-11-16 10:59:00 UTC  DEBUG rpc  Response: {"jsonrpc":"2.0","result":"0x7ee9392ac7a5a0bad08b3943eb9bf8aa4bf02b6f","id":0}.
2018-11-16 10:59:00 UTC  TRACE rpc  Request: {"params":["0x7ee9392ac7a5a0bad08b3943eb9bf8aa4bf02b6f","ThisIsATestnetPassphrase",0],"method":"personal_unlockAccount","jsonrpc":"2.0","id":1}.
2018-11-16 10:59:00 UTC  DEBUG rpc  Response: {"jsonrpc":"2.0","error":{"code":-32602,"message":"Invalid params: invalid type: integer `0`, expected a 0x-prefixed, hex-encoded number of length 32."},"id":1}.

Elixir-omg:

mix run --no-start -e  '
   contents = OMG.Eth.DevHelpers.prepare_env!() |> OMG.Eth.DevHelpers.create_conf_file()
   "~/config.exs" |> Path.expand() |> File.write!(contents)
 '
** (MatchError) no match of right hand side value: {:error, %{"code" => -32602, "message" => "Invalid params: invalid type: integer `0`, expected a 0x-prefixed, hex-encoded number of length 32."}}
    test/support/dev_helpers.ex:109: OMG.Eth.DevHelpers.unlock_fund/1
    test/support/dev_helpers.ex:70: OMG.Eth.DevHelpers.create_and_fund_authority_addr/0
    test/support/dev_helpers.ex:44: OMG.Eth.DevHelpers.prepare_env!/1
    (stdlib) erl_eval.erl:670: :erl_eval.do_apply/6
    (stdlib) erl_eval.erl:878: :erl_eval.expr_list/6
    (stdlib) erl_eval.erl:404: :erl_eval.expr/5
    (stdlib) erl_eval.erl:438: :erl_eval.expr/5
    (stdlib) erl_eval.erl:122: :erl_eval.exprs/5
    (elixir) lib/code.ex:192: Code.eval_string/3
    (elixir) lib/enum.ex:737: Enum."-each/2-lists^foreach/1-0-"/2
    (elixir) lib/enum.ex:737: Enum.each/2
    (mix) lib/mix/tasks/run.ex:132: Mix.Tasks.Run.run/5
    (mix) lib/mix/tasks/run.ex:76: Mix.Tasks.Run.run/1
    (mix) lib/mix/task.ex:314: Mix.Task.run_task/3
    (mix) lib/mix/cli.ex:80: Mix.CLI.run_task/2
    (elixir) lib/code.ex:677: Code.require_file/2

personal_unlockAccount in https://github.com/omisego/elixir-omg/blob/7949ef4d2892feaa73dd247331cb492d5e12380d/apps/omg_eth/test/support/dev_helpers.ex#L109 is sending a decimal 0 instead of a hex encoded value. This needs to be 0x0 for compatibility with Parity.

https://wiki.parity.io/JSONRPC

Exiting utxos show up in the `get_utxos` result

introduced around when we decided to nevers stop pulling exit events from the root chain (to keep validating them, in case there's a block withholding attack).

This causes the (utxo adding) deposit event or a (utxo adding) block to be processed after the (utxo removing) exit event is handled.

Spotted by @kevsul for standard exits, probably bites the same for all flavors of MoreVP events.

2 ideas to fix come to mind:
1/ add more EthereumEventListeners that fill the WatcherDB out (which in turn serves the get_utxos endpoint)
2/ speak to WatcherDB from ExitProcessor rather than directly from ethereum events

Probably 2/ is more elegant. Also 1/ would require the revamped RootChainCoordinator to work at all

Watcher creates a large error message when headers cannot be handled

Watcher:

2019-01-30 08:57:54.703 [error] module=Plug.Adapters.Cowboy function=onresponse/4 ⋅Cowboy returned 400 and there are no headers in the connection.

This may happen if Cowboy is unable to parse the request headers,
for example, because there are too many headers or the header name
or value are too large (such as a large cookie).

You can customize those values when configuring your http/https
server. The configuration option and default values are shown below:

    protocol_options: [
      max_header_name_length: 64,
      max_header_value_length: 4096,
      max_headers: 100,
      max_request_line_length: 8096
    ]

Full logs:

2019-01-30 04:18:05,760 INFO    :Starting Launcher
2019-01-30 04:18:05,768 INFO    :Service type to launch is Elixir Watcher
2019-01-30 04:18:05,768 INFO    :Starting launch process for build c1453859efc1077cfbf6cffcf22f294eadaaae17
2019-01-30 04:18:05,780 INFO    :Connected to the Ethereum client
2019-01-30 04:18:05,780 INFO    :Ethereum client is b'{"jsonrpc":"2.0","id":67,"result":"Geth/v1.8.21-stable/linux-amd64/go1.11.4"}\n'
omg_rpc: generated priv/static/swagger.json
omg_watcher: generated priv/static/swagger.json
2019-01-30 04:18:08,803 INFO    :Elixir mix compile successful
2019-01-30 04:18:08,803 INFO    :Using pre-deployed contract on network RINKEBY
2019-01-30 04:18:08,803 INFO    :Writing config_watcher.exs
2019-01-30 04:18:08,838 INFO    :Deleted Watcher LevelDB data
2019-01-30 04:18:12,007 INFO    :Initialised Watcher chain database
2019-01-30 04:18:17,084 INFO    :Watcher Postgres instance initialised
2019-01-30 04:18:17,085 INFO    :Launcher process complete
2019-01-30 04:18:19.962 [info] module=OMG.Watcher.Application function=start_watcher_supervisor/0 ⋅Started application OMG.Watcher.Application⋅
2019-01-30 04:18:20.206 [info] module=OMG.API.EthereumEventListener function=init/1 ⋅Starting EthereumEventListener for depositor⋅
2019-01-30 04:18:20.224 [info] module=OMG.DB function=exit_infos/1 ⋅Reading exits' info, this might take a while. Allowing 60000 ms⋅
2019-01-30 04:18:20.231 [info] module=OMG.DB function=in_flight_exits_info/1 ⋅Reading in flight exits' info, this might take a while. Allowing 60000 ms⋅
2019-01-30 04:18:20.232 [info] module=OMG.DB function=competitors_info/1 ⋅Reading competitors' info, this might take a while. Allowing 60000 ms⋅
2019-01-30 04:18:20.234 [info] module=OMG.API.EthereumEventListener function=init/1 ⋅Starting EthereumEventListener for exit_processor⋅
2019-01-30 04:18:20.237 [info] module=OMG.API.EthereumEventListener function=init/1 ⋅Starting EthereumEventListener for exit_finalizer⋅
2019-01-30 04:18:20.239 [info] module=OMG.API.EthereumEventListener function=init/1 ⋅Starting EthereumEventListener for exit_challenger⋅
2019-01-30 04:18:20.241 [info] module=OMG.API.EthereumEventListener function=init/1 ⋅Starting EthereumEventListener for in_flight_exit_processor⋅
2019-01-30 04:18:20.243 [info] module=OMG.API.EthereumEventListener function=init/1 ⋅Starting EthereumEventListener for piggyback_processor⋅
2019-01-30 04:18:20.244 [info] module=OMG.API.EthereumEventListener function=init/1 ⋅Starting EthereumEventListener for competitor_processor⋅
2019-01-30 04:18:20.246 [info] module=OMG.API.EthereumEventListener function=init/1 ⋅Starting EthereumEventListener for challenges_responds_processor⋅
2019-01-30 04:18:20.248 [info] module=OMG.API.EthereumEventListener function=init/1 ⋅Starting EthereumEventListener for piggyback_challenges_processor⋅
2019-01-30 04:18:20.250 [info] module=OMG.API.EthereumEventListener function=init/1 ⋅Starting EthereumEventListener for ife_exit_finalizer⋅
2019-01-30 04:18:20.351 [info] module=Phoenix.Endpoint.CowboyHandler function=start_link/3 ⋅Running OMG.Watcher.Web.Endpoint with Cowboy using http://0.0.0.0:7434⋅
2019-01-30 04:18:20.362 [info] module=OMG.DB function=utxos/1 ⋅Reading UTXO set, this might take a while. Allowing 600000 ms⋅
2019-01-30 04:18:20.365 [info] module=OMG.API.State function=init/1 ⋅Started State, height: 0, deposit height: 0⋅
2019-01-30 08:57:54.703 [error] module=Plug.Adapters.Cowboy function=onresponse/4 ⋅Cowboy returned 400 and there are no headers in the connection.

This may happen if Cowboy is unable to parse the request headers,
for example, because there are too many headers or the header name
or value are too large (such as a large cookie).

You can customize those values when configuring your http/https
server. The configuration option and default values are shown below:

    protocol_options: [
      max_header_name_length: 64,
      max_header_value_length: 4096,
      max_headers: 100,
      max_request_line_length: 8096
    ]

Move omg_rpc supervision tree from omg_api to omg_rpc

The purpose of this piece of work is to have clean application borders. One issue is that OMG.RPC.Web.Endpoint (which is part of omg_rpc) supervision tree is manually included in the OMG.API.Sup. We should leave the umbrella application supervision tree in the appropriate app (omg_rpc).

But because omg_rpc is a dependency to omg_api it means the endpoints will be open before the application boots up. There's a couple of approaches to solve that.

1. Omg rpc application setting that gets read in a plug before any controllers are hit
2. Dynamic routes.
   https://gist.github.com/chrismccord/1c6b6fb086d5432d4e08

It would make sense to try and do this as a Stacking Theory for Systems Design where components would subscribe to alarms and others would react to that. In this case I imagine something like:

• omg_api starts and raises an alarm :omg_api_boot
• omg_rpc boots and loads all routes, but routes return not ready. Another approach is not to load the routes and postpone it for later -> https://gist.github.com/chrismccord/1c6b6fb086d5432d4e08
• When omg_api finishes it's boot process, it clears the alarm
• omg_rpc is subscribed to alarms of type :omg_api_boot and since the alarms is cleared, it would enable the api endpoints
• in case omg_api has other issues, it raises an alarm as well, and that would potentially disable the routes (depends on the type of the alarm)

I would like us to review the usage of OMG.RPC.Web.Encoding it seems like it's not named properly and it's used is some tests around the umbrella (watcher,api).

Question: Do we want two health check routes that would respond to infrastructure in terms of when a particular deployment becomes operational? Something like:
GET http://childchain|watcher/health 200 OK when application is completely booted and relevant alarms cleared.
GET http://childchain|watcher/health 503 Service Unavailable when application is in boot process or not able to boot.

New Challenger cannot challenge invalid exits from deposits

Can only be reproduced on staging, but after #357 is resolved or worked around (by disabling byzantine-chain halts).

exiting_utxopos = 40002000000000
~c(echo '{"utxo_pos": #{exiting_utxopos}}' | http POST localhost:7435/utxo.get_challenge_data) |>  :os.cmd()

will crash the Challenger with:

2019-01-11 11:27:46.752 [error] ⋅GenServer OMG.DB.LevelDBServer terminating
** (ArgumentError) argument error
    :erlang.bit_size(:not_found)
    (omg_db) lib/leveldb_core.ex:91: OMG.DB.LevelDBCore.key/2
    (elixir) lib/enum.ex:1314: Enum."-map/2-lists^map/1-0-"/2
    (omg_db) lib/leveldb_server.ex:71: OMG.DB.LevelDBServer.handle_call/3
    (stdlib) gen_server.erl:636: :gen_server.try_handle_call/4
    (stdlib) gen_server.erl:665: :gen_server.handle_msg/6
    (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Last message (from #PID<0.638.0>): {:blocks, [:not_found, <<147, 50, 249, 146, 60, 2, 8, 0, 139, 153, 44, 165, 193, 230, 90, 233, 133, 116, 16, 196, 72, 144, 95, 25, 14, 225, 149, 61, 248, 136, 214, 239>>]}⋅

because in https://github.com/omisego/elixir-omg/blob/master/apps/omg_watcher/lib/challenger.ex#L40 it is assumed that the utxo-creating block is not a deposit block.

Worth noting, that the utxo-creating block is only used to figure out the owner of the exit. It might be much easier to fix by leveraging the contract (calling exits and possibly getStandardExitId or its elixir counterpart), instead of going through blocks.

EDIT. Even more simply, owner can be taken from the ExitInfo stored in OMG.DB

Protocol.UndefinedError - enumerating sth in transaction.submit?

Message

  protocol Enumerable not implemented for <<59, 2, 183, 56, 142, 245, 12, 130, 103, 40, 127, 169, 160, 207, 222, 39, 152, 66, 125, 147, 170, 16, 225, 175, 119, 219, 2, 115, 185, 134, 34, 228, 47, 8, 137, 157, 41, 115, 214, 191, 239, 141, 75, 45, 144, 79, 102, 227, 227, 188, ...>>. This protocol is implemented for: Ecto.Adapters.SQL.Stream, Postgrex.Stream, DBConnection.Stream, DBConnection.PrepareStream, Stream, Function, HashDict, Map, Date.Range, List, Range, HashSet, MapSet, IO.Stream, File.Stream, GenEvent.Stream

Backtrace (last 10 lines)

(elixir) /root/deb/elixir_1.8.0-1/lib/elixir/lib/enum.ex:1: Enumerable.impl_for!/1
(elixir) /root/deb/elixir_1.8.0-1/lib/elixir/lib/enum.ex:141: Enumerable.reduce/3
(elixir) lib/enum.ex:311: Enum.all?/2
(omg_api) lib/state/transaction/signed.ex:54: OMG.API.State.Transaction.Signed.reconstruct/2
(omg_api) lib/core.ex:48: OMG.API.Core.recover_tx/1
(omg_api) lib/api.ex:31: OMG.API.submit/1
(omg_rpc) lib/web/controllers/transaction.ex:28: OMG.RPC.Web.Controller.Transaction.submit/2
(omg_rpc) lib/web/controllers/transaction.ex:15: OMG.RPC.Web.Controller.Transaction.action/2
(omg_rpc) lib/web/controllers/transaction.ex:15: OMG.RPC.Web.Controller.Transaction.phoenix_controller_pipeline/2
(omg_rpc) lib/web/endpoint.ex:15: OMG.RPC.Web.Endpoint.instrument/4

Bump propcheck

propcheck 1.1.3 RETIRED!
(invalid) Invalid setup macros, use 1.1.4 instead

Better separation of concerns between child chain and watcher

The purpose of this piece of work is to be able to cleanly deploy two applications - childchain and watcher.
We can achieve that by having clean application borders.

I propose that we utilize Publish–subscribe pattern through out the repository.

1. OMG application needs to be free from either childchain (omg_api) or watcher (omg_watcher)
2. OMG.State is coupled with OMG.EventerAPI witch calls OMG.Watcher.Eventer. This makes OMG modules coupled with the Watcher application.
3. OMG.State is coupled with OMG.BlockQueueAPI witch calls OMG.API.BlockQueue.Server. This makes OMG modules coupled with the Childchain application.

One additional thing to be sure to cover - subscriptions need to be persisted between process restarts by either application configuration or Process.monitor (and re-subscription in case of failure).

Arbitrary Data Transaction Field

The eWallet team and @kfichter would like to have a metadata field as part of the transaction format. This would be an arbitrary byte field. It opens up multiple use cases:

• allows multiple eWallets to coordinate by internal wallet ids
• can run game actions on-chain (e.g. recording chess moves)
• storing immutable data alongside a transaction

The field does not have to be very wide. Please discuss concerns here and come to a decision.

@T-Dnzt @robinclart @whoisjeremylam @kevsul @paulperegud @pdobacz

Watcher Design

We need to have a discussion about the Watcher at a high level since there's a lot of confusion about the vision of what purpose it serves, which leads us to confusion about things like #156.

Rather than tackling individual features ad-hoc, let's talk about how we envision the future of the Watcher.

People have characterized it as:

• A verifier (blocks, withholding, transaction consistency, root chain submissions)
• A full node (non-mining)
• A light node
• An convenience API gateway to the child chain
• A client/wallet

This specifics of what it does and what clients it serves (servers, browsers, wallets, etc.) can lead feature requirements (like whether we want CORS support).

As to actual functional requirements, here's what we need on the integrations end:

Proposed requirements

Block explorer APIs (each endpoint should have filtering capabilities):

• Getting all blocks
• Getting a block and its transactions
• Getting all transactions (paginated, per address, per list of addresses)
• Get specific transaction (by id)
• Getting UTXOs (paginated, per address, per list of addresses?)
• Get balance by address

eWallet APIs:

• Build transaction
• Submit signed transaction
• Get specific transaction (by correlation field)

Events:

• Subscribe to Websocket event endpoints notifying the following events (probably for a list of addresses?):
  • should exit due to [some fault]
  • Standard Exit (MVP)
    • Exit Challenge
  • In-flight exit (MoreVP)
    • Delegated?
    • Exit challenge
    • Challenge response
  • Exited successfully
  • new transaction submitted
  • new deposit
  • transaction shows up in a block
  • new block
  • ...we probably need to add to this list...

UTXO Management

• Merging / splitting / recommendations

Security

• Key management
• Verify consistency

Deployment

• Support multiple instances of watchers per node (many eWallets, many Watchers, 1 plasma node)

This is the beginning of a design discussion. Please drop your opinions in here.

/cc @whoisjeremylam @robinclart @omisego/blockchain @T-Dnzt @Pongch @kevsul @kfichter

Watcher does not apply blocks

After running watcher for a couple of hours it then stops to sync. It fails to apply one block.

The investigation showed that is is caused by the fact that submission of the block is missing.

Can't find sources for jakebunce/contractexchanger

@JBunCE It is used in docker-compose.yml. Is it closed source?

Problem with Browser interaction with Childchain and Watcher due to CORS issue

I got the following errors,

Failed to load http://localhost:4000/account/utxo?address=0x0b547a50e73f3b91741539ede4474ed1d36522e9: No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://localhost:8080' is therefore not allowed access.

It appears CORS is not setup on the server side, so the browser cannot query the watcher.

Got same problem locally on my machine as well as the staging server

Large number of transactions results in significantly larger database write latency

There's a correlation between the number of transactions and the database write latency:

2019-02-11 15:03:39.259 [info] module=OMG.Watcher.DB.Transaction function=update_with/1 ⋅Block #20000 persisted in DB done in 209.214ms⋅
2019-02-11 15:03:39.491 [info] module=OMG.Watcher.DB.Transaction function=update_with/1 ⋅Block #21000 persisted in DB done in 215.262ms⋅
2019-02-11 15:03:39.657 [info] module=OMG.Watcher.DB.Transaction function=update_with/1 ⋅Block #22000 persisted in DB done in 148.284ms⋅
2019-02-11 15:03:39.893 [info] module=OMG.Watcher.DB.Transaction function=update_with/1 ⋅Block #23000 persisted in DB done in 10.555ms⋅
2019-02-11 15:03:40.160 [info] module=OMG.Watcher.DB.Transaction function=update_with/1 ⋅Block #24000 persisted in DB done in 241.434ms⋅
2019-02-11 15:03:40.619 [info] module=OMG.Watcher.DB.Transaction function=update_with/1 ⋅Block #25000 persisted in DB done in 433.303ms⋅
2019-02-11 15:03:41.079 [info] module=OMG.Watcher.DB.Transaction function=update_with/1 ⋅Block #26000 persisted in DB done in 434.907ms⋅
2019-02-11 15:03:41.528 [info] module=OMG.Watcher.DB.Transaction function=update_with/1 ⋅Block #27000 persisted in DB done in 422.638ms⋅
2019-02-11 15:03:41.965 [info] module=OMG.Watcher.DB.Transaction function=update_with/1 ⋅Block #28000 persisted in DB done in 413.638ms⋅
2019-02-11 15:03:42.432 [info] module=OMG.Watcher.DB.Transaction function=update_with/1 ⋅Block #29000 persisted in DB done in 443.488ms⋅
2019-02-11 15:03:42.879 [info] module=OMG.Watcher.DB.Transaction function=update_with/1 ⋅Block #30000 persisted in DB done in 422.256ms⋅
2019-02-11 15:03:43.321 [info] module=OMG.Watcher.DB.Transaction function=update_with/1 ⋅Block #31000 persisted in DB done in 421.294ms⋅
2019-02-11 15:03:43.784 [info] module=OMG.Watcher.DB.Transaction function=update_with/1 ⋅Block #32000 persisted in DB done in 434.074ms⋅
2019-02-11 15:03:44.246 [info] module=OMG.Watcher.DB.Transaction function=update_with/1 ⋅Block #33000 persisted in DB done in 437.981ms⋅
2019-02-11 15:03:44.609 [info] module=OMG.Watcher.DB.Transaction function=update_with/1 ⋅Block #34000 persisted in DB done in 333.133ms⋅
2019-02-11 15:03:44.667 [info] module=OMG.Watcher.DB.Transaction function=update_with/1 ⋅Block #35000 persisted in DB done in 28.46ms⋅

2019-02-11 15:02:53.669 [info] module=OMG.Watcher.BlockGetter.Core function=log_downloading_blocks/2 ⋅Child chain seen at block #36000. Downloading blocks [9000]⋅
2019-02-11 15:02:53.671 [info] module=OMG.Watcher.BlockGetter.Core function=log_downloading_blocks/2 ⋅Child chain seen at block #36000. Downloading blocks [10000]⋅
2019-02-11 15:02:53.673 [info] module=OMG.Watcher.BlockGetter.Core function=log_downloading_blocks/2 ⋅Child chain seen at block #36000. Downloading blocks [11000]⋅
2019-02-11 15:02:53.675 [info] module=OMG.Watcher.BlockGetter.Core function=log_downloading_blocks/2 ⋅Child chain seen at block #36000. Downloading blocks [12000]⋅
2019-02-11 15:02:53.676 [info] module=OMG.Watcher.BlockGetter.Core function=log_downloading_blocks/2 ⋅Child chain seen at block #36000. Downloading blocks [13000]⋅
2019-02-11 15:02:53.677 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #11000 2C07BD87F040B000... with 1 txs⋅
2019-02-11 15:02:53.678 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #10000 AAD017AFC894292A... with 22 txs⋅
2019-02-11 15:02:53.678 [info] module=OMG.Watcher.BlockGetter.Core function=log_downloading_blocks/2 ⋅Child chain seen at block #36000. Downloading blocks [14000]⋅
2019-02-11 15:02:53.680 [info] module=OMG.Watcher.BlockGetter.Core function=log_downloading_blocks/2 ⋅Child chain seen at block #36000. Downloading blocks [15000]⋅
2019-02-11 15:02:53.682 [info] module=OMG.Watcher.BlockGetter.Core function=log_downloading_blocks/2 ⋅Child chain seen at block #36000. Downloading blocks [16000]⋅
2019-02-11 15:02:53.683 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #9000 C02207E9FA2BC469... with 77 txs⋅
2019-02-11 15:02:53.684 [info] module=OMG.Watcher.BlockGetter.Core function=log_downloading_blocks/2 ⋅Child chain seen at block #36000. Downloading blocks [17000]⋅
2019-02-11 15:02:53.685 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #12000 327679F89114206D... with 77 txs⋅
2019-02-11 15:02:53.686 [info] module=OMG.Watcher.BlockGetter.Core function=log_downloading_blocks/2 ⋅Child chain seen at block #36000. Downloading blocks [18000]⋅
2019-02-11 15:02:53.691 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #13000 7E000E3C5EE4F778... with 96 txs⋅
2019-02-11 15:02:53.692 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #14000 CA409A55EFAAA039... with 96 txs⋅
2019-02-11 15:02:53.696 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #16000 72C38A3BD68A8DF5... with 96 txs⋅
2019-02-11 15:02:53.702 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #17000 4C5D00D3B76D9CE2... with 93 txs⋅
2019-02-11 15:02:53.703 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #15000 4EBBE04D16B9E157... with 98 txs⋅
2019-02-11 15:02:53.705 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #18000 97254C7B4C1A614A... with 95 txs⋅
2019-02-11 15:02:53.706 [info] module=OMG.Watcher.BlockGetter.Core function=log_downloading_blocks/2 ⋅Child chain seen at block #36000. Downloading blocks [19000]⋅
2019-02-11 15:02:53.711 [info] module=OMG.Watcher.BlockGetter.Core function=log_downloading_blocks/2 ⋅Child chain seen at block #36000. Downloading blocks [20000]⋅
2019-02-11 15:02:53.716 [info] module=OMG.Watcher.BlockGetter.Core function=log_downloading_blocks/2 ⋅Child chain seen at block #36000. Downloading blocks [21000]⋅
2019-02-11 15:02:53.725 [info] module=OMG.Watcher.BlockGetter.Core function=log_downloading_blocks/2 ⋅Child chain seen at block #36000. Downloading blocks [22000]⋅
2019-02-11 15:02:53.727 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #19000 6BE9860F1FD0F479... with 99 txs⋅
2019-02-11 15:02:53.727 [info] module=OMG.Watcher.BlockGetter.Core function=log_downloading_blocks/2 ⋅Child chain seen at block #36000. Downloading blocks [23000]⋅
2019-02-11 15:02:53.728 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #20000 F3241D619C3D7B3C... with 96 txs⋅
2019-02-11 15:02:53.730 [info] module=OMG.Watcher.BlockGetter.Core function=log_downloading_blocks/2 ⋅Child chain seen at block #36000. Downloading blocks [24000]⋅
2019-02-11 15:02:53.734 [info] module=OMG.Watcher.BlockGetter.Core function=log_downloading_blocks/2 ⋅Child chain seen at block #36000. Downloading blocks [25000]⋅
2019-02-11 15:02:53.734 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #23000 D84F24E5C332CCC6... with 2 txs⋅
2019-02-11 15:02:53.734 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #21000 240574F0EE46020B... with 90 txs⋅
2019-02-11 15:02:53.737 [info] module=OMG.Watcher.BlockGetter.Core function=log_downloading_blocks/2 ⋅Child chain seen at block #36000. Downloading blocks [26000]⋅
2019-02-11 15:02:53.738 [info] module=OMG.Watcher.BlockGetter.Core function=log_downloading_blocks/2 ⋅Child chain seen at block #36000. Downloading blocks [27000]⋅
2019-02-11 15:02:53.743 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #22000 712F3EF9C6038B1C... with 63 txs⋅
2019-02-11 15:02:53.746 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #24000 34DD7FD70DC793B0... with 106 txs⋅
2019-02-11 15:02:53.753 [info] module=OMG.Watcher.BlockGetter.Core function=log_downloading_blocks/2 ⋅Child chain seen at block #36000. Downloading blocks [28000]⋅
2019-02-11 15:02:53.758 [info] module=OMG.Watcher.BlockGetter.Core function=log_downloading_blocks/2 ⋅Child chain seen at block #36000. Downloading blocks [29000]⋅
2019-02-11 15:02:53.759 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #26000 46AF59054866D4F3... with 191 txs⋅
2019-02-11 15:02:53.762 [info] module=OMG.Watcher.BlockGetter.Core function=log_downloading_blocks/2 ⋅Child chain seen at block #36000. Downloading blocks [30000]⋅
2019-02-11 15:02:53.764 [info] module=OMG.Watcher.BlockGetter.Core function=log_downloading_blocks/2 ⋅Child chain seen at block #36000. Downloading blocks [31000]⋅
2019-02-11 15:02:53.768 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #25000 2E1E25FFDE5A07C6... with 197 txs⋅
2019-02-11 15:02:53.776 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #27000 089B0891660614C3... with 186 txs⋅
2019-02-11 15:02:53.780 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #29000 EEF2DDFAD0E2D211... with 198 txs⋅
2019-02-11 15:02:53.783 [info] module=OMG.Watcher.BlockGetter.Core function=log_downloading_blocks/2 ⋅Child chain seen at block #36000. Downloading blocks [32000]⋅
2019-02-11 15:02:53.783 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #31000 327D3A897BFBB5C6... with 190 txs⋅
2019-02-11 15:02:53.786 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #28000 DD9CFCF393C681AE... with 191 txs⋅
2019-02-11 15:02:53.788 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #30000 5929024EEE16D033... with 185 txs⋅
2019-02-11 15:02:53.809 [info] module=OMG.Watcher.BlockGetter.Core function=log_downloading_blocks/2 ⋅Child chain seen at block #36000. Downloading blocks [33000]⋅
2019-02-11 15:02:53.814 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #32000 66FF865865D7860A... with 192 txs⋅
2019-02-11 15:02:53.823 [info] module=OMG.Watcher.BlockGetter.Core function=log_downloading_blocks/2 ⋅Child chain seen at block #36000. Downloading blocks [34000]⋅
2019-02-11 15:02:53.834 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #33000 6164576504BE169E... with 198 txs⋅
2019-02-11 15:02:53.838 [info] module=OMG.Watcher.BlockGetter.Core function=log_downloading_blocks/2 ⋅Child chain seen at block #36000. Downloading blocks [35000]⋅
2019-02-11 15:02:53.843 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #35000 8D17F725DFB84497... with 10 txs⋅
2019-02-11 15:02:53.847 [info] module=OMG.Watcher.BlockGetter.Core function=validate_download_response/5 ⋅Validating block #34000 84F26E7BA2626738... with 154 txs⋅

We could consider using https://hexdocs.pm/ecto/Ecto.Repo.html#c:insert_all/3

Watcher isn't Watching so no UTXO data is returned

Deposits have been recognised by the Childchain:

2019-02-01 07:02:58.956 [info] module=OMG.API.State.Core function=deposit/2  Recognized deposits [%{amount: 100, blknum: 1, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, eth_he
ight: 3791698, owner: <<148, 74, 129, 190, 236, 172, 145, 128, 39, 135, 251, 207, 185, 118, 127, 203, 248, 29, 177, 245>>}]
2019-02-01 07:05:59.446 [info] module=OMG.API.State.Core function=deposit/2  Recognized deposits [%{amount: 100, blknum: 2, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, eth_he
ight: 3791710, owner: <<148, 74, 129, 190, 236, 172, 145, 128, 39, 135, 251, 207, 185, 118, 127, 203, 248, 29, 177, 245>>}]
2019-02-01 07:09:28.446 [info] module=OMG.API.State.Core function=deposit/2  Recognized deposits [%{amount: 100, blknum: 3, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, eth_he
ight: 3791724, owner: <<148, 74, 129, 190, 236, 172, 145, 128, 39, 135, 251, 207, 185, 118, 127, 203, 248, 29, 177, 245>>}]
2019-02-01 07:10:13.946 [info] module=OMG.API.State.Core function=deposit/2  Recognized deposits [%{amount: 100, blknum: 4, currency: <<0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>, eth_he
ight: 3791727, owner: <<148, 74, 129, 190, 236, 172, 145, 128, 39, 135, 251, 207, 185, 118, 127, 203, 248, 29, 177, 245>>}]

No log message about a deposit in the Watcher. Output from restarted service:

2019-02-01 07:14:28,740 INFO    :Starting Launcher
2019-02-01 07:14:28,747 INFO    :Service type to launch is Elixir Watcher
2019-02-01 07:14:28,747 INFO    :Starting launch process for build fb3ca10f523cc9add9da516021a59e5e45d67bf7
2019-02-01 07:14:28,757 INFO    :Connected to the Ethereum client
2019-02-01 07:14:28,757 INFO    :Ethereum client is b'{"jsonrpc":"2.0","id":67,"result":"Geth/v1.8.22-stable/linux-amd64/go1.11.5"}\n'
omg_rpc: generated priv/static/swagger.json
omg_watcher: generated priv/static/swagger.json
2019-02-01 07:14:31,552 INFO    :Elixir mix compile successful
2019-02-01 07:14:31,552 INFO    :Using pre-deployed contract on network RINKEBY
2019-02-01 07:14:31,552 INFO    :Writing config_watcher.exs
2019-02-01 07:14:31,561 INFO    :Deleted Watcher LevelDB data
2019-02-01 07:14:34,662 INFO    :Initialised Watcher chain database
2019-02-01 07:14:38,684 INFO    :Watcher Postgres instance initialised
2019-02-01 07:14:38,684 INFO    :Launcher process complete
2019-02-01 07:14:42.133 [info] module=OMG.Watcher.Application function=start_watcher_supervisor/0  Started application OMG.Watcher.Application
2019-02-01 07:14:42.475 [info] module=OMG.API.EthereumEventListener function=init/1  Starting EthereumEventListener for depositor
2019-02-01 07:14:42.508 [info] module=OMG.DB function=exit_infos/1  Reading exits' info, this might take a while. Allowing 60000 ms
2019-02-01 07:14:42.520 [info] module=OMG.DB function=in_flight_exits_info/1  Reading in flight exits' info, this might take a while. Allowing 60000 ms
2019-02-01 07:14:42.521 [info] module=OMG.DB function=competitors_info/1  Reading competitors' info, this might take a while. Allowing 60000 ms
2019-02-01 07:14:42.524 [info] module=OMG.API.EthereumEventListener function=init/1  Starting EthereumEventListener for exit_processor
2019-02-01 07:14:42.526 [info] module=OMG.API.EthereumEventListener function=init/1  Starting EthereumEventListener for exit_finalizer
2019-02-01 07:14:42.528 [info] module=OMG.API.EthereumEventListener function=init/1  Starting EthereumEventListener for exit_challenger
2019-02-01 07:14:42.530 [info] module=OMG.API.EthereumEventListener function=init/1  Starting EthereumEventListener for in_flight_exit_processor
2019-02-01 07:14:42.531 [info] module=OMG.API.EthereumEventListener function=init/1  Starting EthereumEventListener for piggyback_processor
2019-02-01 07:14:42.533 [info] module=OMG.API.EthereumEventListener function=init/1  Starting EthereumEventListener for competitor_processor
2019-02-01 07:14:42.535 [info] module=OMG.API.EthereumEventListener function=init/1  Starting EthereumEventListener for challenges_responds_processor
2019-02-01 07:14:42.537 [info] module=OMG.API.EthereumEventListener function=init/1  Starting EthereumEventListener for piggyback_challenges_processor
2019-02-01 07:14:42.538 [info] module=OMG.API.EthereumEventListener function=init/1  Starting EthereumEventListener for ife_exit_finalizer
2019-02-01 07:14:42.677 [info] module=Phoenix.Endpoint.CowboyHandler function=start_link/3  Running OMG.Watcher.Web.Endpoint with Cowboy using http://0.0.0.0:7434
2019-02-01 07:14:42.695 [info] module=OMG.DB function=utxos/1  Reading UTXO set, this might take a while. Allowing 600000 ms
2019-02-01 07:14:42.698 [info] module=OMG.API.State function=init/1  Started State, height: 0, deposit height: 0

Client receives no UTXO data:

go run plasma_cli.go                                                                                           
INFO[2019-02-01T14:18:25+07:00] 200 OK
INFO[2019-02-01T14:18:25+07:00] UTXOs for address 0x944A81BeECac91802787fBCFB9767FCBf81db1f5
{Version:1.0 Success:true Data:[]}

Performance metrics

Pulling from @JBunCE's comment in https://www.pivotaltracker.com/story/show/159842559/comments/199927466

Applications:

• How long does it take to create a Plasma block, how large is the block queue? Does the block queue have the potential to lead to a memory leak and data loss if a pod goes out of memory?
• What's the I/O from LevelDB for writing blocks to disc?
• What's the impact of moving from LevelDB to RocksDB?
• Crypto operations like timing hash functions

Database:

• Block write latency to Postgres
• UTXO set read latency

Etheruem:

• How long does it take to broadcast a block to Ethereum?
• How would a change of Ethereum client impact our services?

Development setup

I'm trying to set up my local env.

After compiling the contracts, I was able to run make geth and make env.

Using this issue to track progress and get help.

/cc @pdobacz

Fix child chain servers gas price selection mechanism

It seems to be behaving as intended but the cadence of reducing/bumping the gas price seems to be wrong, see childchain's logs around any block submission. They follow the pattern:
1/ gas price goes down to minimal during doing nothing (probably when "no txs->no blocks" happen, which was not the case when the gas price mechanism was implemented)
2/ when there's time to submit, every 500ms the gas price doubles, quickly reaching maximum

It seems that the spot where the gas price recalculation takes place in block_queue.ex is not the right one. We probably want to make the logic of that more explicit, and rely less on the cadence of stuff from BlockQueue.Core being called by the handle_infos there.

Additional application metrics

Count and measure times for:

• transactions
  • success
  • failure
• block submits
• deposits
  • deposit amount
• exits
  • standard
    • started
    • challenged
    • success
  • in-flight
    • started
    • challenged
    • success

This list is not necessarily comprehensive. Feel free to add other things to measure.