Coder Social home page Coder Social logo

casper-network / casper-node Goto Github PK

View Code? Open in Web Editor NEW
389.0 34.0 217.0 61.69 MB

Reference client for CASPER protocol

Home Page: https://casper.network

License: Apache License 2.0

Rust 91.01% Makefile 0.08% Shell 5.86% TypeScript 2.45% JavaScript 0.17% Python 0.43%
cspr casper casper-network blockchain rust

casper-node's Introduction

LOGO

casper-node

Reference node for the Casper Blockchain Protocol.

Casper Blockchain

Casper is the blockchain platform purpose-built to scale opportunity for everyone. Building toward blockchain’s next frontier, Casper is designed for real-world applications without sacrificing usability, cost, decentralization, or security. It removes the barriers that prevent mainstream blockchain adoption by making blockchain friendly to use, open to the world, and future-proof to support innovations today and tomorrow. Guided by open-source principles and built from the ground up to empower individuals, the team seeks to provide an equitable foundation made for long-lasting impact. Read more about our mission at: https://casper.network/network/casper-association

Current Development Status

The status on development is reported during the Community calls and is found here

The Casper MainNet is live.

Specification

Get Started with Smart Contracts

Community

Running a casper-node from source

Pre-Requisites for Building

  • CMake 3.1.4 or greater
  • Rust
  • libssl-dev
  • pkg-config
  • gcc
  • g++
  • recommended wasm-strip (used to reduce the size of compiled Wasm)
# Ubuntu prerequisites setup example
apt update
apt install cmake libssl-dev pkg-config gcc g++ -y
# the '-s -- -y' part ensures silent mode. Omit if you want to customize
curl https://sh.rustup.rs -sSf | sh -s -- -y

Setup

Before building a node, prepare your Rust build environment:

make setup-rs

The node software can be compiled afterwards:

cargo build -p casper-node --release

The result will be a casper-node binary found in target/release. Copy this somewhere into your PATH, or substitute target/release/casper-node for casper-node in all examples below.

Running one node

To run a validator node you will need to specify a config file and launch the validator subcommand, for example

casper-node validator /etc/casper-node/config.toml

The node ships with an example configuration file that should be setup first. There is also a template for a local chainspec in the same folder.

For launching, the following configuration values must be properly set:

Setting Description
network.known_addresses Must refer to public listening addresses of one or more currently-running nodes. If the node cannot connect to any of these addresses, it will panic. The node can be run with this referring to its own address, but it will be equivalent to specifying an empty list for known_addresses - i.e. the node will run and listen, but will be reliant on other nodes connecting to it in order to join the network. This would be normal for the very first node of a network, but all subsequent nodes should normally specify that first node's public listening address as their known_addresses.

The node will not run properly without another node to connect to. It is recommended that multiple nodes are run.

Running multiple nodes on one machine

There is a tool which automates the process of running multiple nodes on a single machine.

Note that running multiple nodes on a single machine is normally only recommended for test purposes.

Configuration

In general nodes are configured through a configuration file, typically named config.toml. This file may reference other files or locations through relative paths. When it does, note that all paths that are not absolute will be resolved relative to config.toml directory.

Environment overrides

Some environments may call for overriding options through the environment. In this scenario, the NODE_CONFIG environment variable can be used. For example: alternatively expressed as

export NODE_CONFIG=consensus.secret_key_path=secret_keys/node-1.pem;network.known_addresses=[1.2.3.4:34553, 200.201.203.204:34553]
casper-node validator /etc/casper-node/config.toml

Note how the semicolon is used to separate configuration overrides here.

Other environment variables

To set the threshold at which a warn-level log message is generated for a long-running reactor event, use the env var CL_EVENT_MAX_MICROSECS. For example, to set the threshold to 1 millisecond:

CL_EVENT_MAX_MICROSECS=1000

To set the threshold above which the size of the current scheduler queues will be dumped to logs, use the CL_EVENT_QUEUE_DUMP_THRESHOLD variable. For example, to set the threshold to 10000 events:

CL_EVENT_QUEUE_DUMP_THRESHOLD=10000

This will dump a line to the log if the total number of events in queues exceeds 10000. After each dump, the threshold will be automatically increased by 10% to avoid log flooding.

Example log entry:

Current event queue size (11000) is above the threshold (10000): details [("FinalitySignature", 3000), ("FromStorage", 1000), ("NetworkIncoming", 6500), ("Regular", 500)]

Logging

Logging can be enabled by setting the environment variable RUST_LOG. This can be set to one of the following levels, from lowest priority to highest: trace, debug, info, warn, error:

RUST_LOG=info cargo run --release -- validator resources/local/config.toml

If the environment variable is unset, it is equivalent to setting RUST_LOG=error.

Log message format

A typical log message will look like:

Jun 09 01:40:17.315 INFO  [casper_node::components::rpc_server rpc_server.rs:127] starting HTTP server; server_addr=127.0.0.1:7777

This is comprised of the following parts:

  • timestamp
  • log level
  • full module path (not to be confused with filesystem path) of the source of the message
  • filename and line number of the source of the message
  • message

Filtering log messages

RUST_LOG can be set to enable varying levels for different modules. Simply set it to a comma-separated list of module-path=level, where the module path is as shown above in the typical log message, with the end truncated to suit.

For example, to enable trace level logging for the network module in components, info level for all other modules in components, and warn level for the remaining codebase:

RUST_LOG=casper_node::components::network=trace,casper_node::comp=info,warn

Logging network messages and tracing events

Special logging targets exist in net_in and net_out which can be used to log every single network message leaving or entering a node when set to trace level:

RUST_LOG=net_in::TRACE,net_out::TRACE

All messages in these logs are also assigned a unique ID that is different even if the same message is sent to multiple nodes. The receiving node will log them using the same ID as the sender, thus enabling the tracing of a message across multiple nodes provided all logs are available.

Another helpful logging feature is ancestor logging. If the target dispatch is set to at least debug level, events being dispatched will be logged as well. Any event has an id (ev) and may have an ancestor (a), which is the previous event whose effects caused the resulting event to be scheduled. As an example, if an incoming network message gets assigned an ID of ev=123, the first round of subsequent events will show a=123 as their ancestor in the logs.

Changing the logging filter at runtime

If necessary, the filter of a running node can be changed using the diagnostics port, using the set-log-filter command. See the "Diagnostics port" section for details on how to access it.

Debugging

Some additional debug functionality is available, mainly allowed for inspections of the internal event queue.

Diagnostics port

If the configuration option diagnostics_port.enabled is set to true, a unix socket named debug.socket by default can be found next to the configuration while the node is running.

Interactive use

The debug.socket can be connected to by tools like socat for interactive use:

socat readline unix:/path/to/debug.socket

Entering help will show available commands. The set command allows configuring the current connection, see set --help.

Example: Collecting a consensus dump

After connecting using socat (see above), we set the output format to JSON:

set --output=json

A confirmation will acknowledge the settings change (unless --quiet=true is set):

{
  "Success": {
    "msg": "session unchanged"
  }
}

We can now call dump-consensus to get the latest era serialized in JSON format:

dump-consensus
{
  "Success": {
    "msg": "dumping consensus state"
  }
}
{"id":8,"start_time":"2022-03-01T14:54:42.176Z","start_height":88,"new_faulty" ...

An era other than the latest can be dumped by specifying as a parameter, e.g. dump-consensus 3 will dump the third era. See dump-consensus --help for details.

Example: Dumping the event queue

With the connection set to JSON output (see previous example), we can also dump the event queues:

dump-queues
{
  "Success": {
    "msg": "dumping queues"
  }
}
{"queues":{"Regular":[],"Api":[],"Network":[],"Control":[],"NetworkIncoming":[]
}}{"queues":{"Api":[],"Regular":[],"Control":[],"NetworkIncoming":[],"Network":
[]}}{"queues":{"Network":[],"Control":[],"Api":[],"NetworkIncoming":[],"Regular
":[]}}

Empty output will be produced on a node that is working without external pressure, as the queues will be empty most of the time.

Non-interactive use

The diagnostics port can also be scripted by sending a newline-terminated list of commands through socat. For example, the following sequence of commands will collect a consensus dump without the success-indicating header:

set -o json -q true
dump-consensus

For ad-hoc dumps, this can be shortened and piped into socat:

echo -e 'set -o json -q true\ndump-consensus' | socat - unix-client:debug.socket > consensus-dump.json

This results in the latest era being dumped into consensus-dump.json.

Running a client

See the client README.

Running a local network

See the nctl utility README.

Running on an existing network

To support upgrades with a network, the casper-node is installed using scripts distributed with the casper-node-launcher.

casper-node's People

Contributors

afck avatar alexanderlimonov avatar alsrdn avatar bors[bot] avatar casperlabs-bors-ng[bot] avatar darthsiroftardis avatar dwerner avatar edhastingscasperassociation avatar fizyk20 avatar fraser999 avatar georgepisaltu avatar goral09 avatar hanswestman74 avatar henrytill avatar jacek-casper avatar linclelinkpart5 avatar luis-casper avatar marc-casperlabs avatar mpapierski avatar mparlikar avatar nilz3ro avatar rafal-ch avatar real-felix avatar rob-casper avatar robj avatar sacherjj avatar samuelschlesinger avatar tomvasile avatar xcthulhu avatar zacshowa avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

casper-node's Issues

People can't sync [ DELTA 8 ]

Many guys experience sync issues. I observe network and found Scheduler Queue Total Count constantly increasing:

image

See spike on gossiper here:

image

Which correlated with peers been joining my node and then disconnected in same frequence here:

image

Where is also a glitch in metrics which can indecate restart of my validator unit or can be related to this issue we have presiously, where metrics with all endpoints can't be served properly and fall off periodically.

image

  • Update:

This error recently popups in my log:

{
  "timestamp": "Jan 10 05:15:47.654",
  "level": "DEBUG",
  "fields": {
    "message": "connection error: connection closed before message completed"
  },
  "target": "hyper::server::conn::spawn_all"
}

Query status takes very long time.

After network made above 100 validators query status end point takes ages to respond.

Reproducing error:

curl -s http://127.0.0.1:8888/status

Expected:

Fast replay

  • Update 1

After first very long wait queries back to normal, fast replay.

  • Update 2

This not only happen with /status query, but also present on Prometheus /metrics

/metrics:

image
image

/status

image

  • Update 3

People reporting issues with deploys, takes a while to be able to succeed, expected errors:

FailedToGetResponse(reqwest::Error { kind: Request, url: Url { scheme: "http", host: Some(Ipv4(127.0.0.1)), port: Some(7777), path: "/rpc", query: None, fragment: None }, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 111, kind: ConnectionRefused, message: "Connection refused" })) })', client/src/deploy/put.rs:56:31
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace

node panicked: Failed to download linear chain.

After recent crash and no success with join it again I try to run validator from scratch.

Service exit error:

Main process exited, code=killed, status=6/ABRT

Valdator log is overloaded and produced about 4G since last 30 minutes, mostly:

message received","msg":"payload: AddressGossiper::(gossip-response
message received","msg":"payload: AddressGossiper::(gossip(gossiped-address

Warnings present in a log:

"WARN","fields":{"message":"Finality signatures not handled in joiner reactor"},"target":"casper_node::reactor::joiner","span":{"ev":7206551,"name":"dispatch events"},"spans":[{"ev":7206551,"name":"crank"},{"ev":7206551,"name":"dispatch events"}]}

"WARN","fields":{"message":"network announcement ignored.","other":"Consensus(Protocol { era_id.0: 214, .. })"},"target":"casper_node::reactor::joiner","span":{"ev":7239826,"name":"dispatch events"},"spans":[{"ev":7239826,"name":"crank"},{"ev":7239826,"name":"dispatch events"}]}

Most of the warning spam is:

"WARN","fields":{"message":"NodeId::Tls(7e9d..fa72): outgoing connection failed","peer_address":

Just two warnings about:

{"timestamp":"Dec 26 14:23:37.920","level":"WARN","fields":{"message":"large event size, consider reducing it or boxing","event_size":"160"},"target":"casper_node::reactor"}

{"timestamp":"Dec 26 14:23:38.245","level":"WARN","fields":{"message":"large event size, consider reducing it or boxing","event_size":"544"},"target":"casper_node::reactor"}

  • Full error message output:
   0: casper_node::panic_hook
   1: core::ops::function::Fn::call
   2: std::panicking::rust_panic_with_hook
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panicking.rs:597:17
   3: std::panicking::begin_panic::{{closure}}
   4: std::sys_common::backtrace::__rust_end_short_backtrace
   5: std::panicking::begin_panic
   6: <casper_node::components::linear_chain_sync::LinearChainSync<I> as casper_node::components::Component<REv>>::handle_event
   7: <casper_node::reactor::joiner::Reactor as casper_node::reactor::Reactor>::dispatch_event
   8: casper_node::reactor::Runner<R>::crank::{{closure}}
   9: casper_node::cli::Cli::run::{{closure}}
  10: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  11: std::thread::local::LocalKey<T>::with
  12: tokio::runtime::enter::Enter::block_on
  13: tokio::runtime::thread_pool::ThreadPool::block_on
  14: tokio::runtime::context::enter
  15: tokio::runtime::handle::Handle::enter
  16: casper_node::main
  17: std::sys_common::backtrace::__rust_begin_short_backtrace
  18: std::rt::lang_start::{{closure}}
  19: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/core/src/ops/function.rs:259:13
      std::panicking::try::do_call
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panicking.rs:381:40
      std::panicking::try
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panicking.rs:345:19
      std::panic::catch_unwind
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panic.rs:396:14
      std::rt::lang_start_internal
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/rt.rs:51:25
  20: main
  21: __libc_start_main
  22: _start

node panicked: Failed to download linear chain.

License matters, clarification on usage, and GPL violations

I've been following CasperLabs' work since I heard a talk about it in DevCon V. I really like the CBC protocol and its designs. However, on license matters, I think there's something I must bring up, as I couldn't find discussions on this elsewhere. There are things that have to be fixed (GPL violations), and things that many developers and me would really appreciate clarifications, so that we can properly utilize the codebase, and avoid any "misuse" defined in your license terms.

License matters

If you knew MongoDB, you know that "open source" is a somewhat sensitive word -- as people expect codebases under an "open source" license would comply with the open source definition. OSI reviews provided some reasons why they think the current license is not appropriately "open source". While that's only according to their OSI's definitions, I couldn't find any responses from CasperLabs on how those critics are addressed, and what's CasperLabs' definition of open source. Without this, the word "open source" in the license title "CapserLabs Open Source License" is going to be a source of trouble when the software is made aware by the wider open source community.

The license also contains some reference to "FLOSS", which is supposed to mean free/libre and open source software. The current license also heavily deviates from what the community commonly understand it is.

A recommendation from mine would be if the term "open source" can be removed from the license title and preamble. Why not simply call it "free and open"? No one will question that whether the license is "free and open" at all, but many will question whether the license is "open source" or "FLOSS".

Clarification on usage

I also hope if there can be a more concrete summary page on what is allowed and what is not allowed under the current license terms, as after reading the whole license, I still find it quiet confusing. Not only does this benefit developers like me, but also ecosystems and enterprises heavily.

GPL violations

The license this codebase use (COSLv1) is an open-codebase license. As it's not compatible with GPL, no code in this codebase shall depend on it. However, currently there's one violation of it:

  • colour, licensed under GPL-3.0.

That dependency has to be removed from the current codebase to avoid the violation. Given it's only a simple dependency, I think this would be relatively easy to do.

Node dead after start [ DELTA 6 ]

Trying to join delta-6 recently, after couple of minutes node crash with provided error:

   0: casper_node::panic_hook
   1: core::ops::function::Fn::call
   2: std::panicking::rust_panic_with_hook
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panicking.rs:597:17
   3: std::panicking::begin_panic_handler::{{closure}}
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panicking.rs:499:13
   4: std::sys_common::backtrace::__rust_end_short_backtrace
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/sys_common/backtrace.rs:141:18
   5: rust_begin_unwind
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panicking.rs:495:5
   6: std::panicking::begin_panic_fmt
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panicking.rs:437:5
   7: casper_node::components::consensus::<impl casper_node::components::Component<REv> for casper_node::components::consensus::era_supervisor::EraSupervisor<I>>::handle_event
   8: <casper_node::reactor::validator::Reactor as casper_node::reactor::Reactor>::dispatch_event
   9: casper_node::cli::Cli::run::{{closure}}
  10: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  11: std::thread::local::LocalKey<T>::with
  12: tokio::runtime::enter::Enter::block_on
  13: tokio::runtime::thread_pool::ThreadPool::block_on
  14: tokio::runtime::context::enter
  15: casper_node::main
  16: std::sys_common::backtrace::__rust_begin_short_backtrace
  17: std::rt::lang_start::{{closure}}
  18: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/core/src/ops/function.rs:259:13
      std::panicking::try::do_call
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panicking.rs:381:40
      std::panicking::try
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panicking.rs:345:19
      std::panic::catch_unwind
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panic.rs:396:14
      std::rt::lang_start_internal
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/rt.rs:51:25
  19: main
  20: __libc_start_main
  21: _start

panicked at 'fatal error [node/src/components/consensus/era_supervisor.rs:811]: Consensus shutting down due to inability to participate in the network', node/src/effect.rs:423:9
  • Update

Probably getting this from nodes which are on previous delta still.

{
  "timestamp": "Dec 30 09:32:32.937",
  "level": "INFO",
  "fields": {
    "message": "NodeId::Tls(3da7..bf59): dropping connection to NodeId::Tls(4037..0b1e) due to genesis config hash mismatch",
    "our_hash": "42edd6b09c101a65d7aa3626bf0811cb39fa4d2d3957d8a28868edd4aa8d1a9e",
    "their_hash": "0e9f4c541abeb97a34708f732b6d0acc7345dbda394ae115ffef213461d1a892"
  },
  • Update 2

Attached logs are in pure JSON format, can be easily to operate, this is full cycle from start to crash.

Inside zip is tar.xz archive, GitHub only support zip, sorry for it

delta-6.zip

Block provided by BlockAdded event does not match block specification

The block specification in the JSON-RPC API shows the era report containing lists of rewards and validator weights. This shows up as expected if a block is fetched, e.g.:

          "era_report": {
            "equivocators": [],
            "inactive_validators": [],
            "rewards": [
              {
                "amount": 1202458594726,
                "validator": "01026ca707c348ed8012ac6a1f28db031fadd6eb67203501a353b867a08c8b9a80"
              },
              {
                "amount": 19956734910,
                "validator": "010427c1d1227c9d2aafe8c06c6e6b276da8dcd8fd170ca848b8e3e8e1038a6dc8"
              },

where rewards is an array of reward objects. However, if a block is received on the events stream with the BlockAdded event the rewards and next_era_validator_weights are returned as maps with the key being the validator public key.

This discrepancy makes life difficult to use a standard process for parsing blocks, and appears to serve no purpose. Could the block presented with the BlockAdded event use the same format at that returned by the JSON-API?

Avoid restarting event stream event ID from 0 on each restart

Currently, the event stream indexes every event it produces.  The numbering starts at 0 and increases by 1 for every subsequent event.  However, if the node stops and restarts (which it would do e.g. during an upgrade) then the numbering starts from 0 again.

We want to persist the indexing and on the restart, continue from where we left off.

Node cant start and communicate with local api after / is full

I have installed casperlabs in / mountpoint.
The node was running as a validator and the mount point became 100% full and the node crashed. I have cleared the mount point and tried restarting the node but it fils with the below error.

CHain : Delta-10

0: casper_node::panic_hook
1: core::ops::function::Fn::call
2: std::panicking::rust_panic_with_hook
at rustc/f76ecd0668fcdb289456cdc72a39ad15467cc454/library/std/src/panicking.rs:595:17
3: std::panicking::begin_panic_handler::{{closure}}
at rustc/f76ecd0668fcdb289456cdc72a39ad15467cc454/library/std/src/panicking.rs:497:13
4: std::sys_common::backtrace::__rust_end_short_backtrace
at rustc/f76ecd0668fcdb289456cdc72a39ad15467cc454/library/std/src/sys_common/backtrace.rs:141:18
5: rust_begin_unwind
at rustc/f76ecd0668fcdb289456cdc72a39ad15467cc454/library/std/src/panicking.rs:493:5
6: std::panicking::begin_panic_fmt
at rustc/f76ecd0668fcdb289456cdc72a39ad15467cc454/library/std/src/panicking.rs:435:5
7: casper_node::effect::EffectBuilder::load_state::{{closure}}::{{closure}}
8: <core::future::from_generator::GenFuture as core::future::future::Future>::poll
9: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
10: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
11: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
12: <core::future::from_generator::GenFuture as core::future::future::Future>::poll
13: <core::future::from_generator::GenFuture as core::future::future::Future>::poll
14: <std::panic::AssertUnwindSafe as core::ops::function::FnOnce<()>>::call_once
15: tokio::runtime::task::raw::poll
16: std::thread::local::LocalKey::with
17: tokio::runtime::thread_pool::worker::Context::run_task
18: tokio::runtime::thread_pool::worker::Context::run
19: tokio::macros::scoped_tls::ScopedKey::set
20: tokio::runtime::thread_pool::worker::run
21: tokio::loom::std::unsafe_cell::UnsafeCell::with_mut
22: tokio::runtime::task::core::Core<T,S>::poll
23: <std::panic::AssertUnwindSafe as core::ops::function::FnOnce<()>>::call_once
24: tokio::runtime::task::harness::Harness<T,S>::poll
25: tokio::runtime::blocking::pool::Inner::run
26: tokio::runtime::context::enter
27: std::sys_common::backtrace::__rust_begin_short_backtrace
28: core::ops::function::FnOnce::call_once{{vtable.shim}}
29: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once
at rustc/f76ecd0668fcdb289456cdc72a39ad15467cc454/library/alloc/src/boxed.rs:1328:9
<alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once
at rustc/f76ecd0668fcdb289456cdc72a39ad15467cc454/library/alloc/src/boxed.rs:1328:9
std::sys::unix::thread::Thread::new::thread_start
at rustc/f76ecd0668fcdb289456cdc72a39ad15467cc454/library/std/src/sys/unix/thread.rs:71:17
30: start_thread
31: clone

panicked at 'could not deserialize state from storage type name "core::option::Option<casper_node::components::block_proposer::deploy_sets::BlockProposerDeploySets>" err InvalidTagEncoding(61)', node/src/effect.rs:1252:13
Error: "/var/lib/casper/bin/1_0_0/casper-node" "validator" "/etc/casper/1_0_0/config.toml" exited with error

The problem and solution of node startup failure + peer is 0

node status,Attempt to restart but failed
1637059325a93741b1ab3fd5a1eac75

check peer is 0
image

check casper-node.stderr.log,Prompt is related to the config.toml
27e2b6ef9180831cdbb87e2e1ea5023

check casper-node.log ,Found that a node is down
681847895f8a8bacd3dda3457951051

solution:
After deleting the node that reported the error, the service started normally, with more than 40 peers

Checksum for target account hash

Implement CEP-57.

There is already a draft PR #1557

Needs the following touch-ups before it is merged:

  • Tests fixed since @afck and I changed the scheme to use our hash entropy efficiently
  • Rebase to dev
  • Optimizations
    • Replace blake2bhash which returns a Vec with something that returns [u8; 32] so we avoid using the heap
    • Reduce the number of passes over the hex string in decode to at most 2
    • Make it so decode doesn’t do any heap allocations
    • Simplify for loops to use iteration instead
  • Remove all hex external crate dependencies
  • Benchmarks: run cargo bench --bench transfer_bench -- --save-baseline 'dev' on dev branch, then on this branch run cargo bench --bench transfer_bench -- --baseline 'dev'

Unable to obtain historical stake

I'm attempting to obtain historical (balance+stake) information from the JSON-RPC interface. I can obtain historical balance (which is currently always 0 as all balance is staked) and current stake (from the state_get_auction_info method) but not historical stake.

Is historical stake available somewhere? Or some other method to calculate an account's (balance+stake), for example obtaining per-era reward data?

Gossiper on rampage

Observing last 6 hours on my monitoring solution, I notice table items are significantly increased.
First thoughts it was someone spamming allot of deploys, but observing deploys count I figure out this is not the issue.

Here is some screenshots from my monitor ( 6 hour frame ):

Items finished:
image

Items current:
image

Deploys onward:
image

Table items finished:
image

Unpleasant consequences:

CPU utilization:
image

Memory usage:
image

Interrupts/s:
image

On load average we can observe strange behavior where in same 6 hours time frame 5 and 15 minutes decrease, but small momentum 1 minute spikes is growing.

image

Some stepped memory decrease ( is upside down in zabbix ):
image

Main process exited, code=killed, status=9/KILL

Happen Dec 26 02:52:25 UTC

I noticed Scheduler Queue overload, peak happen 1:00 UTC one hour before node crash:

image

DELTA BALANCER: scheduler_queue_api_count: 0
DELTA BALANCER: scheduler_queue_network_count: 47.3333
DELTA BALANCER: scheduler_queue_network_incoming_count: 4947216.0417
DELTA BALANCER: scheduler_queue_regular_count: 23.125
DELTA BALANCER: scheduler_queue_total_count: 4947286.5
  • Update 1

Trying to recover and restart node so far unsuccessfully:

Main process exited, code=killed, status=6/ABRT

Node can't sync above "height": 4215,

Key issue while joining delta testnet

While attempting to join the delta testnet, I provided public hex key 01329cc15cb92fca5d5ce6d7fcc881010161547f452e2caad4dff565708b00b8c7, which was successfully funded. Upon launching the node and attempting to bond, I received some warnings of this form:

Nov  5 14:51:39 casper-node casper-node[7048]: {"timestamp":"Nov 05 14:51:39.169","level":"INFO","fields":{"message":"9be8..2d7b: connection closed","peer_id":"2725..4368","peer_address":"135.181.73.218:34858"},"target":"casper_node::components::small_network"}
Nov  5 14:51:39 casper-node casper-node[7048]: {"timestamp":"Nov 05 14:51:39.169","level":"WARN","fields":{"message":"9be8..2d7b: outgoing connection closed","peer_id":"2725..4368","peer_address":"135.181.73.218:34553"},"target":"casper_node::components::small_network"}
Nov  5 14:53:49 casper-node casper-node[7048]: {"timestamp":"Nov 05 14:53:49.348","level":"WARN","fields":{"message":"9be8..2d7b: outgoing connection failed","peer_id":"725c..21c8","peer_address":"135.181.73.218:34553","err":"failed to send message"},"target":"casper_node::components::small_network"}
Nov  5 14:57:36 casper-node casper-node[7048]: {"timestamp":"Nov 05 14:57:36.031","level":"WARN","fields":{"message":"9be8..2d7b: outgoing connection failed","peer_id":"99c4..8c92","peer_address":"135.181.73.218:34553","err":"failed to send message"},"target":"casper_node::components::small_network"}
Nov  5 15:01:36 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:01:36.250","level":"WARN","fields":{"message":"9be8..2d7b: outgoing connection failed","peer_id":"4b5b..adef","peer_address":"135.181.73.218:34553","err":"failed to send message"},"target":"casper_node::components::small_network"}
Nov  5 15:06:30 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:06:30.501","level":"WARN","fields":{"message":"9be8..2d7b: receiving message failed, closing connection","err":"Connection reset by peer (os error 104)","peer_id":"4712..58c3"},"target":"casper_node::components::small_network"}
Nov  5 15:06:30 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:06:30.501","level":"INFO","fields":{"message":"9be8..2d7b: connection closed","peer_id":"4712..58c3","peer_address":"135.181.73.218:43550"},"target":"casper_node::components::small_network"}
Nov  5 15:06:30 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:06:30.501","level":"WARN","fields":{"message":"9be8..2d7b: outgoing connection closed","peer_id":"4712..58c3","peer_address":"135.181.73.218:34553"},"target":"casper_node::components::small_network"}
Nov  5 15:11:32 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:11:32.363","level":"WARN","fields":{"message":"9be8..2d7b: receiving message failed, closing connection","err":"Connection reset by peer (os error 104)","peer_id":"6e1a..8a85"},"target":"casper_node::components::small_network"}
Nov  5 15:11:32 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:11:32.363","level":"INFO","fields":{"message":"9be8..2d7b: connection closed","peer_id":"6e1a..8a85","peer_address":"164.68.114.22:40472"},"target":"casper_node::components::small_network"}
Nov  5 15:11:32 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:11:32.364","level":"WARN","fields":{"message":"9be8..2d7b: outgoing connection closed","peer_id":"6e1a..8a85","peer_address":"164.68.114.22:34553"},"target":"casper_node::components::small_network"}
Nov  5 15:13:42 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:13:42.144","level":"WARN","fields":{"message":"9be8..2d7b: outgoing connection failed","peer_id":"b56a..6ef1","peer_address":"164.68.114.22:34553","err":"failed to send message"},"target":"casper_node::components::small_network"}
Nov  5 15:15:42 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:15:42.292","level":"WARN","fields":{"message":"9be8..2d7b: outgoing connection failed","peer_id":"f9ce..270e","peer_address":"164.68.114.22:34553","err":"failed to send message"},"target":"casper_node::components::small_network"}
Nov  5 15:16:15 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:16:15.752","level":"WARN","fields":{"message":"9be8..2d7b: receiving message failed, closing connection","err":"Connection reset by peer (os error 104)","peer_id":"3937..faf0"},"target":"casper_node::components::small_network"}
Nov  5 15:16:15 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:16:15.753","level":"INFO","fields":{"message":"9be8..2d7b: connection closed","peer_id":"3937..faf0","peer_address":"135.181.2.250:41402"},"target":"casper_node::components::small_network"}
Nov  5 15:16:15 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:16:15.753","level":"WARN","fields":{"message":"9be8..2d7b: outgoing connection closed","peer_id":"3937..faf0","peer_address":"135.181.2.250:34553"},"target":"casper_node::components::small_network"}
Nov  5 15:17:48 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:17:48.079","level":"WARN","fields":{"message":"9be8..2d7b: outgoing connection failed","peer_id":"e8f1..99eb","peer_address":"164.68.114.22:34553","err":"failed to send message"},"target":"casper_node::components::small_network"}
Nov  5 15:19:48 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:19:48.185","level":"WARN","fields":{"message":"9be8..2d7b: outgoing connection failed","peer_id":"3024..ecc5","peer_address":"164.68.114.22:34553","err":"failed to send message"},"target":"casper_node::components::small_network"}
Nov  5 15:21:48 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:21:48.511","level":"WARN","fields":{"message":"9be8..2d7b: outgoing connection failed","peer_id":"7ddf..85d6","peer_address":"164.68.114.22:34553","err":"failed to send message"},"target":"casper_node::components::small_network"}
Nov  5 15:23:49 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:23:49.251","level":"WARN","fields":{"message":"9be8..2d7b: outgoing connection failed","peer_id":"fe6a..d28b","peer_address":"164.68.114.22:34553","err":"failed to send message"},"target":"casper_node::components::small_network"}
Nov  5 15:35:32 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:35:32.681","level":"WARN","fields":{"message":"9be8..2d7b: receiving message failed, closing connection","err":"Connection reset by peer (os error 104)","peer_id":"1a63..5169"},"target":"casper_node::components::small_network"}
Nov  5 15:35:32 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:35:32.681","level":"INFO","fields":{"message":"9be8..2d7b: connection closed","peer_id":"1a63..5169","peer_address":"135.181.2.250:37742"},"target":"casper_node::components::small_network"}
Nov  5 15:35:32 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:35:32.681","level":"WARN","fields":{"message":"9be8..2d7b: outgoing connection closed","peer_id":"1a63..5169","peer_address":"135.181.2.250:34553"},"target":"casper_node::components::small_network"}
Nov  5 15:42:28 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:42:28.365","level":"WARN","fields":{"message":"9be8..2d7b: receiving message failed, closing connection","err":"Connection reset by peer (os error 104)","peer_id":"8cd6..a450"},"target":"casper_node::components::small_network"}
Nov  5 15:42:28 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:42:28.365","level":"INFO","fields":{"message":"9be8..2d7b: connection closed","peer_id":"8cd6..a450","peer_address":"164.68.114.22:41120"},"target":"casper_node::components::small_network"}
Nov  5 15:42:28 casper-node casper-node[7048]: {"timestamp":"Nov 05 15:42:28.365","level":"WARN","fields":{"message":"9be8..2d7b: outgoing connection closed","peer_id":"8cd6..a450","peer_address":"164.68.114.22:34553"},"target":"casper_node::components::small_network"}

Which soon turned into a ton of errors of this form:

Nov  5 17:00:13 casper-node casper-node[7048]: {"timestamp":"Nov 05 17:00:13.753","level":"ERROR","fields":{"message":"invalid incoming message to consensus instance","sender":"63dc..34cb","error":"asymmetric key error: failed to construct Ed25519 signature from []"},"target":"casper_node::components::consensus::era_supervisor"}
Nov  5 17:00:13 casper-node casper-node[7048]: {"timestamp":"Nov 05 17:00:13.754","level":"ERROR","fields":{"message":"invalid incoming message to consensus instance","sender":"f1f7..7af1","error":"asymmetric key error: failed to construct Ed25519 signature from []"},"target":"casper_node::components::consensus::era_supervisor"}
Nov  5 17:00:13 casper-node casper-node[7048]: {"timestamp":"Nov 05 17:00:13.754","level":"ERROR","fields":{"message":"invalid incoming message to consensus instance","sender":"6ab1..2807","error":"asymmetric key error: failed to construct Ed25519 signature from []"},"target":"casper_node::components::consensus::era_supervisor"}
Nov  5 17:00:13 casper-node casper-node[7048]: {"timestamp":"Nov 05 17:00:13.754","level":"ERROR","fields":{"message":"invalid incoming message to consensus instance","sender":"d353..5ddf","error":"asymmetric key error: failed to construct Ed25519 signature from []"},"target":"casper_node::components::consensus::era_supervisor"}
Nov  5 17:00:13 casper-node casper-node[7048]: {"timestamp":"Nov 05 17:00:13.754","level":"ERROR","fields":{"message":"invalid incoming message to consensus instance","sender":"0269..7cfd","error":"asymmetric key error: failed to construct Ed25519 signature from []"},"target":"casper_node::components::consensus::era_supervisor"}
Nov  5 17:00:13 casper-node casper-node[7048]: {"timestamp":"Nov 05 17:00:13.754","level":"ERROR","fields":{"message":"invalid incoming message to consensus instance","sender":"da45..ead8","error":"asymmetric key error: failed to construct Ed25519 signature from []"},"target":"casper_node::components::consensus::era_supervisor"}
Nov  5 17:00:13 casper-node casper-node[7048]: {"timestamp":"Nov 05 17:00:13.754","level":"ERROR","fields":{"message":"invalid incoming message to consensus instance","sender":"ff54..4652","error":"asymmetric key error: failed to construct Ed25519 signature from []"},"target":"casper_node::components::consensus::era_supervisor"}
Nov  5 17:00:13 casper-node casper-node[7048]: {"timestamp":"Nov 05 17:00:13.755","level":"ERROR","fields":{"message":"invalid incoming message to consensus instance","sender":"1be7..3688","error":"asymmetric key error: failed to construct Ed25519 signature from []"},"target":"casper_node::components::consensus::era_supervisor"}
Nov  5 17:00:13 casper-node casper-node[7048]: {"timestamp":"Nov 05 17:00:13.764","level":"ERROR","fields":{"message":"invalid incoming message to consensus instance","sender":"cae0..79ee","error":"asymmetric key error: failed to construct Ed25519 signature from []"},"target":"casper_node::components::consensus::era_supervisor"}
Nov  5 17:00:13 casper-node casper-node[7048]: {"timestamp":"Nov 05 17:00:13.775","level":"ERROR","fields":{"message":"invalid incoming message to consensus instance","sender":"fc9b..46a9","error":"asymmetric key error: failed to construct Ed25519 signature from []"},"target":"casper_node::components::consensus::era_supervisor"}

Which ended in a panic of this form:

Nov  5 18:06:21 casper-node casper-node[24645]:    0: casper_node::panic_hook
Nov  5 18:06:21 casper-node casper-node[24645]:    1: core::ops::function::Fn::call
Nov  5 18:06:21 casper-node casper-node[24645]:    2: std::panicking::rust_panic_with_hook
Nov  5 18:06:21 casper-node casper-node[24645]:              at rustc/0cd7ff7ddfb75a38dca81ad3e76b1e984129e939/src/libstd/panicking.rs:530:17
Nov  5 18:06:21 casper-node casper-node[24645]:    3: rust_begin_unwind
Nov  5 18:06:21 casper-node casper-node[24645]:              at rustc/0cd7ff7ddfb75a38dca81ad3e76b1e984129e939/src/libstd/panicking.rs:437:5
Nov  5 18:06:21 casper-node casper-node[24645]:    4: std::panicking::begin_panic_fmt
Nov  5 18:06:21 casper-node casper-node[24645]:              at rustc/0cd7ff7ddfb75a38dca81ad3e76b1e984129e939/src/libstd/panicking.rs:391:5
Nov  5 18:06:21 casper-node casper-node[24645]:    5: casper_node::effect::EffectBuilder<REv>::fatal::{{closure}}
Nov  5 18:06:21 casper-node casper-node[24645]:    6: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
Nov  5 18:06:21 casper-node casper-node[24645]:    7: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
Nov  5 18:06:21 casper-node casper-node[24645]:    8: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
Nov  5 18:06:21 casper-node casper-node[24645]:    9: <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
Nov  5 18:06:21 casper-node casper-node[24645]:   10: tokio::runtime::task::raw::poll
Nov  5 18:06:21 casper-node casper-node[24645]:   11: std::thread::local::LocalKey<T>::with
Nov  5 18:06:21 casper-node casper-node[24645]:   12: tokio::runtime::thread_pool::worker::Context::run_task
Nov  5 18:06:21 casper-node casper-node[24645]:   13: tokio::runtime::thread_pool::worker::Context::run
Nov  5 18:06:21 casper-node casper-node[24645]:   14: tokio::macros::scoped_tls::ScopedKey<T>::set
Nov  5 18:06:21 casper-node casper-node[24645]:   15: tokio::runtime::thread_pool::worker::run
Nov  5 18:06:21 casper-node casper-node[24645]:   16: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
Nov  5 18:06:21 casper-node casper-node[24645]:   17: <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
Nov  5 18:06:21 casper-node casper-node[24645]:   18: tokio::runtime::task::harness::Harness<T,S>::poll
Nov  5 18:06:21 casper-node casper-node[24645]:   19: tokio::runtime::blocking::pool::Inner::run
Nov  5 18:06:21 casper-node casper-node[24645]:   20: tokio::runtime::context::enter
Nov  5 18:06:21 casper-node casper-node[24645]:   21: tokio::runtime::handle::Handle::enter
Nov  5 18:06:21 casper-node casper-node[24645]:   22: std::sys_common::backtrace::__rust_begin_short_backtrace
Nov  5 18:06:21 casper-node casper-node[24645]:   23: core::ops::function::FnOnce::call_once{{vtable.shim}}
Nov  5 18:06:21 casper-node casper-node[24645]:   24: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once
Nov  5 18:06:21 casper-node casper-node[24645]:              at rustc/0cd7ff7ddfb75a38dca81ad3e76b1e984129e939/src/liballoc/boxed.rs:1081:9
Nov  5 18:06:21 casper-node casper-node[24645]:       <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once
Nov  5 18:06:21 casper-node casper-node[24645]:              at rustc/0cd7ff7ddfb75a38dca81ad3e76b1e984129e939/src/liballoc/boxed.rs:1081:9
Nov  5 18:06:21 casper-node casper-node[24645]:       std::sys::unix::thread::Thread::new::thread_start
Nov  5 18:06:21 casper-node casper-node[24645]:              at rustc/0cd7ff7ddfb75a38dca81ad3e76b1e984129e939/src/libstd/sys/unix/thread.rs:87:17
Nov  5 18:06:21 casper-node casper-node[24645]:   25: start_thread
Nov  5 18:06:21 casper-node casper-node[24645]:   26: clone
Nov  5 18:06:21 casper-node casper-node[24645]: panicked at 'fatal error [node/src/components/consensus/era_supervisor.rs:724]: Consensus shutting down due to inability to participate in the network', node/src/effect.rs:366:9

Bid contract deployment with an error

I've tried to deploy a contract from newly fresh compiled sources but the deployment was completed with an error.
I've tried to use the contract that was assembled when starting Delta-1 - everything works fine

The same error was in delta-2
"error_message": "Interpreter error: host module doesn't export function with name casper_call_contract"

Validator crash after some transactions been processed trough.

Been sending some small spam trough my node, was a sequence 1000 transaction.
Observing event on my monitor it was approximatly 870 pass only.

image

image

  • Crash message:
   0: casper_node::panic_hook
   1: core::ops::function::Fn::call
   2: std::panicking::rust_panic_with_hook
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panicking.rs:597:17
   3: std::panicking::begin_panic_handler::{{closure}}
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panicking.rs:499:13
   4: std::sys_common::backtrace::__rust_end_short_backtrace
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/sys_common/backtrace.rs:141:18
   5: rust_begin_unwind
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panicking.rs:495:5
   6: std::panicking::begin_panic_fmt
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panicking.rs:437:5
   7: <casper_node::components::gossiper::Gossiper<T,REv> as casper_node::components::Component<REv>>::handle_event
   8: <casper_node::reactor::validator::Reactor as casper_node::reactor::Reactor>::dispatch_event
   9: casper_node::cli::Cli::run::{{closure}}
  10: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  11: std::thread::local::LocalKey<T>::with
  12: tokio::runtime::enter::Enter::block_on
  13: tokio::runtime::thread_pool::ThreadPool::block_on
  14: tokio::runtime::context::enter
  15: casper_node::main
  16: std::sys_common::backtrace::__rust_begin_short_backtrace
  17: std::rt::lang_start::{{closure}}
  18: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/core/src/ops/function.rs:259:13
      std::panicking::try::do_call
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panicking.rs:381:40
      std::panicking::try
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panicking.rs:345:19
      std::panic::catch_unwind
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/panic.rs:396:14
      std::rt::lang_start_internal
             at rustc/25f6938da459a57b43bdf16ed6bdad3225b2a3ce/library/std/src/rt.rs:51:25
  19: main
  20: __libc_start_main
  21: _start

panicked at 'internal error: entered unreachable code: can't have gossiped if we don't hold the complete data', node/src/components/gossiper.rs:269:17
  • Latest log entry content

{"timestamp":"Jan 04 12:46:15.800","level":"DEBUG","fields":{"event":"deploy gossiper: check gossip timeout for deploy-hash(d008..134f) with NodeId::Tls(bad4..0f7d)","q":"Regular"},"target":"casper_node::reactor","span":{"ev":37307377,"name":"dispatch events"},"spans":[{"ev":37307377,"name":"crank"},{"ev":37307377,"name":"dispatch events"}]}

  • Log attached

tx-crash.log

  • Update

Join back was very easy, no issue.

  • Update

Network dead, we not be able to produce any blocks, nodes crashing with:

panicked at 'fatal error [node/src/components/consensus/era_supervisor.rs:811]: Consensus shutting down due to inability to participate in the network', node/src/effect.rs:423:9

Use different namespaces for different networks.

Currently a node uses a single directory to hold its LMDB database files and various "unit" files created by consensus. To avoid problems if a node is run against e.g. the testnet and later the mainnet, we need to create a subdirectory in the storage folder which is named as per the given network, i.e. it will use the chainspec.network.name value for the subdir name.

There are a few aspects to this task:

  • need to pass the network name in via Storage::new(). This will be done similarly to the protocol_version: it's also specified in the chainspec, and the initializer reactor (which constructs the Storage component) accesses the chainspec by calling the ChainspecLoader component.
  • the ContractRuntime component's constructor should be changed. Currently it takes a copy of the Storage component's config (storage_config: WithDir<StorageConfig>), but only needs that to deduce the path in which to create its own LMDB files. We should replace that variable with a simple PathBuf or &Path which will provided via a new getter on the Storage component. The initializer reactor has access to the newly-constructed Storage just before it constructs the ContractRuntime.
  • We need to ensure when nodes upgrade from the current version to this new one that they move their existing database files. This code should only need to exist while we're transitioning from the currently running version of node to the next one, so the simplest way to achieve this is to have the the Storage component move the 4 relevant files in its constructor. The files are data.lmdb, data.lmdb-lock, storage.lmdb and storage.lmdb-lock. They just need to be moved from what is currently root to the new subdir of root. That code should have a TODO added to help remind us to remove it after the next version is published, as we don't want this behaviour to persist after the upgrade in which this change is introduced.
  • All existing tests should be updated if required.
  • Should add a new test to the existing suite of Storage unit tests to exercise the behaviour of moving the LMDB files. Note that the data.lmdb and data.lmdb-lock files need not be actual LMDB files for the purposes of this test. They're the DB files backing global state (owned by ContractRuntime), but the test only needs to ensure they're moved appropriately - they won't be actually read as DB files. We tend to use tempfile::tempdir() for testing such filesystem-related things.

error[E0412]: cannot find type `__rlimit_resource_t` in crate `libc`

First, thanks for working on this product!

When following the build instructions I get

Compiling warp-json-rpc v0.2.0
error[E0412]: cannot find type `__rlimit_resource_t` in crate `libc`
  --> node/src/utils/rlimit.rs:23:32
   |
23 |     const LIBC_RESOURCE: libc::__rlimit_resource_t;
   |                                ^^^^^^^^^^^^^^^^^^^ not found in `libc`

error[E0412]: cannot find type `__rlimit_resource_t` in crate `libc`
  --> node/src/utils/rlimit.rs:31:32
   |
31 |     const LIBC_RESOURCE: libc::__rlimit_resource_t = libc::RLIMIT_NOFILE;
   |                                ^^^^^^^^^^^^^^^^^^^ not found in `libc`

error: aborting due to 2 previous errors

For more information about this error, try `rustc --explain E0412`.
error: could not compile `casper-node`

CLType of `{"ByteArray":32}` returned in deploy

When querying the deploy with hash f81d0a32890b47a5dfbb55c7bffe75d2a9301c442f7857b1908409b8e9921e83 on mainnet returns a JSON structure with the following:

        "session": {
          "Transfer": {
            "args": [
              ...
              [
                "target",
                {
                  "cl_type": {
                    "ByteArray": 32
                  },
                  "bytes": "94664ce59fa2e6eb0cc69d270fc91dd9dd2ba02a1f7964c69f036ef4a68bb96f",
                  "parsed": "94664ce59fa2e6eb0cc69d270fc91dd9dd2ba02a1f7964c69f036ef4a68bb96f"
                }
              ],
              ...
            ]
          }
        },

however cl_type is meant to be an enum rather than an object. Given the byte array size can be inferred from the bytes field can this be altered so the cl_type is just a simple ByteArray value as per other CL types?

Current [ DELTA 6 ] have peers connected from [ DELTA 5 ]

Current seed node have 7 peers connected, where one of this peer i check he is from DELTA 5 and he have about 30 peers connected from previous network. Possibility of cross chain talk.

{
  "api_version": "1.0.0",
  "chainspec_name": "casper-delta-5",
  "genesis_root_hash": "a665..29c6",
  "peers": [
    {
      "node_id": "NodeId::Tls(08ba..cc65)",
      "address": "157.90.21.155:35000"
    },
    {
      "node_id": "NodeId::Tls(08d4..0273)",
      "address": "168.119.250.90:35000"
    },
    {
      "node_id": "NodeId::Tls(1237..8332)",
      "address": "159.69.218.20:35000"
    },
    {
      "node_id": "NodeId::Tls(17d5..2f0f)",
      "address": "168.119.123.213:35000"
    },
    {
      "node_id": "NodeId::Tls(193d..23ab)",
      "address": "157.90.26.198:35000"
    },
    {
      "node_id": "NodeId::Tls(2fc9..7c20)",
      "address": "116.202.24.91:35000"
    },
    {
      "node_id": "NodeId::Tls(35b2..a27e)",
      "address": "78.47.110.240:35000"
    },
    {
      "node_id": "NodeId::Tls(3e9f..87e4)",
      "address": "116.203.182.37:35000"
    },
    {
      "node_id": "NodeId::Tls(4037..0b1e)",
      "address": "88.99.171.225:35000"
    },
    {
      "node_id": "NodeId::Tls(4214..1f70)",
      "address": "135.181.40.95:35000"
    },
    {
      "node_id": "NodeId::Tls(4a9b..2834)",
      "address": "88.99.120.29:35000"
    },
    {
      "node_id": "NodeId::Tls(52f0..b54c)",
      "address": "94.130.224.158:35000"
    },
    {
      "node_id": "NodeId::Tls(5a06..7fe9)",
      "address": "168.119.173.204:35000"
    },
    {
      "node_id": "NodeId::Tls(5e61..d73f)",
      "address": "135.181.42.149:35000"
    },
    {
      "node_id": "NodeId::Tls(64b6..aec8)",
      "address": "116.203.206.169:35000"
    },
    {
      "node_id": "NodeId::Tls(64c9..abb4)",
      "address": "88.99.36.131:35000"
    },
    {
      "node_id": "NodeId::Tls(6758..caac)",
      "address": "49.12.96.112:35000"
    },
    {
      "node_id": "NodeId::Tls(6d85..298b)",
      "address": "78.46.121.69:35000"
    },
    {
      "node_id": "NodeId::Tls(75e5..f7d6)",
      "address": "78.47.9.189:35000"
    },
    {
      "node_id": "NodeId::Tls(874f..9bcb)",
      "address": "95.217.154.113:35000"
    },
    {
      "node_id": "NodeId::Tls(90eb..7a13)",
      "address": "95.216.147.88:35000"
    },
    {
      "node_id": "NodeId::Tls(9435..f7d9)",
      "address": "78.47.18.35:35000"
    },
    {
      "node_id": "NodeId::Tls(98b6..dd10)",
      "address": "78.46.186.163:35000"
    },
    {
      "node_id": "NodeId::Tls(a9e7..4b5e)",
      "address": "168.119.59.12:35000"
    },
    {
      "node_id": "NodeId::Tls(ac63..e930)",
      "address": "168.119.169.105:35000"
    },
    {
      "node_id": "NodeId::Tls(b27e..72d6)",
      "address": "168.119.177.37:35000"
    },
    {
      "node_id": "NodeId::Tls(b328..035e)",
      "address": "168.119.126.225:35000"
    },
    {
      "node_id": "NodeId::Tls(b44b..ad4e)",
      "address": "78.47.131.222:35000"
    },
    {
      "node_id": "NodeId::Tls(dc8d..7985)",
      "address": "116.202.106.191:35000"
    },
    {
      "node_id": "NodeId::Tls(e39e..bab8)",
      "address": "78.46.198.83:35000"
    },
    {
      "node_id": "NodeId::Tls(ea95..f095)",
      "address": "95.217.213.84:35000"
    },
    {
      "node_id": "NodeId::Tls(f887..28cd)",
      "address": "157.90.26.194:35000"
    },
    {
      "node_id": "NodeId::Tls(fbd5..e151)",
      "address": "168.119.59.64:35000"
    },
    {
      "node_id": "NodeId::Tls(fef3..6370)",
      "address": "195.201.148.78:35000"
    }
  ],
  "last_added_block_info": {
    "hash": "6225f5a1130929f4e5f59f96d96c8436c007c67a949b3d1b63672d81287c5bab",
    "timestamp": "2020-12-27T09:03:14.944Z",
    "era_id": 221,
    "height": 4396,
    "state_root_hash": "d735ceeb4e9ec2f31a9f19e63cdab4445e5df85a7f805d69f30f6c2fd9dfd3f2",
    "creator": "01b952f866b554dbb80b420348bffcdc1033b5f97444e47dc8630db2d05da255d6"
  },
  "build_version": "0.3.1"
}
{
  "api_version": "1.0.0",
  "chainspec_name": "casper-delta-6",
  "genesis_root_hash": "9eb4..aef6",
  "peers": [
    {
      "node_id": "NodeId::Tls(1905..f268)",
      "address": "129.146.102.188:35000"
    },
    {
      "node_id": "NodeId::Tls(2455..759c)",
      "address": "129.146.58.40:35000"
    },
    {
      "node_id": "NodeId::Tls(3601..bb91)",
      "address": "161.97.136.153:56588"
    },
    {
      "node_id": "NodeId::Tls(3d16..a5d8)",
      "address": "158.101.3.185:35000"
    },
    {
      "node_id": "NodeId::Tls(59da..01e6)",
      "address": "18.191.239.36:35000"
    },
    {
      "node_id": "NodeId::Tls(7a60..f706)",
      "address": "101.36.118.242:35000"
    },
    {
      "node_id": "NodeId::Tls(f03f..1d84)",
      "address": "18.144.176.168:35000"
    }
  ],
  "last_added_block_info": {
    "hash": "2a48f3e41a2ee7eaf1b2d04eb8b1b3afa90091bc597872c858445fe328191b74",
    "timestamp": "2020-12-30T12:12:07.552Z",
    "era_id": 18,
    "height": 308,
    "state_root_hash": "ffd1515f875a1e27a19db90176ce01cfb6b7368ce5c2ac0be7838755abd4e596",
    "creator": "013f774a58f4d40bd9b6cce7e306e53646913860ef2a111d00f0fe7794010c4012"
  },
  "build_version": "0.4.0"
}
  • Update 3

Current peers situation

 STATUS     IP ADDRESS       TRUSTED HASH           CHAIN NAME       ERA   HEIGH     VERSION
--------------------------------------------------------------------------------------------
 Trusted    18.144.176.168   c13a2 .... baa532      casper-delta-6    19    318      0.4.0
 Trusted    13.57.200.251    c13a2 .... baa532      casper-delta-6    19    318      0.4.0
--------------------------------------------------------------------------------------------
 HOST       127.0.0.1        null                   casper-delta-6  null   null      0.4.0
 Useless    49.12.96.112     6225f .... 7c5bab      casper-delta-5   221   4396      0.3.1
 Available  13.57.200.251    c13a2 .... baa532      casper-delta-6    19    318      0.4.0
 Available  129.146.102.188  c13a2 .... baa532      casper-delta-6    19    318      0.4.0
 Useless    95.216.147.88    6225f .... 7c5bab      casper-delta-5   221   4396      0.3.1
 Available  129.146.58.40    c13a2 .... baa532      casper-delta-6    19    318      0.4.0
 Available  158.101.3.185    c13a2 .... baa532      casper-delta-6    19    318      0.4.0
 Available  18.191.239.36    c13a2 .... baa532      casper-delta-6    19    318      0.4.0
 Useless    135.181.147.7    6225f .... 7c5bab      casper-delta-5   221   4396      0.3.1
 Available  101.36.118.242   c13a2 .... baa532      casper-delta-6    19    318      0.4.0
 Useless    168.119.177.37   6225f .... 7c5bab      casper-delta-5   221   4396      0.3.1
 Available  18.144.176.168   c13a2 .... baa532      casper-delta-6    19    318      0.4.0

Trusted hash: c13a20dc256787c5a64c8dc63636638d0548815f8b728a5d1fa226515abaa532

PeersCount:     12

Useless peers:  4
Blocked access: 0
Good condition: 8
  • Update 4

Ok, so this above happen on the time of the sync, after sync complete situation change and ancient peers been disconnected from the node accordingly:

STATUS     IP ADDRESS       TRUSTED HASH           CHAIN NAME       ERA   HEIGH     VERSION
--------------------------------------------------------------------------------------------
Trusted    18.144.176.168   c13a2 .... baa532      casper-delta-6    19    318      0.4.0
Trusted    13.57.200.251    c13a2 .... baa532      casper-delta-6    19    318      0.4.0
--------------------------------------------------------------------------------------------
HOST       127.0.0.1        c13a2 .... baa532      casper-delta-6    19    318      0.4.0
Available  13.57.200.251    c13a2 .... baa532      casper-delta-6    19    318      0.4.0
Blocked    88.99.120.29                                                                  
Available  18.144.176.168   c13a2 .... baa532      casper-delta-6    19    318      0.4.0

Trusted hash: c13a20dc256787c5a64c8dc63636638d0548815f8b728a5d1fa226515abaa532

PeersCount:     3

Useless peers:  0
Blocked access: 1
Good condition: 2

Peers cross talk possibility for attack vector

Recently peers jump from 50 to 118 for exact 3 minutes. ( speaking dry-run )

image

Start of event: 2021-03-28 07:03:00 UTC
End of event: 2021-03-28 07:06:00 UTC

image

This seen in a log:

"message":"dropping connection due to network name mismatch","our_id":"NodeId::Tls(8735..e0b2)","peer_id":"NodeId::Tls(bf5c..c2d2)","our_network":"\"casper-dryrun\"","their_network":"\"delta-11\""

We aware, some validators run dry-run release on the same IP which been used in Delta's and they are still listed as trusted IP in Delta-11 config.toml. Possible this nodes act as relay. We can remove them from trusted list, but this issue still introcude possibility of exploit.

fatal error via control announcement

Found in the nightly nctl tests: Happens consistently in itst01.

Drone: https://drone-auto-casper-network.casperlabs.io/casper-network/casper-node/202/7/2


cladmin@admin:~$ tail stdout.log
{"timestamp":"May 09 18:05:13.599","level":"DEBUG","fields":{"event":"small network: request: gossip: AddressGossiper::(gossip(gossiped-address 127.0.0.1:34557))","q":"Network"},"target":"casper_node::reactor","span":{"ev":157,"name":"dispatch events"},"spans":[{"ev":157,"name":"crank"},{"ev":157,"name":"dispatch events"}]}
{"timestamp":"May 09 18:05:13.599","level":"DEBUG","fields":{"event":"address gossiper: gossiped gossiped-address 127.0.0.1:34557 to ","q":"Regular"},"target":"casper_node::reactor","span":{"ev":158,"name":"dispatch events"},"spans":[{"ev":158,"name":"crank"},{"ev":158,"name":"dispatch events"}]}
{"timestamp":"May 09 18:05:13.599","level":"DEBUG","fields":{"message":"paused gossiping since no more peers to gossip to","item":"gossiped-address 127.0.0.1:34557"},"target":"casper_node::components::gossiper","span":{"ev":158,"name":"dispatch events"},"spans":[{"ev":158,"name":"crank"},{"ev":158,"name":"dispatch events"}]}
{"timestamp":"May 09 18:05:18.587","level":"DEBUG","fields":{"event":"linear chain: Initialize timeout","q":"Regular"},"target":"casper_node::reactor","span":{"ev":159,"name":"dispatch events"},"spans":[{"ev":159,"name":"crank"},{"ev":159,"name":"dispatch events"}]}
{"timestamp":"May 09 18:05:18.587","level":"INFO","fields":{"message":"hasn't downloaded any blocks in expected time window. Shutting down…"},"target":"casper_node::components::linear_chain_sync","span":{"ev":159,"name":"dispatch events"},"spans":[{"ev":159,"name":"crank"},{"ev":159,"name":"dispatch events"}]}
{"timestamp":"May 09 18:05:18.587","level":"DEBUG","fields":{"event":"control: fatal error [node/src/components/linear_chain_sync.rs:824]: no syncing progress, shutting down…","q":"Control"},"target":"casper_node::reactor","span":{"ev":160,"name":"dispatch events"},"spans":[{"ev":160,"name":"crank"},{"ev":160,"name":"dispatch events"}]}
{"timestamp":"May 09 18:05:18.587","level":"ERROR","fields":{"message":"fatal error via control announcement","file":"node/src/components/linear_chain_sync.rs","line":"824","msg":"no syncing progress, shutting down…"},"target":"casper_node::reactor","span":{"ev":160,"name":"dispatch events"},"spans":[{"ev":160,"name":"crank"},{"ev":160,"name":"dispatch events"}]}
{"timestamp":"May 09 18:05:18.587","level":"INFO","fields":{"message":"shutting down socket, no longer accepting incoming connections","our_id":"NodeId::Tls(0254..87d6)"},"target":"casper_node::components::small_network"}
{"timestamp":"May 09 18:05:18.589","level":"INFO","fields":{"message":"exiting casper-node","exit_code":"101"},"target":"casper_node"}
{"timestamp":"May 09 18:05:18.591","level":"WARN","fields":{"message":"failed running \"/home/cladmin/cl/git/casper-node/utils/nctl/assets/net-1/nodes/node-5/bin/1_0_0/casper-node\" \"validator\" \"/home/cladmin/cl/git/casper-node/utils/nctl/assets/net-1/nodes/node-5/config/1_0_0/config.toml\"","exit_status":"exit code: 101"},"target":"casper_node_launcher::utils"}


Log files invalid format

Periodically jq refuse to parse with the following error

parse error: Invalid numeric literal at line ...

This is the case of the issue:
image

Another problem I seen is this, often make grep thinks is a binary file

log_issu

This above artifacts possible coming from incorrect log rotation.

Crash during catchup with trusted hash

message.txt

See the attached file for log from the node.
Crash can be reproduced by installing everything on a clean box and syncing the node with trusted hashes.

Tested: Around 20 Resyncs
Crashes: 6
Tested with different trusted hashes - Crash is not always happening (So hash is valid)
One of the hashes tested with: 3b0e4ae85f096b0be2e28402c3f32c471d0f5f4a470d2709c731e0febf3d26ec
Environment: Ubuntu 18.04 (Docker - can provide build scripts)

Weird behavior on genesis generation. After "Consensus(Shutdown)" node shutdown.

On the start of the Delta - 3 exact on the moment of genesis generation, my validator lost peers.

Status replay:

{
  "api_version": "1.0.0",
  "chainspec_name": "casper-delta-3",
  "genesis_root_hash": "e81d..b4c0",
  "peers": {},
  "last_added_block_info": null,
  "build_version": "1.6.0"
}

Expected:

{
    "api_version": "1.0.0",
    "chainspec_name": "casper-delta-3",
    "genesis_root_hash": "e81d..b4c0",
    "peers": {
        "45cf..1364": "13.57.242.68:35000",
        "4e5d..4e53": "194.19.235.78:35000",
        "716d..9a06": "167.99.74.47:35000",
        "db09..0b9b": "54.183.27.75:35000",
        "e52c..55ac": "207.180.221.204:35000"
    },
    "last_added_block_info": null,
    "build_version": "1.6.0"
}
  1. Error
24: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once
             at rustc/0cd7ff7ddfb75a38dca81ad3e76b1e984129e939/src/liballoc/boxed.rs:1081:9
      <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once
             at rustc/0cd7ff7ddfb75a38dca81ad3e76b1e984129e939/src/liballoc/boxed.rs:1081:9
      std::sys::unix:🧵:Thread:🆕:thread_start
             at rustc/0cd7ff7ddfb75a38dca81ad3e76b1e984129e939/src/libstd/sys/unix/thread.rs:87:17
  25: start_thread
  26: clone
  1. Error

"DEBUG","fields":{"event":"consensus: Shutdown if current era is inactive","q":"Regular"},"target":"casper_node::reactor","span":{"ev":8053,"name":"dispatch events"},"spans":[{"ev":8053,"name":"crank"},{"ev":8053,"name":"dispatch events"}]}
{"timestamp":"Nov 20 22:05:00.003","level":"TRACE","fields":{"event":"Consensus(Shutdown)","q":"Regular"},"target":"casper_node::reactor","span":{"ev":8053,"name":"dispatch events"},"spans":[{"ev":8053,"name":"crank"},{"ev":8053,"name":"dispatch events"}]}


P.S. Peers was present before genesis time.

Node Crash [ DELTA 8 ]

Based on:

I've found the "scheduler_queue_total_count" to be a more useful one tbh. It shows the current total backlog of these kind of events waiting to be processed. We aim for it to always be very low - ideally single or double digits. When we saw the recent OOM issues, before the node crashed, that value would invariably skyrocket to hundreds of thousands - meaning we had a huge backlog of events

Here is the moment of actual crash: ( 2021-01-10 14:11:52 UTC )

image

Since we sure this is OOM most luckily:

image

Here is no doubts, the reason for node to loose ability providing us with HTTP and RPC end point is coming from the same source. We can observe it on mostly every graph, as example I post graphs below.

From here we can see, issue start accumulating before the actual catastrophe ( 2021-01-10 06:27:00 UTC )

image

  • Update 1

I been tracking network activity, and here is some in case can be usefull.

image

From here we can see, since my crawler still crawling after node colapse, new IP addresses still joining chain.

At moment of crash my node have 69 peers connected, auction order book have 65 bids and network coun 132 unique IP addresses. My validator crash happen when 133 addresses been count, it was exactly on the moment when new IP join network, not sure if this can be related to some specific missconfigured node joining.

  • Update 2

Last IP's joined network:

130 95.216.224.235
131 95.217.0.233
132 95.217.154.113
133 95.217.154.200
134 95.217.213.84
135 95.217.84.84
136 95.217.94.83

  • Update 3

List of current "still alive peers":

 104.197.195.68  Era: 101       Height: 2647  
 104.236.61.140  Era: 101       Height: 2647  
 107.22.247.79   Era: 101       Height: 2647  
 116.202.106.191 Era: 83        Height: 2277  
 116.202.24.91   Era: 80        Height: 2205  
 116.203.157.37  Era: 101       Height: 2647  
 116.203.206.169 Era: 101       Height: 2647  
 116.203.214.160 Era: 101       Height: 2647  
 116.203.254.245 Era: 101       Height: 2647  
 116.203.69.88   Era: 101       Height: 2647  
 129.146.102.188 Era: 101       Height: 2647  
 129.146.102.28  Era: 101       Height: 2647  
 129.146.137.190 Era: 101       Height: 2647  
 129.146.166.34  Era: 101       Height: 2647  
 129.146.17.202  Era: 101       Height: 2647  
 129.146.243.139 Era: 101       Height: 2647  
 129.146.58.40   Era: 101       Height: 2647  
 129.146.83.137  Era: 101       Height: 2647  
 134.209.16.172  Era: 101       Height: 2647  
 135.181.129.210 Era: 101       Height: 2647  
 135.181.143.233 Era: 97        Height: 2605  
 135.181.163.79  Era: 101       Height: 2647  
 135.181.192.149 Era: 101       Height: 2647  
 135.181.25.45   Era: 101       Height: 2647  
 135.181.30.191  Era: 101       Height: 2647  
 135.181.35.187  Era: 101       Height: 2647  
 135.181.40.95   Era: 101       Height: 2647  
 135.181.89.76   Era: 101       Height: 2647  
 135.181.97.186  Era: 101       Height: 2647  
 13.68.180.213   Era: 101       Height: 2647  
 138.197.65.212  Era: 101       Height: 2647  
 143.110.225.98  Era: 101       Height: 2647  
 144.76.199.44   Era: 101       Height: 2647  
 144.91.109.12   Era: 101       Height: 2647  
 157.90.19.142   Era: 101       Height: 2647  
 157.90.26.194   Era: 101       Height: 2647  
 158.101.3.185   Era: 101       Height: 2647  
 158.247.204.136 Era: 101       Height: 2647  
 161.97.124.55   Era: 101       Height: 2647  
 161.97.152.70   Era: 101       Height: 2647  
 161.97.82.77    Era: 101       Height: 2647  
 161.97.96.132   Era: 101       Height: 2647  
 167.86.75.50    Era: 101       Height: 2647  
 168.119.123.213 Era: 101       Height: 2647  
 168.119.173.204 Era: 101       Height: 2647  
 168.119.179.150 Era: 101       Height: 2647  
 168.119.59.64   Era: 101       Height: 2647  
 172.105.13.176  Era: 101       Height: 2647  
 173.212.205.2   Era: 101       Height: 2647  
 194.19.235.78   Era: 101       Height: 2647  
 195.201.148.78  Era: 96        Height: 2565  
 195.201.220.118 Era: 101       Height: 2647  
 207.180.198.56  Era: 101       Height: 2647  
 207.180.245.251 Era: 101       Height: 2647  
 3.19.76.184     Era: 101       Height: 2647  
 34.121.43.83    Era: 101       Height: 2647  
 34.65.104.178   Era: 101       Height: 2647  
 35.198.176.85   Era: 101       Height: 2647  
 35.233.165.80   Era: 101       Height: 2647  
 40.75.89.48     Era: 101       Height: 2647  
 46.4.15.204     Era: 101       Height: 2647  
 49.12.109.102   Era: 101       Height: 2647  
 49.12.66.192    Era: 97        Height: 2605  
 49.12.96.112    Era: 101       Height: 2647  
 52.247.126.146  Era: 101       Height: 2647  
 52.247.219.167  Era: 101       Height: 2647  
 78.46.121.69    Era: 101       Height: 2647  
 78.46.186.163   Era: 101       Height: 2647  
 78.47.10.119    Era: 101       Height: 2647  
 78.47.110.240   Era: 101       Height: 2647  
 78.47.9.189     Era: 101       Height: 2647  
 88.198.227.113  Era: 101       Height: 2647  
 88.99.120.29    Era: 101       Height: 2647  
 88.99.36.131    Era: 101       Height: 2647  
 94.130.224.158  Era: 101       Height: 2647  
 95.216.147.88   Era: 101       Height: 2647  
 95.216.164.122  Era: 101       Height: 2647  
 95.217.154.113  Era: 101       Height: 2647  
 95.217.154.200  Era: 101       Height: 2647  
 95.217.213.84   Era: 101       Height: 2647  
 95.217.84.84    Era: 101       Height: 2647 

Dead peers:

103.212.121.38                               
116.202.17.49                                
116.203.157.213                              
116.203.182.37                               
135.181.118.39                               
135.181.147.7                                
135.181.152.29                               
135.181.42.149                               
13.57.200.251                                
144.76.111.74                                
144.91.123.104                               
144.91.83.59                                 
157.90.21.155                                
157.90.26.198                                
158.101.12.238                               
159.146.54.127                               
159.69.218.20                                
161.97.136.153                               
161.97.143.67                                
167.99.74.47                                 
168.119.126.225                              
168.119.169.105                              
168.119.177.31                               
168.119.177.37                               
168.119.250.90                               
168.119.49.220                               
168.119.59.12                                
173.212.200.158                              
173.249.31.34                                
18.144.176.168                               
18.217.179.179                               
20.57.161.70                                 
206.189.4.64                                 
207.180.199.157                              
207.180.213.119                              
207.180.243.57                               
34.122.124.107                               
34.123.112.124                               
34.75.153.42                                 
35.236.56.221                                
35.239.228.18                                
45.61.52.142                                 
46.4.91.24                                   
62.171.165.37                                
78.46.198.83                                 
78.46.200.17                                 
78.47.131.222                                
78.47.18.35                                  
78.47.218.195                                
88.198.200.113                               
88.99.171.225                                
88.99.227.165                                
95.216.224.235                               
95.217.0.233                                 
95.217.94.83

Last delta-8 geomap:

delta-8

Geodata CSV attached
delta-8_geo_data.txt

Add test for crashed-node reconnection regression

#1413 fixed a regression that caused a node not to reconnect to a crashed node. The conditions are somewhat tough to replicate outside of our integration, but it could be possible to add these to the networking tests using testing::Network. We should definitely try to replicate this bug (by temporarily turning off the fix) and add a test for it.

timestamp format is invalid for key `network.timestamp` at line 208 column 1

hello, when running-one-node), I get this error

Mar 17 02:45:30.638 INFO  [casper_node::cli cli.rs:149] node starting up; version=0.9.0-cb1d20ad
Mar 17 02:45:30.638 WARN  [casper_node::reactor reactor.rs:437] large event size, consider reducing it or boxing; event_size=272
Error: chainspec error: decoding from TOML error: timestamp format is invalid for key `network.timestamp` at line 208 column 1

Caused by:
    0: decoding from TOML error: timestamp format is invalid for key `network.timestamp` at line 208 column 1
    1: timestamp format is invalid for key `network.timestamp` at line 208 column 1
➜  casper-node pwd
/etc/casper-node
➜  casper-node ls
chainspec.toml  config.toml

chainspec.toml and config.toml copy from chainspec.toml and config.toml

[network]
# Human readable name for convenience; the genesis_hash is the true identifier.  The name influences the genesis hash by
# contributing to the seeding of the pseudo-random number generator used in contract-runtime for computing genesis
# post-state hash.
name = 'casper-example'
# Timestamp for the genesis block.  This is the beginning of era 0. By this time, a sufficient majority (> 50% + F/2 —
# see finality_threshold_percent below) of validator nodes must be up and running to start the blockchain.  This
# timestamp is also used in seeding the pseudo-random number generator used in contract-runtime for computing genesis
# post-state hash.
timestamp = '${TIMESTAMP}'

How should I do?

" Failed sending Request" error when executing transfer subcommand

When i try to run :

cargo run --release -- transfer \
    --node-address=http://localhost:50101 \
    --secret-key=../utils/nctl/assets/net-1/nodes/node-3/keys/secret_key.pem \
    --amount=1234567 \
    --target-account=$PUBLIC_KEY \
    --chain-name=casper-net-1 \
    --payment-amount=10000

I got this error:

Failed sending Request(Request { jsonrpc: "2.0", method: "account_put_deploy", params: Some(Map({"deploy": Object({"approvals": Array([Object({"signature": String("01701aa67c643dd16098e830151b5e6555acb83bb58a1fa75324df9d99f2c8ed646b93d410f6ae4a561bcc94d95e6b4b36b9adcc8a7df2b4e0fa9d43ca1137b50d"), "signer": String("016222cc3d4dc35c387d3d2deff214674be8f750a921b97667efadf6b0c6826226")})]), "hash": String("65e0e5a356038e088dca524734fd61aba7dc92229d88fdd8fc6b9753e015e7fc"), "header": Object({"account": String("016222cc3d4dc35c387d3d2deff214674be8f750a921b97667efadf6b0c6826226"), "body_hash": String("889297a958b5385f092f1918bd3b2f04507286076ad9124148505339703d6a21"), "chain_name": String("casper-net-1"), "dependencies": Array([]), "gas_price": Number(1), "timestamp": String("2021-05-06T20:08:21.389Z"), "ttl": String("1h")}), "payment": Object({"ModuleBytes": Object({"args": Array([Array([String("amount"), Object({"bytes": String("021027"), "cl_type": String("U512"), "parsed": String("10000")})])]), "module_bytes": String("")})}), "session": Object({"Transfer": Object({"args": Array([Array([String("amount"), Object({"bytes": String("0387d612"), "cl_type": String("U512"), "parsed": String("1234567")})]), Array([String("target"), Object({"bytes": String("fa522ee1af717bcd71259d336666017000eb3f659346806f4cdcdab848724bd4"), "cl_type": Object({"ByteArray": Number(32)}), "parsed": String("fa522ee1af717bcd71259d336666017000eb3f659346806f4cdcdab848724bd4")})]), Array([String("id"), Object({"bytes": String("00"), "cl_type": Object({"Option": String("U64")}), "parsed": Null})])])})})})})), id: Num(-799331664064629152) })

My local network is running and the output of nctl-status is the flowing:

validators-1:casper-net-1-node-1    RUNNING   pid 6266, uptime 5:24:14
validators-1:casper-net-1-node-2    RUNNING   pid 6267, uptime 5:24:14
validators-1:casper-net-1-node-3    RUNNING   pid 6268, uptime 5:24:14
validators-2:casper-net-1-node-4    RUNNING   pid 6309, uptime 5:24:12
validators-2:casper-net-1-node-5    RUNNING   pid 6310, uptime 5:24:12
validators-3:casper-net-1-node-10   STOPPED   Not started
validators-3:casper-net-1-node-6    STOPPED   Not started
validators-3:casper-net-1-node-7    STOPPED   Not started
validators-3:casper-net-1-node-8    STOPPED   Not started
validators-3:casper-net-1-node-9    STOPPED   Not started

Any idea how to fix this issue please ?

Dictionary

This task is about implementing support for improved local keys, which intends to resolve past problems that lead to their removal. Contract developers will use this to store data in a separate keyspace prefixed by the URef address provisioned to a user by the system.

CEP for reference: casper-network/ceps#39

error[E0412]: cannot find type `__rlimit_resource_t` in crate `libc`

When running:
$ cargo build -p casper-node --release

I'm getting:

Compiling casper-node v0.9.0
error[E0412]: cannot find type `__rlimit_resource_t` in crate `libc`
--> node/src/utils/rlimit.rs:23:32
|
23 |     const LIBC_RESOURCE: libc::__rlimit_resource_t;
|                                ^^^^^^^^^^^^^^^^^^^ not found in `libc`

error[E0412]: cannot find type `__rlimit_resource_t` in crate `libc`
--> node/src/utils/rlimit.rs:32:32
|
32 |     const LIBC_RESOURCE: libc::__rlimit_resource_t  = 7;
|                                ^^^^^^^^^^^^^^^^^^^ not found in `libc`

error: aborting due to 2 previous errors

For more information about this error, try `rustc --explain E0412`.
error: could not compile `casper-node`

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.