Ansible playbooks for deployment POA Network nodes on EC2 or any Linux (Ubuntu 16.04) hosting. Includes master of ceremony, validator, bootnode, explorer, netstat roles
License: MIT License
Things to prepare:
threads with processing_threads in node.tomlparity db kill ? resync ?Things to do:
https://d1h4xl4cr1h0mo.cloudfront.net/v1.9.2/x86_64-unknown-linux-gnu/parity3604a030388cd2c22ebe687787413522106c697610426e09b3c5da4fe70bbd33Wondering if this https://github.com/debops/ansible-ferm module will benefit our security policies.
Manage iptables firewall using ferm
Currently each node uses its own user to run processes. Bootnode uses bootnode and moc uses moc user. Each user has its own keyfile. There is superuser root.
Login in as root is considered as bad security practice. Deployment playbooks should run as ordinary user with sudo privileges. No root account configuration should be necessary.
I see no value for having different users on different hosts. The proposal is to have the same user running all the processes.
Playbooks put public key for the operator to the remote host so they can SSH as a poa user with their private key.
To support multiple cloud providers deployment playbooks should clearly separate configuration management code and cloud resources provisioning.
To achieve this I propose
group_vars/all to a separate file-access roles to the -aws-access roles/aws subfolderThe role should allow having archiving for
At the moment, archiving node on kovan/mainent runs with parameters
RUST_BACKTRACE=1 parity --pruning=archive --fat-db on --cache-size-db 12000 --min-peers 5 --max-peers 10 --snapshot-peers 500 --pruning-history 1200 --no-ui --no-dapps --auto-update=all --no-discovery --allow-ips=public --no-periodic-snapshot
we should use parameters
Looks like in tasks we have two ways to define working path. Smth like
{{ home }}/eth-netstats
and
/home/{{ username }}/eth-netstats
Is there reason two have them both?
@phahulin
it is recommended to run with --usd-per-tx 0 because Parity cannot calculate the correct POA-USD rate.
[mining]
usd_per_tx = "0"
Is this OK?
jhl@johnny-lenovo:~/poa-network/deployment-playbooks/group_vars$ git branch
Is this OK?
jhl@johnny-lenovo:~/poa-network/deployment-playbooks/group_vars$ git branch
Now we have two files - all.example and all.network. The point of this issue is to refactor all.example file to explicitly specify there all necessary variables for deployment. It will become a kind of instruction file for those who want to create their own network.
Some templates have {{ansible_distribution_release}} variable declaration, without spaces. Looks like we should find and change all such variables to {{ some_var_name }}. Doesn't affect funtionality actually, but code will look better.
From the point of simplicity it will be a good idea to explicitly define which variables are used by roles. To do that I propose to create defaults/main.yml files in each role, where will be all the necessary (for that role) variables defined.
When launched as a dependency from another role those defaults variables will be overwritten by group_vars/ folder variables, so it's safe to define them.
As well, we will be able to launch dependent roles separately.
Solution:
README.md file should have steps in order to run ec2.yml book. For example:
It should tell a user to setup AWS keys first in groups_var folder:
https://github.com/oraclesorg/deployment-playbooks/blob/master/group_vars/all#L15-L16
Question#1:
What is https://github.com/oraclesorg/deployment-playbooks/blob/master/group_vars/all#L24?
Where do I find it? Does it show up when I create EC2 instance?
Question#2:
Does the script assumes to have
https://github.com/oraclesorg/deployment-playbooks/blob/master/group_vars/all#L18
ssh keypairname already being set in aws keys? if not, is there an instruction on how to do so?
Question#3:
Any difference between files.pub vs ssh_bootnode.pub? Where each one is used?
https://github.com/oraclesorg/deployment-playbooks/tree/master/files
Questoin#4:
When you run ec2.yml how to verify that everything has completed as expected? Please provide verification steps.
Question#5:
When you run sites.yml how to verify that everything has completed as expected? Please provide verification steps.
Some templates has unnecessary Environment=MYVAR=myval declaration, which should be cleaned up. The list of templates below:
As I can see the difference is in the group variables files and a single template file.
One may create two example files for different network types. Then parameterize template so it will be the same for both networks and put the configuration to the variables file. That way only one branch is needed which is easier to maintain and integrate with terraform provisioning.
Is there any reason for maintaining two additional branches that I am not aware of?
Some variables may be moved to the role level and set up with reasonable defaults.
Variables that control access to the node:
allow_validator_ssh: true
allow_validator_p2p: true
Play variables:
nginx_headers: "on"
PROXY_PORT: "8545"
username: "bootnode"
users:
- name: "bootnode"
home: "/home/bootnode"
Proposed structure is:
roles
|-bootnode
|-defaults
|-main.yml
For better time sync http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html
Problem:
ssh/22 for themtcp/30303 open to the internetSolution:
add variable allow_ssh: true/false
if it's true, ssh/22 is enabled on the security group. the default value is true
add variable allow_p2p: true/false
if it's true, tcp/30303 is enabled on a security group. the default values is true
Currenly git task downloads the latest version of code committed to GitHub repo.
git:
repo: "https://github.com/{{ MAIN_REPO_FETCH }}/chain-explorer"
dest: "{{ home }}/chain-explorer"
Proposal is to fix the version to a specific commit like this:
git:
repo: "https://github.com/{{ MAIN_REPO_FETCH }}/chain-explorer"
dest: "{{ home }}/chain-explorer"
version: "acee07c"
The main reason is reproducible deployments. When I deploy from the same configuration I expect to get the same result. If I rely on the latest version of the code in the repository then my deployment in 1 month will be different from my deployment today.
This scenario is hard to troubleshoot. Also fixing the version allows controlled dependency upgrade that is visible in the commit history.
Problem:
Solution:
Issue:
TASK [Install python] ************************************************************************************************
The authenticity of host '13.56.182.154 (13.56.182.154)' can't be established.
ECDSA key fingerprint is SHA256:8f91Tzugvp0uA+5mRrqxpVkIWMhH753u5kiU3K9Hvfc.
Are you sure you want to continue connecting (yes/no)? fatal: [13.56.182.154]: FAILED! => {"failed": true, "msg": "Timeout (12s) waiting for privilege escalation prompt: "}
It waits for an answer and it doesn't respond yes to it
Each node creates log directories and configures log directories permissions.
It is a good candidate for the refactoring to a separate role.
https://github.com/poanetwork/deployment-playbooks/blob/master/roles/bootnode/tasks/main.yml#L2-L16
Deployment playbooks support one base OS image at this time - Ubuntu 16.04. This issue requests support for CentOS 7.X as base image for POA nodes.
To add CentOS 7.x support:
apt to yum based on ansible_os_family variableExample of maintaining different set of tasks for different platforms:
# file: tasks.yml
- import_tasks: tasks_ubuntu.yml
when: ansible_os_family == "Debian"
- import_tasks: tasks_centos.yml
when: ansible_os_family == "RedHat"
Problem:
there is only one thread for bootnode which is not enough
Solution:
increase to 4 using parameter json-threads=4
--jsonrpc-threads THREADS Turn on additional processing threads in all RPC servers.
Setting this to non-zero value allows parallel cpu-heavy queries
execution. (default: 0)
Problem: Now we have the same security group for different networks. While tested Sokol I updated a security group and it affected Core. 😳
Solution: Add a prefix to security group, e.g. core- and sokol-
At this moment list of packages that this role installs looks like this:
- bc
- haveged
- rsync
- iotop
- dstat
- sysstat
- htop
- lbzip2
- pigz
- unzip
- zip
- mtr
- tcpdump
- openssh-client
- sudo
- mc
- net-tools
- screen
- git
- cloud-utils
- build-essential
- nload
They fall into different categories but it is impossible to tell what is the exact purpose of each package.
Sysadmin tools mtr and tcpdump are typical examples. Operator uses these packages to troubleshoot issues on the node. If drop these packages node is still operational.
POA Components dependencies If drop these packages some of poa components will stop working. When poa component stops dependency on this packet it should be removed from the system.
Ansible modules dependencies. Ansible modules depend on these packages to do their tasks. If the module is not needed then this package should not be installed.
The proposal:
preconf roleCurrent playbooks look like this
roles:
- usermanager
- nodejs
- poa-logrotate
- poa-parity
- poa-pm2
- poa-netstats
- validator
- validator-access
What they should look like is this
roles:
- validator
Only a single role that defines node type.
If one wants to co-locate two roles on a single node it is possible and clear:
roles:
- validator
- explorer
To achieve this use Ansible role dependencies
Noticed that Google is being used for DNS services and only google (8.8.8.8 and 8.8.4.4) inside of /group_vars/all. Although I do this myself, perhaps add another public DNS services provider...just in case google is unreachable? (Granted, if google's unreachable - we should probably be ducking and covering.)
Steps:
When I run ansible-playbook with image: "ami-9d04e4e5" I get:
TASK [Launch instance] ***********************************************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg": "Instance creation failed => InvalidAMIID.NotFound: The image id '[ami-9d04e4e5]' does not exist"}Solution:
Please explain to the user how to create an AMI
Vagrant complains about implicit compatibility mode selection.
Vagrant has automatically selected the compatibility mode '2.0'
according to the Ansible version installed (2.4.3.0).
Alternatively, the compatibility mode can be specified in your Vagrantfile:
https://www.vagrantup.com/docs/provisioning/ansible_common.html#compatibility_mode
To make it explicit add the following line to the Vagrantfile:
node.vm.provision :ansible do |ansible|
...
ansible.compatibility_mode = "2.0"
...
While installing pyOpenSSL module there is a chance, that not all necessary packages already installed.
To avoid this i propose the following code changes for nginx role:
Current:
- name: Ensure python OpenSSL dependencies are installed.
pip:
name: pyOpenSSL
state: present
Fix:
- name: Ensure OpenSSL dependencies are installed (Ubuntu)
package:
name: {{ item }}
state: present
with_items:
- build-essential
- libssl-dev
- libffi-dev
- python-dev
when: ansible_os_family == "Debian"
- name: Ensure OpenSSL dependencies are installed (CentOS)
package:
name: {{ item }}
state: present
with_items:
- gcc
- libffi-devel
- openssl-devel
- python-devel
when: ansible_os_family == "RedHat"
- name: Install OpenSSL module
pip:
name: pyOpenSSL
state: present
Update instructions: https://github.com/poanetwork/poa-devops/blob/master/docs/Update-parity-version.md
Migration playbook (https://github.com/poanetwork/poa-devops/blob/master/roles/upd-parity-version/tasks/main.yml):
cors section for bootnodesUpdate nodes (I round - less critical nodes):
Update defaults in deployment playbooks repository (#86):
https://d1h4xl4cr1h0mo.cloudfront.net/v1.9.2/x86_64-unknown-linux-gnu/parity, checksum to 3604a030388cd2c22ebe687787413522106c697610426e09b3c5da4fe70bbd33Update nodes (II round - public rpc nodes and MoC):
Update nodes (III round - part of validators)
Notify BINANCE?
Update nodes (IV round - other bootnodes and validators):
Update nodes (V round - backup)
Currently all the node types plays are defined in site.yml. This issue suggests splitting the site.yml file into a set of files like validator.yml. Then these files are imported to the main file with import_playbook statement. This structure is modular and can be easily integrated with Terraform.
For example, to provision only the validator node one runs
ansible-playbook validator.yml
Current validator.yml moves to the aws subfolder as it contains aws-related code.
Prefix each role with the installation_path variable and make sure role does not install anything outside of this path.
This step simplifies different roles collocation on the same vm and playbook dependency on the single home variable.
Some configuration files are owned by root, some are owned by corresponding role-user (moc, bootnode, etc). They all should probably be owned by role-user.
Currently -access roles contain code for configuration of EC2 firewall and system firewall. Moving to new cloud providers like Azure require splitting node configuration from provider configuration.
ufw.yml file into the role itself, like from moc-access to mocmain.yml file in this role-access role to the -aws-access role-aws-access role to the EC2-related playbooks and skip dependency from the moc roleThese suggestions apply to all five node types.
If the SSH keys are password protected, ansible-playbook fails with
TASK [hf-spec-change : Shutdown poa-netstats service] *****************************************************************************
fatal: [52.191.165.235]: FAILED! => {"changed": false, "msg": "Unable to stop service poa-netstats: Failed to stop poa-netstats.service: Interactive authentication required.\nSee system logs and 'systemctl status poa-netstats.service' for details.\n"}
to retry, use: --limit @/home/mm/poa-devops/site.retry
The workaround is to use non-password-protected ssh keys, but that's a security vulnerability if the control system is compromised. Suggest looking into allowing interactive auth during deployment.
Best, MM
In the current configuration node.toml files are downloaded from https://github.com/poanetwork/deployment-azure repository. Then additional parameters are added via text replacement or appending.
It would be better to store them directly in playbook as templates: everything will be kept in one place && it's much cleaner - with text replacement it's not clear what the resulting file looks like.
Update instructions: https://github.com/poanetwork/poa-devops/blob/master/docs/Update-parity-version.md
Parity binary for orchestrator (https://github.com/phahulin/parity/tree/v1.10.0-disable-parity-whisper-extensions)
Not sure about /api/health endpoint - doesn't work without dapps enabled, maybe re-enable them (openethereum/parity-ethereum#8245, #108)
Update default version in playbook config (#106)
Launch second MoC node on parity 1.9.2 and CentOS
Update nodes (round I):
Ping Andrew Cravenho to check poa explorer
Update nodes (round II):
Update nodes (round III):
Check consensus
Update archiving nodes (round IV):
If machine fails to start during make test the tests show the error and then try to start the next machine.
The desired behavior is to fail on the first error.
The possible solution is
- vagrant up $$i || exit 1; \
+ vagrant up $$i; \
https://github.com/debops/ansible-fail2ban
I'm curious if it will help to secure our nodes
@VitalyZnachenok
I am working on PR that will update playbooks to pass Ansible-lint. Linter checks the code against set of rules and generates errors in case of their violation.
I would suggest running linter as a pre-commit hook so each commit is validated.
Also run ansible-playbook --syntax-check to verify syntax and show deprecation warnings.
Running virtual machine in the cloud just to test the change in the playbook is time consuming.
Better way is to test changes locally before applying them to master branch. There are two possible ways of doing this:
vagrant with VirtualBox & Ansible provisionersvagrant with Docker & Ansible provisionersBoth options use Ubuntu images similar to those used in the cloud.
The second option is faster and require less resources.
Does this project use some kind of local testing already?
The workflow will look like:
vagrant up runs Docker container and applies playbook to this containervagrant down tears down the containerpre-commit hook runs linter and syntax checks)As a possible solution to #74 one may create two host groups in the inventory - sokol and core. Then add sokol-related variables to the group_vars/sokol file and core-related variables to the group_vars/core file.
Variables will live in a single branch and may be applied based on what deployment group hosts belong to. That makes possible maintaining both sokol and core deployments from a single playbook.
group_vars/all should be renamed to group_vars/poa which is applied to the poa group. poa group includes sokol and core as children.
This issue discusses variable naming conventions. Now there is some structure in naming like bootnode_orchestrator but it is used for subset of variables and is not fully consistent.
All variables must have prefix defining their scope. Possible scopes for node types are validator, moc, explorer, netstat, bootnode. Possible scopes for the network-level variables are test and prod. It is possible to use role name as scope for low-level roles like parity_.
Suffix describes the variable type as Ansible does not have strict type system. It is optional and should be used when the type of the variable is not clear from the context. Example: parity_binary_url.
All variables that are not intended to be modified by the user should be moved to the role level /vars folder. Example: NODE_PWD: "node.pwd", NODE_SOURCE_DEB.
User should be able to configure role just looking at its defaults folder.
All variables should be lower-case.
AWS-related variables must be in a separate file.
When running the playbook I get an error:
$ ansible-playbook -i hosts site.yml
...
TASK [usermanager : Create users] ******************************************************
failed: [bootnode/0] (item={u'name': u'bootnode'}) => {"changed": false, "item": {"name": "bootnode"}, "msg": "useradd: Permission denied.\nuseradd: cannot lock /etc/passwd; try again later.\n", "name": "bootnode", "rc": 1}
I expect this task to use sudo or the whole play run with become: True.
Ansible connects to the remote machine as user poa. This user has passwordless sudo access:
poa@netstat:~$ sudo whoami
root
As fix I add become: True to the whole play:
- hosts: bootnode
become: True
vars:
...
Currently all the variables are in the group_vars/all file.
The first type of vars should be moved to the role level vars folder and the second to the default folder.
Variables on playbook level will be specific to the environment (sokol or core) and contain only required ones. That way it will be easier for the user to configure the deployment.
I can't find the usage of several variables defined in the all.network file. Could you please describe their purpose and where are they used?
snmp_syslocation: "USA"
snmp_ipsubnet: "172.16.0.0/16"
nameservers:
- "8.8.8.8"
- "8.8.4.4"
ntpservers:
- "server 0.us.pool.ntp.org"
- "server 1.us.pool.ntp.org"
- "server 2.us.pool.ntp.org"
- "server 3.us.pool.ntp.org"
https://github.com/poanetwork/deployment-playbooks/blob/master/group_vars/all.network#L7-L18
Let's consider explorer role.
explorer role depends on poa-parity:
---
dependencies:
..
- { role: poa-parity }
But poa-parity itself has dependency on explorer (and netstats) by using template roles/poa-parity/templates/poa-chain-explorer.j2.
The scope of this issue to resolve such circular dependencies. Node-level roles depend on other roles. But this roles never depend on the node roles - only roles down the hierarchy. Instead second-level roles get configuration and configure the templates accordingly.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.