josephmachado / bitcoinmonitor

Near real time ETL to populate a dashboard.

Hi 👋

I write about data engineering | data modeling | software engineering | system design

• 🔭 I write data engineering articles on Start data engineering

• 👨‍💻 All of my projects are available at https://github.com/josephmachado

• 📝 I regularly write articles on https://www.startdataengineering.com/

• 💬 Ask me about Data engineering, SQL, Databases, Data pipelines, Data infrastructure

• 📫 How to reach me https://www.startdataengineering.com/contact

• 📄 Know about my experiences https://www.linkedin.com/in/josephmachado1991/

Connect with me:

Languages and Tools:

bitcoinmonitor's Issues

SSH hangs after authenticated

From the Makefile (cloud-metabase) I removed the below part open http://localhost:3001 because It was giving me the following error: /bin/bash: open: command not found

cloud-metabase:
	terraform -chdir=./terraform output -raw private_key > private_key.pem && chmod 600 private_key.pem && ssh -o "IdentitiesOnly yes" -i private_key.pem ubuntu@$$(terraform -chdir=./terraform output -raw ec2_public_dns) -N -v -v -f -L 3001:$$(terraform -chdir=./terraform output -raw ec2_public_dns):3000 && rm private_key.pem

Also, ssh tunnel refusing connections with "channel 2" so I added -v -v as in this thread

After running the make cloud-metabase command, I get the following message:

OpenSSH_8.2p1 Ubuntu-4ubuntu0.5, OpenSSL 1.1.1f  31 Mar 2020
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolving "ec2-18-205-163-192.compute-1.amazonaws.com" port 22
debug2: ssh_connect_direct
debug1: Connecting to ec2-xx.xxx.xxx.xxx.compute-1.amazonaws.com [xx.xxx.xxx.xxx] port 22.
debug1: Connection established.
debug1: identity file private_key.pem type -1
debug1: identity file private_key.pem-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.9p1 Ubuntu-3
debug1: match: OpenSSH_8.9p1 Ubuntu-3 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to ec2-xx.xxx.xxx.xxx.compute-1.amazonaws.com:22 as 'ubuntu'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c
debug2: host key algorithms: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[email protected],ssh-ed25519,[email protected],rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
debug2: ciphers stoc: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,[email protected],zlib
debug2: compression stoc: none,[email protected],zlib
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,[email protected],diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
debug2: ciphers stoc: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,[email protected]
debug2: compression stoc: none,[email protected]
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:G8FDUPgMnfKFC27I4q5bnyvEEVwHqfDfe26uONkarNQ
The authenticity of host 'ec2-1xx.xxx.xxx.xxx.compute-1.amazonaws.com (xx.xxx.xxx.xxx)' can't be established.
ECDSA key fingerprint is SHA256:G8FDUPgMnfKFC27I4q5bnyvEEVwHqfDfe26uONkarNQ.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'ec2-xx.xxx.xxx.xxx.compute-1.amazonaws.com,xx.xxx.xxx.xxx' (ECDSA) to the list of known hosts.
debug2: set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: private_key.pem  explicit
debug2: pubkey_prepare: done
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,[email protected],ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[email protected],[email protected]>
debug1: kex_input_ext_info: [email protected] (unrecognised)
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Trying private key: private_key.pem
debug2: we sent a publickey packet, wait for reply
debug1: Authentication succeeded (publickey).
Authenticated to ec2-xx.xxx.xxx.xxx.compute-1.amazonaws.com ([xx.xxx.xxx.xxx]:22).
debug1: Local connections to LOCALHOST:3001 forwarded to remote address ec2-xx.xxx.xxx.xxx.compute-1.amazonaws.com:3000
debug1: Local forwarding listening on ::1 port 3001.
debug2: fd 4 setting O_NONBLOCK
debug1: channel 0: new [port listener]
debug1: Local forwarding listening on 127.0.0.1 port 3001.
debug2: fd 5 setting O_NONBLOCK
debug1: channel 1: new [port listener]
debug2: fd 3 setting TCP_NODELAY
debug1: Requesting [email protected]
debug1: forking to background
debug1: Entering interactive session.
debug1: pledge: network
root@user:/home/user1/BitcoinMonitor# debug1: client_input_global_request: rtype [email protected] want_reply 0
debug1: Remote: /home/ubuntu/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
root@user:/home/user1/BitcoinMonitor#