Illegal instruction on Athlon, same code works fine on Xeon.

$ ./bin/hsd
[info] (chain) Chain is loading.
[info] (chain) Checkpoints are enabled.
[info] (chaindb) Opening ChainDB...
Illegal instruction (core dumped)

$ cat /proc/cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 16
model : 6
model name : AMD Athlon(tm) II X2 270 Processor
stepping : 3
microcode : 0x10000c8
cpu MHz : 800.000
cache size : 1024 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 2
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 5
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt nodeid_msr hw_pstate vmmcall npt lbrv svm_lock nrip_save
bugs : tlb_mmatch fxsave_leak sysret_ss_attrs null_seg spectre_v1 spectre_v2
bogomips : 6800.26
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 48 bits physical, 48 bits virtual
power management: ts ttp tm stc 100mhzsteps hwpstate

processor : 1
vendor_id : AuthenticAMD
cpu family : 16
model : 6
model name : AMD Athlon(tm) II X2 270 Processor
stepping : 3
microcode : 0x10000c8
cpu MHz : 800.000
cache size : 1024 KB
physical id : 0
siblings : 2
core id : 1
cpu cores : 2
apicid : 1
initial apicid : 1
fpu : yes
fpu_exception : yes
cpuid level : 5
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt nodeid_msr hw_pstate vmmcall npt lbrv svm_lock nrip_save
bugs : tlb_mmatch fxsave_leak sysret_ss_attrs null_seg spectre_v1 spectre_v2
bogomips : 6800.26
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 48 bits physical, 48 bits virtual
power management: ts ttp tm stc 100mhzsteps hwpstate

Hi all, I tried to bid the TLD “detoo” in the testnet but ended up in limbo: I could not tell if I won or lost the auction. More specifically, I could not update the domain (assuming I won), nor could I redeem the lost bids (assuming I lost).

Here are the actions done:

• Opened an auction at height 4380:

{
  "version": 0,
  "height": 4380,
  "value": 0,
  "address": "ts1qxc98du0yyw2nm7twqu7jxrp8wapu4vdf0f9w0p",
  "covenant": {
    "type": 2,
    "action": "OPEN",
    "items": [
      "ab0c9758d97cde4d5051e6149922f9163771347093144749516eac5317e224d6",
      "00000000",
      "6465746f6f"
    ]
  },
  "coinbase": false,
  "hash": "12686799a04328f84f3436a33ffce9123a8c0c6e216842aa60fbca72340722fe",
  "index": 0
}

• First bid from my wallet #1:

{
  "version": 0,
  "height": 4546,
  "value": 10000000,
  "address": "ts1qux0u2ufxss6r23eu3clsd49aptqal868fr0ue8",
  "covenant": {
    "type": 3,
    "action": "BID",
    "items": [
      "ab0c9758d97cde4d5051e6149922f9163771347093144749516eac5317e224d6",
      "1c110000",
      "6465746f6f",
      "74f4412e9597b133b89f43cae23ca52e6438b67243e48167f669a91ace6203f8"
    ]
  },
  "coinbase": false,
  "hash": "aefdae7e9b3e819348703adabbff06048b52e34f2241f094a9868f8f1ca1f231",
  "index": 0
}

• Second bid from my wallet #2:

{
  "version": 0,
  "height": 4575,
  "value": 8000000,
  "address": "ts1qcwjav2cxs03kmqw3rruwnuhnlkezkyz8gszr7v",
  "covenant": {
    "type": 3,
    "action": "BID",
    "items": [
      "ab0c9758d97cde4d5051e6149922f9163771347093144749516eac5317e224d6",
      "1c110000",
      "6465746f6f",
      "0da9a35a6cce738c89bdcc015d3ca0992f35ad94c9dc1680fb55daaf96b097eb"
    ]
  },
  "coinbase": false,
  "hash": "03c7c646bd68831f89be005aee7753d6b2f0d6772f66d62d24d7e73ff3512b53",
  "index": 0
}

• Revealed the second bid from my wallet #2:

{
  "version": 0,
  "height": 4744,
  "value": 4000000,
  "address": "ts1qcwjav2cxs03kmqw3rruwnuhnlkezkyz8gszr7v",
  "covenant": {
    "type": 4,
    "action": "REVEAL",
    "items": [
      "ab0c9758d97cde4d5051e6149922f9163771347093144749516eac5317e224d6",
      "1c110000",
      "a0917e698f710aa416a6c32d1f8845180f3a6ddb75d19743ee096163862d0588"
    ]
  },
  "coinbase": false,
  "hash": "e27ae33f7539b55543641b4581940c72b327bf86e4072b2c13aecbd344da3cdd",
  "index": 0
}

• Revealed the first bid from my wallet #1:

{
  "version": 0,
  "height": 4746,
  "value": 5000000,
  "address": "ts1qux0u2ufxss6r23eu3clsd49aptqal868fr0ue8",
  "covenant": {
    "type": 4,
    "action": "REVEAL",
    "items": [
      "ab0c9758d97cde4d5051e6149922f9163771347093144749516eac5317e224d6",
      "1c110000",
      "615cd11f85fc01ef9ea99d7d69bd7513a701c6c154c1a25102ea4d4ef7d9a32a"
    ]
  },
  "coinbase": false,
  "hash": "3071ac33bc04b1077d8b965aa0fde8734db8992043bdecbb6d41dac8f2d86be6",
  "index": 0
}

• Wait for reveal stage over

• Failed to register from my wallet #1 (the tx never gone through):

{
  "version": 0,
  "height": -1,
  "value": 0,
  "address": "ts1qux0u2ufxss6r23eu3clsd49aptqal868fr0ue8",
  "covenant": {
    "type": 6,
    "action": "REGISTER",
    "items": [
      "ab0c9758d97cde4d5051e6149922f9163771347093144749516eac5317e224d6",
      "1c110000",
      "000019000906036e7331076578616d706c6503636f6d00010203040000000000000000000000000000000000",
      "b1ef2d9b7ffeecb1a8e539bfd1bee8f21f4c2ce0a24ed27a62ff1bfe319b3cc2"
    ]
  },
  "coinbase": false,
  "hash": "c3394db57a9c97cbc43dbf86577c214afd73e1aeb856c27af78da1d1b75a1efd",
  "index": 0
}

• Failed to register from my wallet #2 (the tx never gone through either):

{
  "version": 0,
  "height": -1,
  "value": 0,
  "address": "ts1qcwjav2cxs03kmqw3rruwnuhnlkezkyz8gszr7v",
  "covenant": {
    "type": 6,
    "action": "REGISTER",
    "items": [
      "ab0c9758d97cde4d5051e6149922f9163771347093144749516eac5317e224d6",
      "1c110000",
      "000019000906036e7331076578616d706c6503636f6d00010203040000000000000000000000000000000000",
      "619108b1ae8c8f6ebcad7cca15826c2d323e5806244ca531e5ab98bdec47718f"
    ]
  },
  "coinbase": false,
  "hash": "e67eba42314a234d4cc4b49016bfd0e89a3f293f23b15eb3364ad2d3e6fdc767",
  "index": 0
}

At this point I assumed maybe I have lost both bids, but I couldn’t redeem any of the wallets, either. I would get an error message “No reveals to redeem”. Did I miss something in the process?

Cloned the repo and tried to build on my Ubuntu 16.04 system.

alan@robby:~/src/hsd⟫ npm install --production
audited 930 packages in 5.098s
found 0 vulnerabilities

alan@robby:~/src/hsd⟫ bin/hsd 
Bindings for bcrypto were not built.
Please build them before continuing.
1 alan@robby:~/src/hsd⟫ node --version
v10.8.0

Description

I was testing recovering funds and names from a seed phrase and observed some strange behavior. Despite being able to manage names that I knew the wallet owned, 1) hsw-cli rpc getnames returned an empty list, and 2) hsw-cli rpc getnameinfo XXX returned a different response than hsd-cli rpc getnameinfo XXX. hsd-cli returned the correct name state (e.g. domain was in the BIDDING state), and hsw-cli returned an incorrect name state (e.g. domain was CLOSED).

Reproducing

Unfortunately I do not have a minimum example. But I will try to explain my steps in sufficient detail.

1. Acquire some funds and names on full node A (explicitly referring to the full node to emphasize starting from a fresh chain).
2. Check to see the names this wallet has interacted with: hsw-cli rpc getnames ==> nonempty.
3. On full node B, hsw-cli mkwallet NAME --mnemonic "seed phrase".
4. Check the wallet's balance hsw-cli --id NAME --account default get ==> 0 as expected
5. Rescan the tx db hsw-cli rescan 0 and wait for it to finish. (NOTE: is this all that's necessary to restore a wallet? If there are other necessary steps, then I did not do them)
6. Now check the wallet's balance hsw-cli --id NAME --account default get ==> the original amount from step 1, as expected.
7. Try to ask full node B for the names its wallet owns: hsw-cli rpc getnames ==> empty, unlike in step 2.

I suspect this is related, but I also observed inconsistencies in running the following commands on full node B:

1. hsw-cli rpc getnameinfo XXX ==> reported XXX to be closed
2. hsd-cli rpc getnameinfo XXX ==> reported XXX to be bidding, the correct answer

Details

In the interest of providing as much helpful information as possible, I should also note that these full nodes contained a modified lookahead parameter (here: ./lib/wallet/account.js:53,836) because the default value of 10 was insufficient for a full recovery. I would not expect this to lead to the issues I observed, but you never know.

I also tried recovering the wallet on totally clean, freshly sync'd full nodes and also full nodes that were already up-to-date and running for a while. Neither behaved as expected for the getnames rpc call, and I do not recall if both nodes returned inconsistent results for the getnameinfo calls or if it was just the freshly sync'd full node that had trouble.

Attempting to set up a full node but coming across errors that mention that it is unable to connect to certain IP addresses.

Here is a log.
https://gist.github.com/luongthomas/f35c761a0ddefcd0cee891a25b385574

I"m running this on MacOS High Sierra Version 10.13.6
npm version 6.2.0

Please let me know if you need additional details.

Erroring out at this installation step.

Command
$ npm install --production

Error

node-gyp rebuild

CXX(target) Release/obj.target/unbound/src/node_unbound.o
../src/node_unbound.cc:1:10: fatal error: 'unbound.h' file not found
#include <unbound.h>
^~~~~~~~~~~
1 error generated.
make: *** [Release/obj.target/unbound/src/node_unbound.o] Error 1
gyp ERR! build error
gyp ERR! stack Error: make failed with exit code: 2
gyp ERR! stack at ChildProcess.onExit (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:258:23)
gyp ERR! stack at emitTwo (events.js:126:13)
gyp ERR! stack at ChildProcess.emit (events.js:214:7)
gyp ERR! stack at Process.ChildProcess._handle.onexit (internal/child_process.js:198:12)
gyp ERR! System Darwin 17.7.0
gyp ERR! command "/usr/local/bin/node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /Users/pk/Documents/code/handshake/hsd/node_modules/unbound
gyp ERR! node -v v8.11.3
gyp ERR! node-gyp -v v3.6.2
gyp ERR! not ok

We originally went with 2.5 minute block times because it seemed like the trendy choice for bitcoin-based currencies, and it's technically been proven to work in practice. Our retargetting is a port of zcash's digishield implementation, so porting their retargetting as well as their block parameters seemed logical.

In retrospect, I think 2.5 minutes is simply unsafe. Even 5 minutes is questionable. If we want to be rock solid we would pick 10 minutes, but due to the nature of our protocol, we need higher transaction throughput than that, and I'm more comfortable with a 5 minute block time than I am with doubling the block size at the moment (though, the block size will no doubt have to be increased in the future).

version 0.0.0

No matter what nblocks is, the response is -1.

curl $url \
  -X POST \
  --data '{
    "method": "estimatefee",
    "params": [ '$nblocks' ]
  }'

curl $url \
  -X POST \
  --data '{
    "method": "estimatesmartfee",
    "params": [ '$nblocks' ]
  }'

After opening an auction but not bidding on it during the bidding period, the auction is now closed, and seems unable to be opened again. Do we have to wait the 27 days until it expires to reopen it?

> hsw-rpc getnameinfo alicepaysbob

{
  "name": "alicepaysbob",
  "nameHash": "3d61e5b16e22c646df9c993a73eba63a9c9c5463feeced11d5915e7d944ea5e8",
  "state": "CLOSED",
  "height": 7192,
  "renewal": 7192,
  "owner": {
    "hash": "0000000000000000000000000000000000000000000000000000000000000000",
    "index": 4294967295
  },
  "value": 0,
  "highest": 0,
  "data": "",
  "transfer": 0,
  "revoked": 0,
  "claimed": false,
  "weak": false,
  "stats": {
    "renewalPeriodStart": 7192,
    "renewalPeriodEnd": 15832,
    "blocksUntilExpire": 7366,
    "daysUntilExpire": 25.58
  }
}

> hsw-rpc getauctioninfo alicepaysbob

{
  "name": "alicepaysbob",
  "nameHash": "3d61e5b16e22c646df9c993a73eba63a9c9c5463feeced11d5915e7d944ea5e8",
  "state": "CLOSED",
  "height": 7192,
  "renewal": 7192,
  "owner": {
    "hash": "0000000000000000000000000000000000000000000000000000000000000000",
    "index": 4294967295
  },
  "value": 0,
  "highest": 0,
  "data": "",
  "transfer": 0,
  "revoked": 0,
  "claimed": false,
  "weak": false,
  "stats": {
    "renewalPeriodStart": 7192,
    "renewalPeriodEnd": 15832,
    "blocksUntilExpire": 7366,
    "daysUntilExpire": 25.58
  },
  "bids": [],
  "reveals": []
}

> hsw-rpc sendopen alicepaysbob
Already sent an open for: alicepaysbob.

> hsw-rpc sendbid alicepaysbob 0.00001 0.00001
Name has not reached the bidding phase yet.

So, we're going to be dropping Cuckoo Cycle in favor of a simple hashcash PoW. SHA3 is the obvious choice, but it's likely that vanilla SHA3 ASICs already exist somewhere. In order to create a level playing field for hardware manufacturers right out of the gate, we need to do some funky stuff.

Both SHA3 and BLAKE2b are customizable. BLAKE2b is probably "more" customizable in the sense that adding a key alters the hash state in such a way that it can't be replicated by concatenating data with a preimage. Keccak's cSHAKE is a different story. Someone with an existing SHA3 ASIC just needs to prepend a few bytes and change the final padding -- the latter part may be possible to do with a chip's microcontrollers.

Here's a very simple idea which should break any existing ASIC:

X = BLAKE2b256(H, K="HNS")
Y = cSHAKE256(H, N="HNS", S="")
Z = SHA3(X || Y)

Where Z is the final PoW hash.

We could also XOR X and Y with the previous block hash or something like that. Whatever we do, it should probably be changed slightly before mainnet. Maybe move the nonce itself into the key and personalization string:

H = header without nonce
P = previous blockhash
X = BLAKE2b256(H, K=nonce)
Y = cSHAKE256(H, N="HNS", S=nonce)
X = X xor P
Y = Y xor P
Z = SHA3(X || Y)

Ideas are welcome. Anything sufficiently convoluted enough to break existing ASICs is useful.

I originally thought I had implemented a merklix tree, but after talking to some people, it seems this is a different data structure entirely. I still think it's sort of like a base-2 trie. I'm not sure if it's a trie or a tree anymore, but whatever it is, it's scalable and it works much better than the ethereum-style patricia trie (faster, less storage, smaller proofs).

https://github.com/bcoin-org/btrie/tree/master/research/merklix

Aside from being inspired by Amaury's merklix tree idea, I think it also has similarities to Todd's "merbinner" tree and Bram's MerkleSet project. I'm still trying to grasp a bit of Bram's optimizations, but I think we maybe have a similar approach?

cc @josephpoon @Roasbeef @hschoenburg

Ideas

1. Urkel Trees
2. MKX Tree (why "MKX"? Not sure... Sort of like a merklix tree but not... also the name is available in npm)
3. FFMT - Flat-File Merkle Tree

We're currently only testing auctions/names with the more low-level Chain object. We should be doing full integration tests.

This will come in handy for filter commitments, I imagine.

This requires a lot more engineering and makes reorgs more of a DoS vector, but it's design is more elegant in the abstract: having a 2-phase window where the first 88 blocks causes all names in the window to be placed in a queue -- 88 blocks later they are committed to the tree. In other words, there's another 88 block "lag" after the first 88 block window.

We probably won't do this due to the complexity involved, but I think it makes for a more robust protocol. Keeping this open for further thoughts.

Running from master, also tried a "working" commit 7f6e6ba hard-coded in bpanel.

$ hsd
module.js:557
    throw err;
    ^

Error: Cannot find module 'goosig'
    at Function.Module._resolveFilename (module.js:555:15)
    at Function.Module._load (module.js:482:25)
    at Module.require (module.js:604:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/Users/matthewzipkin/Desktop/work/hsd/lib/primitives/airdropkey.js:13:13)
    at Module._compile (module.js:660:30)
    at Object.Module._extensions..js (module.js:671:10)
    at Module.load (module.js:573:32)
    at tryModuleLoad (module.js:513:12)
    at Function.Module._load (module.js:505:3)

I even manually installed goosig:
npm install goosig

$ npm list goosig
[email protected] /Users/matthewzipkin/Desktop/work/install/dist/app/libs/hsd
└── [email protected] 

...but strangely this does not fix the error

Now that we have DNSSEC ownership proofs for claiming reserved names, we've run
into a few problems:

• A large number of existing DS records use SHA1
• A large number of existing RSA DNSKEYs use SHA1
• A large number of existing RSA DNSKEYs use 1024 bit moduli

All three of those cases are insecure. Anyone with 1,000+ GPUs can collide SHA1
signatures. And while the biggest RSA modulus ever factored was 768 bits, it's
seriously not long before 1024 bits is the next to go down, and when an
attacker is given the economic incentive of owning every .com domain in the
alexa top 100k, it's even less long.

Long story short, we cannot allow SHA1 or 1024 bit RSA moduli in ownership
proofs, however, this creates a new problem: parent zones currently using
insecure algorithms may refuse to upgrade their keys or sign off on new keys.

The DS problem isn't huge, since every referral in the root zone has SHA256 DS
records already, and ICANN can't justify removing SHA256 now that SHA1 has been
compromised.

For the DNSKEYs, we can combat this by "implicitly" upgrading keys to SHA256
during the verification. Whenever the owner of the name wants to add SHA256
sigs, they can upgrade their keys without getting permission from the parent
zone.

The real problem is the use of 1024 bit RSA moduli. We cannot do anything to
mitigate this without those zones upgrading their keys. For example, com's
zone-signing key is 1024 bits ($ dig com DNSKEY). This is a huge issue since
com houses roughly half of the top 100k alexa names. The best nameholders
can do is beg verisign to upgrade their zone-signing key to 2048, and if that
doesn't work, then they're out of luck.

I'm not going to get into whether these large TLD holders/resellers have
incentives to block such an upgrade, for now let's just assume that they refuse
to upgrade their keys for whatever reason and consider the possible solutions.

Possible solutions:

1. Coin vote - Implement a minimal coin voting system to determine
   ownership of reserved names that have not yet been claimed. To initiate a
   vote, 0.1-1% of the total coin supply must be locked up in a single covenant.
   This covenant includes a root of a merklelized map of name-hash->address as
   a proposal. Regular transactions can then vote by including the proposal's
   TXID in their outputs. The community member who makes the proposal would
   presumably publish the full list somewhere for people to audit before voting.
   At the end of a given period, votes are summed up, ownership is assigned
   and can be redeemed with a merkle proof.

2. Hardcoding Cloudflare's KSK - Cloudflare is smart enough to use
   cryptographically strong keys, and lots of names currently use their DNS
   services. Cloudflare allows two kinds of DNSSEC setups, one via NS referrals,
   and one via CNAME. The CNAME one is interesting because it requires a
   recursive query to resolve a cloudflare.net name (see $ dig www.ietf.org.
   as an example). We can hardcode cloudflare.net's KSK in the consensus rules
   and anyone using cloudflare can still redeem their name. This would involve
   creating a separate proof which allows cloudflare CNAMEs (CNAMEs are not
   currently allowed in ownership proofs).

3. Miner consensus - We allow 1024 bit keys to start, and if a 1024 bit
   modulus is ever cracked, miners can set a version bit enabling a soft-fork to
   require 2048 bit keys -- existing 1024 bit names must be re-redeemed with a
   2048 bit key. This option would require us to disable transfers for any name
   redeemed with a 1024 bit key up until the end of the claim period (3 years).
   This is to prevent a thief from stealing a 1024 bit name and selling it
   before the soft-fork is activated. Worst case, a miner cartel could
   prematurely activate the soft-fork, severely inconveniencing everyone
   (though, the soft-fork enabling could also be soft-forked out later on if
   people decide they don't like it).

I'm partial to #3 personally, since it seems like the cleanest and easiest to
implement. Although #1 is nice since it theoretically allows the community to
solve the problem (assuming no sybil attacks), it does add a lot of complexity.
#2 gives a lot of control and responsibility to cloudflare (though, they
technically already have it assuming .com used a 2048 bit key) -- the real
downside is here that this option only covers a certain portion of the alexa
top 100k.

Thoughts?

cc @josephpoon @boymanjor

I think it would be useful to have a registered coin type for hns to speed
up community adoption. I believe that all that needs to happen is a concensus
around a specific number and a pull request with that information submitted
to the repo below:

https://github.com/satoshilabs/slips/blob/master/slip-0044.md

Expected: npm install works with python3

Actual: npm install throws an error

[kus@dev hskd]$ npm install

> [email protected] install /home/kus/hskd/node_modules/unbound
> node-gyp rebuild

gyp ERR! configure error 
gyp ERR! stack Error: Python executable "/usr/bin/python3" is v3.6.6, which is not supported by gyp.
gyp ERR! stack You can pass the --python switch to point to Python >= v2.5.0 & < 3.0.0.
gyp ERR! stack     at PythonFinder.failPythonVersion (/home/kus/.nvm/versions/node/v10.8.0/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:501:19)
gyp ERR! stack     at PythonFinder.<anonymous> (/home/kus/.nvm/versions/node/v10.8.0/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:483:14)
gyp ERR! stack     at ChildProcess.exithandler (child_process.js:279:7)
gyp ERR! stack     at ChildProcess.emit (events.js:182:13)
gyp ERR! stack     at maybeClose (internal/child_process.js:962:16)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:249:5)
gyp ERR! System Linux 4.17.9-200.fc28.x86_64
gyp ERR! command "/home/kus/.nvm/versions/node/v10.8.0/bin/node" "/home/kus/.nvm/versions/node/v10.8.0/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /home/kus/hskd/node_modules/unbound
gyp ERR! node -v v10.8.0
gyp ERR! node-gyp -v v3.7.0
gyp ERR! not ok 
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] (node_modules/unbound):
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] install: `node-gyp rebuild`
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: Exit status 1

audited 930 packages in 4.602s
found 0 vulnerabilities

Python3 is the CURRENT version of python.

nodejs/node-gyp#746 (comment)

Our upstream dependency gyp is Python 2 only, and as it's now EoL (Google won't be maintaining it) it's unlikely that it will be updated to support Python 3.

nodejs/node-gyp#1092

Do we need gyp? I'll go one step further, do we need node? Can we use something else please?

In the current version of hsd; bcuckoo is defaulted to 0.6.0, but 0.5.0 is the latest on npm. Just a heads up

  curl localhost:13037 \
  -X POST \
  --data '[{
    "method": "getrawtransaction",
    "params": ["5f5f9167fb5dbef837976eb80d7c598ace3e792b976813ec22f90f71cc1cf922", true]
  }]'

response with

{
  "error": {
    "message": "Not found."
  }
}

Maybe there should be something like this in /lib/node/http.js initRouter function:

    this.use(async (req, res) => {
      if (req.method === 'POST' && req.path.length === 0) {
        if (Array.isArray(req.body)) {
          // Do something
        };
      };
    });

The additional airdropkey file found in the primitives library includes the goosig library. The module is missing in the current package.json

This includes auction events, etc. Should be useful if someone wants to build a web registrar on top of HSKD.

Currently, only the JSON-rpc interface has been modified to accommodate names. We can do better.

Bitcoin's merkle tree hashes odd nodes with themselves (or nodes without a right sibling, I should say), leading to the duplicate txid vulnerability (a duplicate txid placed after an odd leaf ends up creating the same root as the valid block's commitment).

Switching to the type of merkle tree described in RFC6962 would be preferrable (it simply propagates odd nodes up a level without hashing them). It also has some other benefits: i.e. the ability to prove a leaf vs. an internal node.

We could also require the leaves to be sorted for non-existence proofs.

edit: RFC6962: https://tools.ietf.org/html/rfc6962#section-2.1

Based on the comment in lib/mempool/fees.js:

 * Ported from:
 * https://github.com/bitcoin/bitcoin/blob/master/src/policy/fees.cpp`

and these 2 files being very similar from a quick glance:

• https://github.com/handshake-org/hsd/blob/master/lib/mempool/fees.js
• https://github.com/bcoin-org/bcoin/blob/master/lib/mempool/fees.js

it appears as if handshake is using the same transaction fee structure as bitcoin.

Has any thought and consideration gone into improving the transaction fee system to help ease volatility in the fee market?

Vitalik has come out with some interesting work on transaction fee markets recently, see: zcash/zcash#3473

Curious to start a discussion around any pros/cons around different types of fee markets to see if some of Vitalik's ideas would be useful in the handshake protocol

If I call "getnameinfo" on a domain that was won with only 1 bid, value is always returning as 0 - was just checking to see if this is expected behavior.

If so, I think it could potentially be helpful to return value === highest in the case of single bid domains.

Using SHA3-256. This seems like the safest option for DoS protection with regards to the name tree. SHA3 is slow and as far as I know, there are no ASICs for it.

Are claimed names on testnet going to be ported over to mainnet when it launches? Essentially, if I claim a name in an auction bid, win it, and thus have the rights to it on testnet, when mainnet launches, will that still hold true? Just going through all the docs now, so if this is answered elsewhere please disregard and point me that way! Cheers

diff --git a/bin/hsd b/bin/hsd
index 5f478501..68e56c79 100755
--- a/bin/hsd
+++ b/bin/hsd
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash

rl=0
daemon=0
diff --git a/bin/hsw b/bin/hsw
index 6a56b2f2..8606de8c 100755
--- a/bin/hsw
+++ b/bin/hsw
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash

rl=0
daemon=0

It's time to start redesigning the claim system to be completely
permissionless. The CLAIM covenant should take a DNSSEC proof, stemming from
the ICANN KSK (2010 for now, including support for 2017 when the DNSKEY RRSIG
finally rolls over), all the way down to a final TXT record which commits to an
address. The output's address must be equal to the signed TXT record. This
allows any reserved name nameholder to claim their name in a decentralized way
by adding a TXT record to their name's zone.

The tricky part comes when we have to maintain the root zone. We could devise
a system where, prior to the claim of root zone name, anyone can publish all
records for the referral of any name in the root zone file. This would keep the
existing root zone up to date until the nameholder claims their name. This is
tricky, as it would require the chain tracking signature timestamps, ensuring
that no one could revert the data to the older state.

The problem: glue records in the existing root zone are unsigned. There's
currently no way to prove that a server actually serves a specific zone.

So, if we can't do that, we have a plan B: if the resolver receives an absence
proof for a name in the existing root zone (say, com is currently unclaimed),
the resolver does a validated resolution to get the referral for the name,
removes ICANN's signatures and replaces them with our own. This will allow the
root zone to transparently roll over as more TLDs claim their names on
handshake.

Claim Design

I've laid out some simple design ideas for the claiming process...

Required data for claiming google.com (example)

Zone 0 ()

• KSK-2017 DS (implicit)

Zone 1 (.)

• KSK-2017 DNSKEY (implicit)
• Root ZSK DNSKEY (optionally implicit)
• RRSIGS from . for . DNSKEYs (optionally implicit)
• All DS records for com.
• RRSIG from . for com. DS records.

Zone 2 (com.)

• KSK DNSKEY
• ZSK DNSKEY
• DNSKEY RRSIGs
• All DS records for google.com.
• RRSIG from com. for google.com. DS records.

Zone 3 (google.com.)

• KSK DNSKEY
• ZSK DNSKEY
• DNSKEY RRSIGs
• All TXT records from google.com
  • Final TXT record should have a commitment to
    a handshake address, in the form of:
    hns-claim: SP [version (decimal)] SP [hash (hex)]
    • Example: hns-claim: 0 20e8e23af6e597b7e69f8a28c8bef1509b9bf098
    • Anything not in this exact format is invalid.
• RRSIG for TXT records

Average Rdata sizes with RSA and SHA256

• SHA256 DS record = 36 bytes
• RSASHA256 RRSIG = 275 bytes
• RSASHA256 DNSKEY = 264 bytes
• RSASHA256 DNSKEY (4096) = 520 bytes
• Claim TXT record = 54 bytes

Size with implicit records (best case):

• 2 DS records (36 * 2)
• 4 DNSKEYs (264 * 4)
• 5 RRSIGs (275 * 5)
• 1 TXT Record (54 * 1)
• Total: 2557 bytes

Size without implicit records (best case):

• 2 DS records (36 * 2)
• 5 DNSKEYs (264 * 5)
• 6 RRSIGs (275 * 6)
• 1 TXT Record (54 * 1)
• Total: 3096 bytes

Size Notes

In reality, due to massive RRSets of DNSKEYs, this will probably range from 3kb
- 10kb. The first two RRSets for com alone are 2157 bytes! Due to the nature
of DNSSEC, the entire RRSet must be present.

$ dig.js @a.root-servers.net . DNSKEY +dnssec +nord | grep rcvd
;; MSG SIZE  rcvd: 1414
$ dig.js @a.gtld-servers.net com DNSKEY +dnssec +nord | grep rcvd
;; MSG SIZE  rcvd: 743

Finally, the final TXT record proof may be large if the nameholder doesn't have
direct access to their ZSKs and also currently has a number of TXT records
already setup. =/

Algo Notes

Support only the mainstream algos. Having algo bloat in the protocol is a
disaster and creates more attack vectors. I'm sure you'd be hard-pressed to
find anything currently in the wild which uses edwards curves or GOST.

• Public key algo support for: RSA, P256, P384
  • No consensus support for: DSA, GOSTECC, ed25519, or goldilocks (ed448)
• Hash algo support for: SHA1, SHA256, SHA384, SHA512
  • No consensus support for: GOST94

Bcrypto supports RSA, P256, and P384 through openssl, and ed25519 through
elliptic. Node.js only recently began building with a more modern openssl
which exposes the new RSA/EC api. For safety, we may need to require the full
node to only successfully boot on node.js >=10.0.0 (for the people who really
want to run on older versions, they can modify the boot code, but we shouldn't
support it as a feature). The bcrypto EC and RSA code should also be audited
for buffer overflows and other bugs.

If anyone finds a single name on the reserved list that uses DSA, ed25519, or
ed448 and only that algo, we should support them after all. In my own
experiences wandering/digging around different zones, I have yet to see a
single DNSSEC setup which uses these algorithms.

Proof Notes

Signatures

DNSSEC signature timestamps will have to be validated with MTP. They also
expire, so that has some concerns for the mempool. Realistically, we could
probably get by without validating the timestamps at all. It's unlikely someone
would have an hns-claim record for their name before HNS is even launched or
announced.

TXT records

Require TXT records for the final commitment. Have a TXT record commit to an
address. Using TXT records is necessary for one major reason: a lot of the name
owners on the top 100k may not have direct access to the ZSKs. Most every
registrar's interface allow users to add arbitrary TXT records. This allows
them to create a valid proof without having direct access to the private key
(or presumably will allow them once their tld/registrar supports dnssec).

Simplicity

The DNSSEC proofs will be required to be extremely simple. It must be a
simple DS->DNSKEY chain for each zone cut. Nothing fancy, no CNAME glue as that
would allow people to steal names, and no wildcards for simplicity.

Proof Compression

DNSSEC proofs can be compressed. For a lot of RRSets, we don't necessarily care
about a good portion of the data. Say we only one one record from the entire
RRSet -- depending on how the records are sorted, any record before our
desired record could be excluded from the proof. In its place would be the
serialized state of the SHA1/SHA256 hash up until that point. I figure this
could reduce proof size by 50% or so.

hsd version: 0.0.0

Reproduce steps:

1. Get block by hash

curl MY_NODE \
  -X POST \
  --data '{
    "method": "getblock",
    "params": ["2ef4796311bee5ed806a8f5fc39ce504df77a35148b765ff313f50fd61f668de", true, false]
  }'

And we got:

{
  "result": {
    "hash": "2ef4796311bee5ed806a8f5fc39ce504df77a35148b765ff313f50fd61f668de",
    "confirmations": 4,
    "strippedsize": 416,
    "size": 448,
    "weight": 1696,
    "height": 2729,
    "version": 0,
    "versionHex": "00000000",
    "merkleroot": "c29eac8d1f659f88b24b6d7f785035f174e2c5610832e49e90f57803ceb7d8bb",
    "treeroot": "29faeb37169ac1ca027b196aa757c2da334ba3f0d4a7d060f015ee2f89bd98df",
    "reservedroot": "0000000000000000000000000000000000000000000000000000000000000000",
    "solution": [
      222041,
      576372,
      685364,
      1215339,
      1233328,
      1637621,
      1717694,
      1778494,
      1861115,
      1976774,
      2095832,
      2210835,
      2359596,
      2957521,
      3018868,
      3227690,
      3358198,
      3418168,
      3587186,
      3865488,
      3910604,
      4004144,
      4021577,
      4199737,
      4562581,
      4789507,
      4962891,
      5063286,
      5268351,
      5388752,
      5412080,
      5980137,
      6415042,
      6433540,
      6474751,
      6564636,
      6585625,
      7236251,
      7252854,
      7359484,
      7897167,
      8188845
    ],
    "coinbase": [
      "6d696e656420627920687364",
      "a4abac83daa0a704",
      "0000000000000000"
    ],
    "tx": [
      "ffe734415e0214d05da27cfaf1f831bc1839a541ba1a822467fe28c7cc0175ce"
    ],
    "time": 1533712185,
    "mediantime": 1533710933,
    "bits": 527229860,
    "difficulty": 1.4012917040736978e-07,
    "chainwork": "000000000000000000000000000000000000000000000000000000000014c281",
    "previousblockhash": "8ac93f11e7a298fc52deb3581fc75236e72e3f7120fc5b942e73432976581191",
    "nextblockhash": "303baab91317e7b769051a835fb7d1ac0936b7e4320316d28ca33f3db5102347"
  },
  "error": null,
  "id": null
}

2. Get raw transaction of the transaction in the block we got in step 1

curl MY_NODE \
  -X POST \
  --data '{
    "method": "getrawtransaction",
    "params": ["ffe734415e0214d05da27cfaf1f831bc1839a541ba1a822467fe28c7cc0175ce", true]
  }'

We got:

{
  "result": null,
  "error": {
    "message": "Transaction not found.",
    "code": -1
  },
  "id": null
}

Neutrino support i.e BIP158 would be useful for light clients. Would there be an interest to add them to hskd?

If so we can commit the filters to the block, which would simplify filter verification for the clients.

This is necessary for our phony DNSSEC. I have a prototype here: https://github.com/bcoin-org/bcrypto/blob/master/src/ecdsa/ecdsa.c

Completely untested, but should have it working by mainnet launch.

This is good, as the ultimate goal for bcrypto is to not ever have to require crypto, bn.js, or elliptic when using the native bindings -- basically, implement the node.js crypto API the way it should have been.

Getting the following syntax error when running ./bin/hsd:

...
[info] (http) Node HTTP server listening on 127.0.0.1 (port=13037).
[info] (ns) Root nameserver listening on port 15349.
Error: syntax error
    at Unbound.setOption (/home/ubuntu/hsd/node_modules/unbound/lib/unbound.js:31:18)
    at UnboundResolver.open (/home/ubuntu/hsd/node_modules/bns/lib/resolver/ub.js:203:13)
    at RecursiveServer.open (/home/ubuntu/hsd/lib/dns/server.js:564:20)
    at FullNode.open (/home/ubuntu/hsd/lib/node/fullnode.js:253:19)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7)

Seems to be related to libunbound, although I have libunbound-dev installed.

System Information:

$ node -v
v10.8.0
$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.4 LTS
Release:	16.04
Codename:	xenial

I think the following specs are probably necessary:

• Specification for our Root Zone DNS-message-like serialization
• Specification for covenant behavior (although, this is touched on a bit in the whitepaper)
• Specification for a general overview, with bitcoin assumed to be the base protocol

For SPV wallets wishing to manage names, there needs to be some way for a full node to prove the height at which bidding opened for an auction (this is necessary to calculate the auction's current state!).

I see two ways to do this:

1. Another merklix-style tree header commitment which stores all open auction heights (merklix-style -- still in the process of naming our tree structure).
2. Much simpler: a regular merkle tree commitment in the header of all name hashes that opened for bidding in that particular block.

The first one is pretty much out of the question, as it creates a massive DoS vector and means we would need to add some consensus limits to the number of bids per block.

The second one is do-able, through some trickery. I actually think the regular merkle commitment could be in the coinbase to save precious header space.

Logic:

1. As covenants are being verified, collect a list of name hashes of auctions that just opened for bidding. Name hashes are indexed and stored for fast retrieval (the creation of this list can hopefully be optimized for block assembly). These can also be pruned later.
2. Name hashes are hashed up into a root (using an RFC6962 merkle tree hopefully).
3. The root is placed in the coinbase (or header) as a consensus rule.

Proof Logic:

1. Requesting peer requests proof of bid height with message GETBIDHEIGHT containing a single namehash.
2. Responding peer looks up the current auction state, grabs the opening bid height, gets the current mainchain block header for that height.
3. Using this block header, they construct a merkle proof to the coinbase TXID (a simple path of the siblings down the tree -- no need for the crazy bip37 logic). A second merkle proof is constructed from the name root. The coinbase TX itself must be included in the final proof.
4. Responding peer responds with BIDHEIGHT consisting of [height, coinbase-merkle-proof, coinbase, name-merkle-proof].
5. Requesting peer verifies the proof against their main-chain header at that particular height, but now the tricky part: they need to check that the height is within a reasonable distance from the current tip. If the height is a year old, the node is obviously lying to them and serving old data.

Having it in the coinbase saves on header space, but makes the proofs much bigger. On second thought, maybe just put the merkle root in the header?

I get the following error when running ./bin/hsd --ns-port 5300 after starting hsd

[info] (chain) Chain is loading.
[info] (chain) Checkpoints are enabled.
[info] (chaindb) Opening ChainDB...
Error: IO error: lock /Users/pranay/.hsd/testnet/chain/LOCK: Resource temporarily unavailable

See: bcoin-org/bcoin#533

A lot of PRs have been merged into bcoin recently. Some are cleanups/fixes that we should have in the HSKD codebase as well. We'll have to look at some commit histories.

Likewise, I think I added a number of helper functions to HSKD that could also be useful in bcoin.

Adding this to the todo list.

It would be great to have ledger support for handshake, I see that
https://github.com/LedgerHQ/blue-app-btc supports many utxo based
coins, maybe we can compile a list of TODOs to make this happen

• Agree on bip44 coin type (see #30)
• Create handshake gif for device display
• Submit PR with working code to github.com/LedgerHQ/blue-app-btc

Anything else?

dgram.js:82
  throw new errors.TypeError('ERR_SOCKET_BAD_TYPE');
  ^

TypeError [ERR_SOCKET_BAD_TYPE]: Bad socket type specified. Valid types are: udp4, udp6
    at newHandle (dgram.js:82:9)
    at new Socket (dgram.js:118:16)
    at Object.createSocket (dgram.js:138:10)
    at Object.createSocket (/home/box/handshake/hskd/node_modules/budp/lib/udp.js:154:24)
    at new Base (/home/box/handshake/hskd/node_modules/bns/lib/net.js:34:23)
    at new Server (/home/box/handshake/hskd/node_modules/bns/lib/net.js:243:5)
    at new DNSServer (/home/box/handshake/hskd/node_modules/bns/lib/server.js:40:19)
    at new RootServer (/home/box/handshake/hskd/lib/dns/server.js:41:5)
    at new FullNode (/home/box/handshake/hskd/lib/node/fullnode.js:145:15)
    at Object.<anonymous> (/home/box/handshake/hskd/bin/node:24:14)

Fixed by checking out HEAD~1 (5e72972)

Description

Handshake uses a Vickrey auction, so the winner should only pay the second highest bid. With Handshake, bidders lose access to their funds for the duration of the auction. Only when the auction closes can losers sendredeem to recover their bid amount. Winners can sendupdate to register their name, which creates a REGISTER output and a NONE output containing their (1st bid - 2nd bid) refund. However, the wallet client erroneously reports both of those outputs as being locked up, effectively making the winner pay their full 1st highest bid.

To Reproduce

1. Make 2 wallets, A and B each with 100 HNS.
2. rpc selectwallet A
3. open NAME
4. bid 50, 75 lockup
5. rpc selectwallet B
6. bid 25, 50 lockup
7. rpc selectwallet A
8. reveal 50
   a. (75-50) HNS returned to wallet A
9. rpc selectwallet B
10. reveal 25
    a. (50-25) HNS returned to wallet B
11. redeem NAME
    a. 25 HNS returned to wallet B
    b. B still has ~100 HNS with 0 locked up, as expected
12. rpc selectwallet A
13. update NAME '{}'
    a. REGISTER output with 25 tokens (this is the 2nd highest bid price)
    b. NONE output with 25 tokens (this is the 1st - 2nd price)
14. get --id=A
    a. A has 100 HNS with 50 HNS still locked up
    b. expected: A has 100 HNS with 25 HNS locked up

Details

I ran all of these steps locally on simnet, but have observed this same behavior on testnet. Here is the result of the winner's sendupdate:

"outputs": [
    {
      "value": 25000000,
      "address": "ss1qe6xyedmfg2aujv32xnf4fwlqptvmhyqtfxlq3k",
      "covenant": {
        "type": 6,
        "action": "REGISTER",
        "items": [
          "a0a65c5733a6fe998b2163843cb38e750b79fdb01319d87b61e49d3990fc5740",
          "d2090000",
          "00000000",
          "1d24ff48bcc846484f66b6ac9b85f22f30f82684db4573edd8ee4e74956f82c6"
        ]
      }
    },
    {
      "value": 24995680,
      "address": "ss1qgnvhypauavy9hc2ksk9zev0fjrxyyc7nckmy94",
      "covenant": {
        "type": 0,
        "action": "NONE",
        "items": []
      }
    }
  ]

Here is the balance result of get --id=A:

"balance": {
  "account": -1,
  "tx": 4,
  "coin": 4,
  "unconfirmed": 99987140,
  "confirmed": 99987140,
  "lockedUnconfirmed": 50000000,
  "lockedConfirmed": 50000000
}

Next, I exported A's wallet seed and created a new wallet based on that seed. Then, I rescan the db to bring the new wallet up-to-date. Here is its balance report:

"balance": {
  "account": -1,
  "tx": 4,
  "coin": 4,
  "unconfirmed": 99987140,
  "confirmed": 99987140,
  "lockedUnconfirmed": 0,
  "lockedConfirmed": 50000000
}

In the event that there is some unforseen optimization in cuckoo cycle, or the algorithm ends up being broken somehow, we will end up like bitcoin -- a hash-based proof-of-work. One thing is certain: if that happens, we don't want to wind up being a blake2 currency as ASICs for blake2 currently exist, and the majority of them are controlled by a single ASIC company.

Using SHA3 for the final solution hash seems like a good failsafe.

cc @josephpoon

$ docker run -it --rm quay.io/kargakis/hsd --help
See the hsd wiki at:
https://github.com/handshake-org/hsd/wiki.

Visiting that link redirects me back to https://github.com/handshake-org/hsd.

Could not get certain configurations down to the wallet plugin. Took me a while but realized it was because wallet/plugin.js looks for hsw.conf not wallet.conf like in bcoin.