stacks-network / stacks-core Goto Github PK

Hello, we are specifying a protocol for a canonical content registry. You can read a detailed proposal here: https://github.com/mine-code/canonical-content-registry. We are interested in building it on top of Blockstore to do the following:

1. Store unique CCID (canonical content identifier) in Blockstore
2. Append file instance metadata to CCID (see examples in canonical-content-registry)
3. Tag a CCID with a nametag
4. Sign a statement stating a nametag belongs to an Openname profile
5. Append arbitrary metadata to CCID (hashtags, related CCIDs, etc)
6. Ability to easily interpret the above by a new node in the DHT

I wanted to open this issue to start a discussion about extending Blockstore to support writing metadata in the format above. What are the systems current limitations and what could be possible steps to extend blockstore?

Reading the README should provide some context. Let me know if I can clarify any of the ideas, and feel free to open CCR related issues on that repo.

@muneeb-ali @shea256 @jessewalden @moudy

For DNSChain to resolve domains in blockstore, we need to know the TLDs that are supported.

You can't say "any domains" because you cannot use existing ones (you will break the internet if you try that).

So what TLDs should DNSChain query blockstore for?

Options:

1. preorder name, reserve name, preregister name
2. claim name, register name, reveal name
3. update name, update name value, tag name, stamp name, label name
4. transfer name

The number of names in existence should be manageable for a while. And as the number of names gets too large for this call to manage, this could eventually be paginated, with the option to skip pages.

Problem:

With the current design, names are purchased by paying tribute with mining fees. The problem here, though, is that miners can effectively register domain names for free because they're simply paying themselves the mining fees. This could cause the incentives in the system to break down.

Here, I'm proposing an alternative way of executing and interpreting name operations...
with the purpose of reducing data requirements and network costs.

(note: just a proposal - not sure if this is the way to go, since it definitely has drawbacks)

Right now, we embed the operation instructions directly in the OP_RETURN data. That is, the data contains a code for the operation, and then the parameters for that operation. For example, if it's a name update, we specify it's an update, then we include the name that's being updated, and then we include the hash of the data to be associated with the name.

Advantages:

• all you need to read and interpret the sequence of operations is the sequence of relevant OP_RETURN nulldata embedded directly in the blockchain
• to get a basic view of the namespace, you don't need to pull data from outside sources, like DHTs or APIs

Disadvantages:

• data embedded in the blockchain scales with the number of operations
• if the data associated with a name needs to be updated frequently, the network costs can get super high

New method:

Instead of including the operation instructions directly in the OP_RETURN nulldata...

You format the operation, then sign it with the private key of the owner, then hash it and include that hash in the blockchain, making sure the operation itself is available in a DHT.

If you want to include 10 operations for 10 different names owned by 10 different private keys, you can create the 10 signed operations, bundle them together, then hash and include the bundle in the blockchain, and serve the bundle from a DHT

Advantages:

• cheaper
• lower data requirements
• lets you do thousands of updates with a single bitcoin transaction

Disadvantages:

• a single bit of data being unavailable means that the whole view can't be built
• requires more work for nodes - they have to do more DHT lookups
• less elegant
• probably something else I haven't thought of

blockstore-cli preorder swiftonsecurity 598cabac1a4aaff109bb2374cdc66e415a8993dfb37f681df64809072f3cf8be
{
"error": "Exception: Not enough inputs for transaction.",
"traceback": [
"Traceback (most recent call last):",
" File "/usr/local/lib/python2.7/dist-packages/blockstore/blockstored.py", line 253, in jsonrpc_preorder",
" blockchain_client=blockchain_client, testset=True)",
" File "/usr/local/lib/python2.7/dist-packages/blockstore/lib/operations/preorder.py", line 41, in broadcast",
" nulldata = build(",
" File "/usr/local/lib/python2.7/dist-packages/coinkit/transactions/network.py", line 136, in embed_data_in_blockchain",
" fee=fee, change_address=change_address, format=format)",
" File "/usr/local/lib/python2.7/dist-packages/coinkit/transactions/network.py", line 107, in make_op_return_tx",
" fee=fee, format=format)",
" File "/usr/local/lib/python2.7/dist-packages/coinkit/transactions/outputs.py", line 45, in make_op_return_outputs",
" "value": calculate_change_amount(inputs, send_amount, fee)",
" File "/usr/local/lib/python2.7/dist-packages/coinkit/transactions/outputs.py", line 20, in calculate_change_amount",
" raise Exception('Not enough inputs for transaction.')",
"Exception: Not enough inputs for transaction."
]
}

Thinking through the ways that we can ship a working, simple first release (our "pre-pre-ultra-test-alpha-RFC"). This should involve making things as simple as possible.

Some points of discussion:

1. Should salt be removed from preorders and registrations? See #37
2. Is the consensus hash absolutely necessary? Should it be kept in the first release? (It significantly complicates things). Perhaps we should get some outside guidance on the necessary properties for us to maintain a stable consensus.
3. Should the namespace-specific pricing policy be incorporated in? Or should it be simple in the first release, with the opportunity to open a discussion before its implemented? See #20
4. Should we ship the first release with a non-miner-attackable name price payment scheme? If so, then we could make a super simple fix by switching from mining fee to burn. The other option is go take the time to implement one of the miner-attack-resistant methods we discussed, but that'll be a larger project. See #25
5. How many nodes and DHTs should we have up online?

I run blockstored start --foreground , after blockstored go to block 499349 ,It stop and exit.

Log info:
block_nameops []
Processing block 499347
block_nameops []
Processing block 499348
block_nameops []
Processing block 499349

Exiting blockstored server

On branch Master I'm getting the error:

from coinkit import bin_double_sha256, hex_to_bin_reversed, bin_to_hex_reversed
ImportError: cannot import name bin_double_sha256

when running opennamed

I have coinkit==0.7.5

@shea256 have you seen this error before? Also, we might need to switch to pybitcoin since that is the open-source repo now

the current code (in opennamed.py) assumes OS X / UNIX style backslash in directory path names, this will not work for Windows and needs a fix

Currently, there are two types of incentives for running DHT nodes:

a) Someone running blockstored, is automatically a full DHT node as well (but I can imagine lightweight implementations where people try to turn that functionality off)

b) Companies (such as Onename) providing public DHT servers

It'll be great to brainstorm better incentives for running DHT nodes. Also, to combat SPAM there could be mechanisms where the "storage policy" at DHT nodes requires proof of "something", where that something could be a blockstore transaction.

Blockstore team,

New issue today. Created my own bitcoin node (107.191.41.5:8332) to get around the load balancing issue on btcd.onename.com. Confirmed that the node had an up to date blockchain. Installed blockstore with pip and then started up blockstored. Used a local host address (127.0.0.1) and completed the rest of blockstored's config questions.

Every time I run this blockstored processes blocks 343883 through 343887 and then exits:

When I register a key-value pair, first preorder the name, I get a error information.

blockstore-cli preorder swiftonsecurity N876Ywf8bL5JWMPbHvjHP2AaSF1Ap8Wg4qAn9AAiQbmYFWPh2vqe
{
"error": "UnboundLocalError: local variable 'secret_exponent' referenced before assignment",
"traceback": [
"Traceback (most recent call last):",
" File "/usr/local/lib/python2.7/dist-packages/blockstore/blockstored.py", line 253, in jsonrpc_preorder",
" blockchain_client=blockchain_client, testset=True)",
" File "/usr/local/lib/python2.7/dist-packages/blockstore/lib/operations/preorder.py", line 28, in broadcast",
" hash160 = BitcoinPrivateKey(private_key).public_key().hash160()",
" File "/usr/local/lib/python2.7/dist-packages/coinkit/privatekey.py", line 61, in init",
" if not is_secret_exponent(secret_exponent, self._curve.order):",
"UnboundLocalError: local variable 'secret_exponent' referenced before assignment"
]
}

The current algorithm for calculating the price of a name is as follows:

def calculate_name_price(name):
    # establish the base price
    price = PRICE_FOR_1LETTER_NAMES
    # adjust the price by a factor X for every character beyond the first
    price /= PRICE_DROP_PER_LETTER**(len(name)-1)
    # apply other rules
    if has_numerics(name) or has_underscores_or_dashes(name):
        # for names with numerics or special chars, reduce the price further
        price /= PRICE_DROP_FOR_NON_ALPHABETIC
    else:
        # for alphabetic names, enforce a price floor
        if price < ALPHABETIC_PRICE_FLOOR:
            price = ALPHABETIC_PRICE_FLOOR
    # return the calculated price
    return price

How could this be improved? What other options do we have?

Currently, you can register a key-value pair on blockstore and perform lookups on the pair by entering the key and getting back the value.

This works great for certain applications like registering user identities, but it runs into problems when other types of objects need to be registered in the blockchain.

For example, in order to validate/timestamp/prove the existence of a billion documents in the blockchain, this would require a lot of transactions, a lot of data embedding, and a lot of money, to such a ridiculous extent that this process would not be feasible.

A way to get around this would be to pack the registration of multiple objects into a single transaction. A single transaction/key-value pair registration could have a single unique key associated with the merkle root hash of a merkle tree, where each item in the merkle tree is the hash of an object to be registered.

So to register 128 image files on the blockchain, you could do the following:

1. hash all 128 image files
2. build a merkle tree from the hashes
3. register a unique name
4. include in the value/JSON file a list of file object descriptors, where each descriptor has a locally unique subdomain and the hash of the file
5. include in the value a root URL where the files can be served

To perform lookups, you would need to provide the domain and the subdomain of the packed file.

NOTE: the files can't realistically be stored in the DHT as that would lead to an insane amount of DHT bloat (imagine storing 1B 100KB avg. images). We're thinking of controlling bloat by imposing a restriction of a bitcoin transaction for each stored item (with a max size), and packing multiple files into a single transaction wouldn't be compatible with this system.

Here's an example of how this would work:

Let's say I have an album of 128 images from my birthday party and I want to register them in the blockchain.

And let's say their filenames are as follows:

2015-05-19-birthday-1
2015-05-19-birthday-2
2015-05-19-birthday-3
...

To do this, I first register "ryan-may2015" in a namespace for image storage.

Next, I create a JSON file and include that as the value associated with the name (where the JSON file goes into the DHT and the hash of the file is put in the blockchain):

{
    "fileservers": ["https://cdn.shea.io/albums/may2015/"],
    "files": {
        "birthday-1": "ca978112ca1bbdcafac231b39a23dc4da786eff8147c4e72b9807785afee48bb",
        "birthday-2": "3e23e8160039594a33894f6564e1b1348bbd7a0088d42c4acb73eeaed59c009d",
        "birthday-3": "2e7d2c03a9507ae265ecf5b5356885a53393a2029d241394997265a1a25aefc6",
        ...
    }
}

Last, I make sure to store my files in my personal content-addressed fileserver.

That means my file 2015-05-19-birthday-1 would be stored here:

https://cdn.shea.io/albums/may2015/ca978112ca1bbdcafac231b39a23dc4da786eff8147c4e72b9807785afee48bb

Then, I can perform lookups as follows:

blockstore-cli lookup ryan-may2015.birthday-1

...and I would get back the raw file.

This new blockstore lookup would do the following:

1. lookup the name "ryan-may2015" in the index and get the associated hash
2. lookup the JSON file by the hash in the DHT
3. find the entry in the JSON file with the key "birthday-1" and grab the hash
4. issue a request to the fileserver and inspect the response to make sure the file returned matches the hash
5. return the response to the client

Yay, I just registered an entire photo album in the blockchain!

I think we could realistically store 100-1K files in each transaction, and maybe 10K if we pushed the data limits of the DHT.

Thoughts @denisnazarov @muneeb-ali @jessewalden @moudy?

Two options:

A. make "name renewal" a separate operation from name registration

Pros: it's more explicit, and it encapsulates the functionality better. Also, if we add "admin" functionality, we can allow admins to renew names.
Cons: one extra call to worry about.

B. interpret a name renewal as a name registration on an already registered name (given that the owners match up and the operation is considered valid)

Pros: the end user has to worry about less calls.
Cons: if we add support for name "admins", it's unclear how we would allow admins to renew names on behalf of users.

Thanks @bmuller for the suggestion. This can get rid of the zerorpc requirement as well (twisted is already a requirement). zerorpc also requires an additional installation step on Ubuntu/Debian. This can lead to less code as well. One downside I can think of is that event-driven code is slightly harder to follow.

DHT storage is currently not durable (i.e., key-value pairs are not stored on disk and a DHT node will lose all data on a reboot). This might be fine when the DHT network becomes large, but for bootstrapping the network we want to ensure that we don't lose data in the DHT e.g., if AWS reboots the VMs that Onename is running.

On a related note, we need more DHT nodes by different organizations running on different data centers.

For the durable storage, I'm thinking of just hooking in MongoDB or SQLite. Benefit of MongoDB is that our data is JSON anyway. Benefit of SQLite is that users won't have to install any DB.

hey guys,

when i run ** blockstore-cli preorder XXX privkey ** , terminal shows :

{
"error": "UnboundLocalError: local variable 'secret_exponent' referenced before assignment",
"traceback": [
"Traceback (most recent call last):",
" File "/usr/local/lib/python2.7/dist-packages/blockstore/blockstored.py", line 253, in jsonrpc_preorder",
" blockchain_client=blockchain_client, testset=True)",
" File "/usr/local/lib/python2.7/dist-packages/blockstore/lib/operations/preorder.py", line 28, in broadcast",
" hash160 = BitcoinPrivateKey(private_key).public_key().hash160()",
" File "/usr/local/lib/python2.7/dist-packages/coinkit/privatekey.py", line 61, in init",
" if not is_secret_exponent(secret_exponent, self._curve.order):",
"UnboundLocalError: local variable 'secret_exponent' referenced before assignment"
]
}

what's wrong ?
thanks ;)

Currently the preorder hash is calculated as follows: hash(name + salt + scriptPubKey)

Matt Corallo points out that hash(name + scriptPubKey) is equivalent if the scriptPubKey has never been used before.

This would simplify things AND would save us a bunch of space in the name registration transaction (about 16 bytes, I believe).

The only thing we'd have to do is make it clear that people shouldn't reuse pubkeys (they shouldn't be doing so anyway) unless they want to risk that someone will try to dictionary attack them.

Running blockstored on ubuntu 14.04. Using onename's default bitcoind server.

Blockstored creates its initial index and then proceeds to process blocks. Each block processing step takes 30seconds to 2minutes.

Here is the console output I receive after running for 10 minutes:
jonathan@mark1:~$ blockstored start
Starting blockstored server ...
Do you have your own bitcoind server? (yes/no): no
Using default bitcoind server at btcd.onename.com
Creating initial index ...
Processing block 343883
block_nameops []
Processing block 343884
block_nameops []
Processing block 343885
block_nameops []
Processing block 343886
block_nameops []
Processing block 343887
block_nameops []
Processing block 343888

Any thoughts on what might be causing this?

Thanks,

Jonathan

Users should definitely be informed that their Chain API keys (including the secret key) as well as their bitcoind username/password are being stored in plaintext in ~/.blockstore/blockstore.ini.

In the current system, all namespaces simply exist and don't need to be created. If you want to start operating in a namespace, you simply use a prefix that has the namespace properties in it (like pricing rules and whether names expire or not) and a human-readable identifier of length 1 (in order to distinguish it from other namespace with the same properties).

I'd like to explore a new system, whereby namespaces are explicitly defined and then they can be used from that point on. In this system, names could only be registered in existing namespaces.

Namespaces would be created with a namespace creation operation with the following effects:

1. the properties of the namespace are laid out, including the identifier of the namespace, the name pricing rules, and the name expiration rules
2. a namespace creation fee is paid (to prevent spam)
3. the namespace is added to the list of existing namespaces

Here is an example of a possible namespace creation operation as part of this system:

CREATE NAMESPACE
Operation fee: 1 BTC
TLD: .cool
Name lifetime: 1 year
Baseline name price: 0.01 BTC
Name auctioning: false
Name price decay per letter: 2x

There are 26^4 or 0.5M possible 4-letter TLDs. This means that 0.5M bitcoins would have to be spent to initialize all of the possible namespaces. People would be wise to choose TLDs that don't conflict with existing ICANN TLDs. Further, TLDs don't have to be used in practice. At minimum they act as a unique identifier for the namespace.

To be clear, if a namespace is defined it is not owned by anyone. The person who defines the namespace simply gets to set the properties of that namespace.

See this for details

Trying to experiment with blockstore.

Using default bitcoind server at btcd.onename.com
ERROR: Cannot connect to bitcoind

If a name gets either lost or stolen (by either the private key owner being lost or the private key being compromised), it's important to convey to the world that the name is no longer under the control of the owner.

If we could have a mechanism to remove a compromised name from the system (a flavor/variant of key revocation), then we could prevent the impersonation of websites, user identities, and certificates upon name theft.

More on key revocation, for reference: http://www.ac.uk.pgp.net/pgpnet/secemail/q4/node19.html

In order to make this work, we would need to do the following:

1. add a name operation for revoking names that can only be executed by the owner of the name
2. recommend that as soon as someone registers a name, they create and sign an anyone-can-pay transaction that revokes the name, then hide the signed transaction somewhere safe, in a separate location from the key (a third party could help out with this)
3. recommend the following: as soon as a name is stolen, the pre-signed name revocation transaction should be recovered and broadcasted
4. recommend the following: if a private key owner is compromised but the name hasn't yet been stolen, transfer the name to a new private key owner

This system closely parallels the key revocation system that PGP and other crypto identity systems have implemented, with a few variations to make it work with names as property owned by private keys.

Would help greatly if you could add like a --demo or --mock flag to just run the json rpc server with some dummy data that can sent/retrieved. This mode would bypass the requirement for a bitcoind server.

As a simple first start, just some dummy data (that's properly formatted) for the lookup rpc command would help.

Not super high priority of course, but if it's easy to do and you can do it quick we can add support for blockstore to DNSChain 0.5.0 this weekend. Otherwise it'll make it into the next release.

Feathercoin Core 0.9.3 wallet have support OP_RETURN , I think FTC and BTC are the same. How to support Feathercoin?
I change RPC port 8332 to 9337, blockstore start OK and blockstore-cli Error.

blockstore-cli preorder swiftonsecurity N876Ywf8bL5JWMPbHvjHP2AaSF1Ap8Wg4qAn9AAiQbmYFWPh2vqe
{
"error": "UnboundLocalError: local variable 'secret_exponent' referenced before assignment",
"traceback": [
"Traceback (most recent call last):",
" File "/usr/local/lib/python2.7/dist-packages/blockstore/blockstored.py", line 253, in jsonrpc_preorder",
" blockchain_client=blockchain_client, testset=True)",
" File "/usr/local/lib/python2.7/dist-packages/blockstore/lib/operations/preorder.py", line 28, in broadcast",
" hash160 = BitcoinPrivateKey(private_key).public_key().hash160()",
" File "/usr/local/lib/python2.7/dist-packages/coinkit/privatekey.py", line 61, in init",
" if not is_secret_exponent(secret_exponent, self._curve.order):",
"UnboundLocalError: local variable 'secret_exponent' referenced before assignment"
]
}

Starting with DHT calls, options are:

1. For storing data to DHT dht_store, dht_put, something without "dht_" ? This call will take a and return True/False and a e.g.,

dht_store <value> 

2. For reading data from DHT dht_read, dht_get, something without "dht_"> This call will take a and return or None e.g.,

dht_get <hash>

Add an option to set a messenger address in your profile.

It doesn't have to be specific per se, as in "Subspace address" or "Bitmessage address", but it could just be a generic field like "Messenger address".

It would be useful for mapping a messenger address to an openname and allowing clients that use it to verify the person using the profile on the messenger is the owner.

I suggest we do the following:

1. check the bitcoin.conf file for config info
2. add a call for re-inputting the config info
3. stop prompting the user to re-input the config info upon the starting of bitcoind and failure to connect (the former makes this make even more sense)