Coder Social home page Coder Social logo

lbryio / lbrycrd Goto Github PK

View Code? Open in Web Editor NEW
2.6K 51.0 177.0 89.78 MB

The blockchain that provides the digital content namespace for the LBRY protocol

Home Page: https://lbry.com

License: MIT License

Shell 0.91% QMake 0.01% Python 15.39% Makefile 1.25% C++ 67.82% C 10.93% HTML 0.23% Objective-C++ 0.04% Java 0.32% M4 2.15% CMake 0.26% Assembly 0.30% Dockerfile 0.06% Sage 0.32%
blockchain cryptocurrency bitcoin lbry decentralized c-plus-plus

lbrycrd's Introduction

LBRYcrd - The LBRY blockchain

Build Status MIT licensed

LBRYcrd uses a blockchain similar to bitcoin's to implement an index and payment system for content on the LBRY network. It is a fork of bitcoin core. In addition to the libraries used by bitcoin, LBRYcrd also uses icu4c.

Please read the lbry.tech overview for a general understanding of the LBRY pieces. From there you could read the LBRY spec for specifics on the data in the blockchain.

Table of Contents

  1. Installation
  2. Usage
    1. Examples
    2. Data directory
  3. Running from Source
    1. Ubuntu with pulled static dependencies
    2. Ubuntu with local shared dependencies
    3. MacOS (cross-compiled)
    4. MacOS with local shared dependencies
    5. Windows (cross-compiled)
    6. Use with CLion
  4. Contributing
  5. Mailing List
  6. License
  7. Security
  8. Contact

Installation

Latest binaries are available from https://github.com/lbryio/lbrycrd/releases. There is no installation procedure; the CLI binaries will run as-is and will have any uncommon dependencies statically linked into the binary. The QT GUI is not supported. LBRYcrd is distributed as a collection of executable files; traditional installers are not provided.

Usage

The lbrycrdd executable will start a LBRYcrd node and connect you to the LBRYcrd network. Use the lbrycrd-cli executable to interact with lbrycrdd through the command line. Command-line help for both executables are available through the "--help" flag (e.g. lbrycrdd --help). Examples:

Examples

Run ./lbrycrdd -server -daemon to start lbrycrdd in the background.

Run ./lbrycrd-cli -getinfo to check for some basic information about your LBRYcrd node.

Run ./lbrycrd-cli help to get a list of all commands that you can run. To get help on specific commands run ./lbrycrd-cli [command_name] help

Test locally:

./lbrycrdd -server -regtest -txindex  # run this in its own window
./lbrycrd-cli -regtest generate 120   # mine 20 spendable coins
./lbrycrd-cli -regtest claimname my_name deadbeef 1 # hold a name claim with 1 coin
./lbrycrd-cli -regtest generate 1     # get that claim into the block
./lbrycrd-cli -regtest listnameclaims # show owned claims
./lbrycrd-cli -regtest getclaimsforname my_name # show claims under that name
./lbrycrd-cli -regtest stop           # kill lbrycrdd
rm -fr ~/.lbrycrd/regtest/            # destroy regtest data

For further understanding of a "regtest" setup, see the local stack setup instructions here: https://lbry.tech/resources/regtest-setup

The CLI help is also browsable online at https://lbry.tech/api/blockchain

Data directory

Lbrycrdd will use the below default data directories (changeable with -datadir):

Windows:  %APPDATA%\lbrycrd
Mac:      ~/Library/Application Support/lbrycrd
Unix:     ~/.lbrycrd

The data directory contains various things such as your default wallet (wallet.dat), debug logs (debug.log), and blockchain data. You can optionally create a configuration file lbrycrd.conf in the default data directory which will be used by default when running lbrycrdd.

For a list of configuration parameters, run ./lbrycrdd --help. Below is a sample lbrycrd.conf to enable JSON RPC server on lbrycrdd.

rpcuser=lbry
rpcpassword=xyz123456790
daemon=1
server=1
txindex=1

Running from Source

The easiest way to compile is to utilize the Docker image that contains the necessary compilers: lbry/build_lbrycrd. This will allow you to reproduce the build as made on our build servers. In this sample we map a local lbrycrd folder and a local ccache folder inside the image:

git clone https://github.com/lbryio/lbrycrd.git
cd lbrycrd
docker run -v "$(pwd):/lbrycrd" --rm -v "${HOME}/ccache:/ccache" -w /lbrycrd -e CCACHE_DIR=/ccache lbry/build_lbrycrd packaging/build_linux_64bit.sh

Some examples of compiling directly:

Ubuntu with pulled static dependencies

sudo apt install build-essential git libtool autotools-dev automake pkg-config bsdmainutils curl ca-certificates
git clone https://github.com/lbryio/lbrycrd.git
cd lbrycrd
./packaging/build_linux_64bit.sh
./src/test/test_lbrycrd

Other Linux distros would be similar. The build shell script is fairly trivial; take a peek at its contents.

Ubuntu with local shared dependencies

Note: using untested dependencies may lead to conflicting results.

sudo add-apt-repository ppa:bitcoin/bitcoin
sudo apt-get update
sudo apt-get install libdb4.8-dev libdb4.8++-dev libicu-dev libssl-dev libevent-dev \
                     build-essential git libtool autotools-dev automake pkg-config bsdmainutils curl ca-certificates \
                     libboost-system-dev libboost-filesystem-dev libboost-chrono-dev libboost-test-dev libboost-thread-dev

# optionally include libminiupnpc-dev libzmq3-dev

git clone https://github.com/lbryio/lbrycrd.git
cd lbrycrd
./autogen.sh
./configure --enable-static --disable-shared --with-pic --without-gui CXXFLAGS="-O3 -march=native"
make -j$(nproc)
./src/lbrycrdd -server ...

MacOS (cross-compiled)

sudo apt-get install clang llvm git libtool autotools-dev automake pkg-config bsdmainutils curl ca-certificates \
                     libboost-system-dev libboost-filesystem-dev libboost-chrono-dev libboost-test-dev libboost-thread-dev

git clone https://github.com/lbryio/lbrycrd.git
cd lbrycrd
# download MacOS SDK from your favorite source
mkdir depends/SDKs
tar ... extract SDK to depends/SDKs/MacOSX10.11.sdk
./packaging/build_darwin_64bit.sh

Look in packaging/build_darwin_64bit.sh for further understanding.

MacOS with local shared dependencies

brew install boost berkeley-db@4 icu4c libevent
# fix conflict with gawk pulled first:
brew reinstall readline
brew reinstall gawk

git clone https://github.com/lbryio/lbrycrd.git
cd lbrycrd/depends
make NO_QT=1
cd ..
./autogen.sh
CONFIG_SITE=$(pwd)/depends/x86_64-apple-darwin15.6.0/share/config.site ./configure --enable-static --disable-shared --with-pic --without-gui --enable-reduce-exports CXXFLAGS=-O2
make -j$(sysctl -n hw.ncpu)

Windows (cross-compiled)

Compiling on MS Windows (outside of WSL) is not supported. The Windows build is cross-compiled from Linux like so:

sudo apt-get install build-essential git libtool autotools-dev automake pkg-config bsdmainutils curl ca-certificates \
                     g++-mingw-w64-x86-64 mingw-w64-x86-64-dev

update-alternatives --set x86_64-w64-mingw32-g++ /usr/bin/x86_64-w64-mingw32-g++-posix

git clone https://github.com/lbryio/lbrycrd.git
cd lbrycrd
./packaging/build_windows_64bit.sh

If you encounter any errors, please check doc/build-*.md for further instructions. If you're still stuck, create an issue with the output of that command, your system info, and any other information you think might be helpful. The scripts in the packaging folder are simple and will grant extra light on the build process as needed.

Use with CLion

CLion has not traditionally supported Autotools projects, although some progress on that is now in the works. We do include a cmake build file for compiling lbrycrd. See contrib/cmake. Alas, CLion doesn't support external projects in cmake, so that particular approach is also insufficient. CLion does support "compile_commands.json" projects. Fortunately, this can be easily generated for lbrycrd like so:

pip install --user compiledb
./autogen.sh && ./configure --enable-static=no --enable-shared --with-pic --without-gui CXXFLAGS="-O0 -g" CFLAGS="-O0 -g" # or whatever normal lbrycrd config
compiledb make -j10

Then open the newly generated compile_commands.json file as a project in CLion. Debugging is supported if you compiled with -g. To enable that you will need to create a target in CLion by going to File -> Settings -> Build -> Custom Build Targets. Add an empty target with your choice of name. From there you can go to "Edit Configurations", typically found in a drop-down at the top of the editor. Add a Custom Build Application, select your new target, select the compiled file (i.e. test_lbrycrd or lbrycrdd, etc), and then add any necessary command line parameters. Ensure that there is nothing in the "Before launch" section.

Contributing

Contributions to this project are welcome, encouraged, and compensated. For more details, see https://lbry.tech/contribute

We follow the same coding guidelines as documented by Bitcoin Core, see here. To run an automated code formatting check, try: git diff -U0 master -- '*.h' '*.cpp' | ./contrib/devtools/clang-format-diff.py -p1. This will check any commits not on master for proper code formatting. We try to avoid altering parts of the code that is inherited from Bitcoin Core unless absolutely necessary. This will make it easier to merge changes from Bitcoin Core. If commits are expected not to be merged upstream (i.e. we broke up a commit from Bitcoin Core in order to use a single feature in it), the commit message must contain the string "NOT FOR UPSTREAM MERGE".

The master branch is regularly built and tested, but is not guaranteed to be completely stable. Releases are created regularly to indicate new official, stable release versions.

Testing and code review is the bottleneck for development; we get more pull requests than we can review and test on short notice. Please be patient and help out by testing other people's pull requests, and remember this is a security-critical project where any mistake might cost people lots of money. Developers are strongly encouraged to write unit tests for new code and to submit new unit tests for old code. Unit tests are compiled by default and can be run with src/test/test_lbrycrd

The Travis CI system makes sure that every pull request is built, and that unit and sanity tests are automatically run. See https://travis-ci.org/lbryio/lbrycrd

Testnet

Testnet is maintained for testing purposes and can be accessed using the command ./lbrycrdd -testnet. If you would like to obtain testnet credits, please contact [email protected] or [email protected] .

It is easy to solo mine on testnet. (It's easy on mainnet too, but much harder to win.) For instructions see SGMiner and Mining Contributions

Mailing List

We maintain a mailing list for notifications of upgrades, security issues, and soft/hard forks. To join, visit https://lbry.com/forklist.

License

This project is MIT licensed. For the full license, see LICENSE.

Security

We take security seriously. Please contact [email protected] regarding any security issues. Our PGP key is here if you need it.

Contact

The primary contact for this project is @BrannonKing ([email protected])

lbrycrd's People

Contributors

achow101 avatar ajtowns avatar brannonking avatar cozz avatar domob1812 avatar empact avatar fanquake avatar gavinandresen avatar gmaxwell avatar instagibbs avatar jnewbery avatar jonasschnelli avatar jtimon avatar kallewoof avatar ken2812221 avatar laanwj avatar luke-jr avatar meshcollider avatar morcos avatar non-github-bitcoin avatar paveljanik avatar petertodd avatar practicalswift avatar promag avatar pstratem avatar ryanofsky avatar sdaftuar avatar sipa avatar thebluematt avatar theuni avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

lbrycrd's Issues

dumpprivkey returning a 52-character base58 string starting with '5'

Based on https://en.bitcoin.it/wiki/Private_key#Base58_Wallet_Import_format, The Wallet Import Format key is expected to be a 51-character base58 string, but dumpprivkey is returning a 52-character string which cannot be properly converted to a base58 address based on the public key.

Just for reference, https://en.bitcoin.it/wiki/Wallet_import_format shows steps for converting a private key to/from base58.

Here's an example I tested with:

ubuntu@lbry:~# curl -X POST -d '{"method": "getnewaddress", "params": []}' http://rpc:[email protected]:9245
{"result":"bGdQJgLwXSZ6eajyjgvqz9aMZ2RqkNbD1y","error":null,"id":null}
ubuntu@lbry:~# curl -X POST -d '{"method": "dumpprivkey", "params": ["bGdQJgLwXSZ6eajyjgvqz9aMZ2RqkNbD1y"]}' http://rpc:[email protected]:9245
{"result":"5GBWLGQu5NiaCNLzTtLMTGMVfAkmB6pBqFt9nYvpNXcGxsqCWY3h","error":null,"id":null}

I have release v0.12.0.4 installed on both Ubuntu and Windows 10. Here's my getinfo output.

{"result":{"version":129900,"protocolversion":70013,"walletversion":60000,"balance":0.00000000,"blocks":190158,"timeoffset":0,"connections":12,"proxy":"","difficulty":190641.6861993672,"testnet":false,"keypoololdest":1497023671,"keypoolsize":101,"paytxfee":0.00000000,"relayfee":0.00001000,"errors":"This is a pre-release test build - use at your own risk - do not use for mining or merchant applications"},"error":null,"id":null}

git clone --depth=1 and git reset --hard; git push -f?

Hi,

This is a huge repository coming from Bitcoin initially. Are you planning to cherry pick things from upstream Bitcoin/BitcoinXT? If so, does having the commit history help?

I wonder if it would be worth it to reset the repo to the state where you started modifying it for lbrycrd. Would really shrink it down, possibly with no side-effects (maybe something legally?).

This is very minor and nitpicky, but I'm also curious of a good reason not to, because it could be something I've not though of and should know.

Thanks,
Teran

memory usage 1

Do preliminary work in checking places where the claimtrie uses memory, verify that the empty nodes are the main problem, check if there is some quick optimization that will improve memory usage.

After this is done, create new issues in this epic for the rest of the work.

Feature Request: sendfromaddress command that keeps change in original address

Edit: If working on this issue, please see PR#82 and either use this as a starting place or at least see the comments.

I'd like a way to send credits from a specific address and keep the change in that address. Jack and I talked about calling it sendfromaddress. I'm not in love with that name (there are already several sendfrom/sendto commands) but I can't thin of a better name. This would be really helpful for maintaining public addresses that have a specific purpose.

Example:

  • Address bFromAddress123 has 5 LBC. Address bRecipeintAddress456 is empty.
  • I run ./lbrycrd-cli sendfromaddress bFromAddress123 bRecipientAddress456 2.3
  • Now bRecipientAddress456 has 2.3LBC and bFromAddress123 has 2.6999575LBC (or whatever 2.7 - txn fee is).

Need to make claims (update/ and name claim) in hex

We need a way for users to input hex claims value in lbryrd-cli / lbrycrd RPC (command claimname and updateclaim) so that protobuf claims can be made through lbrycrd

It's probably sufficient to accept hex as default.

Case-insensitive claims

LBRY URLs ought to be case-insensitive, the way traditional domains are. Two claims with names with different case-sensitivity should be considered having the same name.

lbrycrd-cli Info still says "BTC"

from @theunrealtok

error message:
getwalletinfo
Returns an object containing various wallet state info.

Result:
{
"walletversion": xxxxx, (numeric) the wallet version
"balance": xxxxxxx, (numeric) the total confirmed balance of the wallet in BTC
"unconfirmed_balance": xxx, (numeric) the total unconfirmed balance of the wallet in BTC
"immature_balance": xxxxxx, (numeric) the total immature balance of the wallet in BTC
"txcount": xxxxxxx, (numeric) the total number of transactions in the wallet
"keypoololdest": xxxxxx, (numeric) the timestamp (seconds since GMT epoch) of the oldest pre-generated key in the key pool
"keypoolsize": xxxx, (numeric) how many new keys are pre-generated
"unlocked_until": ttt, (numeric) the timestamp in seconds since epoch (midnight Jan 1 1970 GMT) that the wallet is unlocked for transfers, or 0 if the wallet is locked
"paytxfee": x.xxxx, (numeric) the transaction fee configuration, set in BTC/kB
}

Build error on FreeBSD

  CXX      test/test_test_lbrycrd-rpc_wallet_tests.o
  AR       libbitcoin_util.a
  CXXLD    lbrycrdd
libbitcoin_server.a(libbitcoin_server_a-init.o): In function `CleanupBlockRevFiles()':
/usr/local/include/boost/filesystem/path.hpp:139: undefined reference to `boost::filesystem::path_tra
libbitcoin_util.a(libbitcoin_util_a-util.o): In function `SetupEnvironment()':
/home/null1/lbrycrd/src/util.cpp:743: undefined reference to `boost::filesystem::path::imbue(std::loc
/home/null1/lbrycrd/src/util.cpp:744: undefined reference to `boost::filesystem::path::imbue(std::loc
libbitcoin_util.a(libbitcoin_util_a-util.o): In function `boost::program_options::detail::basic_confi
/usr/local/include/boost/program_options/detail/config_file.hpp:161: undefined reference to `boost::p
libbitcoin_util.a(libbitcoin_util_a-util.o): In function `boost::program_options::detail::basic_confi<std::string> > const&, bool)':
/usr/local/include/boost/program_options/detail/config_file.hpp:145: undefined reference to `boost::p, std::allocator<std::string> > const&, bool)'
libbitcoin_wallet.a(libbitcoin_wallet_a-db.o): In function `CDBEnv::Verify(std::string const&, bool (
/home/null1/lbrycrd/src/wallet/db.cpp:157: undefined reference to `Db::verify(char const*, char const
libbitcoin_wallet.a(libbitcoin_wallet_a-db.o): In function `CDBEnv::Salvage(std::string const&, bool,unsigned char> > >, std::allocator<std::pair<std::vector<unsigned char, std::allocator<unsigned char>
/home/null1/lbrycrd/src/wallet/db.cpp:180: undefined reference to `Db::verify(char const*, char const
collect2: error: ld returned 1 exit status
Makefile:2605: recipe for target 'lbrycrdd' failed
gmake[2]: *** [lbrycrdd] Error 1

I think I've had that with and without --with-incompatible-bdb. I am running two jobs as well, but not sure if that's causing the problem.

CClaimTrieCache is caching Nodes inefficiently during updates

lbrycrd uses space inefficient map and set structure to cache Nodes (cacheing nodes is done when updating the claim trie structure to keep track of which node should or should not be updated)

CClaimTrieCache.cacheHashes -> (hashMapType, std::map<std::string, uint256> , stores hashes for each nodes
CClaimTrieCache.cache -> (nodeCacheType, std::map<std::string, CClaimTrieNode*>
CClaimTrieCache.block_originals -> (nodeCacheType)
CClaimTrieCache.dirtyHashes -> (std::setstd::string)
CClaimTrie.dirtyNodes -> (nodeCacheType)

This is problematic for caching large name claims (max length 255) since each subnode gets put into the map/set (i.e, claim "test" must cache "t", "te" ,"tes" , "test") thus 32 kilobytes alone (255*256 /2) is used to store the string keys for a max length name claim in the cache. We could improve the memory usage by utilizing a trie structure.

We actually do not have a base template trie class although ClaimTrie itself is a trie structure. It would make sense to create this base template tire class so that a) ClaimTrie can inherit from it and b) we can use it instead of map and set structure used to cache nodes.

Having a base trie class will also have the benefit of improving code readability and testability.

"This is a pre-release test build...."

Seeing this in lbrycrd-cli getinfo on latest build from master:

"This is a pre-release test build - use at your own risk - do not use for mining or merchant applications"

I think it may need a version bump somewhere.

Lbrycrd claimtrie unit test improvements

Old tests for the claimtrie ( https://github.com/lbryio/lbrycrd/blob/master/src/test/claimtrie_tests.cpp ) should be merged with new tests which contains fixtures and functions to make the expansion of unit tests easier: https://github.com/lbryio/lbrycrd/blob/master/src/test/claimtriebranching_tests.cpp

The new tests could also be improved more, it is mainly testing for the best claim, and we should test other things in the claim trie such as the number of claims in the trie, check if the claimtrie root merkle hash is being updated, check that supports are present in the claim trie directly, etc..

Improving lbrycrd binary install

I am working on documentation for installs of the main pieces of the LBRY system. I use the following environment for testing:

Lubuntu 16.04.3 LTS - Ubuntu derived, takes half the disk space due to graphics diet.
apt update/apt upgrade so it's current.
tmux - share screens and log activity to text file, so I can watch a developer do an install.

When I download and unzip the lbrycrd binaries for Linux that part works fine, but there is no README to explain what to do, no example config, and no lbrycrdd.service file for systemd.

This is what we did for the daemon's configuration:

cat > ~/.lbrycrd/lbrycrd.conf
port=9246
bind=127.0.0.1

rpcallowip=127.0.0.1
rpcbind=127.0.0.1
rpcport=9245
rpcuser=lbryrpc
rpcpassword=securepassw0rd

server=1
txindex=1
ctrl-d

And I didn't record the contents of /etc/systemd/system/lbrycrdd.service and I've since recycled the VPS where we did that, so I do not have a copy of that text handy.

Update Bitcoin codebase to support Atomic Swaps/decentralized OTC exchange

In the time when centralized exchanges are on the fall, in fear of strict regulation - It is time to enable support for decentralized and trustless on-chain and off-chain OTC (over-the-counter) exchanges between cryptocurrencies. By updating the codebase of LBRY to the newest codebase of Bitcoin, we will be able to support Atomic Swaps.
https://github.com/decred/atomicswap
Decred (https://Decred.org) made a repository here. We can fork it.

Here is a picture made by Particl, illustrating how it works: https://image.ibb.co/jcomwm/atomic_swap.png

I will also write about these issues at https://LBRY.Community

aborted on ubuntu 14.04

lbrycrdd: main.cpp:2387: bool ConnectBlock(const CBlock&, CValidationState&, CBlockIndex*, CCoinsViewCache&, CClaimTrieCache&, bool): Assertion `hashPrevBlock == trieCache.getBestBlock()' failed.

Tried binaries and now I also got same error even when compiled myself.

Add minconf param to claimtrie RPC commands

The following commands need a minconf param to specify how many confirmations are required (aka how many blocks back to look). Default minconf should be 6, like other commands (e.g. listnameclaims).

  • getclaimsforname
  • getclaimsintrie
  • getclaimtrie
  • getvalueforname

Include non-winning claims in the claimtrie hash

Problem

Right now, only the winning claim for each name is included in the hash of the claimtrie. Therefore resolving non-winning claims cannot be validated by SPV clients.

Solution

Include all claims in the claimtrie hash.

Effective amount in getvalueforname rpc command does not return correct value

RPC command getvalueforname does not return correct value for "effective amount"

This is because effective amount is not serialized when serializing CClaimValue ( https://github.com/lbryio/lbrycrd/blob/master/src/claimtrie.h#L50 ). Thus reading nEffectiveAmount will not return proper value until a reorder is initiated by the claimtrie.

Effective amount should be calculated by obtaining its supports. nEffectiveAmount should be a protected variable , with CClaimValue having CClaimTrie as a friend class.

build failed on mac

When I build lbrycrd on mac, there was error like this:

Undefined symbols for architecture x86_64:
"memory_cleanse(void*, unsigned long)", referenced from:
CBlockHeader::GetPoWHash() const in libbitcoinconsensus_la-block.o
std::__1::enable_if<(__is_forward_iterator<char const*>::value) && (is_constructible<char, std::__1::iterator_traits<char const*>::reference>::value), std::__1::__wrap_iter<char*> >::type std::__1::vector<char, zero_after_free_allocator >::insert<char const*>(std::__1::__wrap_iter<char const*>, char const*, char const*) in libbitcoinconsensus_la-block.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Add/reorganize unit tests for CClaimTrieCache/CClaimTrie

We have decent coverage for integration testing the CClaimTrieCache together with CClaimTrie but we do not have any unit tests for testing them separately.

Especially for CClaimTrieCache, adding unit tests would help developers understand and debug the class. We'll have better confidence on the correctness of the code going forward.

Acceptance Criteria

Definition of Done

  • Tested against acceptance criteria
  • Tested against the assumptions of the user story
  • The project builds without errors
  • Unit tests are written and passing
  • Tests on devices/browsers listed in the issue have passed
  • QA performed & issues resolved
  • Refactoring completed
  • Any configuration or build changes documented
  • Documentation updated
  • Peer Code Review performed

Lbrycrd can crash when a support and abandon is on the same block

It's possible to crash lbrycrd if two conditions are met

a) the claim is the only claim that exists for that claim's node in the claimtrie and any parent node up to the root node of the claimtrie. (i.e, the claim is for the name "zed" and there is no other claim on any name that starts with the letter "z")

b) The claim is abandoned and supported in the same block, or the claim is abandoned and a support of the claim is also abandoned.

This was due to a bug in the function reorderTrieNode which is called when reordering the claims in a node when a support is detected. In order to find that particular node for a given block update, it has to check the cache for any node updates which contains updated information for the nodes if claims in the node are deleted or updated. There was a bug where reorderTrieNode, failed to check the cache on the root node, thus any update cache entry for the root node (this would result when a children on a root node is deleted, for this to happen every single claim on a children of a root node need to be deleted) would be ignored when searching for the node to reorder. Thus it would find the old node before the udpate/delete and would attempt to reorder it and store it in the cache. This would fail , when we flush the block and update the claim trie and there is a cache entry for a node that should have been deleted, and lbrycrd will crash.

Firstly, to prevent malicious actors from using this exploit, I'm going to make claims at every children node off of the root node (we have a lot of claims on standard ascii characters so we don't need to do that for names that starts with "a" for example, but we need to make claims on non-ascii characters), so that condition a) cannot be met (this has been done on 12/28/2017).

Secondly, we are going to roll a fix for it which will prevent the crash from happening and the claimtrie will be updated as expected. After the fix is applied, this new lbrycrd node can accept blocks where the above scenario occurs. Since the old lbrycrd node cannot accept blocks where the above scenario occurs, this will technically be a hard fork , although the condition for this hard fork is not likely to cause a split due to the first action we've taken.

lbrycrdd created invalid transaction, thinks address is empty even though its not

2016-10-12 03:02:18 keypool added key 1639, size=101
2016-10-12 03:02:18 keypool reserve 1539
2016-10-12 03:02:18 CommitTransaction:
CTransaction(hash=d5e2963d45, ver=1, vin.size=1, vout.size=2, nLockTime=59807)
    CTxIn(COutPoint(ffed835a3d, 0), scriptSig=473044022018e0ee20c8474e, nSequence=4294967294)
    CTxOut(nValue=824.99874871, scriptPubKey=76a914c11400b5d35876bef008839c)
    CTxOut(nValue=25.00000000, scriptPubKey=76a914e37301b31113436528931a41)
2016-10-12 03:02:18 keypool keep 1539
2016-10-12 03:02:18 AddToWallet d5e2963d45dbf17a6eea65967de1a9dc44fe47f74420e3b73a3da62308e2aa76  new
2016-10-12 03:02:18 AddToWallet d5e2963d45dbf17a6eea65967de1a9dc44fe47f74420e3b73a3da62308e2aa76  
2016-10-12 03:02:18 Relaying wtx d5e2963d45dbf17a6eea65967de1a9dc44fe47f74420e3b73a3da62308e2aa76
2016-10-12 03:02:43 keypool added key 1640, size=101
2016-10-12 03:02:43 keypool reserve 1540
2016-10-12 03:02:43 CommitTransaction:
CTransaction(hash=2dc2153fa3, ver=1, vin.size=1, vout.size=2, nLockTime=59807)
    CTxIn(COutPoint(d5e2963d45, 0), scriptSig=47304402206c4c0e0f3e6f35, nSequence=4294967294)
    CTxOut(nValue=25.00000000, scriptPubKey=76a914f012b6449f34caf0be5b5a25)
    CTxOut(nValue=799.99870228, scriptPubKey=76a91482292715dab1cd680e6efe97)
2016-10-12 03:02:43 keypool keep 1540
2016-10-12 03:02:43 AddToWallet 2dc2153fa379701c6de8ede9d17760483627857501e2b45f59427e353ec574a6  new
2016-10-12 03:02:43 CommitTransaction(): Error: Transaction not valid
2016-10-12 03:02:59 incrementBlock: nCurrentHeight (before increment): 59808
2016-10-12 03:02:59 UpdateTip: new best=d1621fd8943436c946ace601571a6784308440f6f369843c35b8df88baf4d9ab height=59808 version=0x20000000 log2_work=63.798004 tx=719930 date='2016-10-12 03:02:50' progress=1.000000 cache=2.9MiB(9076tx)

near the bottom, it looks like an invalid transaction was created
it was created using a script, but the script just calls lbrycrd-cli sendtoaddress
anyway, the transaction was supposed to spent from the address bWLB555cGfRcW4eCPBQgPScmocZoMPNGCd, which is where the previous transaction (d5e2963d45dbf17a6eea65967de1a9dc44fe47f74420e3b73a3da62308e2aa76) sent its change
however, the wallet now thinks that bWLB555cGfRcW4eCPBQgPScmocZoMPNGCd is empty
even though the invalid transaction was never accepted by the network
i bet i could fix this by exporting the address, creating a brand new wallet, then reimporting the address
(though i havent tried that yet)
but the bug should probably be fixed first. plus it may be something we want to send upstream, since its probably not related to anything we changed in bitcoin
let me know if you need help reproducing this. i don't have the invalid transaction that was attempted, but i can send you the wallet file and more logs
i can maybe reconstruct the command that the script tried to run

reproducible_build.sh problems Ubuntu 16.04

Seems to be issues using reproducible_build.sh on Ubuntu 16.04 , I get below problems on a fresh 16.04 install running "reproducible_build.sh -c -t"

libbitcoin_wallet.a(libbitcoin_wallet_a-rbf.o): In function IsRBFOptIn(CTransaction const&, CTxMemPool&)': /home/treantprotector/Documents/lbrycrd/src/policy/rbf.cpp:39: undefined reference to CTxMemPool::CalculateMemPoolAncestors(CTxMemPoolEntry const&, std::set<boost::multi_index::detail::bidir_node_iterator<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::index_node_base<CTxMemPoolEntry, std::allocator > > > > > > >, CTxMemPool::CompareIteratorByHash, std::allocator<boost::multi_index::detail::bidir_node_iterator<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::index_node_base<CTxMemPoolEntry, std::allocator > > > > > > > > >&, unsigned long, unsigned long, unsigned long, unsigned long, std::__cxx11::basic_string<char, std::char_traits, std::allocator >&, bool) const'

reproducible_build.sh worked after running..

sudo apt-get install libevent-dev
sudo apt-get install libssl-dev

I believe it is actually a libevent problem , seems that https://github.com/lbryio/lbrycrd/blob/master/reproducible_build.sh#L300 does not checkout a specific version .. libvevent-dev is version 2.0.21-stable: https://packages.ubuntu.com/search?keywords=libevent-dev

And the version under depends https://github.com/lbryio/lbrycrd/blob/master/depends/packages/libevent.mk is libevent 2.0.22-stable

Tying to verify this..

Separate CClaimTrie into trie implementation and ClaimTrie / Write base template trie class

The CClaimTrie class, which implements the Claim Trie, has a tight integration between the trie data structure and the logic involved in claims.

It would be nice if we had a base template trie class which the CClaimTrie class either inherits from, or used as a member variable (CClaimTrie would either way use a trie class where the node are CClaimTrieNodes). This will improve code readability and allow us to write unit tests exclusively for the trie class to verify its correctness instead of relying on integration tests.

Also, it will allow easier implementation of #106, as we should just be able to switch out the trie class while keeping the logic for claims intact.

Custom crypto = scary scary

I know this is inherited from the BTC fork, however I'd like to point out the obvious issue of having custom crypto code. It's basically rule #1 of any project, never reinvent crypto. Looking at the basic crypto requirements in lbrycrd it seems this could more easily be resolved with OpenSSL, gnutls, etc (Given that the requirement effectively seems to be hash/hmac centric)

Soo would you consider nuking or allowing external hash lib providers (configure.ac options?)

Remove claim expiration

As discovered by @jimmykiselak

Expired claims are not kept track of by claimtrie.cpp , thus they are not updateable using OP_UPDATE_CLAIM. Name owners can only reclaim the expired claim through OP_CLAIM_NAME , which means that all supports must be updated as well.

Expired claims should be kept track of, so they can be updated.

This is a hard fork and will require reindexing. We have roughly till the end of August to fix this issue in a simple way (see related issue #20 ) . Otherwise there needs to be exceptions made for early blocks.

2 pools (yiimp based) reporting Rejects / claimtrie issues?

From @tzarebczan on July 24, 2017 16:23

So we are getting reports from a couple pools about rejects leading to them not finding blocks. Something about our claimtrie. However the large suprnova pool seems to deal with it.

reject and debug logs from @reymus pool (quantumrage) - using same yiimp implementation as Epsylon. http://quantumrage.com/files/errorlogs.tar.gz

BCT Thread where he mentioned the claimtrie issue:
https://bitcointalk.org/index.php?topic=770064.msg20328909#msg20328909

@finer9 - can you reach out to Epsylon for his logs - could help potentially.

Copied from original issue: lbryio/internal-issues#11

Change algorithm to prevent ASICs from centralizing mining power.

With the production and monopolization Baikal has done with the new ASIC miner they have made, (It costs 9,000-10,000$) I think this conflicts with the ideology of LBRY.
https://asicminermarket.com/product/baikal-miner-giant-b/
Mining will be less decentralized, and more reserved for the rich. Also LBRY will be less secure and more prone to attacks before more producers have made more ASICs, and mining power has become more distributed.

This isn't the vision of LBRY, is it?
https://spee.ch/6/Screen-Shot-2018-01-26-at-130322.png <--- See the rise in mining power. (blue line) this graph was collected from https://www.nicehash.com/algorithm/lbry ... You can see mining power has gone up about 10x in about 10 days.

I will be writing more about this issue at https://LBRY.Community soon. It is sad to see all the people using their graphic cards are made obsolete, in the shadow of an exclusive ASIC miner, costing almost 10,000$

I made a relevant Twitter POLL: https://twitter.com/LBRYCommunity/status/957535338267897856

Regtest does not reject dust transactions

This transaction is rejected as being dust on lbrycrd_main:

{ 
  "txid": "6a78ecf9cc7ed8ce700075ebc0f6598aec0e3d969703c5d8ec3dbc6d115fb2fc",
  "size": 192,
  "version": 1,
  "locktime": 0,
  "vin": [
    {
      "txid": "7992634f28066c38a89b6e44ae81db23eceed663180bd2bd2604c6c7db351a10",
      "vout": 0,
      "scriptSig": {
        "asm": "3045022100dc3fc3d565ef852e94d05f69b852c1d36b1079603a5f42aaff2cae49a08495a402205530fe6170096e7d33ab6e44d88836824bebe787e0780b52c6d0eec3b6de9bef[ALL] 03be327adddbf705c863115dfaad45f3dceab48bccf58406b74a33595695867d0a",
        "hex": "483045022100dc3fc3d565ef852e94d05f69b852c1d36b1079603a5f42aaff2cae49a08495a402205530fe6170096e7d33ab6e44d88836824bebe787e0780b52c6d0eec3b6de9bef012103be327adddbf705c863115dfaad45f3dceab48bccf58406b74a33595695867d0a"
      },
      "sequence": 4294967295
    }
  ],
  "vout": [
    {
      "value": 0.00000000,
      "n": 0,
      "scriptPubKey": {
        "asm": "OP_DUP OP_HASH160 81a05dc24d35355bc880ed9998e1cc57e191bca0 OP_EQUALVERIFY OP_CHECKSIG",
        "hex": "76a91481a05dc24d35355bc880ed9998e1cc57e191bca088ac",
        "reqSigs": 1,
        "type": "pubkeyhash",
        "addresses": [
          "bQYfyRFjJ1xwx3qqtdb9htdHuv45KE8dAT"
        ]
      }
    }
  ]
}

while this regtest transaction is accepted:

  "txid": "4d9a274b782bb97bc685f6115e8eb8deaf873e4e9018e5d0ec832c8063d0400b",
  "size": 192,
  "version": 1,
  "locktime": 0,
  "vin": [
    {
      "txid": "0ad723f58eb5687c1ea8ff7f4c61a46167432012bf5f167d2b8e0a269e895874",
      "vout": 0,
      "scriptSig": {
        "asm": "3045022100b57b72c009bfa1e4bc84e67db045aa59383832328ac1a32261bcaabeaab0cb590220447778474c1b7edf97301dc22ba83ee44ca9cb497d505e3cc99e48bdd183ac9f[ALL] 02588081807c5a7e6f1486e99ba53dade8b80139e154a93f8d1a574df91cd4fbc1",
        "hex": "483045022100b57b72c009bfa1e4bc84e67db045aa59383832328ac1a32261bcaabeaab0cb590220447778474c1b7edf97301dc22ba83ee44ca9cb497d505e3cc99e48bdd183ac9f012102588081807c5a7e6f1486e99ba53dade8b80139e154a93f8d1a574df91cd4fbc1"
      },
      "sequence": 4294967295
    }
  ],
  "vout": [
    {
      "value": 0.00000000,
      "n": 0,
      "scriptPubKey": {
        "asm": "OP_DUP OP_HASH160 2c2f6e664a3c59565b4bb5e271962ecdecfbddbb OP_EQUALVERIFY OP_CHECKSIG",
        "hex": "76a9142c2f6e664a3c59565b4bb5e271962ecdecfbddbb88ac",
        "reqSigs": 1,
        "type": "pubkeyhash",
        "addresses": [
          "mjYar5L8Re5JJvY5JwVQkvPuzJi5F4jsCR"
        ]
      }
    }
  ]
}

This makes testing the transactions produced by lbryum difficult since the dust rules for the test environment do not match the live network.

Build error on MacOS

MacOS 10.11.5

./autogen.sh and ./configure went well. With "make" - I am getting the following error:

Making all in src
CXX crypto/libbitcoinconsensus_la-hmac_sha256.lo
CXX crypto/libbitcoinconsensus_la-hmac_sha512.lo
CXX crypto/libbitcoinconsensus_la-ripemd160.lo
CXX crypto/libbitcoinconsensus_la-sha1.lo
CXX crypto/libbitcoinconsensus_la-sha256.lo
CXX crypto/libbitcoinconsensus_la-sha512.lo
CXX libbitcoinconsensus_la-arith_uint256.lo
CXX consensus/libbitcoinconsensus_la-merkle.lo
CXX libbitcoinconsensus_la-hash.lo
CXX primitives/libbitcoinconsensus_la-block.lo
CXX primitives/libbitcoinconsensus_la-transaction.lo
CXX libbitcoinconsensus_la-pubkey.lo
CXX script/libbitcoinconsensus_la-bitcoinconsensus.lo
CXX script/libbitcoinconsensus_la-interpreter.lo
CXX script/libbitcoinconsensus_la-script.lo
CXX script/libbitcoinconsensus_la-script_error.lo
CXX libbitcoinconsensus_la-uint256.lo
CXX libbitcoinconsensus_la-utilstrencodings.lo
gcc -I. -g -O2 -Wall -Wextra -Wno-unused-function -c src/gen_context.c -o gen_context.o
gcc gen_context.o -o gen_context
./gen_context
CC src/libsecp256k1_la-secp256k1.lo
CCLD libsecp256k1.laCXXLD libbitcoinconsensus.la
clang: warning: argument unused during compilation: '-pthread'
clang: warning: argument unused during compilation: '-pthread'
Undefined symbols for architecture x86_64:
"memory_cleanse(void_, unsigned long)", referenced from:
CBlockHeader::GetPoWHash() const in libbitcoinconsensus_la-block.o
std::__1::enable_if<(__is_forward_iterator::value) && (is_constructible<char, std::__1::iterator_traits<char const*>::reference>::value), std::__1::__wrap_iter<char*> >::type std::__1::vector<char, zero_after_free_allocator >::insert<char const*>(std::__1::wrap_iter<char const*>, char const, char const) in libbitcoinconsensus_la-block.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [libbitcoinconsensus.la] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all-recursive] Error 1

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.