second-state / smart-contract-search-engine

Takes a link to a smart contract's raw ABI file and an RPC URL and then indexes all instances of that smart contract

Smart Contract Search Engine

100% Ethereum compatible

If you would like to set up a smart contract search engine one or more of your contracts on Ethereum, Ethereum Classic or any other Ethereum compatible network, please get in touch so we can get you up and running.

---

Easy to use API

You can now access all of the data within the smart contract search engine via a handy API called es-ss.js. This allows your DApp to query all information simply by writing a little Javascript or Node.js code.

---

Example of a DApp which uses the smart contract search engine and the API

General search

You can search for smart contracts on the blockchain without even installing a wallet or Chrome extension. Simply click "Search" to get all results, or type in contract addresses and text to filter the results.

---

Personalized search using predefined queries

If you want to take it a step further, you can download the CyberMiles Chrome Extension and intimately search the block chain in the context of your very own accounts. Here is a quick demonstration.

Pre defined queries

You can click any of the 3 pre defined queries. The results are based on the address in your browsers wallet (i.e. CyberMiles Venus Chrome Extension)

---

Search using plain text

You can also search the system using plain text; for example try typing the words "Cup Cake" into the search box as shown below.

---

Search using address

You can search the system using the address of:

• a contract
• an owner
• a player
• a winner

---

smart-contract-search-engine's Issues

Add utilities section for batch processing

There are some cases where we need to sweep through the index and make a change. There are also time when we may want to deliberately populate an index with data (outside of the program's usual functionality i.e. during setup).
A utilities folder has been created at the following URL
https://github.com/second-state/smart-contract-search-engine/tree/master/python/utilities
At present this folder has a script to programatically upload ABI and bytecode as well as to add a new field to every record in an entire index (add the quality field to the common index)

Document the difference local.js and global.js

We used to just have a single secondStateJS.js file.
This search engine can run in two modalities: local and global.
When running in a local single user environment the local.js file (previously known as secondStateJS.js) can be used.
When running in a global multi user environment (served to the world) the global.js file (which talks to the io.py API) can be used.
This must be documented i.e. the index.html file will need to reflect the appropriate js file in the head.

Switch on new MainNet (new server & Elasticsearch domain with new indices has already been created)

In readiness for a code release, a new MainNet instance has been created and a new index/harvesting process is already underway.

There is one single domain name under which the MainNet instance is to be run. All that is required at go-live time is the following steps.

1. Re-associate the MainNet's ElasticIP to the new instance.

2. Initiate the lets encrypt SSL(HTTPS) certificate by following these instructions.

https://github.com/second-state/smart-contract-search-engine/blob/master/documentation/global_mode.md#ssl-https-using-lets-encrypt

4. Restart the machine and test

Add convention for ABI

We need a convention in order for the search engine to autonomously harvest many different ABI versions of the same DApp.

The python/config.ini of this search engine must adhere to the following convention.
dappname_version
Here is a real config example.

[abis]
fairplay_v1 = https://raw.githubusercontent.com/CyberMiles/smart_contracts/master/FairPlay/v1/dapp/FairPlay.abi

As you can see from the above configuration example, the key conforms to dappname_version i.e. fairplay_v1.

The harvester/indexer of this search engine will store all contracts in the "fairplay" index. Each record in the indices will have a version attribute stored within.

Add a “quality” field

We should add a “quality” field for each contract.

• It starts from 50 when the contract is harvested.
• If the contract can be confirmed to be legit by the admin, it increases to 100.
• If contract is confirmed by the admin to be scam / spam / test, it drops to 0.

For contracts that do not have admin confirmation, a quality score maybe computed automatically.

The client can filter search results based on the quality score.

Update secondStateJS.js to process DApp versioning

Instead of hard coding in the DApp endpoint URL, use the new versioning data from the index to dynamically construct the DApp endoint URL.

For example, the original code (line 388 of secondStateJS.js)

var playUrl = "https://cybermiles.github.io/smart_contracts/FairPlay/v1/dapp/play.html?contract=" + value._source.contractAddress;

will need to change to include the version (v1) as a variable (in the same way that it currently handles the _source.contractAddress; this is easy)

Update frontend when new versioning of contracts is done

We are now

• hashing the ABI
• hashing the bytecode
• hashing both the hash of the ABI and the hash of the bytecode strings into its own single hash

We are doing this so that any combination of versioning is deterministic. For example if a user has the bytecode and or ABI to a contract they can hash it and then search the API.

Also, this is the most robust way to handle versioning of contracts where some contract instances will have shared ABI but unique bytecode etc.

The frontend DApps which are currently using the search engine and API will have to update their Javascript to perform a quick conversion. For example instead of getting the version from the data like this

var version = value._source.dappVersion

They will need to have code like this (which converts the unique hashes to the appropriate version)

if (value._source.abiBytecodeSha3 == "0x12345") {
    var version = "v1";
}

Allow for indexing all contracts (not just ones which are in the config.ini)

Instead of indexing known contracts (using the config.ini ABIs for directions) we will harvest every contract on the network.

List items in reverse date order by default

Create Python equivalent of secondStateJS.js

If the search engine is being used in a local/single user environment (accessing an Elasticsearch ES with no access control restrictions) the JS will work out of the box.

When placing the Elasticsearch behind access control mechanisms i.e. in the global multi-user environment the Python will perform all of the Elasticsearch work using AWS Config (signing all communication between the web server and the Elasticsearch index).

The Python will also be callable via an end user's browser and/or by a remote machine. So on one hand we are improving the security of the search engine, but on the other hand we are also creating a host of pre-defined API endpoints to the index (the end users will never access Elasticsearch directly in this scenario, the host of pre-defined Python functions are the gatekeeper to the Elasticsearch instance).

Demonstrate the advantage of this parallel approach

More fundamentally, articulate that the dynamic smart contract approach is ideal for enterprise in terms of rapid development, faster and less complex global deployment, faster synchronization to the point where activities are taking place in real time.

Facilitate multiple ABI hashes per contract instance

There will be cases where a contract instance is compatible with more than one ABI i.e. a larger contract could have an ABI but a smaller part of the contract could fully implement another ABI (ERC20) implementation.

We can re-write the harvester to store not just the hash of the single ABI (which is discovers during the first pass/index), but instead test all of the ABI code and store a list of hashes; a list of abiSha3 hashes to which the contract instance is compatible with.

There will only every be a single bytecodeSha3 so we can still produce a unique contract instance version by adding all of the abiSha3s plus the bytecodeSha3.

The added flexibility of this new feature will be that end users/API calls can query the search engine and find contracts which are compatible with certain ABI implementations. For example, "show me all of the contract instances which are ERC20 compliant".

Create simple predefined search queries

1. Giveaways I created
2. Giveaways I participated in
3. Giveaways I won

Whilst the end user could manually paste in addresses and click "search", the above predefined searches will automatically fetch the user's address from their Venus browser (wallet) extension.
This will require some web3 work in the index html, however the additional code is simple and can be easily implemented.

Discuss the possibilities of a "Dashboard" where contract creators are invited to upload ABI and bytecode

This frontend page could quickly validate the ABI and bytecode against the contract address which the user has selected as being theirs.
The web page can achieve this by creating a web3 contract instance using the ABI and verify that the bytecode is the same as what is stored in the web3.eth.getTransaction(TransactionHex) transactionData.input.
Once this is verified, the ABI and bytecode will be loaded into the "all" index and then the continual multi-threaded status update can go about calling the public functions of the contract over and over.

each searched result repeatedly appear in the page

For example, the two giveaways shown in the page are the same. (tested in this demo)

Update indices in default code - to adhere to new convention

Update all instances of fairplay to fairplay_v1 and reindex the testnet.

Document how to restart the `~/startup2.sh` when the config.ini is updated.

If a new static ABI is added to the config.ini, the ~/startup2.sh will need to re-run the harvester so that the new ABI can be read into the program and threaded accordingly.

There must be a process for this from a system PID perspective as well as documentation on how to do this.

We will also need to facilitate, the removal of a contract if an ABI is removed from the config.ini

Add search API methods into web3-lity.js

We need a unified JavaScript library for all Second State products. The blockchain RPC library is web3-lity, and I suggest we add the search engine web services into it as well.

Allow for destructed contracts

Show only the last known state of a contract, which has since been destructed.
Display a flag which shows that the contract has been destructed.

Enable HTTPS

Update configuration and documentation to facilitate https at the Apache2 (port 80) endpoint where both the browser UI and machine2machine API reside.

Add bytecode to configuration

The search engine will fetch the bytecode from the original transaction and compare this with the bytecode in the config. If the two bytecodes match then the search engine will provide a URL to the official bytecode.
The same is done for the ABI.

Find the most robust method of rejoining an RPC endpoint

Update io.py to use config.ini for index=""

This can wait until the new architecture which will see the format of contract name and version change (when the "all" index is in full effect)

Store cached function data against ABI & address (as a hash) rather than contract instance address only

Some contract instances will have multiple ABIs i.e. contracts with their actual ABI and then internal ABI association (a smaller ERC20 ABI inside a larger contract). When indexing these contract instances with nested ABIs we need a way to map the contract data (the result of calling all of the pure/view functions) with the contract instance. This can be achieved by creating a unique key (a hash of the contract instance address and also the appropriate ABI)

Facilitate multiple harvesters

The idea behind this is as follows

• shard the work load so that each of them harvest a few ABIs
• Have an "in_progress" field in the "all" index (where the ABIs are stored) which indicates whether that unique ABI is currently being processed by an individual harvester
• At the beginning of all harvesting, each of the harvesters can grab one unprocessed ABI (from the "all" index), sleep for a few seconds and then go back for seconds and thirds and so on
• If there is only one harvester it will eventually just end up with all of the ABIs (keeping in mind that it alone will process these via multi-threading so this will be quite efficient anyway)
• If there are many harvesters they will all have a percentage of the overall available unique ABIs to harvest
• With regards to the harvester going offline, a simple watcher script (JS or python) could test the epoch against current time and switch the "in-progress" field from true to false
• This means of course that the harvester would periodically update the epoch field to ensure that the watcher did not set the "in-progress" field to false

Error harvesting Ethereum contracts with blank name in the output

This is an example of a blank name which is present in Maker contract ABI

{'constant': True, 'inputs': [], 'name': 'owner', 'outputs': [{'name': '', 'type': 'address'}], 'payable': False, 'stateMutability': 'view', 'type': 'function'}

Note the 'name': '' in the outputs list.

The above ABI entry causes the following error

web3.exceptions.BadFunctionCallOutput: Could not decode contract function call name return data b'' for output_types ['bytes32']

Update harvest.py to include new DApp versioning fields

Ensure that the harvest.py now includes dappVersion (which the secondStateJS.js can reference when constructing the DApp endpoint URL).

Consider adding a "Play" feature to the FairPlay frontend

It is possible to get the current date/time from the end user's browser.

// Current date object
var d = new Date(); 

// Epoch i.e. 1557016841960
d.getTime();

// Human readable version i.e. Sun May 05 2019 10:40:41 GMT+1000 (Australian Eastern Standard Time)
var currentDateTime = new Date(d.getTime()); 

Once we have this we can compare the current time with the giveaway's expiry time.

if (currentDateTime > endDate){
    console.log("Giveaway complete");
} else if(currentDateTime < endDate){
    console.log("Giveaway is still running");
}

In cases where the giveaway is still running, we could have a collapsable section called "Play" which would allow the user to execute the smart contract's play function.

The web page could simply default to the account which is injected into the browser (the end user's Venus instance).

Changes to variables in contract versions

Have to ensure that all data is indexed dynamically.
For example we are now using
theStatus = functionData['info'][0]
As apposed to either
functionData['info']["0"]
or
functionData['status']

Events - provide the event logs of a particular contract

Do this in a general / generic way so that it can be portable across all contracts.
Ultimately we are showing "who" did "what" at "which" time.

Having the event logs for the life of the contract allows for additional features in the frontend such as displaying usage over time, trends data visualization and so forth.

Consider adding a checkbox to index.html (Display only current contracts)

We now have a new index attribute called "complete" this could be used to filter search engine results in the frontend (in the event that a user only wants to see currently active product giveaways.)

This will require index.html and secondStateJS.js update.

Change the way we store player_addrs and winner_addrs

If we store the player_addrs and winner_addrs in a list (or as Elasticsearch calls it an ES Array) then we can dynamically ask the list if it contains a certain value (an address).
At present the way we store multiple values means that we have to create an ES query that explicitly asks every item (in the field with multiple values) if it contains the address in question. Whilst this is acceptable when there are only 50 or 100 players/winners, it will not be an ideal solution if the numbers increase into many hundreds or thousands.

Once the data structure is changed to use an internal list, the frontend code can be adjusted to suit the new data structure.

I will update the necessary code and then also document how to interact with an internal list so that others who are querying the API can use the new and improved solution.

Javascript config

At present all of the Python configuration is inside the config.ini, this is really neat.

The Javascript needs a similar system. It may be possible for the Javascript to call the Python (via Flask and fetch the config which would keep everything in one place)

Allow object to be passed into harvest.py (full, topup and state)

The harvest.py script will eventually access the "all" index. The "all" index which is created by the harvest_all.py script is a master list of every contract (regardless of whether an ABI is present). Users can upload ABIs to the "all" index via an interface (which will validate the ABI safely). Once the ABI is available for a particular contract instance, the search engine can call all of the public functions and create a rich index of the state of the contract. This not mean that the search engine has to process every contract though. In the cases where a particular instance of the search engine just needs to harvest 1/2 dozen contracts it can still query the "all" index for the authoritative information such as contract address, ABI, Bytecode etc. but the search engines query does not have to return everything; just what it wants. This can be achieved by passing in a query object like this

{"query": {"match": {"contractAddress": "0x0fEB15a0F7029b0F4aF355aa2Fc3CFa6Df7C8483"}}}'

Add message when no results are returned

The results section of the search page will need to display a simple message in the event that no results are returned i.e. "Sorry, no results found for that query"

Create entirely separate process for status update

We are adding a feature which sets a flag (should the record in the index be constantly updated or has the still-alive contract reached a point where there is no longer any interaction)

This needs to be a separate process because weaving this into the main indexing means that the main indexing is not flexible enough for all contracts (restricted to that particular contracts variable names etc.)

The reality is that this task can be done using Javascript or Python and can run on a separate lightweight system.

For the giveaway contracts the logic would be something like this

doc = {}
outerData = {}
theStatus = "get this from ES (return the status and the itemid so we can perform an update)"
if theStatus == 0:
    outerData['requiresUpdating'] = "yes"
elif theStatus == 1:
    outerData['requiresUpdating'] = "no"
doc["doc"] = outerData
indexResult = self.updateDataInElastic(_esIndex, itemId, json.dumps(doc))

Update harvest.py to process config.ini abi for DApp versioning

Create new code to parse the key of each abi
When given the following ...

[abis]
fairplay_v1 = https://raw.githubusercontent.com/CyberMiles/smart_contracts/master/FairPlay/v1/dapp/FairPlay.abi

The harvest.py will know that the index name is "fairplay" and that the version to be stored is "v1"

Create a tool (Python script) that can produce the contract version hash

We hash the ABI of a contract (abiSha3)
We also hash the Bytecode of a contract (bytecodeSha3)
We combine these to create a unique identifier (abiSha3BytecodeSha3)

This is a great way to deterministically identify a unique instance of a contract (in cases where different unique contract instances can share the same ABI but have different Bytecode etc.)

Whilst the search engine does all of this automatically as part of its normal operation, it would also be great to be able to generate the hashes manually. This would be useful for frontend developers who want to know what value to set as the dappVersion in their display etc.

Create a more robust abiSha3 hash

It is possible that ABIs, which are uploaded to the search engine, could have their functions and events in different orders.
Whilst the check for ABI compatibility would still pass, the abiSha3 hash would be different.

We already strip out tabs and extra spaces etc. to ensure that hashes are more consistent and reliable, however it would be a really good idea to also sort the keys in the ABI data structure so that we can formulate a more robust ABI hash.

Config language analyzer in ElasticSearch

We will need to support semi-natural language queries like this:

https://qbox.io/blog/tutorial-search-plurals-analyzers-elasticsearch

Configure the io.py file to use config.ini (AWS ES credentials)

Provide simultaneous/parallel updating of all search engine data

The idea is to allow all ABIs in the config to be processing in parallel.
Further, for each of the contract instances (which are an instantiation of the ABIs master contract) are to be processed in parallel.
Even further, each of the subsets of data inside a contract instance (i.e. one of the individual fields such as "player_addrs" etc) are to be processed in parallel.

Update harvest.py to include flag for outdated contracts

Tasks:

• 1. Ensure harvestFull and harvestTopup will set a new index attribute called "complete" to True or False, based on the existing smart contract's "status" value.
• 2. Ensure that the harvestStateUpdate functionality only fetches records where "complete" is set to False.
• 3. Ensure that the harvestStateUpdate will update the index attribute called "comeplete" to True or False, based on the current/existing smart contract's "status" value.

If the status of a contract reflects that the product giveaway has been drawn and that no other activity will ever take place, we need some data to reflect this. Otherwise the software which updates the indices will continue to recheck contract[s] which are not ever going to update. This will improve efficiency and speed.

Suggest an attribute such as complete=True/complete=False

The harvestFull and harvestTopup will set complete to True or False during the first time the smart contract instance is indexed (it will always be set) depending on the contract instances "status" variable.

status of 0 -> complete = False
status of 1 -> complete = True

The harvestStateUpdate will only fetch records from the index which have complete set to False. During the update the harvestStateUpdate will adjust the value of "complete" accordingly (again based on the status)

Discuss how the interface will allow users to search different indexes

Different ABIs are different contracts.
The data for each unique contract is indexed in its unique ES Index.

Would like to find out what users would prefer, check boxes to search different indexes (different versions of the contract) or just a completely separate html file (which is skinned slightly differently to make it obvious that the user is using another version of the DApp)

https://github.com/second-state/smart-contract-search-engine/blob/master/js/secondStateJS.js#L3

Create Python utilities for common/shared functions

At present, there are a few Python files which perform the harvesting tasks. There is obviously overlap (operations common to all files) and as such the Python code needs to be altered to adhere to the DRY principle.
The best way to do this is to create a Python utilities file, which is called by the existing files. This way none of the original file names (which are currently set in cron jobs etc.) will change.

Create documentation on how to query API

Users can visit the search engine and search using words and addresses. In addition to this machines can also query the search engine programmatically.

For example
Curl
Using curl in the command line

curl -X GET "http://54.252.157.165/api/data1" -H 'Content-Type: application/json' -d' {"query": {"match": {"contractAddress": "0x0fEB15a0F7029b0F4aF355aa2Fc3CFa6Df7C8483"}}}'

Javascript

_data = {
    "query": {
        "match_all": {}
    }
}
var _dataString = JSON.stringify(_data);
$.ajax({
    url: "http://54.252.157.165/api/data2",
    type: "POST",
    data: _dataString,
    dataType: "json",
    contentType: "application/json",
    success: function(response) {
        console.log(response);
    },
    error: function(xhr) {
        console.log("Get items failed");
    }
});

For example the Javascript code above will return the following

Provide check for wallet's network setting

Provide an alert if the user is on a search engine configured for the mainnet but their Chrome browser extension is connected to the testnet etc.

Document how to deploy the io.py file (Flask)

At present this is being started in the command line

python3.6 python/io.py

This needs to be automated (including checks when/if Flask ever stops)

Allow user to remove one of their own indices from the search engine

Document setting up a two or three node Elasticsearch cluster

This will provide a way for users to create their own search engine instance using free and open source software (as an option, instead of having to use AWS for example)