bootstrapsp / miu

Developing with Hyperledger Indy made easy.

Miu

Miu is a gRPC based python server enabling developers to easily connect and develop Hyperledger Indy based clients using Hyperledger Indy SDK.

Overview

Miu introduces all the goodness of gRPC and writing Indy client in your favourite programing language, the ones supported by gRPC. Extending the list of programing languages in which the Indy SDK is currently provided see libindy SDK

Motivation

Being able to code Indy clients in Golang. :D

Concept

Miu is implemented in gRPC python using the Indy SDK for Python. This now allows us to build Hyperledger Indy SDK based clients using gRPC which can communicate with Miu in high performant Protobuf protocol.

Design Overview

Here's a quick concept overview Miu-Approach

Current state with this release

Current release covers implementation for most of the modules from Indy SDK, precisely speaking :

• Anoncreds
• Blob Storage
• Crypto
• DID
• Ledger
• NonSecret
• Pairwise
• Pool
• Wallet

Project structure

grpc_server folder contains Indy SDK module implementation in python

identityLayer folder contains all the proto files describing the gRPC structure with RPCs and messages

config.json used for configuring the indy node IP address and the pool transaction genesis file.

Note: You'll need to add pool_transactions_genesis file first before configuring in the config.json

Installation and trying out

Pre-req

You'll need Indy node / Server before going through with steps defined below. See How to install Test network.

Once ready use following steps to start with Miu setup

1. Install python 3

sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt-get update
sudo apt-get install python3.6

2. Clone miu repository

3. CD into the location where you have cloned

4. Create virtualenv:

python3 -m venv my_venv

5. Activate virtualenv

    source ./my_venv/bin/activate

6. Install Dependencies

pip install -r requirements.txt

7. Run the miu interface

python server.py

This will start the server on localhost:50051. You then will have to connect your gRPC client to this.

What's next

Of course we'd appreciate help to further enhance this project. Post this alpha release we are looking to work on

• Fixing bugs to stablize this release
• Writing test cases for the Miu interface for automated testing
• Enhanced logging
• And more...

miu's Issues

Multiple 'No positional arguments allowed'.No Handling for specific IndyError

Describe the bug
After performing calls to Indy library and trying to send result back to gRPC client, Miu throws errors, because of not specifying arguments name for protobuf. Also there is no possibility to determine if error is IndyError and send more appropriate error message.
see below
To Reproduce

1. run grpc server Miu
   2.use this code

	conn, err := grpc.Dial("localhost:50051", grpc.WithInsecure())
	if err != nil {
		panic(err)
	}

	//create  poolServiceClient service

	poolServiceClient := identitylayer.NewPoolServiceClient(conn)
	pcr, err := poolServiceClient.CreatePoolLedgerConfig(context.Background(), &identitylayer.CreatePoolLedgerConfigRequest{ConfigName: poolName, Config: &identitylayer.ConfigCreatePoolLedger{GensisTxn: "/path_to_genesis"}})

	if err != nil {
		panic(err)
	}
	fmt.Printf("Pool creation result %d \n", pcr.GetErrorCode())

3. Run gRPC client to connect to Miu
4. Get 'positional argument error'

Expected behavior

1. get 0 as error Code if pool created successfully.
2. get 306 error Code if pool has already been created
3. get Internal error Code in case other error happenned

Screenshots

Desktop (please complete the following information):

• OS: MacOS 10.13.6

Additional context
It's quite common issue in plenty of methods. So PR for IndyError handler and fixing 'no positional arguments allowed' is extremely needed

DidSetKeyMetadata returns incorrect response

Describe the bug
rpc DidSetKeyMetadata (DidSetKeyMetadataRequest) returns (DidSetKeyMetadataRequest) {} should return DidSetKeyMetadataResponse instead of DidSetKeyMetadataRequest.

DidSetKeyMetadataResponse defines the ErrorCode which should be returned as an outcome of setting the KeyMetaData

DidSetKeyMetadataResponse is defined in the proto however not used.

To Reproduce
No steps to reproduce. This is just incorrect definition of RPC method

Expected behavior
Should return DidSetKeyMetadataResponse containing appropriate error code.

gRPC Crypto client exmaple

What this feature is about?
This is to provide Crypto related rpc testing while allowing users of this library to quickly get the idea how client can be built using protofile delivered with this project.

What will be delivered under this module
Deliver gRPC based client side code, preferably written in language other than Python to allow for diversity and giving a chance how client will be written and how will it interact with the server.
This module should provide a gRPC client that invokes almost all the rpcs that are exposed by the server.

Return Wallet / Pool Handler of already opened wallet / pool together with Message 'AlreadyOpened'

Is your feature request related to a problem? Please describe.
When you're attempting to open wallet / pool that has already been opened it's not possible to receive opened entity handle, so it's not possible to use it, even if you have rights.
indy-cli allows to use wallet if it was opened before.

Describe the solution you'd like
We could add simple layer of memory storage for opened Pools and opened Wallets. If wallet / pool is opened - return wallet handle from memory with message that wallet was opened before, of course if user has right credentials to wallet.

example

where openPools is memory dictionary.

Describe alternatives you've considered
List of pools could return configs and handles if they're opened.

Additional context
This feature will help to use the same wallet from several client service instances without storing handle on service side.

Enable TLS connection between server & client

Is your feature request related to a problem? Please describe.
Currently clients connecting to the Miu libary make insecure connection - this is obviously not idea.

Describe the solution you'd like
Using the standard TLS implementation for gRPC, client(s) should be able to make TLS connections to Miu

gRPC NonSecret client example

What this feature is about?
This is to provide NonSecret related rpc testing while allowing users of this library to quickly get the idea how client can be built using protofile delivered with this project.

What will be delivered under this module
Deliver gRPC based client side code, preferably written in language other than Python to allow for diversity and giving a chance how client will be written and how will it interact with the server.
This module should provide a gRPC client that invokes almost all the rpcs that are exposed by the server.

Test Miu on ARM 32/64 platform

Test Miu on ARM 32/64 platforms like Raspi, to find what effort is needed in making it low powered device ready.

Why
This will allow seamless experience on variety of platforms for Miu.

gRPC Ledger example client

What this feature is about?
This is to provide Ledger related rpc testing while allowing users of this library to quickly get the idea how client can be built using protofile delivered with this project.

What will be delivered under this module
Deliver gRPC based client side code, preferably written in language other than Python to allow for diversity and giving a chance how client will be written and how will it interact with the server.
This module should provide a gRPC client that invokes almost all the rpcs that are exposed by the server.

Common IO Error IndyError, when creating configPool

Describe the bug
When attempting to create config pool from gRPC client, Miu throws error, see below
To Reproduce
Steps to reproduce the behavior:

1. Start Miu pythong grpc_server/server.py
2. Provide require input for the gRPC client to be able to create pool, here's e.g. from go based gRPC client

func CreatePoolLedgerConfig(conn *grpc.ClientConn) {
cplc := pb.NewPoolServiceClient(conn)

ctx, cancel := context.WithTimeout(context.Background(), time.Minute)

defer cancel()

createPoolLedgerConf, err := cplc.CreatePoolLedgerConfig(ctx, &pb.CreatePoolLedgerConfigRequest{
	ConfigName: "ScuridPool1",
	Config:     &pb.ConfigCreatePoolLedger{
		GensisTxn: genTx,
	},
})

3. Run gRPC client to connect to Miu on :5001 port
4. See error

DEBUG:indy.pool:create_pool_ledger_config: >>> config_name: 'ScuridPool1', config: '{"genesis_txn": REDACTED
DEBUG:indy.libindy:do_call: >>> name: indy_create_pool_ledger_config, args: (c_char_p(139674895488016), c_char_p(139674752459008), <CFunctionType object at 0x7f089858c5c0>)
DEBUG:indy.libindy:do_call: Function indy_create_pool_ledger_config returned err: 0
DEBUG:indy.libindy:do_call: <<<
INFO:indy.libindy.native.indy.commands: src/commands/mod.rs:133 | PoolCommand command received
INFO:indy.libindy.native.pool_command_executor: src/commands/pool.rs:62 | Create command received
DEBUG:indy.libindy.native.indy.commands.pool: src/commands/pool.rs:138 | create >>> name: "ScuridPool1", config: Some(PoolConfig { genesis_txn: "REDACTED)
DEBUG:indy.libindy:_indy_callback: >>> command_handle: 4, err 114, args: ()
DEBUG:indy.libindy:_indy_callback: <<<
DEBUG:indy.libindy:_indy_loop_callback: >>> command_handle: 4, err 114, args: ()
WARNING:indy.libindy:_indy_loop_callback: Function returned error 114
DEBUG:indy.libindy:_indy_loop_callback <<<
ERROR:pool_service:Exception Occurred @ CreatePoolLedgerConfig ------
ERROR:pool_service:ErrorCode.CommonIOError
ERROR:root:Exception calling application: No positional arguments allowed
Traceback (most recent call last):
File "/home/scurid/goingTheRightSide/miu/pyserverforindy/grpc_server/pool_service.py", line 67, in CreatePoolLedgerConfig
resp = await indy_pool.create_pool_ledger_config(config_name, config)
File "/home/scurid/goingTheRightSide/miu/pyserverforindy/scurid/lib/python3.5/site-packages/indy/pool.py", line 39, in create_pool_ledger_config
create_pool_ledger_config.cb)
File "/usr/lib/python3.5/asyncio/futures.py", line 361, in iter
yield self # This tells Task to wait for completion.
File "/usr/lib/python3.5/asyncio/tasks.py", line 296, in _wakeup
future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
indy.error.IndyError: ErrorCode.CommonIOError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "grpc_server/server.py", line 97, in _call_behavior
return await behavior(argument, context), True
File "/home/scurid/goingTheRightSide/miu/pyserverforindy/grpc_server/pool_service.py", line 72, in CreatePoolLedgerConfig
return identitylayer_pb2.CreatePoolLedgerConfigResponse(resp)
TypeError: No positional arguments allowed

Expected behavior
Should create pool config

Screenshots
-NA-

Desktop (please complete the following information):

• OS: ubuntu 16

Smartphone (please complete the following information):
-NA-

Additional context
-NA-

gRPC DID client example

What this feature is about?
This is to provide DID related rpc testing while allowing users of this library to quickly get the idea how client can be built using protofile delivered with this project.

What will be delivered under this module
Deliver gRPC based client side code, preferably written in language other than Python to allow for diversity and giving a chance how client will be written and how will it interact with the server.
This module should provide a gRPC client that invokes almost all the rpcs that are exposed by the server.

gRPC Pool Client example

What this feature is about?
This is to provide Pool related rpc testing while allowing users of this library to quickly get the idea how client can be built using protofile delivered with this project.

What will be delivered under this module
Deliver gRPC based client side code, preferably written in language other than Python to allow for diversity and giving a chance how client will be written and how will it interact with the server.
This module should provide a gRPC client that invokes almost all the rpcs that are exposed by the server.

Dockerize Miu

What this feature is about?
Provide simple/basic dockerfile for Miu to help quickly run this library.

What will be delivered
Dockerfile with basic gRPC configuration.

gRPC Anoncreds client example

What this feature is about?
This is to provide Anoncreds related rpc testing while allowing users of this library to quickly get the idea how client can be built using protofile delivered with this project.

What will be delivered under this module
Deliver gRPC based client side code, preferably written in language other than Python to allow for diversity and giving a chance how client will be written and how will it interact with the server.
This module should provide a gRPC client that invokes almost all the rpcs that are exposed by the server.

keyJson has incorrect datastructure

Describe the bug
CreateKey method in DID request message DidCreateKeyRequest{
int64 WalletHandle = 1;
string KeyJson = 2;

KeyJson should be a complex dataType with Seed and Crypto as string

To Reproduce
NA

Expected behavior
Proto should have definition :

message DidCreateKeyRequest{
  int64 WalletHandle = 1;
  KeyJson KeyJson = 2;
}

message KeyJson{
string seed = 1;
string crypto = 2;
}

Additional context
There are more than one usage of KeyJson , i.e. there are multiple services like from Crypto, DID that uses KeyJson message. Therefore, individual messages should be created rather than re-using them between services.

This will require proto change as well as update on server and client side.

Invalid Common structure when creating DID

Describe the bug
While attempting to create DID from gRPC client, Miu throws Invalid Common Structure.

@sagar-88 here's the full detailed structure

To Reproduce
Steps to reproduce the behavior:

1. Start miu server
   python grpc_serverserver.py
2. Have a grpc client ready with following input for the CreateAndStoreDID ,
   Following snippet is from a Go client

a, createDIDErr := d.CreateAndStoreMyDid(ctx, &pb.CreateAndStoreMyDidRequest{
WalletHandle: 2, // should be dynamic
DidJson: &pb.CreateAndStoreMyDidJsonMessage{
Did: "Ardhnaarishwar",
Seed: "Demo",
CryptoType: "ed25519",
Cid: false,
},
}

3. Start the client to connect to 5001 port on which server is running
4. See error

DEBUG:indy.did:create_and_store_my_did: >>> wallet_handle: 2, did_json: '{"seed": "Demo", "did": "Ardhnaarishwar", "crypto_type": "ed25519", "cid": null}'
DEBUG:indy.did:create_wallet: Creating callback
DEBUG:indy.libindy:create_cb: >>> cb_type: <class 'ctypes.CFUNCTYPE..CFunctionType'>
DEBUG:indy.libindy:create_cb: <<< res: <CFunctionType object at 0x7f089858c818>
DEBUG:indy.libindy:do_call: >>> name: indy_create_and_store_my_did, args: (c_int(2), c_char_p(139674892393840), <CFunctionType object at 0x7f089858c818>)
DEBUG:indy.libindy:do_call: Function indy_create_and_store_my_did returned err: 0
DEBUG:indy.libindy:do_call: <<<
INFO:indy.libindy.native.indy.commands: src/commands/mod.rs:137 | DidCommand command received
INFO:indy.libindy.native.indy.commands.did: src/commands/did.rs:124 | CreateAndStoreMyDid command received
DEBUG:indy.libindy.native.indy.commands.did: src/commands/did.rs:189 | create_and_store_my_did >>> wallet_handle: 2, my_did_info_json: MyDidInfo { did: Some("Ardhnaarishwar"), seed: Some("Demo"), crypto_type: Some("ed25519"), cid: None }
DEBUG:indy.libindy:_indy_callback: >>> command_handle: 3, err 113, args: (b'', b'')
DEBUG:indy.libindy:_indy_callback: <<<
DEBUG:indy.libindy:_indy_loop_callback: >>> command_handle: 3, err 113, args: (b'', b'')
WARNING:indy.libindy:_indy_loop_callback: Function returned error 113
DEBUG:indy.libindy:_indy_loop_callback <<<
ERROR:did_service:Exception occurred @ CreateAndStoreMyDid----
ERROR:did_service:ErrorCode.CommonInvalidStructure

Expected behavior
Should create a DID

Screenshots
-NA-

Desktop (please complete the following information):

• OS: ubuntu 16

Smartphone (please complete the following information):
-NA-

Additional context
-NA-

gRPC Wallet client example

What this feature is about?
This is to provide Wallet related rpc testing while allowing users of this library to quickly get the idea how client can be built using protofile delivered with this project.

What will be delivered under this module
Deliver gRPC based client side code, preferably written in language other than Python to allow for diversity and giving a chance how client will be written and how will it interact with the server.
This module should provide a gRPC client that invokes almost all the rpcs that are exposed by the server.

ListPools doesn't return list of pools for now.

Describe the bug
ListPools doesn't return list of pools and ListPoolsResponse message has only ErrorCode field.

To Reproduce
Steps to reproduce the behavior:
1.run grpc server Miu
2.use this code

conn, err := grpc.Dial("localhost:50051", grpc.WithInsecure())
	if err != nil {
		panic(err)
	}

//create  poolServiceClient service
poolServiceClient := identitylayer.NewPoolServiceClient(conn)
l2, err := poolServiceClient.ListPools(context.Background(), &identitylayer.ListPoolsRequest{})
	if err != nil {
		panic(err)
	}
	fmt.Printf("Number of pools %d \n", l2.GetErrorCode())

3.Run gRPC client to connect to Miu
4.Get 'ErrorCode' as a result of method execution.

Expected behavior
4. Get list of created pool as indy-cli returns.

Screenshots

Desktop (please complete the following information):

• OS: MacOS 10.13.6

gRPC Pairwise client example

What this feature is about?
This is to provide Pairwise related rpc testing while allowing users of this library to quickly get the idea how client can be built using protofile delivered with this project.

What will be delivered under this module
Deliver gRPC based client side code, preferably written in language other than Python to allow for diversity and giving a chance how client will be written and how will it interact with the server.
This module should provide a gRPC client that invokes almost all the rpcs that are exposed by the server.