cloudwan / gohan

Gohan is an API Gateway Server written by Go that makes it easy for developers to create and maintain REST-style API

Gohan : API gateway server

Gohan is an API Gateway Server written by Go that makes it easy for developers to create and maintain REST-style API. An API Gateway Server has a benefit consolidating various API operations such as authentication, authorization based on policies, logging and input validation on a single place on top of so-called microservice architecture.

Gohan also makes transactional operations, involving multiple micro services, easy to operate. An approach Gohan using is quite simple. Persistent REST API Resources in the RDBMS using transaction, then sync resource status with backend microservices using etcd or MySQL binlog API. Using well-proven RDBMs transaction, we can protect correctness of resources. A strategy let backend microservices sync with correct resource data in the RDBMS makes entire system fault-torrent from various RPC failures. Note that Developers should design _ backend microservices idempotent manner, to handle the cases the same RPC invoked multiple times.

see [Pet Store Example] (./etc/example_schema.yaml)

Documentation

Gitbook

• Installation
• Development
• Schema
• Namespace
• Database
• Policy
• Extension
• JavaScript Extension
• Gohan Script Extension
• CLI
• Legacy features

gohan's Issues

Support OAUTH

Support OAUTH for authentication

Support watch API

Support watch API such as etcd provides.

go-etcd get DEPRECATED

We need to upgrade etcd client

Set DB SetMaxIdleConns

We need to configure the max connection for mysql in the db using
db.SetMaxIdleConns(N)

Support to set config parameter from env variable

Support to set config parameter from env variable

Apply go template with environment parameter.
like this.

address: {{ .GOHAN_IP}}:{{ .GOHAN_PORT}}

etcd v3 api support

Support for etcd v3 api ( this will be supposed in Jan/2016)

Using this version, we can remove ugly event table implementation.

• Keep the event history until compaction
• Replace CAS/CAD with multi-object Txn
• Pagenation
• protobuf

https://github.com/coreos/etcd/blob/master/Documentation/rfc/v3api.md

We may also support mode for etcd only mode since now it's supporting transaction.

Customize sync key per resources

We will support customizing sync key with template support.

sync_key: "{{.resource.node_id}}/server/{{.resource.id}}"

Usecase: Scheduling

Each node will watch certain path
/node1/servers/server_1
/node2/servers/server_1

In this case we will add this in schema config

sync_key: "{{.resource.node_id}}/server/{{.resource.id}}"

client crashes on empty/unset environment variables

Errors from environment variable parsing are ignored, causing the client to crash when no/invalid values are provided

Problematic cases:
One of GOHAN_{REGION,SCHEMA_URL,SERVICE_NAME} unset or empty
One of GOHAN_{OUTPUT_FORMAT,VERBOSITY} (from #77) having an invalid value

Testing for the problem:

cmd=(gohan client event list)
for i in GOHAN_{CACHE_{PATH,SCHEMAS,TIMEOUT},REGION,SCHEMA_URL,SERVICE_NAME}
do
    echo "--------- Unset $i:"
    env -u $i $cmd
    echo
    echo "--------- $i"='"":'
    env $i="" $cmd
    echo
done

# Variables introduced in #77
for i in GOHAN_{VERBOSITY,OUTPUT_FORMAT}
do
    echo "--------- Unset $i:"
    env -u $i $cmd
    echo
    echo "--------- $i"='<null>:'
    env $i='<null>' $cmd
    echo
    echo "--------- $i=wrong:"
    env $i=wrong $cmd
    echo
done

use go-bindata

We can embed base extensions & schemas using go-bindata
https://github.com/jteeuwen/go-bindata

Support import in JS extension

Support import in JS extension

Schema inheritance

Allow inherit schema in order to write JSON schema efficiently.

schemas:
  - id: base
    type: abstract
    schema:
   ...

- schemas
  - id: network
  - extends:  # list of abstract schemas
     -
     - 

When we found this, we will do this when we load schemas.

• merge properties
• append propertiesOrder in the order of with

Control value displayed in list view for relationships

When editing the schema for an item, you can define something like the following:

description: Device Type
permission:
  - create
  - update
relation: device_type
title: Device Type
type: string
id: device_type_id
required: false

The schema device references a relation to the device_type schema, which provides a drop down of Device Types to select from in the edit view.

However, in the list view, you see the device_type.id value. It would be preferred to control what is shown in the list view, ideally showing the device_type.description field (or similar), rather than the UUID generated for the device_type entry.

I found the relation_property setting, and tried several values such as name, device_type_name, etc and nothing modified the output in the list view.

I believe this is an enhancement request, but it would make the output significantly more user friendly.

Error handling for etcd channel creation

We need to handle failure of etcd channel creation to prevent infinite loop.

Support cascade on delete per each property

Support cascade on delete per each property

Broken schema handling

When I've made a schema where in one of the fields type is missing Gohan exit with error:
panic: runtime error: invalid memory address or nil pointer dereference
which is not very helpful.

I think Gohan should print some more meaningful error like "property type is missing from field x on schema y". In current situation on complicated schema is may take quite a while to find where the bug was.

Support transaction type for mysql backend

Support to specify transaction type

We support following transaction level.

READ UNCOMMITTED
READ COMMITTED
REPEATABLE READ
SERIALIZABLE

isolation_level:
   read:  REPEATABLE READ
   create:  SERIALIZABLE
   update: SERIALIZABLE
   delete: SERIALIZABLE

Remove unnecessary transaction from sync

Now, gohan_db_* extensions support to start transactions.
We need to remove unnecessary transaction from sync

export swagger api def

export swagger api def
http://swagger.io/

Parent relationship order matters

When defining a schema, if you wish to add a parent relationship later, the order matters when the GUI is being created.

After creating a new schema registration where the parent object was device, the ability to add a new registration entry to the `device was not possible.

When the parent object was added, it was added as the last object in the object list for that schema. Once I used the schema builder to drag the device_id value to the top of the list, then I had the ability to add entries to the registration schema.

It appears the propertiesOrder is what changed to cause the GUI to provide the input mechanisms.

Working

properties:
  auth_pass:
    description: Authorization Password
    permission:
      - create
      - update
    title: Auth Pass
    type: string
  auth_user:
    description: Authorization User
    permission:
      - create
      - update
    title: Auth User
    type: string
  callerid_name:
    description: CallerID Name
    permission:
      - create
      - update
    title: CID Name
    type: string
  callerid_num:
    description: CallerID Number
    permission:
      - create
      - update
    title: CID Number
    type: string
  device_id:
    description: parent object
    permission:
      - create
      - update
    relation: device
    title: device
    type: string
    unique: false
  id:
    description: ID
    permission:
      - create
    title: ID
    type: string
    view:
      - detail
  label:
    description: Line Label Display
    permission:
      - create
      - update
    title: Label
    type: string
  name:
    description: Name
    permission:
      - create
      - update
    title: Name
    type: string
  tenant_id:
    description: Tenant ID
    permission:
      - create
    title: Tenant ID
    type: string
    view:
      - detail
propertiesOrder:
  - device_id
  - tenant_id
  - id
  - name
  - auth_user
  - auth_pass
  - label
  - callerid_name
  - callerid_num
required:
  - device_id
type: object

Build using gb

This is a feature request / enhancement (and I do plan to look into it and see about submitting a pull request), but the current way of building is slightly awkward. I think it might be better to install this through the gb tool to handle dependency gathering.

https://github.com/constabulary/gb

WebUI login fails when trying to use keystone

I'm trying to setup authentication with the WebUI via keystone, but it just fails with a Error:Data Not Found.

When starting the server, a connection to Keystone is successful (if I use the wrong password for example, I get a CRITICAL and gohan fails to start; when everything is correct, gohan starts as it should).

Here is the console output when trying to login to gohan with Keystone enabled:

14:57:44.666 gohan.server.middleware INFO  Started POST /v2.0/tokens for client 127.0.0.1:40542 data: {"auth":{"passwordCredentials":{"username":"demo","password":"welcome"},"tenantName":"demo"}}
14:57:44.666 gohan.server.middleware DEBUG  Request headers: map[Content-Length:[93] Accept:[application/json, text/javascript, */*; q=0.01] Origin:[https://localhost:9443] User-Agent:[Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36] Content-Type:[application/json] Dnt:[1] Cookie:[org.cups.sid=c524dea0ac5489f42457c22ff312ab19; _ga=GA1.1.279487755.1432391029] Connection:[keep-alive] Referer:[https://localhost:9443/webui/] Accept-Encoding:[gzip, deflate] Accept-Language:[en-GB,en-US;q=0.8,en;q=0.6] X-Requested-With:[XMLHttpRequest]]
14:57:44.666 gohan.server.middleware DEBUG  Request body: {"auth":{"passwordCredentials":{"username":"demo","password":"welcome"},"tenantName":"demo"}}
14:57:44.666 gohan.server.middleware DEBUG  Response headers: map[Access-Control-Allow-Origin:[*] Access-Control-Allow-Headers:[X-Auth-Token, Content-Type] Access-Control-Allow-Methods:[GET,PUT,POST,DELETE] Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
14:57:44.666 gohan.server.middleware DEBUG  Response body: 404 page not found
14:57:44.666 gohan.server.middleware INFO  Completed 404 Not Found in 364.995µs

And here is the relevant connection information:

# keystone configuraion
keystone:
    use_keystone: true
    fake: false
    auth_url: "http://localhost:35357/v2.0"
    user_name: "admin"
    tenant_name: "admin"
    password: "welcome"
    version: "v2.0"

On start up, if I use tshark to look at the HTTP traffic, I see a successful authentication. However, when logging into gohan, I don't see any attempt to connect Keystone at all.

update broken by relation_property

updating an instance of relation_property enabled schema doesn't succeed.

detail is here. https://github.com/jbking/gohan-failure-relation-property-schema

PUT should accept resource creation

PUT should accept resource creation according to RFC

The PUT method requests that the enclosed entity be stored under the supplied Request-URI. If the Request-URI refers to an already existing resource, the enclosed entity SHOULD be considered as a modified version of the one residing on the origin server.
http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html

Weak Dockerfile make build broken

Today, we were going to build a gohan container on the fly.

Sadly, it failed. By make test through make all
The main problem is etcd. It merged a PR 9 hours ago.

etcd-io/etcd@1f97f2d#diff-8663bb0f53797ae80ef14f3e44668a55

It makes etcd crash on bootstrap by a panic. Then a gohan's test failed.

2016-01-08 09:40:32.106330 I | etcdserver: published {Name:default ClientURLs:[http://localhost:2379 http://localhost:4001]} to cluster 7e27652122e8b2ae
2016-01-08 09:40:32.106948 I | etcdserver: setting up the initial cluster version to 2.3
2016-01-08 09:40:32.108872 N | etcdserver: set the initial cluster version to 2.3
panic: interface conversion: net.Conn is *netutil.limitListenerConn, not *net.TCPConn

goroutine 69 [running]:
github.com/coreos/etcd/pkg/transport.(*keepaliveListener).Accept(0xc2081290e0, 0x0, 0x0, 0x0, 0x0)
        /go/src/github.com/coreos/etcd/pkg/transport/keepalive_listener.go:53 +0xe3
net/http.(*Server).Serve(0xc208041a40, 0x2ba95391e860, 0xc2081290e0, 0x0, 0x0)
        /usr/local/go/src/net/http/server.go:1728 +0x92
github.com/coreos/etcd/etcdmain.serveHTTP(0x2ba95391e860, 0xc2081290e0, 0x2ba95391f868, 0xc20810b4e0, 0x0, 0x0, 0x0)
        /go/src/github.com/coreos/etcd/etcdmain/http.go:39 +0x1be
github.com/coreos/etcd/etcdmain.func·006(0x2ba95391e860, 0xc2081290e0)
        /go/src/github.com/coreos/etcd/etcdmain/etcd.go:326 +0x80
created by github.com/coreos/etcd/etcdmain.startEtcd
        /go/src/github.com/coreos/etcd/etcdmain/etcd.go:327 +0x25a8

goroutine 1 [chan receive]:
github.com/coreos/etcd/etcdmain.Main()
        /go/src/github.com/coreos/etcd/etcdmain/etcd.go:194 +0x18d1
main.main()
        /go/src/github.com/coreos/etcd/main.go:29 +0x1b

goroutine 9 [chan receive]:
github.com/coreos/etcd/pkg/logutil.(*MergeLogger).outputLoop(0xc20801da80)
        /go/src/github.com/coreos/etcd/pkg/logutil/merge_logger.go:174 +0x89
created by github.com/coreos/etcd/pkg/logutil.NewMergeLogger
        /go/src/github.com/coreos/etcd/pkg/logutil/merge_logger.go:92 +0xc9

goroutine 17 [syscall]:
os/signal.loop()
        /usr/local/go/src/os/signal/signal_unix.go:21 +0x1f
created by os/signal.init·1
        /usr/local/go/src/os/signal/signal_unix.go:27 +0x35

goroutine 18 [chan receive]:
github.com/coreos/etcd/Godeps/_workspace/src/github.com/golang/glog.(*loggingT).flushDaemon(0x1248620)
        /go/src/github.com/coreos/etcd/Godeps/_workspace/src/github.com/golang/glog/glog.go:879 +0x78
created by github.com/coreos/etcd/Godeps/_workspace/src/github.com/golang/glog.init·1
        /go/src/github.com/coreos/etcd/Godeps/_workspace/src/github.com/golang/glog/glog.go:410 +0x2a7

goroutine 33 [syscall, locked to thread]:
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:2232 +0x1

goroutine 46 [chan receive]:
github.com/coreos/etcd/pkg/logutil.(*MergeLogger).outputLoop(0xc20810a480)
        /go/src/github.com/coreos/etcd/pkg/logutil/merge_logger.go:174 +0x89
created by github.com/coreos/etcd/pkg/logutil.NewMergeLogger
        /go/src/github.com/coreos/etcd/pkg/logutil/merge_logger.go:92 +0xc9

goroutine 50 [select]:
github.com/coreos/etcd/raft.(*node).run(0xc20803a960, 0xc2081081c0)
        /go/src/github.com/coreos/etcd/raft/node.go:275 +0xe8e
created by github.com/coreos/etcd/raft.StartNode
        /go/src/github.com/coreos/etcd/raft/node.go:178 +0x4b6

goroutine 51 [select]:
github.com/coreos/etcd/etcdserver.(*EtcdServer).run(0xc208034680)
        /go/src/github.com/coreos/etcd/etcdserver/server.go:556 +0x4a1
created by github.com/coreos/etcd/etcdserver.(*EtcdServer).start
        /go/src/github.com/coreos/etcd/etcdserver/server.go:429 +0x44c

goroutine 53 [select]:
github.com/coreos/etcd/etcdserver.(*EtcdServer).purgeFile(0xc208034680)
        /go/src/github.com/coreos/etcd/etcdserver/server.go:440 +0x4bc
created by github.com/coreos/etcd/etcdserver.(*EtcdServer).Start
        /go/src/github.com/coreos/etcd/etcdserver/server.go:406 +0x79

goroutine 54 [select]:
github.com/coreos/etcd/etcdserver.monitorFileDescriptor(0xc208040ea0)
        /go/src/github.com/coreos/etcd/etcdserver/metrics.go:81 +0x4d8
created by github.com/coreos/etcd/etcdserver.(*EtcdServer).Start
        /go/src/github.com/coreos/etcd/etcdserver/server.go:407 +0x9a

goroutine 55 [select]:
github.com/coreos/etcd/etcdserver.(*EtcdServer).monitorVersions(0xc208034680)
        /go/src/github.com/coreos/etcd/etcdserver/server.go:1193 +0x3eb
created by github.com/coreos/etcd/etcdserver.(*EtcdServer).Start
        /go/src/github.com/coreos/etcd/etcdserver/server.go:408 +0xb4

goroutine 56 [select]:
github.com/coreos/etcd/etcdserver/etcdhttp.capabilityLoop(0xc208034680)
        /go/src/github.com/coreos/etcd/etcdserver/etcdhttp/capability.go:68 +0x1ad
created by github.com/coreos/etcd/etcdserver/etcdhttp.NewClientHandler
        /go/src/github.com/coreos/etcd/etcdserver/etcdhttp/client.go:61 +0x58

goroutine 58 [select]:
github.com/coreos/etcd/etcdserver.func·002()
        /go/src/github.com/coreos/etcd/etcdserver/raft.go:139 +0xb35
created by github.com/coreos/etcd/etcdserver.(*raftNode).start
        /go/src/github.com/coreos/etcd/etcdserver/raft.go:196 +0x180

goroutine 59 [select]:
github.com/coreos/etcd/etcdserver.func·005()
        /go/src/github.com/coreos/etcd/etcdserver/server.go:506 +0x563
created by github.com/coreos/etcd/etcdserver.(*EtcdServer).startApplier
        /go/src/github.com/coreos/etcd/etcdserver/server.go:524 +0x14a

goroutine 60 [select]:
github.com/coreos/etcd/pkg/fileutil.func·001()
        /go/src/github.com/coreos/etcd/pkg/fileutil/purge.go:72 +0x658
created by github.com/coreos/etcd/pkg/fileutil.PurgeFile
        /go/src/github.com/coreos/etcd/pkg/fileutil/purge.go:78 +0x231

goroutine 61 [select]:
github.com/coreos/etcd/pkg/fileutil.func·001()
        /go/src/github.com/coreos/etcd/pkg/fileutil/purge.go:72 +0x658
created by github.com/coreos/etcd/pkg/fileutil.PurgeFile
        /go/src/github.com/coreos/etcd/pkg/fileutil/purge.go:78 +0x231

goroutine 66 [IO wait]:
net.(*pollDesc).Wait(0xc20810dd40, 0x72, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc20810dd40, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).accept(0xc20810dce0, 0x0, 0x2ba95391b278, 0xc208129330)
        /usr/local/go/src/net/fd_unix.go:419 +0x40b
net.(*TCPListener).AcceptTCP(0xc2080381f0, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/tcpsock_posix.go:234 +0x4e
net.(*TCPListener).Accept(0xc2080381f0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/tcpsock_posix.go:244 +0x4c
github.com/coreos/etcd/pkg/transport.(*rwTimeoutListener).Accept(0xc20812a680, 0x0, 0x0, 0x0, 0x0)
        /go/src/github.com/coreos/etcd/pkg/transport/timeout_listener.go:44 +0x7a
net/http.(*Server).Serve(0xc208041920, 0x2ba95391e7f0, 0xc20812a680, 0x0, 0x0)
        /usr/local/go/src/net/http/server.go:1728 +0x92
github.com/coreos/etcd/etcdmain.serveHTTP(0x2ba95391e7f0, 0xc20812a680, 0x2ba95391f750, 0xc20812db30, 0x45d964b800, 0x0, 0x0)
        /go/src/github.com/coreos/etcd/etcdmain/http.go:39 +0x1be
github.com/coreos/etcd/etcdmain.func·005(0x2ba95391e7f0, 0xc20812a680)
        /go/src/github.com/coreos/etcd/etcdmain/etcd.go:318 +0x5a
created by github.com/coreos/etcd/etcdmain.startEtcd
        /go/src/github.com/coreos/etcd/etcdmain/etcd.go:319 +0x245c

goroutine 67 [IO wait]:
net.(*pollDesc).Wait(0xc20810de20, 0x72, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc20810de20, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).accept(0xc20810ddc0, 0x0, 0x2ba95391b278, 0xc208129360)
        /usr/local/go/src/net/fd_unix.go:419 +0x40b
net.(*TCPListener).AcceptTCP(0xc2080381f8, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/tcpsock_posix.go:234 +0x4e
net.(*TCPListener).Accept(0xc2080381f8, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/tcpsock_posix.go:244 +0x4c
github.com/coreos/etcd/pkg/transport.(*rwTimeoutListener).Accept(0xc20812a7e0, 0x0, 0x0, 0x0, 0x0)
        /go/src/github.com/coreos/etcd/pkg/transport/timeout_listener.go:44 +0x7a
net/http.(*Server).Serve(0xc208041980, 0x2ba95391e7f0, 0xc20812a7e0, 0x0, 0x0)
        /usr/local/go/src/net/http/server.go:1728 +0x92
github.com/coreos/etcd/etcdmain.serveHTTP(0x2ba95391e7f0, 0xc20812a7e0, 0x2ba95391f750, 0xc20812db30, 0x45d964b800, 0x0, 0x0)
        /go/src/github.com/coreos/etcd/etcdmain/http.go:39 +0x1be
github.com/coreos/etcd/etcdmain.func·005(0x2ba95391e7f0, 0xc20812a7e0)
        /go/src/github.com/coreos/etcd/etcdmain/etcd.go:318 +0x5a
created by github.com/coreos/etcd/etcdmain.startEtcd
        /go/src/github.com/coreos/etcd/etcdmain/etcd.go:319 +0x245c

goroutine 68 [IO wait]:
net.(*pollDesc).Wait(0xc20810df00, 0x72, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc20810df00, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).accept(0xc20810dea0, 0x0, 0x2ba95391b278, 0xc208129390)
        /usr/local/go/src/net/fd_unix.go:419 +0x40b
net.(*TCPListener).AcceptTCP(0xc208038200, 0xa869c0, 0x0, 0x0)
        /usr/local/go/src/net/tcpsock_posix.go:234 +0x4e
net.(*TCPListener).Accept(0xc208038200, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/tcpsock_posix.go:244 +0x4c
github.com/coreos/etcd/Godeps/_workspace/src/golang.org/x/net/netutil.(*limitListener).Accept(0xc20812a960, 0x0, 0x0, 0x0, 0x0)
        /go/src/github.com/coreos/etcd/Godeps/_workspace/src/golang.org/x/net/netutil/listen.go:30 +0x9e
github.com/coreos/etcd/pkg/transport.(*keepaliveListener).Accept(0xc208128fb0, 0x0, 0x0, 0x0, 0x0)
        /go/src/github.com/coreos/etcd/pkg/transport/keepalive_listener.go:49 +0x71
net/http.(*Server).Serve(0xc2080419e0, 0x2ba95391e860, 0xc208128fb0, 0x0, 0x0)
        /usr/local/go/src/net/http/server.go:1728 +0x92
github.com/coreos/etcd/etcdmain.serveHTTP(0x2ba95391e860, 0xc208128fb0, 0x2ba95391f868, 0xc20810b4e0, 0x0, 0x0, 0x0)
        /go/src/github.com/coreos/etcd/etcdmain/http.go:39 +0x1be
github.com/coreos/etcd/etcdmain.func·006(0x2ba95391e860, 0xc208128fb0)
        /go/src/github.com/coreos/etcd/etcdmain/etcd.go:326 +0x80
created by github.com/coreos/etcd/etcdmain.startEtcd
        /go/src/github.com/coreos/etcd/etcdmain/etcd.go:327 +0x25a8

goroutine 70 [chan receive]:
github.com/coreos/etcd/pkg/osutil.func·001()
        /go/src/github.com/coreos/etcd/pkg/osutil/interrupt_unix.go:51 +0x68
created by github.com/coreos/etcd/pkg/osutil.HandleInterrupts
        /go/src/github.com/coreos/etcd/pkg/osutil/interrupt_unix.go:72 +0x1f6
Running Suite: Server Suite
===========================
Random Seed: 1452246034 - Will randomize all specs
Will run 39 of 39 specs
•[It] should work
  /go/src/github.com/cloudwan/gohan/server/server_test.go:581
[AfterEach] Server package test
  /go/src/github.com/cloudwan/gohan/server/server_test.go:65

------------------------------
• Failure [0.008 seconds]
Server package test
/go/src/github.com/cloudwan/gohan/server/server_test.go:922
  Sync
  /go/src/github.com/cloudwan/gohan/server/server_test.go:582
    should work [It]
    /go/src/github.com/cloudwan/gohan/server/server_test.go:581

    Expected success, but got an error:
        <*etcd.EtcdError | 0xc209452990>: {
            ErrorCode: 501,
            Message: "All the given peers are not reachable",
            Cause: "Tried to connect to each peer twice and failed",
            Index: 0,
        }
        501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]

    /go/src/github.com/cloudwan/gohan/server/server_test.go:551

    Full Stack Trace
    /go/src/github.com/cloudwan/gohan/server/server_test.go:551 (0x511125)
        func.050: Expect(server.Sync()).To(Succeed())
    /go/src/github.com/cloudwan/gohan/server/server_suite_test.go:50 (0x4f8071)
        TestServer: RunSpecs(t, "Server Suite")
    /usr/local/go/src/testing/testing.go:447 (0x4d913f)
        tRunner: test.F(t)
    /usr/local/go/src/runtime/asm_amd64.s:2232 (0x4a4401)
        goexit:

etcd has responsibility for it. But in gohan's Dockerfile, it could use stable version of etcd.
(I agree golang lacks version specification for go get, but you can clone stable version of etcd into your environment before)

This is not only for etcd. gohan's Dockerfile has some go get dependencies.

FSM Support

Finite state machine is a one of generic pattern in resource management application.
We would like to support this as a kind of metaschema.
state management schema can be used to generate doc & UI & assert illegal update.

This enforcement will be applied on model layer so we can apply this enforcement in cascading model update.

state_management:
 schemas: # list of schema ID
 - $SCHEMA_1
 - $SCHEMA_2
 property_id:  $STATUS # status property
 states:
   $STATUS_NAME: # 
      $ACTION: # allowed action on this status  ( create / update / delete )
          - $NEXT_STATUS1  # list of allowed next status
          - $NEXT_STATUS2

Note a special state "NONE" means it isn't exist in DB.

Example
We can have state_machine in property

 states_machine:
  NONE:
    create:
    - CREATE_IN_PROGRESS
    - ERROR
  ACTIVE:
    update:
    - REBOOTING:
    - UPDATE_IN_PROGERSS
    - ERROR
    delete:
    - DELETE_IN_PROGRESS
    - ERROR
  CREATE_IN_PROGRESS:
    update:
    - ACTIVE
    - ERROR
  UPDATE_IN_PROGRESS:
    update:
    - ACTIVE
    - ERROR
  DELETE_IN_PROGRESS:
    delete:
    - NONE
  REBOOTING:
    update:
    - ACTIVE
    - ERROR
  ERROR:
    delete:
    - DELETE_IN_PROGRESS
    - ERROR
    - FATAL_ERROR
    update:
    - UPDATE_IN_PROGRESS

References:

• FSM https://en.wikipedia.org/wiki/Finite-state_machine

support graceful restart

We can use this lib https://github.com/facebookgo/grace

Deleting schema with data linked to a parent causes crash

When deleting a schema that has data associated with it, which has a parent object, the FOREIGN KEY constraint fails, and causes gohan to crash.

Log:

13:08:07.777 gohan.schema INFO  Loading schema ./provisioning_schema.yaml ...
13:08:07.779 gohan.schema WARNING  Overwriting schema device_type
13:08:07.779 gohan.schema WARNING  Overwriting schema server
13:08:07.780 gohan.schema WARNING  Overwriting schema customer
13:08:07.781 gohan.schema WARNING  Overwriting schema device
13:08:07.782 gohan.db CRITICAL  Error during deleting table:FOREIGN KEY constraint failed

It does seem to delete the schema and remove it from the YAML file, however the database constraint fails, and causes the crash to happen.

Support id pool property

We need to manage pool such as IP address, vlan id allocation and RT allocation.

test

test

etcd watch process should handle 401 error

etcd watch process should handle 401 error.

What's happening in our env.

(1) resources more than 500 rows
(2) boot gohan
(3) state watch process will lock for each
(4) etcd stores up to 1000 history, so we lost some history and got 401 error

Redesign Donburi

• Implement Donburi by pure Go code.
• Jinja2 like expression so that we can make more similar style with Ansible

It looks like we can use Pongo2 lib.
https://github.com/flosch/pongo2

It supports Jinja2 style template & advanced style expressions
https://github.com/flosch/pongo2/blob/master/template_tests/expressions.tpl

Remove licensing check

Now, we OSSed the Gohan, so the current licencing check in Make file won't fit for 3rd party PR.

Sort sub-graph of schemas

We may do something even if we have loops in schemas.
We can divide subgraph of schemas using Flood fill algorithm, then we can apply topological sorting for each subgraph.

Support is_owner_user condition

Support is_owner_user condition

Currently, we only support is_owner which actually means is_owner_tenant.
We will add new condition which limit it to a user.

V8 Support

We manage status of V8 support in this issue.

V8 is 100x faster than Otto JS VM and 10x times slower than pure go.
However, we can only do async callback to invoke go code from JavaScript in v8 (using ry's branch), and this means we need rewrite most of our extension code.

Initial implementation is done, and you can use it when you enable V8.

How to make with v8 support

You need have a env this package works. (see instructions)
https://github.com/ry/v8worker

ENABLE_V8=true make

Support consul for node management & distributed lock

Support consul for node management & distributed lock

register extension for specific schema

add schema_id

util/GetContent forcely obtain content on same path from embeded asset even prefixed with 'file://' scheme

Imagine, the usage in https://github.com/cloudwan/gohan/pull/96/files#diff-7b5720c18b4d0cecba0091d848458b0fR683

etc/templates/swagger.tmpl is bundled in gohan's asset on same path. So, the template in asset is chosen, is not chosen cli argument specified one.
AND even with 'file://' scheme, util/GetContent cut it off before finding. So the template in asset is chosen.

Its okay preferring bundled asset on same path. (Informing about it might be needed.)
However with 'file://' explicitly, it should be obtained from filesystem instead of asset.

Can gohan_http post multipart/form-data?

This is a question about gohan_http which is javascript built-in function.

Is it possible to upload file with gohan_http?
It seems to be able to send only json document.
https://github.com/cloudwan/gohan/blob/master/extension/otto/gohan_util.go#L186

If there are other ways to post file, could you tell me it?

Thank you

Show only child properties once selecting a parent object

Ran into an issue where we're trying to show two drop downs on a page, one of them being a list of parent objects, and the other being the list of child objects to the parent.

The issue is that the second drop down doesn't have a way (that I've found) that allows you to show only child objects for the parent object in the other drop down. Here is an example:

So I have the following schemas:

• Manufacturers
  • Models
  • Firmware (to be added)

Manufacturers is the parent to models (and firmware). When adding a Device to a Customer, the idea was that you would select the Manufacturer, the Model, and then the Firmware. The two dropdowns for Model and Firmware would be populated based on the Manufacturer that is selected.

Currently, all values in the child drop downs show.

Manufacturer

properties:
  description:
    description: Description
    permission:
      - create
      - update
    title: Description
    type: string
  id:
    description: ID
    permission:
      - create
    title: ID
    type: string
    view:
      - detail
  name:
    description: Name
    permission:
      - create
      - update
    title: Name
    type: string
    unique: true
  tenant_id:
    description: Tenant ID
    permission:
      - create
    title: Tenant ID
    type: string
    view:
      - detail
propertiesOrder:
  - id
  - name
  - description
  - tenant_id
required: null
type: object

Model

properties:
  id:
    description: ID
    permission:
      - create
    title: ID
    type: string
    view:
      - detail
  manufacturer_id:
    description: parent object
    permission:
      - create
    relation: manufacturer
    title: manufacturer
    type: string
    unique: false
  name:
    description: Name
    permission:
      - create
      - update
    title: Name
    type: string
  tenant_id:
    description: Tenant ID
    permission:
      - create
    title: Tenant ID
    type: string
    view:
      - detail
propertiesOrder:
  - id
  - name
  - tenant_id
  - manufacturer_id
required:
  - manufacturer_id
type: object

Test Object with Manufacturer and Model

properties:
  id:
    description: ID
    permission:
      - create
    title: ID
    type: string
    view:
      - detail
  manufacturer:
    description: Manufacturer
    permission:
      - create
      - update
    relation: manufacturer
    relation_property: manufacturer
    title: Manufacturer
    type: string
  model:
    description: Model
    permission:
      - create
      - update
    relation: model
    relation_property: model
    title: Model
    type: string
  tenant_id:
    description: Tenant ID
    permission:
      - create
    title: Tenant ID
    type: string
    view:
      - detail
propertiesOrder:
  - id
  - manufacturer
  - model
  - tenant_id
required: null
type: object

Stop extension env on end of API call

Stop extension env on end of API call
otherwise garbage extension may keep running even after API call finished

PANIC on non-admin schema access

We got PANIC on non-admin schema access

[martini] PANIC: runtime error: invalid memory address or nil pointer dereference
/usr/local/go/src/runtime/panic.go:423 (0x40e5719)

Support SAML auth

Support SAML auth for identity management

may be, we can use this package
https://github.com/mattbaird/gosaml

Support conditions by property value in policy

Support conditions by property status value in policy

policy:
  - action: 'read'
    condition:
    - property:
        status:
        - ACTIVE
        - CREATE_IN_PROGRESS
        - UPDATE_IN_PROGRESS
        - DELETE_IN_PROGRESS
        - ERROR
    effect: allow
    id: member
    principal: Member
  - action: 'update'
    condition:
    - property:
        status:  
           ACTIVE: ERROR
    effect: allow
    id: member
    principal: Member
  - action: 'reboot'
    condition:
    - property:
        status: ACTIVE
    effect: allow
    id: member
    principal: Member
  - action: 'delete'
    condition:
    - property:
        status: 
        - ACTIVE
        - ERROR
    effect: allow
    id: member
    principal: Member

Support to generate migration diff from two schema files

Generate migration diff from two schema files

Support PATCH

Some applications such as Keystone uses PATCH request.

We should be able to support PATCH request.
https://tools.ietf.org/html/rfc5789

Allow editing extensions & policy from API & Webui

Allow editing extensions & policy from API & Webui.
I believe this helps local development

make fails with Go1.6

networkRed,NetworkRed)'
SIGQUIT: quit
PC=0x46ac11 m=0

goroutine 0 [idle]:
runtime.futex(0xfbb0c8, 0x0, 0x0, 0x0, 0x0, 0xfba450, 0x0, 0x0, 0x418c24, 0xfbb0c8, ...)
/usr/local/go/src/runtime/sys_linux_amd64.s:302 +0x21
runtime.futexsleep(0xfbb0c8, 0x0, 0xffffffffffffffff)
/usr/local/go/src/runtime/os1_linux.go:40 +0x53
runtime.notesleep(0xfbb0c8)
/usr/local/go/src/runtime/lock_futex.go:145 +0xa4
runtime.stopm()
/usr/local/go/src/runtime/proc.go:1535 +0x10b
runtime.findrunnable(0xc82001b500, 0x0)
/usr/local/go/src/runtime/proc.go:1973 +0x739
runtime.schedule()
/usr/local/go/src/runtime/proc.go:2072 +0x24f
runtime.park_m(0xc82016d800)
/usr/local/go/src/runtime/proc.go:2137 +0x18b
runtime.mcall(0x7ffffcf20170)
/usr/local/go/src/runtime/asm_amd64.s:233 +0x5b

goroutine 1 [chan receive, 9 minutes]:
testing.RunTests(0xbc2c40, 0xf98990, 0x1, 0x1, 0xc820082d01)
/usr/local/go/src/testing/testing.go:583 +0x8d2
testing.(*M).Run(0xc820039f08, 0xfb9fe0)
/usr/local/go/src/testing/testing.go:515 +0x81
main.main()
github.com/cloudwan/gohan/db/_test/_testmain.go:104 +0x253

goroutine 17 [syscall, 9 minutes, locked to thread]:
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1998 +0x1

goroutine 5 [syscall, 9 minutes]:
os/signal.signal_recv(0x0)
/usr/local/go/src/runtime/sigqueue.go:116 +0x132
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
/usr/local/go/src/os/signal/signal_unix.go:28 +0x37

goroutine 6 [semacquire, 9 minutes]:
sync.runtime_Semacquire(0xc8200eaaa4)
/usr/local/go/src/runtime/sema.go:47 +0x26
sync.(_Mutex).Lock(0xc8200eaaa0)
/usr/local/go/src/sync/mutex.go:83 +0x1c4
database/sql.(_Tx).Rollback(0xc8202772c0, 0x0, 0x0)
/usr/local/go/src/database/sql/sql.go:1266 +0x6b
github.com/cloudwan/gohan/db/sql.(_Transaction).Close(0xc82048e980, 0x0, 0x0)
/home/ubuntu/go/src/github.com/cloudwan/gohan/db/sql/sql.go:711 +0x48
github.com/cloudwan/gohan/db_test.glob.func1.3.3()
/home/ubuntu/go/src/github.com/cloudwan/gohan/db/db_test.go:133 +0x32
github.com/onsi/ginkgo/internal/leafnodes.(_runner).runSync(0xc820061020, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/home/ubuntu/go/src/github.com/onsi/ginkgo/internal/leafnodes/runner.go:109 +0x9b
github.com/onsi/ginkgo/internal/leafnodes.(_runner).run(0xc820061020, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/home/ubuntu/go/src/github.com/onsi/ginkgo/internal/leafnodes/runner.go:63 +0xa0
github.com/onsi/ginkgo/internal/leafnodes.(_SetupNode).Run(0xc82002a178, 0x2b9c4ac60180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/home/ubuntu/go/src/github.com/onsi/ginkgo/internal/leafnodes/setup_nodes.go:14 +0x52
github.com/onsi/ginkgo/internal/spec.(_Spec).runSample.func1(0xc8204b1720, 0xc8200c43c0, 0x2b9c4ac60180, 0xc8200d6b40)
/home/ubuntu/go/src/github.com/onsi/ginkgo/internal/spec/spec.go:136 +0x1b1
github.com/onsi/ginkgo/internal/spec.(_Spec).runSample(0xc8200c43c0, 0x0, 0x2b9c4ac60180, 0xc8200d6b40)

goroutine 7 [chan receive, 9 minutes]:
github.com/onsi/ginkgo/internal/specrunner.(_SpecRunner).registerForInterrupts(0xc8202b4480)
/home/ubuntu/go/src/github.com/onsi/ginkgo/internal/specrunner/spec_runner.go:175 +0x1a6
created by github.com/onsi/ginkgo/internal/specrunner.(_SpecRunner).Run
/home/ubuntu/go/src/github.com/onsi/ginkgo/internal/specrunner/spec_runner.go:56 +0x66

goroutine 18 [select, 9 minutes, locked to thread]:
runtime.gopark(0xbc33c8, 0xc820022728, 0xabb4c0, 0x6, 0x18, 0x2)
/usr/local/go/src/runtime/proc.go:262 +0x163
runtime.selectgoImpl(0xc820022728, 0x0, 0x18)
/usr/local/go/src/runtime/select.go:392 +0xa67
runtime.selectgo(0xc820022728)
/usr/local/go/src/runtime/select.go:215 +0x12
runtime.ensureSigM.func1()
/usr/local/go/src/runtime/signal1_unix.go:279 +0x358
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1998 +0x1

goroutine 35 [chan receive, 9 minutes]:
database/sql.(*DB).connectionOpener(0xc8200c74a0)
/usr/local/go/src/database/sql/sql.go:727 +0x45
created by database/sql.Open
/usr/local/go/src/database/sql/sql.go:493 +0x33f

goroutine 11 [chan receive, 9 minutes]:
database/sql.(*DB).connectionOpener(0xc8200c7a20)
/usr/local/go/src/database/sql/sql.go:727 +0x45
created by database/sql.Open
/usr/local/go/src/database/sql/sql.go:493 +0x33f

Support rate limiting

Support rate limiting

Schema editor will accept broken schema

Schema editor shouldn't accept broken schema.
We need to verify the schema before accepting it