hashicorp / go-plugin

Golang plugin system over RPC.

Go Plugin System over RPC

go-plugin is a Go (golang) plugin system over RPC. It is the plugin system that has been in use by HashiCorp tooling for over 4 years. While initially created for Packer, it is additionally in use by Terraform, Nomad, Vault, Boundary, and Waypoint.

While the plugin system is over RPC, it is currently only designed to work over a local [reliable] network. Plugins over a real network are not supported and will lead to unexpected behavior.

This plugin system has been used on millions of machines across many different projects and has proven to be battle hardened and ready for production use.

Features

The HashiCorp plugin system supports a number of features:

Plugins are Go interface implementations. This makes writing and consuming plugins feel very natural. To a plugin author: you just implement an interface as if it were going to run in the same process. For a plugin user: you just use and call functions on an interface as if it were in the same process. This plugin system handles the communication in between.

Cross-language support. Plugins can be written (and consumed) by almost every major language. This library supports serving plugins via gRPC. gRPC-based plugins enable plugins to be written in any language.

Complex arguments and return values are supported. This library provides APIs for handling complex arguments and return values such as interfaces, io.Reader/Writer, etc. We do this by giving you a library (MuxBroker) for creating new connections between the client/server to serve additional interfaces or transfer raw data.

Bidirectional communication. Because the plugin system supports complex arguments, the host process can send it interface implementations and the plugin can call back into the host process.

Built-in Logging. Any plugins that use the log standard library will have log data automatically sent to the host process. The host process will mirror this output prefixed with the path to the plugin binary. This makes debugging with plugins simple. If the host system uses hclog then the log data will be structured. If the plugin also uses hclog, logs from the plugin will be sent to the host hclog and be structured.

Protocol Versioning. A very basic "protocol version" is supported that can be incremented to invalidate any previous plugins. This is useful when interface signatures are changing, protocol level changes are necessary, etc. When a protocol version is incompatible, a human friendly error message is shown to the end user.

Stdout/Stderr Syncing. While plugins are subprocesses, they can continue to use stdout/stderr as usual and the output will get mirrored back to the host process. The host process can control what io.Writer these streams go to to prevent this from happening.

TTY Preservation. Plugin subprocesses are connected to the identical stdin file descriptor as the host process, allowing software that requires a TTY to work. For example, a plugin can execute ssh and even though there are multiple subprocesses and RPC happening, it will look and act perfectly to the end user.

Host upgrade while a plugin is running. Plugins can be "reattached" so that the host process can be upgraded while the plugin is still running. This requires the host/plugin to know this is possible and daemonize properly. NewClient takes a ReattachConfig to determine if and how to reattach.

Cryptographically Secure Plugins. Plugins can be verified with an expected checksum and RPC communications can be configured to use TLS. The host process must be properly secured to protect this configuration.

Architecture

The HashiCorp plugin system works by launching subprocesses and communicating over RPC (using standard net/rpc or gRPC). A single connection is made between any plugin and the host process. For net/rpc-based plugins, we use a connection multiplexing library to multiplex any other connections on top. For gRPC-based plugins, the HTTP2 protocol handles multiplexing.

This architecture has a number of benefits:

• Plugins can't crash your host process: A panic in a plugin doesn't panic the plugin user.

• Plugins are very easy to write: just write a Go application and go build. Or use any other language to write a gRPC server with a tiny amount of boilerplate to support go-plugin.

• Plugins are very easy to install: just put the binary in a location where the host will find it (depends on the host but this library also provides helpers), and the plugin host handles the rest.

• Plugins can be relatively secure: The plugin only has access to the interfaces and args given to it, not to the entire memory space of the process. Additionally, go-plugin can communicate with the plugin over TLS.

Usage

To use the plugin system, you must take the following steps. These are high-level steps that must be done. Examples are available in the examples/ directory.

1. Choose the interface(s) you want to expose for plugins.

2. For each interface, implement an implementation of that interface that communicates over a net/rpc connection or over a gRPC connection or both. You'll have to implement both a client and server implementation.

3. Create a Plugin implementation that knows how to create the RPC client/server for a given plugin type.

4. Plugin authors call plugin.Serve to serve a plugin from the main function.

5. Plugin users use plugin.Client to launch a subprocess and request an interface implementation over RPC.

That's it! In practice, step 2 is the most tedious and time consuming step. Even so, it isn't very difficult and you can see examples in the examples/ directory as well as throughout our various open source projects.

For complete API documentation, see GoDoc.

Roadmap

Our plugin system is constantly evolving. As we use the plugin system for new projects or for new features in existing projects, we constantly find improvements we can make.

At this point in time, the roadmap for the plugin system is:

Semantic Versioning. Plugins will be able to implement a semantic version. This plugin system will give host processes a system for constraining versions. This is in addition to the protocol versioning already present which is more for larger underlying changes.

What About Shared Libraries?

When we started using plugins (late 2012, early 2013), plugins over RPC were the only option since Go didn't support dynamic library loading. Today, Go supports the plugin standard library with a number of limitations. Since 2012, our plugin system has stabilized from tens of millions of users using it, and has many benefits we've come to value greatly.

For example, we use this plugin system in Vault where dynamic library loading is not acceptable for security reasons. That is an extreme example, but we believe our library system has more upsides than downsides over dynamic library loading and since we've had it built and tested for years, we'll continue to use it.

Shared libraries have one major advantage over our system which is much higher performance. In real world scenarios across our various tools, we've never required any more performance out of our plugin system and it has seen very high throughput, so this isn't a concern for us at the moment.

go-plugin's Issues

more examples

Hi,

Would it be possible to see more implementation examples? I'm wrapping my head around on how to create dynamically loaded plugins ;/

plugins left unix socket files after crash

OS: linux

The plugins is abnormally closed, such as SIGKILL, the unix socket file will be left behind, and causing a leak in some case.

why not plugin support "Abstract sockets" to resolve this case.

http://man7.org/linux/man-pages/man7/unix.7.html

Abstract sockets
Socket permissions have no meaning for abstract sockets: the process
umask(2) has no effect when binding an abstract socket, and changing
the ownership and permissions of the object (via fchown(2) and
fchmod(2)) has no effect on the accessibility of the socket.
Abstract sockets automatically disappear when all open references to
the socket are closed.
The abstract socket namespace is a nonportable Linux extension.

python grpc connection fails with commit #52 and beyond

The go to python grpc connection fails with commit #52 and till #51 it is working fine.
This is in relation to project http://github.com/vmware/terraform-provider-vcloud-director

WORKING #51

[root@centos7 go-plugin]# git show --oneline -s
e37881a Allow the server's logger to be passed in via the ServeConfig (#51)
[root@centos7 go-plugin]#

FAILED #52

[root@centos7 go-plugin]# git show --oneline -s
4b3b291 Add a Connection Broker for GRPC plugins (#52)
[root@centos7 go-plugin]# /home/terraform-provider-vcloud-director/go/src/test/test_catalog.sh
2018/01/21 04:50:30 [INFO][PLUGINLOG] __INIT__init
2018/01/21 04:50:30 [INFO][PLUGINLOG] _INIT_TestAccResourceCatalog
=== RUN TestAccResourceCatalogBasic
2018/01/21 04:50:30 [INFO][PLUGINLOG] INIT__LOGGING
2018/01/21 04:50:30 [INFO][PLUGINLOG] _INIT__testAccPreCheck
2018/01/21 04:50:30 [INFO][PLUGINLOG] _DONE__testAccPreCheck
2018-01-21T04:50:30.619-0500 [DEBUG] plugin: starting plugin: path=/usr/bin/sh args="[sh -c python3 /home/terraform-provider-vcloud-director/plugin-python/plugin.py]"
2018/01/21 04:50:30 [INFO][PLUGINLOG] __INIT__func providerConfigure
2018-01-21T04:50:30.620-0500 [DEBUG] plugin: waiting for RPC address: path=/usr/bin/sh
2018-01-21T04:50:31.013-0500 [DEBUG] plugin.sh: INFO:root:Already running / Not starting new server
Error: plugin "PY_PLUGIN" doesn't support gRPC
exit status 1
[root@centos7 go-plugin]#

===============================================================
FAILED #55

[root@centos7 go-plugin]# git show --oneline -s
485ef45 Provide a context for managing the lifecycle of gRPC plugins (#55)
[root@centos7 go-plugin]# /home/terraform-provider-vcloud-director/go/src/test/test_catalog.sh
2018/01/21 04:47:46 [INFO][PLUGINLOG] __INIT__init
2018/01/21 04:47:46 [INFO][PLUGINLOG] _INIT_TestAccResourceCatalog
=== RUN TestAccResourceCatalogBasic
2018/01/21 04:47:46 [INFO][PLUGINLOG] INIT__LOGGING
2018/01/21 04:47:46 [INFO][PLUGINLOG] _INIT__testAccPreCheck
2018/01/21 04:47:46 [INFO][PLUGINLOG] _DONE__testAccPreCheck
2018-01-21T04:47:46.084-0500 [DEBUG] plugin: starting plugin: path=/usr/bin/sh args="[sh -c python3 /home/terraform-provider-vcloud-director/plugin-python/plugin.py]"
2018/01/21 04:47:46 [INFO][PLUGINLOG] __INIT__func providerConfigure
2018-01-21T04:47:46.084-0500 [DEBUG] plugin: waiting for RPC address: path=/usr/bin/sh
2018-01-21T04:47:46.475-0500 [DEBUG] plugin.sh: INFO:root:Already running / Not starting new server
Error: plugin "PY_PLUGIN" doesn't support gRPC
exit status 1
[root@centos7 go-plugin]#

stdout/stderr sync not implemented for gRPC plugins

I might be missing something completely, but it seems that the stdout/stderr sync feature is not implemented for the GRPC server path.

The GRPCServer.Stdout and GRPCServer.Stderr fields are not used anywhere. They are set up however in the same way as in RPCServer in the common codepath in Serve() it's just RPCServer has control over the channel, while in GRPC I guess it is not that obvious how the sync can be implemented. Is my understanding correct?

I think this asymmetry should be documented between netRPC and gRPC! Are there any more differences?

Logs in gRPC plugins can work using the log, hclog packages, they are pointing to the old stderr.

Workaround for getting the output of fmt.Println() for example to stdErr involves saving the old stdErr in a var and using that (for some reason the same trick does not work with stdOut):

var oldOut *os.File
var oldErr *os.File
// Here is a real implementation of KV that writes to a local file with
// the key name and the contents are the value of the key.
type KV struct{}

func (KV) Put(key string, value []byte) error {
	fmt.Fprintf(oldOut,"Hello, I'm the plugin putter.1\n") # this does not work
	fmt.Fprintf(oldErr,"Hello, I'm the plugin putter.2\n") # this works
	fmt.Fprintf(os.Stderr,"Hello, I'm the plugin putter.3\n") # this does not work
	fmt.Fprintf(os.Stdout,"Hello, I'm the plugin putter.4\n") # this does not work
	log.New(os.Stderr, "hihii", 0).Printf("test") # this does not work
	log.Printf("log hello:) - %v %v vs %v %v\n", os.Stdout, os.Stderr, oldOut, oldErr) # this works
	value = []byte(fmt.Sprintf("%s\n\nWritten from plugin-go-grpc", string(value)))
	return ioutil.WriteFile("kv_"+key, value, 0644)
}

func (KV) Get(key string) ([]byte, error) {
	return ioutil.ReadFile("kv_" + key)
}

func main() {
	oldOut = os.Stdout
	oldErr = os.Stderr
	plugin.Serve(&plugin.ServeConfig{
		HandshakeConfig: shared.Handshake,
		Plugins: map[string]plugin.Plugin{
			"kv_grpc": &shared.KVGRPCPlugin{Impl: &KV{}},
		},
		// A non-nil value here enables gRPC serving for this plugin...
		GRPCServer: plugin.DefaultGRPCServer,
	})
}

In comparison the Sync feature does work well in the netRPC plugin when SyncStdout and SyncStderr are set on the plugin.ClientConfig.

panic: runtime error: integer divide by zero while initing the client

2017/10/20 09:54:02  ok starting to client 
2017-10-20T09:54:02.204-0400 [DEBUG] plugin: starting plugin: path=/usr/bin/sh args="[sh -c python3 /home/terraform-provider-vcloud-director/plugin-python/plugin.py]"
2017-10-20T09:54:02.204-0400 [DEBUG] plugin: waiting for RPC address: path=/usr/bin/sh
panic: runtime error: integer divide by zero

goroutine 14 [running]:
google.golang.org/grpc.(*addrConn).resetTransport(0xc4200c4200, 0x0, 0xc42001e780)
	/home/terraform-provider-vcloud-director/go/src/google.golang.org/grpc/clientconn.go:930 +0xf50
google.golang.org/grpc.(*addrConn).connect.func1(0xc4200c4200)
	/home/terraform-provider-vcloud-director/go/src/google.golang.org/grpc/clientconn.go:736 +0x2f
created by google.golang.org/grpc.(*addrConn).connect
	/home/terraform-provider-vcloud-director/go/src/google.golang.org/grpc/clientconn.go:735 +0x400
[root@worker3 test]# 

I see the above error but no specific message why its happening

https://github.com/srinarayanant/terraform-provider-vcloud-director/blob/master/go/src/test/test.go

Above is the link to the source file , used as a test which throws this error.

This was working fine for a while till I had updated rpc interface , but now reverting the changes still the problem persists.

grpc example issue

Following the README in github.com/hashicorp/go-plugin/examples/grpc

$ go version
go version go1.9.3 darwin/amd64

Executing the commands from the README.

# This builds the main CLI
$ go build -o kv

# This builds the plugin written in Go
$ go build -o kv-go-grpc ./plugin-go-grpc

# This tells the KV binary to use the "kv-go-grpc" binary
$ export KV_PLUGIN="./kv-go-grpc"

# Read and write
$ ./kv put hello world

$ ./kv get hello
world

I get the following output

$ ./kv get hello
2018-02-01T13:24:27.656Z [DEBUG] plugin: starting plugin: path=/bin/sh args="[sh -c ./kv-go-grpc]"
2018-02-01T13:24:27.658Z [DEBUG] plugin: waiting for RPC address: path=/bin/sh
2018-02-01T13:24:27.672Z [ERROR] plugin.sh: protocol init: timestamp=2018-02-01T13:24:27.672Z error=""kv" is not a GRPC-compatible plugin"
2018-02-01T13:24:27.674Z [DEBUG] plugin: plugin process exited: path=/bin/sh
Error: plugin exited before we could connect

SIGINT gives panic when parsing log output

When closing the server with CTRL+c, a panic occurs:

panic: interface conversion: interface is float64, not string

goroutine 131 [running]:
panic(0xc524e0, 0xc4209df100)
	/home/bro/programmer/go/go/src/runtime/panic.go:500 +0x1a1
github.com/hashicorp/go-plugin.parseJSON(0xc420445ef0, 0x84, 0x1, 0x1420360, 0xc4209fcbe0)
	/home/bro/programmer/fabriscale/go/ws/src/github.com/hashicorp/go-plugin/log_entry.go:118 +0x4fc
github.com/hashicorp/go-plugin.(*Client).logStderr(0xc4205172d0, 0x1411be0, 0xc420036200)
	/home/bro/programmer/fabriscale/go/ws/src/github.com/hashicorp/go-plugin/client.go:746 +0x277
created by github.com/hashicorp/go-plugin.(*Client).Start
	/home/bro/programmer/fabriscale/go/ws/src/github.com/hashicorp/go-plugin/client.go:544 +0xb3c

'go-getter' integration

I see it on the roadmap, but no other mention. Any chance for this to see daylight?

plugin-go-grpc example keeps creating a new kv-go-grpc sub/process after each run

It seems to commit 0cf1b43, i.e. https://github.com/hashicorp/go-plugin/blob/master/examples/grpc/main.go#L67 causes a new sub/process for kv-go-grpc being created after each run, no matter the run is successful or not.

Also removing os.Exit(0) at L67 results in "[WARN] plugin: plugin failed to exit gracefully".

Please create a release so changes from pull request 111 can be included into vault sdk

Hi.
Please create a release so changes from pull request #111 can be included into vault sdk

Revision 1b0 onward fails to work with vault plugin system

Hey there! When authoring a Vault plugin, if I vendor with any recent version of vault and with revision 1b0f542 or onward of this plugin, the plugin fails to properly communicate with vault.

You can test this by cloning https://github.com/hashicorp/vault-auth-plugin-example, removing the existing vendoring (vendor and Gopkg*) and starting from scratch with this Gopkg.toml:

[[constraint]]
  name = "github.com/hashicorp/vault"
  version = "=1.0.1"

[[override]]
  name = "github.com/hashicorp/go-plugin"
  revision = "1b0f5429182b3157c6f22301a77fde89bab6dbd7"

Failing back to any previous revision and the plugin works as expected. It would appear that the Dial function is failing in:

go-plugin/grpc_broker.go

Line 376 in f444068

return nil, fmt.Errorf("timeout waiting for connection info")

I'm starting here in the event it's not strictly related to vault. Thanks!

Is it possible to extend the life of a plugin function's interface parameter?

When a plugin function accepts an interface argument, supplied by the base process (as in the bidirectional example), it seems that the parameter is only usable until the plugin function returns and after that the grpc connection for that interface closes.

Is there a recommended way to allow that parameter be usable for as long as the plugin is running? So that I can for instance have an Init(service Service) method of the plugin which can be used by the base application to supply an instance of the service interface to the plugin, which can then be called from various go routines after the Init function has returned?

Interrupting parent can result in orphaned plugins hot-looping

Filing this here so we can think about it:

hashicorp/terraform#5043 (comment)

Unfortunately we don't have repro, but we do have one user reporting it.

Need Basic Examples

Hi @mitchellh ,
Actually I have doubts in rpc communication between main application and plugin , it'll be fine if you give some examples.

Adding logging around the exit code when exiting

During debugging of a Terraform run within a container, it's not super clear why a process exited. In the example we had, it was being OOM-killed by the kernel due to memory constraints:

Sep 26 12:01:19 ip-192-168-123-123 kernel: Memory cgroup out of memory: Kill process 2011 (terraform) score 257 or sacrifice child

@radeksimko found that this line seems to be relevant: https://github.com/hashicorp/go-plugin/blob/master/client.go#L571

c.logger.Debug("plugin process exited", "path", cmd.Path)

which meant the logs didn't show the exit process of why the process was killed:

2018-09-26T12:01:48.395Z [DEBUG] plugin: plugin process exited: path=/terraform/.terraform/plugins/linux_amd64/terraform-provider-aws_v1.37.0_x4

It would be beneficial to have an exit code here as well, as having 137 show would help diagnose a SIGTERM due to the OOM killer

Double close of DoneCh in rpc_server

2017/02/13 22:38:36 [DEBUG] plugin: nomad: panic: close of closed channel
2017/02/13 22:38:36 [DEBUG] plugin: nomad: 
2017/02/13 22:38:36 [DEBUG] plugin: nomad: goroutine 81 [running]:
2017/02/13 22:38:36 [DEBUG] plugin: nomad: panic(0x107ae00, 0xc4202b09d0)
2017/02/13 22:38:36 [DEBUG] plugin: nomad: 	/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/panic.go:500 +0x1a1
2017/02/13 22:38:36 [DEBUG] plugin: nomad: github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(*controlServer).Quit(0xc42008e588, 0x1, 0x1a27ec8, 0x0, 0x0)
2017/02/13 22:38:36 [DEBUG] plugin: nomad: 	/home/travis/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/rpc_server.go:118 +0x5a
2017/02/13 22:38:36 [DEBUG] plugin: nomad: reflect.Value.call(0xc420020960, 0xc42008e590, 0x13, 0x11f9fcb, 0x4, 0xc420466ef0, 0x3, 0x3, 0xc420260601, 0xc4202606a0, ...)
2017/02/13 22:38:36 [DEBUG] plugin: nomad: 	/home/travis/.gimme/versions/go1.7.linux.amd64/src/reflect/value.go:434 +0x5c8
2017/02/13 22:38:36 [DEBUG] plugin: nomad: reflect.Value.Call(0xc420020960, 0xc42008e590, 0x13, 0xc420466ef0, 0x3, 0x3, 0x1, 0xc4202e6780, 0x20003)
2017/02/13 22:38:36 [DEBUG] plugin: nomad: 	/home/travis/.gimme/versions/go1.7.linux.amd64/src/reflect/value.go:302 +0xa4
2017/02/13 22:38:36 [DEBUG] plugin: nomad: net/rpc.(*service).call(0xc420211740, 0xc420211700, 0xc42034aab0, 0xc420098800, 0xc420355760, 0xff8160, 0xc4202b0987, 0x181, 0xfe42e0, 0x1a27ec8, ...)
2017/02/13 22:38:36 [DEBUG] plugin: nomad: 	/home/travis/.gimme/versions/go1.7.linux.amd64/src/net/rpc/server.go:383 +0x148
2017/02/13 22:38:36 [DEBUG] plugin: nomad: created by net/rpc.(*Server).ServeCodec
2017/02/13 22:38:36 [DEBUG] plugin: nomad: 	/home/travis/.gimme/versions/go1.7.linux.amd64/src/net/rpc/server.go:477 +0x421
2017/02/13 22:38:36 [DEBUG] plugin: /tmp/nomad-test.UjY/nomad: plugin process exited

Unfortunately do not have reproduction steps or full stack trace as I saw this in travis.

Missing license

I'm checking the project and I don't find any license. Please, can you license it?

Be able to just implement a gRPC only pluginmap

Can't implement pluginmap as a map of grpcplugins

var PluginMap = map[string]plugin.GRPCPlugin

Have to use map[string]plugin.Plugin

and implement client and server funcs for both grpc flavor and rpc flavor.

Consider Apache License?

Would you ever consider moving to an Apache license and/or dual licensing?

host and plugin on different machines?

Could you please clarify for me if go-plugin can be use in a way that the host application and the plugins are on different machines?
Since it uses gRPC or net/rpc, I would expect that it should be doable.
But the net/rpc connection is forced to use unix or 127.0.0.1 on Windows machine

Callback when a plugin dies

Hey all, I'm trying to make the system I'm building more stable, so I'm at the point where my app should know when a plugin crashes, in order to change a state variable.

Apparently there's no mechanism for getting the event (still it's logged), so I use at the moment the Ping() function of the RPC client.

Is there a way to get the plugin lifecycle events to make things a bit more clean? A callback like OnModuleKill would be perfect.
Thanks

feature-request: do not pass the whole os.Environ to plugin

Hey ppl :-D
Thanks for great system!
Have a feature request, on line https://github.com/hashicorp/go-plugin/blob/master/client.go#L523, the host is passing the whole set of the environment variables to the plugins.
I'd like to have the ability to skip it (for security reasons).

Need to be able to attach a debugger to subprocesses.

See #20466 in terraform.

I'm working on some changes to rpc_client.go and client.go to implement this.

GRPCBroker's Dial timeout

Hey
I'm posting after trying for hours, the plugin structure is working fine, but setting up the bidirectional communication for my program is proving to be a real challenge.

I think I understand how this works, but probably I'm missing something important. Basically I've started the plugin, and extracted the broker to use it in the main code (host), it looks like this:

	// Assign the implementation, calls to this are RPC now
	test := raw.(sdk.Test_rpc_container)

	// Bootstrap the connections to the module?
	h := HeartbeatService{}
	hbServiceServer := &sdk.GRPCHBServiceServer{Impl: &h}

	var svr *grpc.Server
	serverFunc := func(opts []grpc.ServerOption) *grpc.Server {
		svr = grpc.NewServer(opts...)
		proto.RegisterHeartbeatServer(svr, hbServiceServer)

		return svr
	}

	go test.Broker.AcceptAndServe(1, serverFunc)

so the Broker should be listening to connections on channel 1. It's hardcoded, yes.

And in the plugin I have this:

func (s *FuncRPCServer) Call(ctx context.Context, req *proto.CallRequest) (*proto.CallResponse, error) {

	conn, err := s.broker.Dial(1)
	if err != nil {
		return nil, errors.New("I HAZ FAIL")
	}

This setup is very close to the bidirectional example, the RPC server creates the channel once the function Call gets called, yet it goes into timeout after 5 secs failing here.

So... this is the point where I ask for help, as the documentation doesn't seem to describe anything particular about this.

Please tell me if you need more details. Thanks for the useful library.

Thanks

Do I need to recompile the host progrma if I write a new plugin?

the plugins map is hard encoded in the client code,is that means I need to recompile the host(client) program if i want to install a new plugin ?

Logging and SIGPIPE when host process dies

Nomad uses go-plugin to spin up various plugins and auxiliary processes, and saw surprising (to us) behavior when host process dies in hashicorp/nomad#5598 .

Nomad uses go-plugin to spin up long-running plugins with lifecycle independent from host, to ease in-place upgrades and reconfiguration, and use the reattachment patterns ReattachConfig supported by this.

However, we observe the following problems after host process is restarted:

1. The plugin gets a SIGPIPE signal upon the next log/Stdout/Stderr write operation. When the host (e.g. go-plugin client) process dies, Stdout/Stderr pipe closes and any write from plugin fails with io.ErrClosedPipe error, and the plugin receive SIGPIPE, typically killing it.

   • note that if plugin explicitly ignores SIGPIPE, hclog may panics on log write failure in https://github.com/hashicorp/go-hclog/blob/6907afbebd2eef854f0be9194eb79b0ba75d7b29/intlogger.go#L370-L373

2. On successful reattachment by a restarted host process, stdout/stderr syncing is lost, and any plugin log lines to Stdout/Stderr are lost.

Nomad works around this by having a dedicated log file for the plugin and not writing to the plugin Stderr in hashicorp/nomad#5598 .

Ideally, go-plugin can makes handling host process restarting and re-attaching better. One possibility might be using fifo files such that plugin can always write to it with some buffer, but this may require clever use of non-blocking flags (to ensure plugin can proceed when fifo buffer is full).

Report serverListener_tcp() errors more clearly

I'm trying to run the system on Windows, and the way it uses to connect to plugins is TCP. I see that the functions refer to two environment variables, PLUGIN_MIN_PORT and PLUGIN_MAX_PORT. If they are undefined, a clueless error is returned.

I kind of understand why they have to be declared as environment variables, it makes good sense, though the error took me some time to find out what it actually was. Please report the absence of these variables more clearly. Or make it configurable in the code.

Thanks

Plain English Protocol Documentation

I might be missing some detail, but I'm having a really difficult time following the code to try and understand the protocol used here. I'm not a Go-lang developer, and I'm trying to implement the protocol used in node for that very reason.

I've gotten as far as the initial handshake and then the code just loses all meaning to me. Too many levels of indirection to other libraries. It might have more meaning to someone more familiar with the Go stack.

Here's what I've been able to follow so far:

• Host spawns/forks process for plugin.
• Plugin checks environment variables for "Magic Cookie" to ensure it's hosted by the parent application.
• Plugin listens for unix/tcp socket in somewhat random location/port.
• Plugin reports over stdout to host with message formatted like:
  • BaseProtocolVersion|AppPluginProtocolVersion|SocketProtocol|SocketAddress

After that I'm lost in multiplexers, redirected stdio streams, and go-specific network implementations. Once the plugin process starts listening on tcp/unix domain socket, I presume that the host application makes a connection to the socket. Who talks next, and what is the protocol?

Is there a flowchart or block diagram somewhere? Anything that describes the control flow for this protocol?

I don't expect a 100-page manual, but some next steps would be helpful. I'd be happy to produce some documentation for other people making this attempt if someone can help me get over this part of the learning curve.

reading plugin stderr read |0: file already closed

I'm seeing intermittent errors being logged from go-plugin with a message of "reading plugin stderr", and an error of "read |0: file already closed". This seems to happen when plugins are terminated.

In looking at https://golang.org/src/os/exec/exec.go?s=17633:17682#L601, and client.go, it seems like there may be a race condition where the reader side of the pipe is closed after the command exits, but the go-plugin client is still trying to read from it. The client's logStderr is looking for an io.EOF error, but in this case I think it's actually getting some variant of the os.ErrClosed error. I'm not familiar enough with the code to understand exactly what's going on/what should be changed, though. Hope this can point someone in the right direction!

Vendored interface types

I have a plugin that wants to use types defined in the host application. The plugin vendors the code from the host application. I'm getting an error at runtime:

reading body gob: name not registered for interface "github.com/me/plugin/vendor/github.com/me/hostapp/plugins.MyInterface"

I am calling gob.Register for the types that implement the interface, but since I'm vendoring they get registered under a different type name and the gob decoding fails.

Are you aware of any projects that have used go-plugin in this manner? I know I can either:

• stop vendoring and build against code in my GOPATH, or
• only use concrete types between the host/plugins instead of interfaces

I just wanted to check to see if others have run into this came up with a different solution.

AutoMTLS does not work with brokering from plugin to host.

With AutoMTLS=true, brokered connections dialed from the plugin back to the host fail with transport: authentication handshake failed: x509: certificate is valid for localhost, not unused.

Additionally, brokering with AutoMTLS requires the use of broker.AcceptAndServe since the autogenerated TLS configuration is not exported and therefore cannot be used to set up a gRPC server to use with the listener returned from broker.Accept.

CleanupClients() does not use hclog

The CleanupClients method is not defined on a type, and thus does not have direct access to a configured logger. It logs a single line using stdlib log:

go-plugin/client.go

Line 235 in a5174f8

log.Println("[DEBUG] plugin: waiting for all plugin processes to complete...")

This is problematic because 1) it is not structured or formatted as we would expect it to be, and 2) it goes to stdout regardless of the actual logger configuration.

The package does manage a list of instantiated clients, each of which has a logger defined on its struct. I can think of two solutions:

• A series of "waiting" and "stopped" messages, specific to each plugin, logged through its own logger
• Pick the first client, if any, and log the single message through there.

Thoughts on the best option? Perhaps you can think of other options? Happy to send a patch assuming we are aligned

import all grpc modules error

when I tried to build the example code with command "go build -o ./plugin/greeter ./plugin/greeter_impl.go", then I got these error info:

go: google.golang.org/[email protected]: unrecognized import path "google.golang.org/genproto" (https fetch: Get https://google.golang.org/genproto?go-get=1: dial tcp 216.239.37.1:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.) go: golang.org/x/[email protected]: unrecognized import path "golang.org/x/sync" (https fetch: Get https://golang.org/x/sync?go-get=1: dial tcp 216.239.37.1:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.) go: golang.org/x/[email protected]: unrecognized import path "golang.org/x/text" (https fetch: Get https://golang.org/x/text?go-get=1: dial tcp 216.239.37.1:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.) go: golang.org/x/[email protected]: unrecognized import path "golang.org/x/sys" (https fetch: Get https://golang.org/x/sys?go-get=1: dial tcp 216.239.37.1:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.) go: golang.org/x/[email protected]: unrecognized import path "golang.org/x/net" (https fetch: Get https://golang.org/x/net?go-get=1: dial tcp 216.239.37.1:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.) go: google.golang.org/[email protected]: unrecognized import path "google.golang.org/grpc" (https fetch: Get https://google.golang.org/grpc?go-get=1: dial tcp 216.239.37.1:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.) go: error loading module requirements

Bidirectional communication

Quote:

Bidirectional communication. Because the plugin system supports complex arguments, 
the host process can send it interface implementations and the plugin can call back
into the host process.

Any example of how can we approach this? Do we need separate (g)rpc connection for that? Is there any project on github that uses this kind of functionality? Also, is it somehow possible to use one plugin from another?

Handling plugin crashes

Hello. I would like to know what is the recommended way of handling plugin crashes (i.e. the plugin process shutting down). I noticed go-plugin doesn't restart the process in this situation, so I was wondering if you are considering adding this feature, and if you are not then I would like to know the rationale behind that decision. Initially my idea was to verify the plugin status before any call, and start it if the process is down, but I wanted to check with you guys first in case I'm missing something. Thanks!

Passing TLS Certificates to plugin

go-plugins can communicate over TLS, but there is no way to pass the TLS certificate to the plugin binary when it is spawned. Afaik the approach to do this currently is wrapping the certificate and key in Vault and passing the wrapped token through an environment variable, but that seems overkill to me. Would passing this over stdout when spawning the plugin be a good alternative? I'm happy to try upgrading the protocol.

Did you miss commit the GRPCBroker

Expose method for detecting if you're inside a plugin

in packer we have a line that looks like

inPlugin := os.Getenv(plugin.MagicCookieKey) == plugin.MagicCookieValue

It would be great if this could be exposed through go-plugin so we can take different actions depending on if we're in a plugin or the main body

Can we version this project?

This way i can lock Go app from changes at this module

Are channels supported in a plugin interface?

I want a plugin interface to return a channel the host can listen on for asynchronous notifications (and vice versa). Is this supported?

TestClient_syncStreams fails randomly

As reported in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=897504
TestClient_syncStreams fails as follows:

--- FAIL: TestClient_syncStreams (0.10s)
        rpc_client_test.go:59: bad:
stdouttest^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@

https://tests.reproducible-builds.org/debian/rb-pkg/buster/amd64/golang-github-hashicorp-go-plugin.html

Basic example seems to be broken (at least on Mac)

I the main program and here is what I'm getting:

bash-3.2$ ./basic
2018-07-05T00:32:05.780+0500 [DEBUG] plugin: starting plugin: path=./plugin/greeter args=[./plugin/greeter]
2018-07-05T00:32:05.784+0500 [DEBUG] plugin: waiting for RPC address: path=./plugin/greeter
2018-07-05T00:32:05.870+0500 [DEBUG] plugin.greeter: message from plugin: foo=bar timestamp=2018-07-05T00:32:05.868+0500
2018-07-05T00:32:05.871+0500 [DEBUG] plugin.greeter: plugin address: address=/var/folders/4v/3yrsz17s1m9bqlj35bgqvyvr0000gn/T/plugin847884065 network=unix timestamp=2018-07-05T00:32:05.871+0500
2018-07-05T00:32:05.877+0500 [DEBUG] plugin.greeter: message from GreeterHello.Greet: timestamp=2018-07-05T00:32:05.876+0500
Hello!
2018-07-05T00:32:05.877+0500 [DEBUG] plugin.greeter: 2018/07/05 00:32:05 [ERR] plugin: plugin server: accept unix /var/folders/4v/3yrsz17s1m9bqlj35bgqvyvr0000gn/T/plugin847884065: use of closed network connection
2018-07-05T00:32:05.878+0500 [DEBUG] plugin: plugin process exited: path=./plugin/greeter

Is "use of closed network connection" expected here? I tried running bidirectional example and there is no problem.

Unable to get go-plugin/GRPC flavor working in container.

Problem first happened with my teams project, but recreated the issue using the go-plugins example to be sure.

Steps:

Container built FROM golang:latest

run interactively

1  go get github.com/hashicorp/go-plugin
2  cd src/github.com/hashicorp/go-plugin/examples/grpc/
3  go build -o kv
4  go build -o kv-go-grpc ./plugin-go-grpc
5  export KV_PLUGIN="./kv-go-grpc"
6  ./kv put hello world

root@b93305f57ca2:/go/src/github.com/hashicorp/go-plugin/examples/grpc# ./kv put hello world
2017-10-02T22:51:35.213Z [DEBUG] plugin: starting plugin: path=/bin/sh args="[sh -c ./kv-go-grpc]"
2017-10-02T22:51:35.214Z [DEBUG] plugin: waiting for RPC address: path=/bin/sh
2017-10-02T22:51:35.220Z [DEBUG] plugin.sh: plugin address: timestamp=2017-10-02T22:51:35.220Z address=/tmp/plugin040334221 network=unix

*It just hangs here indefinitely. (Same behavior as in my current project).

If, I do the same for the basic example and it runs fine.

Thanks!

io.Reader example

Hey, is there any sample code, how to handle io.Reader? Especially when they are a return value of a plugin function.

could you please provide a simple doc

as the title, thank you

defer client.Kill() - terraform provider clean up

Hi Team ,
is there a recommended way to hook the defer client.Kill() to a terraform providers clean up method .

I could use the [func providerConfigure(d *schema.ResourceData) (interface{}, error) ] to initialize the plugin , but will need to kill the client plugin after all the terraform operations are done

Thanks & Regards,
Sri

net/rpc only plugin without grpc bloat

Is it possible to create a net/rpc only implementation (both host and plugin) without bloated by gRPC dependencies?
WIll the binaries always have all modules listed go.sum file?

plugin talk to host

I was able to split the basic example into 2 binaries, I can get the response back to my host from the plugin, but I can't understand how the plugin can call into the host.

SIGTERM for a plug in

Hi!

We built a build system based on plugins and Im writing a facility which gives the user an ability to cancel a build pipeline. For this to happen I need to actually kill the pipeline which is the plugin binary itself.

I thought of creating an endpoint cancels the server, but is there a way from the client to send a kill signal? Or to look for the binary and kill it with os interrupt?

License questions

Given that this is MPL licensed library and go uses static compile, can I use this library in a GPLv3 licensed application written in go?