meyskens / lookout-sdk

SDK for lookout analyzers

lookout-sdk

Toolkit for writing new analyzers for source{d} Lookout.

What Does the SDK Provide?

For the complete documentation of source{d} Lookout, please take a look at https://docs.sourced.tech/lookout.

For detailed information about the different parts of Lookout, and how they interact you can go to the Lookout architecture guide.

lookout-sdk provides:

• proto definitions.
• pre-generated libraries for Golang and Python, offering:
  • an easy access to the DataService API though a gRPC service. Lookout will take care of dealing with Git repositories, UAST extraction, programming language detection, etc.
  • low-level helpers to work around some protobuf/gRPC caveats.
• quickstart examples of an Analyzer that detects language and number of functions (written in Go and in Python).

Caveats

For the gRPC client and server please follow these requirements:

• set a common maximum gRPC message size in gRPC servers and clients. This is required to avoid hitting different gRPC limits when handling UASTs, that can be huge —see grpc/grpc#7927—. To do so use the included helpers in lookout-sdk:
  • go: using pb.NewServer and pb.DialContext.
  • python: using lookout.sdk.grpc.create_server and lookout.sdk.grpc.create_channel.
• support RFC 3986 URI scheme; lookout-sdk includes helpers for this:
  • go: using pb.ToGoGrpcAddress and pb.Listen.
  • python: using lookout.sdk.grpc.to_grpc_address.
• use insecure connection:
  • currently lookout expects to use insecure gRPC connections, as provided by pb.DialContext
  • python: run server using server.add_insecure_port(address) (example).

DataService

When DataService is being dialed, you should:

• turn on gRPC Wait for Ready mode if your analyzer creates a connection to DataServer before it was actually started. This way the RPCs are queued until the chanel is ready:
  • go: using grpc.WaitForReady(true).
  • python: using the wait_for_ready flag.
• golang: reset connection backoff to DataServer on event: if you keep the connection to DataServer open you need to reset the backoff when your analyzer receives a new event. Use the conn.ResetConnectBackoff method in your event handlers. It's needed to avoid broken connections after a lookoutd redeployment. In case of a long restart of lookoutd gRPC server, the backoff timeout may increase so much that the analyzer will not be able to reconnect before it makes the new request to DataServer.

Contributing

Contributions are welcome and very much appreciated 🙌

Please refer to our Contribution Guide for more details.

Community

source{d} has an amazing community of developers and contributors who are interested in Code As Data and/or Machine Learning on Code. Please join us! 👋

• Slack
• Twitter
• Email

Code of Conduct

All activities under source{d} projects are governed by the source{d} code of conduct.

License

Apache License Version 2.0, see LICENSE