aseovic / helidon Goto Github PK

At the moment we only support check method within our HealthService implementation.

We need to add support for other methods inherited from HealthBase, in order to implement full gRPC Health Service spec.

We need to be able to use gRPC interceptors in the CDI micro profile environment. The interceptors should be discoverable and applied to services or methods.

The current thinking is that we can use annotation binding for this in the same way that CDI interceptors work. A gRPC service class can be annotated with custom "interceptor binding" annotations. A gRPC interceptor is also annotated with the same annotation. When the server is bootstrapped and services are discovered the interceptors for the specific annotations should be discovered and applied to the service/method configuration.

There are standard annotations for micro profile metrics that the current Helidon CDI code creates interceptors for. This means that any gRPC service method annotated with one of these annotations will have metrics applied. The issue here is that using CDI interceptors to apply metrics does not really work for asynchronous methods such as gRPC methods (as an aside it would not work for async Jersey JAX-RS methods either) because the metrics will only wrap the actual method call and not the time between the start of the method and sending the async result.

We need to decide whether to do something different for gRPC. This might be difficult given how the rest of the Helidon MP code processes these annotations. We might have to use a different set of annotations for gRPC metrics.

The gRPC security examples use a custom CallCredentials implementation BasicAuthCallCredentials to configure basic auth for the client. As the comment in the PR to oracle/helidon says, this should br removed as it is not actually required. The clients should use a properly created and configured GrpcClientSecurity instance.

We should modify the existing Tracing documentation to:

1. Show Tracer creation independently
2. Show how the Tracer created in 1 above is used to configure both WebServer and GrpcServer.

The gRPC client module's ClientServiceDescriptor and ClientMethodDescriptor classes both have an inner interface called Config. This inner interface's name needs to be changed to Rules.

ClientServiceDescriptor.Config ---> ClientServiceDescriptor.Rules
ClientMethodDescriptor.Config ---> ClientMethodDescriptor.Rules

NOTE: For completeness (and tidiness and elegance) make sure that you change all of the relevant field and parameter names where these classes are used from config to rules.

This work should be done on a branch and merged into the grpc-client branch.

We need to write gRPC Clients doc section, with following sub-sections:

• Using generated gRPC client stubs
• Implementing clients by hand
• Using custom serialization formats
• Configuring clients
• Metrics
• Tracing
• Security

The current code base uses an interface to define a priority client or server interceptor. This should be changed so that priority is based on an int and priority of a specific interceptor can be defined by annotating the class with @Priority or setting the priority when adding the interceptor.

In addition to the gRPC-generated clients that we currently use in examples (GreetClient, StringClient, HealthClient) we should implement the equivalents using our client framework.

We should also implement a client for the GreetServiceJava, which would demonstrate how to use Java serialization between the client and the server.

Once the implementation is complete, we need to document the usage of the client-side Helidon MP gRPC framework, including:

1. The annotations available and their use
2. Client-side proxy generation
3. CDI usage and bootstrapping

Once #6 is completed, we should add support for service and method-level metrics configuration.

It should be possible to add custom annotations to server side service classes and methods and have those annotations processed to add functionality to the service. This is similar to how the Jersey "Feature" plugin mechanism works. For example a SecurityFeature might process the @RolesAllowed annotation to add security configuration to a service.

This issue is related to the MP gRPC server issue #27 and depends on the annotation support issue #52.

Requirements

• Create a "feature" interface to be implemented by features that should allow the feature to modify a service's configuration. For example add interceptors, security, etc.
• Register a "feature", either as a class to instance with the GrpcRouting.
• The feature should be able to configure all of the services registered with the GrpcRouting regardless of which order services and features are registered.
• The feature should have access to the annotated class and methods so that it can apply functionality based on the annotations present.

Success Criteria

• It should be possible to write a feature that can be registered with the GrpcRouting
• The feature should be able to configure every service registered with the GrpcRouting
• The feature should be able to configure every method on every service registered with the GrpcRouting.

The web server security examples include examples on how to use outbound security, the same features are required for outbound gRPC calls. Depends on #26 (allow configuration of client security)

The use case is that an authenticated request is made to a Helidon http endpoint and in the code that handles that endpoint another request is made to an external endpoint. This external endpoint requires authentication.

There are two scenarios:

1. Pass the same credentials through to the external endpoint that were used to access the Helidon endpoint.
2. Override the credentials by specifying new credentials to use when accessing the external endpoint.

Requirements:

1. We need to be able to support calling authenticated http endpoints from gRPC methods in the same way as the existing web server examples do.
2. We need to be able to call authenticated gRPC endpoints from both gRPC methods and web server request handlers using the same patterns as used in the web server examples

Success Criteria:

1. An authenticated Helidon web service method is able to make an outbound call to an authenticated gRPC service passing through the same credentials that were used to make the web request
2. An authenticated Helidon web service method is able to make an outbound call to an authenticated gRPC service using different credentials than those were used to make the web request
3. An non-authenticated Helidon web service method is able to make an outbound call to an authenticated gRPC service using some configured credentials
4. An authenticated Helidon gRPC service method is able to make an outbound call to an authenticated gRPC service passing through the same credentials that were used to make the web request
5. An authenticated Helidon gRPC service method is able to make an outbound call to an authenticated gRPC service using different credentials than those were used to make the web request
6. An non-authenticated Helidon gRPC service method is able to make an outbound call to an authenticated gRPC service using some configured credentials
7. An authenticated Helidon gRPC service method is able to make an outbound call to an authenticated web service passing through the same credentials that were used to make the web request
8. An authenticated Helidon gRPC service method is able to make an outbound call to an authenticated web service using different credentials than those were used to make the web request
9. An non-authenticated Helidon gRPC service method is able to make an outbound call to an authenticated web service using some configured credentials

We should provide a base class that makes it simpler to implement clients, along the lines of what we did in Synoki.

Tasks:

• Copy the service and method descriptor classes (and their test classes) from helidon-grpc-server into helidon-grpc-client renaming them appropriately.

ServiceDescriptor --> ClientServiceDescriptor
MethodDescriptor --> ClientMethodDescriptor

In both cases above change where these classes use io.grpc.ServerInterceptor to use io.grpc.ClientInterceptor

This will then give us the ability to define/build service definitions on the client.

• Port over the base client class io.synoki.grpc.client.AbstractGrpcClient from Synoki gRPC client module https://gitlab-odx.oracledx.com/synoki/synoki-grpc.

Helidon gRPC has no concept of Serilizer so this should be removed from io.synoki.grpc.client.AbstractGrpcClient

The m_methods field can be replaced with a ClientServiceDescriptor field as that is what we will use in the Helidon client to describe a service. This field can be set from the constructor, i.e. the new AbstractGrpcClient should take a ClientServiceDescriptor as its constructor parameter.

When complete we should be able to create an instance of the new AbstractGrpcClient for a given ClientServiceDescriptor and then use methods such as AbstractGrpcClient.blockingUnaryCall to call methods. We will likely need to add a method name parameter to methods like AbstractGrpcClient.blockingUnaryCall so that we can locate the relevant io.grpc.MethodDescriptor from the ClientServiceDescriptor field.

Requirements

Allow users to configure security and pass necessary authN/authZ credentials to the server.

When #20 is complete we will have ClientSeviceDefinition and ClientMethodDefinition classes. We can add methods to these classes to set the CallCredentials to use at the service or method level which will be applied in addition to any CallCredentials set at the Channel level. The CallCredentials should be applied in the order channel followed by service followed by method so that each level can add to or override previously applied credentials.

Success Criteria

1. Calling a method on a service that has no CallCredentials set at either the service or method level should apply the CallCredentials set for the Channel.
2. Calling a method on a service that has CallCredentials set at the service level should apply the service's CallCredentials in addition to any CallCredentials set for the Channel.
3. Calling a method on a service that has CallCredentials set at the service level should apply the service's CallCredentials and be able to override any CallCredentials set for the Channel. For example if the Channel sets a basic auth header it should be possible for the services CallCredentials to override that vale with a different basic auth header.
4. Calling a method on a service that has CallCredentials set at the method level should apply the method's CallCredentials in addition to the service's CallCredentials and in addition to any CallCredentials set for the Channel.
5. Calling a method on a service that has CallCredentials set at the method level should apply the method's CallCredentials and override the service's and channel's CallCredentials. For example if the service sets a basic auth header it should be possible for the channel's CallCredentials to override that vale with a different basic auth header.

The Helidon web server supports ABAC (attribute based access control) but the gRPC server does not currently supply enough information in the security interceptor to make this work.

A lot of the examples of ABAC work using Jersey and annotations but the gRPC server cannot yet support creating a service from annotated classes and methods.

ABAC can also be configured in configuration so this should be supported by the gRPC server so that ABAC rules can be configured globally, for services and, for methods.

See the web server examples under examples/grpc/security-abac

Success Criteria:

1. Start a gRPC server with ABAC security enabled from configuration.
2. Having configured global ABAC rules and registered a service with no ABAC rules; allow access to a method on a service where the caller meets the globally configured ABAC rules.
3. Having configured global ABAC rules and registered a service with no ABAC rules; deny access to a method on a service where the caller does not meet the globally configured ABAC rules.
4. Having configured global ABAC rules and registered a service that overrides the global ABAC rules; allow access to a method on a service where the caller meets the service level configured ABAC rules.
5. Having configured global ABAC rules and registered a service that overrides the global ABAC rules; deny access to a method on a service where the caller does not meet the service level configured ABAC rules.
6. Having configured global ABAC rules and registered a service that overrides the global and service ABAC rules at the method level; allow access to a method on a service where the caller meets the service level configured ABAC rules.
7. Having configured global ABAC rules and registered a service that overrides the global and service ABAC rules at the method level; deny access to a method on a service where the caller does not meet the service level configured ABAC rules.

Requirement:

The MP version of the gRPC server should support discovery and configuration of gRPC health checks.

A discoverable health check would be one of:

• The service class itself implements HealthCheck
• A method on the service class annotated with @Health that returns a HealthCheckResponse.

These changes should go into the ServiceModeller class that builds a ServiceDescriptor from an annotated class. In both of the above cases the instance of the service to use to call the health check method will be provided by the instance Supplier obtains from the ServiceModeller.instanceSupplier() method. Some sort of wrapper health check implementation will be required:

public class WrapperHealthCheck
        implements HealthCheck {
    private final Supplier<?> supplier;
    private final Method method;

    ... constructor ...

    public HealthCheckResponse call() {
        Object instance = supplier.get();
        if (instance instances HealthCheck) {
            return ((HealthCheck) instance).call():
        }
        return (HelathCheckResponse) method.invoke(supplier.get());
    }
}

Completion Criteria:

• It should be possible to write an annotated service implementation that is a CDI bean (annotated with @ ApplicationScoped and implements HealthCheck so that when the ServiceDescriptor created by the ServiceModeller has the service instance as the health check:

@ApplicationScoped
public class MyService
    implements HealthCheck {
}

• It should be possible to write an annotated service implementation that is a CDI bean (annotated with @ ApplicationScoped and has a method annotated with @Health that returns a HealthCheckResponse so that when the ServiceDescriptor created by the ServiceModeller uses a call to that method as the health check:

@ApplicationScoped
public class MyService {
    @ Health
    public HealthCheckResponse health() {
    ... do check ...
    }
}

We need to change documentation to make it clear that the gRPC server doesn't have to be started in order to get all service health checks from it and add them to HealthSupport.

The current documentation example and comments imply that it does.

We should allow users to create a client-side proxy for a gRPC service for a given channel and client-side interface.

The users should be able to inject client-side proxies for a gRPC service using named channel and the service interface declared at the injection point:

@Inject
@Channel("myGrpcServer")
private MyServiceClient client;

The above should end up injecting the result of

GrpcClientHelper.proxy(myGrpcServerChannel, MyServiceClient.class)

Allow tracing to be configured on the client and propagate tracing span information from the client to the server.

Document outbound security: (depends on #29)

Currently docs/src/main/docs/security/03_containers-integration.adoc only shows outbound security for a web request method calling out to another http endpoint. This documentation needs to be enhanced to show gRPC outbound examples:

1. http --> http
2. http --> gRPC
3. gRPC --> http
4. gRPC --> gRPC

We need to look deeper into a number of areas before we can implement Helidon MP, such as:

1. Bootstrapping
2. CDI
3. Annotations
4. Client-side proxies/code generation

Once this exploratory work is completed, we will be able to better define the remaining work.

Document how to create secure gRPC services.

Include the following:

• How to configure and create a secure server.
• How to define security at the service level.
• How to define security at the method level.

Implement GrpcClientConfiguration, which will allow us to configure the client the same way we configure the server using GrpcServerConfiguration.

Add the ability to create a server side gRPC service from annotated classes and interfaces in the same way that Jersey does for web services.

This work relates to the MP server issue #27

Requirements

• Classes or interfaces should be annotated with the @RpcService annotation.
• Use the name field from the annotation to specify the gRPC service name. If the name field is blank use the Class name.
• Methods should be annotated with annotations to represent one of the four supported method types @BidiStreaming, @ClientStreaming, @serverstreaming and @unary.
• Use the name field from the method annotation to specify the gRPC method name. If the name field is blank use the actual method name.
• Detect the request and response types from the method signature.
• Allow the request and response type to be explicitly configured using annotations on the methods

Requirements

The gRPC server should be configurable to be secured with SSL.

The Helidon web server SocketConfiguration contains code to configure SSL so this code can copied to configured SSL in GrpcServerConfiguration the same way and then using the web server SSLContextBuilder to build an SSLContext from the config. This can be used using pretty much the same code that the web server uses to create a Netty JdkSslContext (see the Helidon NettyWebServer class) to pass to the NettyServerBuilder when starting the gRPC server.

See the web server SslTest class for examples of how to create a SSLContext for testing.

Success Criteria:

1. It should be possible to programatically create a GrpcServerConfiguration and configure SSL. For example:

SSLContext  sslContext = ...
GrpcServerConfiguration.build()
    .ssl(sslContext)
    .build();

2. It should be possible to configure SSL via a configuration file.
   For example a configuration file conf-ssl.conf might look like this:

grpcserver {
  ssl: {
    private-key: {
      keystore-resource-path: "certificate.p12"
      keystore-passphrase: "helidon"
    }
}

then create the configuration like this:

Config config = Config.builder().sources(ConfigSources.classpath("config1.conf")).build();
GrpcServerConfiguration serverConfig = config.get("grpcserver").as(GrpcServerConfiguration::create).get();

3. Using a SSL enabled configuration when starting the server should configure Netty to use SSL.
   3.1 It should not be possible to connect a non-ssl client to the server
   3.2 It should be possible to connect a client using SSL to the server

For examples of how to test this the Synoki gRPC project has SSL tests that create an SSL enabled client in io.synoki.grpc.server.TlsIT

We need to refactor builder API to use ServiceDescriptor and MethodDescriptor, which should capture all relevant information necessary to fully configure each service method.

ServiceDescriptor

• service name
• method descriptors
• interceptors that should be applied to all service methods
• default metrics configuration
• health check handler
• service-level security configuration

MethodDescriptor

• method name
• handler
• method-level interceptors
• metrics configuration override
• method-level security configuration

The Webserver examples modules have examples of how to perform secure outbound calls to other http servers. We need to add similar examples for gRPC.

Note: This depends on #20 so that we can use the client API in the examples.

There are four scenarios: (see the io.helidon.security.integration.grpc.OutboundSecurityIT functional test for code that sets up the outbound security).

1. Calling a secure http endpoint from a gRPC endpoint and propagating the existing credentials
2. Calling a secure http endpoint from a gRPC endpoint and overriding the existing credentials
3. Calling a secure gRPC endpoint from a http endpoint and propagating the existing credentials
4. Calling a secure gRPC endpoint from a http endpoint and overriding the existing credentials

The request and response types for a method are optional and should not be in the ClientMethodDescriptor constructor. They need to be moved to the builder in the same way that the server descriptor was refactored.

Requirements

The gRPC client module should not have a dependency on metrics (metrics will be in a new stand-alone module).

Once this is complete it should be possible to remove this dependency from helidon-grpc-client:

<dependency>
    <groupId>io.helidon.metrics</groupId>
    <artifactId>helidon-metrics</artifactId>
    <version>${project.version}</version>
</dependency>

Tasks

Remove the metricType field from both the ClientServiceDescriptor and ClientMethodDescriptor classes as well as corresponding builders and tests.

NOTE: this work needs to be done on a branch and merged into the grpc-client branch.

The gRPC MP server supports creating and configuring services from annotated classes. We need to be able to support using annotations to configure security in the same way that the Helidon web server does, for example @RolesAllowed, @Authenticated, etc...

The existing web server code already does a lot of annotation work. We need to refactor this code to produce a new security/annotation-support module that contains the common code shared between the web server and the gRPC server.

The Helidon team want to use yaml as the format for showing examples of configuration in documentation files.

Go through all gRPC documentation under /docs and README.md files in the various modules (including examples) and make sure configuration is in the correct format.

This work needs to be done on a branch that will be merged into master so that it updated the initial pull request to Oracle/Helidon.

Security in gRPC cannot be created from configuration - this was initially removed when porting over code from the web server and now needs to be put back.

Success Criteria:

1. It should be possible to add the GrpcSecurity interceptor to a GrpcRouting by creating an instance just from configuration, i.e. GrpcSecurity.create(config.get("security"))
2. It should be possible to define default (global) security in configuration, for example:

security {
    grpc-server: {
        # Configuration of integration with grpc server
        # The default configuration to apply to all services not explicitly configured below
        defaults {
            authenticate = true
        }
    }
}

3. It should be possible to define service level security in configuration, for example:

security {
    grpc-server: {
        services: [
            {
                name: "FooService"
                defaults {
                    roles-allowed = ["admin"]
                }
            },
            {
                name: "BarService"
                defaults {
                    roles-allowed = ["admin"]
                }
            }
        ]
    }
}

It should be possible to define method level security in configuration, for example:

security {
    grpc-server: {
        services: [
            {
                name: "FooService"
                defaults {
                    roles-allowed = ["admin"]
                }
                methods: [
                    {
                        name: "DoSomething"
                        roles-allowed = ["admin"]
                    },
                    {
                        name: "DoSomethingElse"
                        roles-allowed = ["user"]
                    }
                ]
            }
        ]
    }
}

We need to remove @author tags from the Javadoc comments throughout the codebase, in order to pass the latest chekstyle rules and be able to merge into oracle/helidon.

The gRPC server module's ServiceDescriptor and MethodDescriptor classes both have an inner interface called Config. This inner interface's name needs to be changed to Rules.

ServiceDescriptor.Config ---> ServiceDescriptor.Rules
MethodDescriptor.Config ---> MethodDescriptor.Rules

NOTE: For completeness (and tidiness and elegance) make sure that you change all of the relevant field and parameter names where these classes are used from config to rules.

This work needs to be done on a branch that will be merged into master so that it updated the initial pull request to Oracle/Helidon.

We should clean up and refactor GrpcServerConfiguration to support same usage patterns (builder, etc.) that the existing web server ServerConfiguration class uses.

In the gRPC code the priority value for both server and client interceptors is an enum. It was commented in the oracle/helidon PR that an enum is too restrictive. This had actually already been realised by us and changed in our grpc-mp branch. This change needs to be ported over to the grpc-server branch so that it can be applied to the PR. This change should also then be applied to the grpc-client branch.

We need to design and implement support for the Helidon MP gRPC server bootstrapping, which will allow us to:

1. Discovered services and interceptors based on the annotations
2. Instantiate them via CDI context/container and inject necessary dependencies into them
3. Start the gRPC server based on the configuration and deploy discovered services into it

Capture client-side metrics at the service or method level, and make them available for collection/scraping.

We should be able to expose them locally via a co-located WebServer, when the client is a long-running process within the same network as the server.

However, we should also consider remote and/or short-running clients, which may not be accessible for scraping by Prometheus, so we should come up with a solution that pushes metrics from the clients to the gRPC server they are connected to, which can then expose them locally along with its own metrics and make them available for scraping.

Once the implementation is complete, we need to document the usage of the server-side Helidon MP gRPC framework, including:

1. The annotations available and their use
2. Valid method signatures for various method types (unary, server streaming, client streaming and bidirectional)
3. CDI usage

We need to be able to configure who can execute particular method.

We need to add Reactive gRPC Server section under the existing Reactive Webserver section in the Helidon documentation, which will have the following sub-sections:

• gRPC Server Introduction
• gRPC Server Configuration
• Routing
• Service Implementation
• Interceptors
• Health Checks
• Metrics

Requirements

The metrics support in Helidon gRPC should be separated out of the helidon-grpc-server module and into a new helidon-grpc-metrics module.

NOTE: This work needs to be done on a branch that will be merged into master so that it updated the initial pull request to Oracle/Helidon.

Once this is complete it should be possible to remove this dependency from helidon-grpc-server:

<dependency>
    <groupId>io.helidon.metrics</groupId>
    <artifactId>helidon-metrics</artifactId>
    <version>${project.version}</version>
</dependency>

Currently it is possible to configure metrics for a service like this:

GrpcRouting routing = GrpcRouting.builder()
                                 .register(new EchoService(), rules -> rules.timed())
                                 .build();

After this change metrics for a service will be configured by adding the metrics interceptor:

GrpcRouting routing = GrpcRouting.builder()
                                 .register(new EchoService(), rules -> rules.intercept(GrpcMetrics.timed()))
                                 .build();

The same is true for method level configuration, for example inside a GrpcService.update() method. For example this:

@Override
public void update(ServiceDescriptor.Config config) {
    config.proto(Greet.getDescriptor())
            .unary("Greet", this::greet, cfg -> cfg.timed()));
}

will become this:

@Override
public void update(ServiceDescriptor.Config config) {
    config.proto(Greet.getDescriptor())
            .unary("Greet", this::greet, cfg -> cfg.intercept(GrpcMetrics.timed()));
}

Tasks

• Create a new module helidon-grpc-metrics under grpc/metrics.
  This new module should depend on:

<dependency>
    <groupId>io.helidon.grpc</groupId>
    <artifactId>helidon-grpc-core</artifactId>
    <version>${project.version}</version>
</dependency>
<dependency>
    <groupId>io.helidon.metrics</groupId>
    <artifactId>helidon-metrics</artifactId>
    <version>${project.version}</version>
</dependency>

as well as required test dependencies. It may also have test dependencies on the gRPC server for integration tests:

<dependency>
    <groupId>io.helidon.grpc</groupId>
    <artifactId>helidon-grpc-server</artifactId>
    <version>${project.version}</version>
    <scope>test</scope>
</dependency>

• Remove the metricType field from both the ServiceDescriptor and MethodDescriptor classes as well as corresponding builders.
• Move the GrpcMetrics class and the GrpcMetricsInterceptorIT and MetricsIT test classes to the new helidon-grpc-metrics module.
• Change any metrics examples to use the new way to configure metrics at the service and method levels using the GrpcMetrics interceptor.
• Change the gRPC metrics documentation under docs/grpc to show the correct way to configure metrics.