Comments (15)
I've created an issue: grpc/grpc-java#11229
from grpc-spring.
Have you tried this config:
from grpc-spring.
Thanks
I haven't tried it yet. I will test it with K8s and let you know the result
from grpc-spring.
Looks like it works. At least now I see that K8s can't kill it for configured period of time.
Additionaly to shutdownGracePeriod =-1
I configured terminationGracePeriodSeconds
for 24h (just for testing).
I also tried to adjust various confs:
grpc:
server:
port: 6565
reflection-service-enabled: true
shutdown-grace-period: -1
enable-keep-alive: true
keep-alive-time: 86400
keep-alive-timeout: 86400
permit-keep-alive-without-calls: true
permit-keep-alive-time: 86400
But after 5 minutes the app is getting killed anyway. I can't find a conf that is responsible for that.
[SpringApplicationShutdownHook] [trace_id=, span_id=]n.d.b.g.s.s.GrpcServerLifecycle : Completed gRPC server shutdown
Looks like it's Spring conf. I will try to experiment with it more
from grpc-spring.
You could add a log line/debug break point here:
To check if the waiting gets interrupted somehow.
from grpc-spring.
Thanks, I'm already looking into such a logic. It's not easy to debug everything with K8s. I will try to add more logs by DEBUG or use Telepresence or something to understand why the service is getting killed after 5 minutes.
from grpc-spring.
Depending on your setup debugging in K8s is easy.
Just expose an additional port or tunnel/port-forward(?) into the container and then connect as usual.
from grpc-spring.
The connection is closed from the K8 side. When I run the server without K8s and send kill -TERM
to the Java process, it waits to close all connections properly. For k8s, the connection is closed, service shut down, and k8s kill container. Need to check ingress timeouts. Or maybe I need to adjust keep-alive
confs as well
from grpc-spring.
Thanks for the update
from grpc-spring.
When running a Java process inside a Docker container, sending a SIGTERM signal (kill -TERM 1) results in immediate termination rather than a graceful shutdown. This issue does not occur when running the same Java process locally and do the same kill.
kubectl exec -it pod_id -- /bin/bash
kill -TERM 1
Locally, it works fine but when server is inside Docker container graceful shutdown doesn't work and I can't understand why localServer.awaitTermination();
immediately kills the server. I don't see any InterruptedException
when I connect via 5005.
from grpc-spring.
I'm running out of ideas. Do you have any ideas on further investigation or narrowing down the scope?
Thanks!
from grpc-spring.
Sorry, unfortunately not.
from grpc-spring.
I think localServer.awaitTermination();
doesn't work as I expect :( . I'm working on implementing a custom ShutDown Hook to check the number of active streams before terminating.
This article describes similar problem https://fedor.medium.com/shutting-down-grpc-services-gracefully-961a95b08f8
Just FYI. Thanks for the help
from grpc-spring.
Maybe also create an issue upstream in grpc-java and link it here.
Maybe they can add a build in variant as well because I cannot imagine that you are the only one having this problem.
from grpc-spring.
Yes, I expected it should already be handled downstream. Creating an issue on grpc-java is a good thing.
from grpc-spring.
Related Issues (20)
- grails can not work HOT 7
- Getting "Could not generate CGLIB subclass of class" error after upgrading to version 3.1.0.RELEASE HOT 1
- @GrpcClient is null spring boot 3 HOT 5
- Hot-reloading TLS certificates HOT 3
- Off-heap memory leak. I'm not sure if it's a grpc-spring problem or a netty problem.
- After the service restarts, requests do not come through.
- Failed to start bean 'shadedNettyGrpcServerLifecycle' or something related to maven build config
- GRPC Client Stream
- Cant create connections when using Eureka and grpc.client.GLOBAL.immediate-connect-timeout at startup
- Spring Framework BeanPostProcessorChecker: warning logs about BeanPostProcessors HOT 2
- "Socket accept failed" after upgrading gPRC Spring version to 3.1.0.RELEASE HOT 1
- io.grpc.StatusRuntimeException: UNAVAILABLE: RST_STREAM closed stream. HTTP/2 error code: NO_ERROR
- 如何修改注册上nacos的元数据key "gRPC_port",希望自定义服务发现策略
- basic auth and server reflection broken with io.grpc >= 1.66.0
- Add support for server reflection v1 HOT 1
- Micrometer Tracing doesn't work with @GrpcClient injection in constructors
- `@GrpcExceptionHandler` doesnot propagate the cause
- gRPC direct executor best practices for non-blocking code
- GrpcClientMetricAutoConfiguration conflicts with MeterRegistryCustomizer due to eager initialization of MeterRegistry
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from grpc-spring.