gRPC connection leak
When a loadbalancer drops the tcp connection (timeout), an extra underlying gRPC connection is set up. Eventually, this starves resources on either the client or the server.
The solution is two fold:
- remove the connection leakage
- implement a ping with timeout < 30 seconds (nginx default)