Avoid using port forward for redis

[rivernews]

In travis, we use port forward to let it connect to redis on our k8 cluster.

However, from time to time, we see the port forward fails, like broken pipe, or anything. But eventually making java scraper not able to connect. The lettuce library, like throwing exception as below:

io.lettuce.core.RedisConnectionException: Unable to connect to localhost:6379
    at io.lettuce.core.RedisConnectionException.create (RedisConnectionException.java:78)
    at io.lettuce.core.RedisConnectionException.create (RedisConnectionException.java:56)
    at io.lettuce.core.AbstractRedisClient.getConnection (AbstractRedisClient.java:234)
    at io.lettuce.core.RedisClient.connectPubSub (RedisClient.java:366)
    at com.shaungc.utilities.PubSubSubscription.<init> (PubSubSubscription.java:60)
    at com.shaungc.javadev.App.main (App.java:27)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:282)
    at java.lang.Thread.run (Thread.java:834)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: localhost/127.0.0.1:6379
Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused
    at io.netty.channel.unix.Errors.throwConnectException (Errors.java:124)
    at io.netty.channel.unix.Socket.finishConnect (Socket.java:243)
    at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect (AbstractEpollChannel.java:660)
    at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect (AbstractEpollChannel.java:637)
    at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady (AbstractEpollChannel.java:524)
    at io.netty.channel.epoll.EpollEventLoop.processReady (EpollEventLoop.java:473)
    at io.netty.channel.epoll.EpollEventLoop.run (EpollEventLoop.java:383)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run (SingleThreadEventExecutor.java:989)
    at io.netty.util.internal.ThreadExecutorMap$2.run (ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run (FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run (Thread.java:834)

There're some SO mentioning that port forward is generally slower than exposing the redis server over k8 service.

As such, in order to increase the reliability of redis service, we want to

• Make redis server exposed to external world traffic on K8
  • Test if we can connect to remote redis server using the domain name, from our local laptop
• Protect redis with password. We probably need to set the password by providing a config file. Let's keep it simple.
  • Test the password is working

[rivernews]

For now, we probably can just re-try the tasks. But in the long term we would like to have this enhancement.

[rivernews]

Current Status

What we may keep explore

• The reason current way is not working might be because we have conflicting components:

  • For redis this kind of tcp service, ingress resource is not needed. Instead, such ingress should be configured at the ingress controller level. So we should:
    • Remove ingress resource of redis-cluster
    • Let ingress controller config the port 6379:redis-cluster/redis-cluster-service:6379
    • You should leave the redis service untouched, looks like there's no problem with that one.
    • Then we may check if we can use redis-cli to connect to it, but there's really no guarantee.
    • 🛑 However, if we don't have ingress rule (resource), then we won't have external dns, which is we don't have custom domain anymore, and cannot use redis.api.shaungc.com. This gets us stuck.

• You can still try the official nginx-ingress tcp setup, then use redis-cli -h clusterIP -p 6379 to try to connect

  • Remember to remove ingress resource in this case, which is simply removing the domain name in our microservice terraform module. No other change required.
  • 🎉 While looking at this external dns related doc, we suddenly recall that our firewall did not open 6378! Then Bingo we can connect now. That article also mentions our configuration - w/o L4 load balancer - which this article has a great sketch graphs depicting different kinds of k8 network, but this article only tell few about using ingress to directly let in external traffic, w/o a load balancer. It does mention the extra cost LB introduces, which is the core of our concern.

• Ideally we can connect through a domain name. In that case, we'll have to look into k8 external dns helm chart.

• [Research] kubernetes/stable/nginx-ingress uses lua to handle upstream, and this affects how tcp service is handled. On the other hand nginxinc seem to still use stream to handle things, at least it provides a stream-snippet configuration so that in case high level tcp service feature doesn't work, we can always make our hands dirty and directly write the stream for our 6379.

• Finally, another path is to change to use Kubespray - install on a bare metal machine, or EC2. So that we have more choice over ingress type, nodeport, etc. But whether if this will allow us to create a domain redis.shaungc.com and connect over tcp 6379, is still unknown and requires more research.

  • Another article demonstrating setting up network resource, install k8, till http request on a AWS EC2.
  • This step is probably more of solving the compute resource - cost issue.

Relevant resources

• Same people configuring the tcp.6379=np/svc:6379 but with no luck, still cannot connect to the service.
• This post pointed it out clear that nginx is using Lua instead of nginx upstream to handle tcp services. The placeholder in nginx.conf is expected. Then the problem becomes: why that lua doesn't work for tcp service?
• Nginx ingress version uses lua for upstream since 0.21. In 0.24, Move some configuration logic from Nginx config to Lua code which possibly removed stream to lua there. Its documentation also mentions that it heavily relies on nginx lua module.
• In original nginx way of handling tcp services, it should be using stream { upstream redis {...} server { ...} } kind of fashion to do this, as described in this thread. This SO asnwer is also talking about something similar.
• Kubernetes nginx ingress does seem to support TCP services, via the configmap 6379: ns/svc:6379 form. This is also most of the post / SO answer when you google "nginx ingress expose redis / tcp services / 6379". But it just doesn't work! Like the github issue above people configured it, and also k8 service & port, but still not working.
  • Many people also refer to the nginx ingress doc to expose TCP service, which basically talks about the same thing.
• Indeed Helm for nginx ingress has a configuration to expose the tcp services as stated in this issue, like --set tcp.8080="default/example-tcp-svc:9000", and that did created an additional configmap containing the tcp ports, etc. This can explain why helm only has a controller.tcp.configMapNamespace but not configmap name, because the chart created the configmap for you, and you just specify ports on the tcp param.
• Helm chart of Kerbernetes/stable/nginx-ingress has a config example of L4, but it's specifically for aws.
• This post pretty much summarized our current way of using Daemonset for nginx, and thus does not require external load balancer from cloud platform, like ELB on AWS.

[rivernews]

Conclusion

We can now wrap up:

• Configure redis a password
• No need for new domain name, just use api.shaungc.com which will always be there. Use it in travis job. You may need to setup secrets for it as well. Then we can get rid of port forwarding.

[rivernews]

Works as expected. Just note that for public redis, we're using 6378.