iptables custom CoreDNS transparent proxy failed

[fff070]

Hello everyone, I have a local xray proxy (tproxy) running with inbound port 12345, and I set up iptables for proxying. Recently I had a requirement to build a k3s cluster. Since k3s uses CoreDNS as the default DNS, after finishing the setup I found that Pods could not use my host machine’s proxy.

When I checked using kubectl logs -n kube-system -l k8s-app=kube-dns, I saw that the upstream DNS was pointing to my wireless router (192.168.124.1:53). So I redirected back to 192.168.124.1:53, and then inside the Pod I ran nslookup www.baidu.com. It returned both IPv6 and IPv4. On the host machine, running nslookup www.baidu.com also returned IPv4 and IPv6, but the only difference was the order — inside the Pod, IPv6 came first.

Next, I used an ubuntu:22.04 Pod:

kubectl run dns-test --image=ubuntu:22.04 -it --rm

Inside the container, I ran apt update, but it kept hanging at:

0% [Connecting to archive.ubuntu.com (2620:2d:4000:1::101)] [Connecting to security.ubuntu.com (2620:2d:4000:1::102)]

Then it failed with:

Failed to fetch http://archive.ubuntu.com/ubuntu/dists/jammy-backports/InRelease  
Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4002:1::102). - connect (101: Network is unreachable)  
Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::102).

However, if I add the iptables rule:

iptables -t mangle -A XRAY -i cni0 -j RETURN

and restart the service, I find that inside the Pod I cannot ping www.google.com, but I can ping www.baidu.com. Also, inside the ubuntu:22.04 Pod, apt update works fine without errors.

If I use:

curl -x socks5://192.168.124.3:10808 www.google.com

I can fetch the webpage data. I know that in this case traffic from cni0 does not go through the xray proxy.

But my requirement is: all Pod traffic, including DNS, must go through xray.

How should I set this up correctly in order to achieve my requirement?

[fff070]

this is my current configuration
#!/bin/bash
ip route add local default dev lo table 100
ip rule add fwmark 1 table 100
iptables -t mangle -N XRAY
iptables -t mangle -A XRAY -d 10.0.0.0/8 -j RETURN
iptables -t mangle -A XRAY -d 100.64.0.0/10 -j RETURN
iptables -t mangle -A XRAY -d 127.0.0.0/8 -j RETURN
iptables -t mangle -A XRAY -d 169.254.0.0/16 -j RETURN
iptables -t mangle -A XRAY -d 172.16.0.0/12 -j RETURN
iptables -t mangle -A XRAY -d 192.0.0.0/24 -j RETURN
iptables -t mangle -A XRAY -d 224.0.0.0/4 -j RETURN
iptables -t mangle -A XRAY -d 240.0.0.0/4 -j RETURN
iptables -t mangle -A XRAY -d 255.255.255.255/32 -j RETURN
iptables -t mangle -A XRAY -d 192.168.124.1 -p tcp --dport 53 -j RETURN
iptables -t mangle -A XRAY -d 192.168.124.1 -p udp --dport 53 -j RETURN
iptables -t mangle -A XRAY -d 192.168.0.0/16 -p tcp ! --dport 53 -j RETURN
iptables -t mangle -A XRAY -d 192.168.0.0/16 -p udp ! --dport 53 -j RETURN
iptables -t mangle -A XRAY -p tcp -j TPROXY --on-port 12345 --tproxy-mark 1
iptables -t mangle -A XRAY -p udp -j TPROXY --on-port 12345 --tproxy-mark 1
iptables -t mangle -A PREROUTING -j XRAY

iptables -t mangle -N XRAY_SELF
iptables -t mangle -A XRAY_SELF -d 10.0.0.0/8 -j RETURN
iptables -t mangle -A XRAY_SELF -d 100.64.0.0/10 -j RETURN
iptables -t mangle -A XRAY_SELF -d 127.0.0.0/8 -j RETURN
iptables -t mangle -A XRAY_SELF -d 169.254.0.0/16 -j RETURN
iptables -t mangle -A XRAY_SELF -d 172.16.0.0/12 -j RETURN
iptables -t mangle -A XRAY_SELF -d 192.0.0.0/24 -j RETURN
iptables -t mangle -A XRAY_SELF -d 224.0.0.0/4 -j RETURN
iptables -t mangle -A XRAY_SELF -d 240.0.0.0/4 -j RETURN
iptables -t mangle -A XRAY_SELF -d 255.255.255.255/32 -j RETURN
iptables -t mangle -A XRAY_SELF -d 192.168.124.1 -p tcp --dport 53 -j RETURN
iptables -t mangle -A XRAY_SELF -d 192.168.124.1 -p udp --dport 53 -j RETURN
iptables -t mangle -A XRAY_SELF -d 192.168.0.0/16 -p tcp ! --dport 53 -j RETURN
iptables -t mangle -A XRAY_SELF -d 192.168.0.0/16 -p udp ! --dport 53 -j RETURN
iptables -t mangle -A XRAY_SELF -m mark --mark 1234 -j RETURN
iptables -t mangle -A XRAY_SELF -p tcp -j MARK --set-mark 1
iptables -t mangle -A XRAY_SELF -p udp -j MARK --set-mark 1
iptables -t mangle -A OUTPUT -j XRAY_SELF