Can't access pod across nodes in canal/Flannel network

[lw8008]

We are using Canal and find access request(like DNS) cross nodes will not work.
With tcpdump, i can see the package send to flannel,1 cni but seems not transfer to pod veth.
then failed with ";; connection timed out; no servers could be reached"
But if the request from the same node with pod(liek DNS), it works.

Current Behavior

Here is the details:

dns pod in node
ccloud@k8s1002:~> kubectl get po -n kube-system -owide |grep dns
coredns-7c5566588d-q7t5c 1/1 Running 0 2d15h 10.42.9.7 k8s2002

dns pod can access in same node k8s1002
k8s2002:~ # nslookup www.google.com 10.42.9.7
Server: 10.42.9.7
Address: 10.42.9.7#53

Name: www.google.com
Address: 216.58.203.100
Name: www.google.com
Address: 2404:6800:4006:809::2004

dns pod can not access cross nodes in same cluster, but ping and traceroute works
k8s1002:> nslookup www.google.com 10.42.9.7
;; connection timed out; no servers could be reached
k8s1002: # ping 10.42.9.7
PING 10.42.9.7 (10.42.9.7) 56(84) bytes of data.
64 bytes from 10.42.9.7: icmp_seq=1 ttl=63 time=0.395 ms
64 bytes from 10.42.9.7: icmp_seq=2 ttl=63 time=0.361 ms
^C
--- 10.42.9.7 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.361/0.378/0.395/0.017 ms

k8s1002:~ # traceroute 10.42.9.7 -p 53 -T
traceroute to 10.42.9.7 (10.42.9.7), 30 hops max, 60 byte packets
1 10.42.9.0 (10.42.9.0) 0.288 ms 0.241 ms 0.316 ms
2 10.42.9.7 (10.42.9.7) 0.299 ms 0.288 ms 0.364 ms

if traceroute with 53/UDP will not work.
k8s1002:~ # traceroute 10.42.9.7 -p 53 -U
traceroute to 10.42.9.7 (10.42.9.7), 30 hops max, 60 byte packets
1 10.42.9.0 (10.42.9.0) 0.279 ms 0.294 ms 0.248 ms
2 * * *
3 * * *

when do the nslookup in node 1002, and tcpdump in node 2002, can see package transfer to 2002 eth0/flannel.1 but seems not go to pod veth
k8s2002:~ # tcpdump -i flannel.1 |grep 10.42.9.7 |grep 10.42.0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on flannel.1, link-type EN10MB (Ethernet), capture size 262144 bytes
22:38:07.158232 IP 10.42.0.0.35179 > 10.42.9.7.domain: 60203+ A? www.google.com. (32)
22:38:09.158186 IP 10.42.0.0.35179 > 10.42.9.7.domain: 60203+ A? www.google.com. (32)
22:38:11.158429 IP 10.42.0.0.35179 > 10.42.9.7.domain: 60203+ A? www.google.com. (32)
k8s2002:~ # tcpdump -i cali764e7c62201 |grep 10.42.9.7 |grep 10.42.0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on cali764e7c62201, link-type EN10MB (Ethernet), capture size 262144 bytes
^C102 packets captured
125 packets received by filter
0 packets dropped by kernel

Other info:
k8s2002:~ # ip route show
default via 10.10.203.1 dev eth0
10.10.203.0/24 dev eth0 proto kernel scope link src 10.10.203.22
10.42.0.0/24 via 10.42.0.0 dev flannel.1 onlink
10.42.1.0/24 via 10.42.1.0 dev flannel.1 onlink
10.42.2.0/24 via 10.42.2.0 dev flannel.1 onlink
10.42.3.0/24 via 10.42.3.0 dev flannel.1 onlink
10.42.5.0/24 via 10.42.5.0 dev flannel.1 onlink
10.42.6.0/24 via 10.42.6.0 dev flannel.1 onlink
10.42.7.0/24 via 10.42.7.0 dev flannel.1 onlink
10.42.8.0/24 via 10.42.8.0 dev flannel.1 onlink
10.42.9.3 dev cali50ac339f6f0 scope link
10.42.9.7 dev cali764e7c62201 scope link
10.42.9.9 dev cali4aa5209a580 scope link
10.42.9.10 dev calida7512ac2f4 scope link
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown

k8s2002:~ # iptables-save |grep 10.42.9.7
-A KUBE-SEP-M665IC26QLRAQYCE -s 10.42.9.7/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-M665IC26QLRAQYCE -p tcp -m tcp -j DNAT --to-destination 10.42.9.7:53
-A KUBE-SEP-O54IDBNUTNUKKZAJ -s 10.42.9.7/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-O54IDBNUTNUKKZAJ -p udp -m udp -j DNAT --to-destination 10.42.9.7:53
-A KUBE-SEP-TRBBYPFCFWMGKQUV -s 10.42.9.7/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-TRBBYPFCFWMGKQUV -p tcp -m tcp -j DNAT --to-destination 10.42.9.7:9153

k8s2002:~ # iptables-save |grep KUBE-SEP-O54IDBNUTNUKKZAJ
:KUBE-SEP-O54IDBNUTNUKKZAJ - [0:0]
-A KUBE-SEP-O54IDBNUTNUKKZAJ -s 10.42.9.7/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-O54IDBNUTNUKKZAJ -p udp -m udp -j DNAT --to-destination 10.42.9.7:53
-A KUBE-SVC-TCOU7JCQXEZGVUNU -j KUBE-SEP-O54IDBNUTNUKKZAJ

k8s2002:~ # ifconfig cali764e7c62201
cali764e7 Link encap:Ethernet HWaddr EE:EE:EE:EE:EE:EE
UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1
RX packets:471949 errors:0 dropped:0 overruns:0 frame:0
TX packets:546133 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:38676965 (36.8 Mb) TX bytes:151108435 (144.1 Mb)

k8s2002:~ # ifconfig flannel.1
flannel.1 Link encap:Ethernet HWaddr 8E:C1:3C:C4:11:27
inet addr:10.42.9.0 Bcast:0.0.0.0 Mask:255.255.255.255
UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1
RX packets:2279537 errors:0 dropped:0 overruns:0 frame:0
TX packets:29946 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:154460830 (147.3 Mb) TX bytes:3360476 (3.2 Mb)

Your Environment

Flannel version
It's Canal. CNI flannel plugin v0.8.0
Backend used (e.g. vxlan or udp):
vxlan
Orchestrator version (e.g. kubernetes, mesos, rkt):
K8s 1.17.4 by Rancher
Operating System and version:
Suse 12SP5
Linux k8s2002 4.12.14-122.32-default #1 SMP Wed Aug 5 12:59:08 UTC 2020 (477c426) x86_64 x86_64 x86_64 GNU/Linux