can not ping from OTBR to MTD

[lancersky2017]

Describe the bug A clear and concise description of what the bug is.
Hi, experts,
My OTBR built on OpenWrt 21.02. After the MTD attach success, I can access to the MTD use ot-ctl ping but system ping not.
There is one OTBR and one MTD in the Thread network.

~# ot-ctl ping fd11:22:0:0:52cd:33bb:5ea4:da42
16 bytes from fd11:22:0:0:52cd:33bb:5ea4:da42: icmp_seq=11 hlim=64 time=14ms
1 packets transmitted, 1 packets received. Packet loss = 0.0%. Round-trip min/avg/max = 14/14.0/14 ms.
Done
~# 
~# ping -6 -I wpan0 fd11:22:0:0:52cd:33bb:5ea4:da42
PING fd11:22:0:0:52cd:33bb:5ea4:da42 (fd11:22::52cd:33bb:5ea4:da42): 56 data bytes
ping: sendto: Permission denied
~# 
~# ot-ctl netdata show
Prefixes:
fd11:22:0:0::/64 paros med 7c00
Routes:
fd9f:4d2e:2419:b87d::/64 s med 7c00
fdd4:1d8c:d98f:b204::/64 s med 7c00
fdda:b85f:3c43:7bbb::/64 s med 7c00
Services:
44970 01 77000500000e10 s 7c00
44970 5d fddead00beef000020ce2622de55c6b7d11f s 7c00
Done

check the route table

~# ip -6 r s t all
default from 240e:3b0:3255:1bad::/64 via fe80::9683:c4ff:fe12:b0d1 dev eth0 proto static metric 512 pref medium
default from 240e:3b0:3255:f6b5::/64 via fe80::9683:c4ff:fe12:b0d1 dev eth0 proto static metric 512 pref medium
default from fd9f:4d2e:2419:b87d::/64 via fe80::9683:c4ff:fe12:b0d1 dev eth0 proto static metric 512 pref medium
default from fdd4:1d8c:d98f:b204::/64 via fe80::9683:c4ff:fe12:b0d1 dev eth0 proto static metric 512 pref medium
default from fdda:b85f:3c43:7bbb::/64 via fe80::9683:c4ff:fe12:b0d1 dev eth0 proto static metric 512 pref medium
64:ff9b::/96 dev tayga-nat64 proto static metric 1024 pref medium
2001:db8:1::1 dev tayga-nat64 proto static metric 1024 pref medium
240e:3b0:3255:1bad::/64 dev eth0 proto static metric 256 pref medium
unreachable 240e:3b0:3255:1bad::/64 dev lo proto static metric 2147483647 pref medium
240e:3b0:3255:f6b5::/64 dev eth0 proto static metric 256 pref medium
unreachable 240e:3b0:3255:f6b5::/64 dev lo proto static metric 2147483647 pref medium
dd15:c315:3634::/64 dev br-lan proto static metric 1024 pref medium
dd15:c315:3634:1::/64 dev br-guest proto static metric 1024 linkdown pref medium
unreachable dd15:c315:3634::/48 dev lo proto static metric 2147483647 pref medium
fd11:22::/64 from 240e:3b0:3255:1bad::/64 via fe80::bc9f:e1ff:fe29:9ed8 dev eth0 proto static metric 512 pref medium
fd11:22::/64 from 240e:3b0:3255:f6b5::/64 via fe80::bc9f:e1ff:fe29:9ed8 dev eth0 proto static metric 512 pref medium
fd11:22::/64 from fd9f:4d2e:2419:b87d::/64 via fe80::bc9f:e1ff:fe29:9ed8 dev eth0 proto static metric 512 pref medium
fd11:22::/64 from fdd4:1d8c:d98f:b204::/64 via fe80::bc9f:e1ff:fe29:9ed8 dev eth0 proto static metric 512 pref medium
fd11:22::/64 from fdda:b85f:3c43:7bbb::/64 via fe80::bc9f:e1ff:fe29:9ed8 dev eth0 proto static metric 512 pref medium
fd11:22::/64 dev wpan0 proto kernel metric 256 pref medium
fd9f:4d2e:2419:b87d::/64 dev eth0 proto static metric 256 pref medium
unreachable fd9f:4d2e:2419:b87d::/64 dev lo proto static metric 2147483647 pref medium
fdd4:1d8c:d98f:b204::/64 dev eth0 proto static metric 256 pref medium
unreachable fdd4:1d8c:d98f:b204::/64 dev lo proto static metric 2147483647 pref medium
fdda:b85f:3c43:7bbb::/64 dev eth0 proto static metric 256 pref medium
unreachable fdda:b85f:3c43:7bbb::/64 dev lo proto static metric 2147483647 pref medium
fdde:ad00:beef::/64 dev wpan0 proto kernel metric 256 pref medium
fe80::/64 dev br-lan proto kernel metric 256 pref medium
fe80::/64 dev tayga-nat64 proto kernel metric 256 pref medium
fe80::/64 dev wlan0 proto kernel metric 256 pref medium
fe80::/64 dev wpan0 proto kernel metric 256 pref medium
default via fe80::9683:c4ff:fe12:b0d1 dev eth0 proto ra metric 1024 expires 1376sec hoplimit 64 pref medium
local ::1 dev lo table local proto kernel metric 0 pref medium
anycast 240e:3b0:3255:1bad:: dev eth0 table local proto kernel metric 0 pref medium
local 240e:3b0:3255:1bad:9683:c4ff:fe26:4506 dev eth0 table local proto kernel metric 0 pref medium
anycast 240e:3b0:3255:f6b5:: dev eth0 table local proto kernel metric 0 pref medium
local 240e:3b0:3255:f6b5:c2:ea3e:3554:86f0 dev eth0 table local proto kernel metric 0 pref medium
local 240e:3b0:3255:f6b5:9683:c4ff:fe26:4506 dev eth0 table local proto kernel metric 0 pref medium
anycast dd15:c315:3634:: dev br-lan table local proto kernel metric 0 pref medium
local dd15:c315:3634::1 dev br-lan table local proto kernel metric 0 pref medium
anycast fd11:22:: dev wpan0 table local proto kernel metric 0 pref medium
local fd11:22::b2e7:d853:d2a0:bca5 dev wpan0 table local proto kernel metric 0 pref medium
anycast fd9f:4d2e:2419:b87d:: dev eth0 table local proto kernel metric 0 pref medium
local fd9f:4d2e:2419:b87d:9683:c4ff:fe26:4506 dev eth0 table local proto kernel metric 0 pref medium
anycast fdd4:1d8c:d98f:b204:: dev eth0 table local proto kernel metric 0 pref medium
local fdd4:1d8c:d98f:b204:9683:c4ff:fe26:4506 dev eth0 table local proto kernel metric 0 pref medium
anycast fdda:b85f:3c43:7bbb:: dev eth0 table local proto kernel metric 0 pref medium
local fdda:b85f:3c43:7bbb:9683:c4ff:fe26:4506 dev eth0 table local proto kernel metric 0 pref medium
anycast fdde:ad00:beef:: dev wpan0 table local proto kernel metric 0 pref medium
local fdde:ad00:beef::ff:fe00:7c00 dev wpan0 table local proto kernel metric 0 pref medium
local fdde:ad00:beef::ff:fe00:fc00 dev wpan0 table local proto kernel metric 0 pref medium
local fdde:ad00:beef::ff:fe00:fc10 dev wpan0 table local proto kernel metric 0 pref medium
local fdde:ad00:beef::ff:fe00:fc11 dev wpan0 table local proto kernel metric 0 pref medium
local fdde:ad00:beef::ff:fe00:fc38 dev wpan0 table local proto kernel metric 0 pref medium
local fdde:ad00:beef:0:20ce:2622:de55:c6b7 dev wpan0 table local proto kernel metric 0 pref medium
anycast fe80:: dev br-lan table local proto kernel metric 0 pref medium
anycast fe80:: dev tayga-nat64 table local proto kernel metric 0 pref medium
anycast fe80:: dev eth0 table local proto kernel metric 0 pref medium
anycast fe80:: dev wlan0 table local proto kernel metric 0 pref medium
anycast fe80:: dev wpan0 table local proto kernel metric 0 pref medium
local fe80::9683:c4ff:fe26:4506 dev eth0 table local proto kernel metric 0 pref medium
local fe80::9683:c4ff:fe26:4507 dev br-lan table local proto kernel metric 0 pref medium
local fe80::9683:c4ff:fe26:4508 dev wlan0 table local proto kernel metric 0 pref medium
local fe80::9c69:4748:292a:c3ed dev wpan0 table local proto kernel metric 0 pref medium
local fe80::e5c1:bc16:60e9:f6e8 dev tayga-nat64 table local proto kernel metric 0 pref medium
multicast ff00::/8 dev br-guest table local proto kernel metric 256 linkdown pref medium
multicast ff00::/8 dev br-lan table local proto kernel metric 256 pref medium
multicast ff00::/8 dev tayga-nat64 table local proto kernel metric 256 pref medium
multicast ff00::/8 dev eth0 table local proto kernel metric 256 pref medium
multicast ff00::/8 dev wlan0 table local proto kernel metric 256 pref medium
multicast ff00::/8 dev wpan0 table local proto kernel metric 256 pref medium

~# ip -6 route get fd11:22:0:0:52cd:33bb:5ea4:da42
fd11:22::52cd:33bb:5ea4:da42 from :: via fe80::9683:c4ff:fe12:b0d1 dev eth0 proto ra src fd9f:4d2e:2419:b87d:9683:c4ff:fe26:4506 metric 1024 hoplimit 64 pref medium

after remove the following rule and the system ping work.

fd11:22::/64 from 240e:3b0:3255:1bad::/64 via fe80::bc9f:e1ff:fe29:9ed8 dev eth0 proto static metric 512 pref medium
fd11:22::/64 from 240e:3b0:3255:f6b5::/64 via fe80::bc9f:e1ff:fe29:9ed8 dev eth0 proto static metric 512 pref medium
fd11:22::/64 from fd9f:4d2e:2419:b87d::/64 via fe80::bc9f:e1ff:fe29:9ed8 dev eth0 proto static metric 512 pref medium
fd11:22::/64 from fdd4:1d8c:d98f:b204::/64 via fe80::bc9f:e1ff:fe29:9ed8 dev eth0 proto static metric 512 pref medium
fd11:22::/64 from fdda:b85f:3c43:7bbb::/64 via fe80::bc9f:e1ff:fe29:9ed8 dev eth0 proto static metric 512 pref medium

Below is some log information about the OTBR process,

Tue Feb 14 01:40:48 2023 user.info otbr-agent[29774]: 05:31:42.097 [I] MeshForwarder-: Sent IPv6 HopOpts msg, len:81, chksum:0000, ecn:no, to:0xffff, sec:yes, prio:normal
Tue Feb 14 01:40:48 2023 user.info otbr-agent[29774]: 05:31:42.097 [I] MeshForwarder-:     src:[fd11:22:0:0:52cd:33bb:5ea4:da42]
Tue Feb 14 01:40:48 2023 user.info otbr-agent[29774]: 05:31:42.097 [I] MeshForwarder-:     dst:[ff03:0:0:0:0:0:0:1]
Tue Feb 14 01:40:48 2023 user.info otbr-agent[29774]: 05:31:42.141 [I] Platform------: Session socket is ready
Tue Feb 14 01:40:48 2023 user.info otbr-agent[29774]: 05:31:42.143 [I] Platform------: > ipaddr mleid
Tue Feb 14 01:40:48 2023 user.info otbr-agent[29774]: 05:31:42.195 [I] MeshForwarder-: Sent IPv6 HopOpts msg, len:81, chksum:0000, ecn:no, to:0xffff, sec:yes, prio:normal
Tue Feb 14 01:40:48 2023 user.info otbr-agent[29774]: 05:31:42.196 [I] MeshForwarder-:     src:[fd11:22:0:0:52cd:33bb:5ea4:da42]
Tue Feb 14 01:40:48 2023 user.info otbr-agent[29774]: 05:31:42.198 [I] MeshForwarder-:     dst:[ff03:0:0:0:0:0:0:1]
Tue Feb 14 01:40:52 2023 user.info otbr-agent[29774]: 05:31:45.908 [I] BorderRouter--: Received Router Advertisement from fe80:0:0:0:9683:c4ff:fe12:b0d1 on infra netif 3
Tue Feb 14 01:40:52 2023 user.info otbr-agent[29774]: 05:31:45.908 [I] NetDataPublshr: Publishing ExternalRoute ::/0
Tue Feb 14 01:40:52 2023 user.info otbr-agent[29774]: 05:31:45.909 [I] NetDataPublshr: ExternalRoute ::/0 - State: NoEntry -> ToAdd
Tue Feb 14 01:40:52 2023 user.info otbr-agent[29774]: 05:31:45.909 [I] NetDataPublshr: ExternalRoute ::/0 (state:ToAdd) in netdata - total:0, preferred:0, desired:10
Tue Feb 14 01:40:52 2023 user.info otbr-agent[29774]: 05:31:45.909 [I] NetDataPublshr: ExternalRoute ::/0 - State: ToAdd -> Adding
Tue Feb 14 01:40:52 2023 user.info otbr-agent[29774]: 05:31:45.909 [I] NetDataPublshr: ExternalRoute ::/0 (state:Adding) - update in 598 msec
Tue Feb 14 01:40:52 2023 user.info otbr-agent[29774]: 05:31:45.909 [I] BorderRouter--: Processing PIO (240e:3b0:3255:1bad::/64, 2592000 seconds)
Tue Feb 14 01:40:52 2023 user.info otbr-agent[29774]: 05:31:45.909 [I] NetDataPublshr: Publishing ExternalRoute 240e:3b0:3255:1bad::/64
Tue Feb 14 01:40:52 2023 user.info otbr-agent[29774]: 05:31:45.909 [I] NetDataPublshr: ExternalRoute 240e:3b0:3255:1bad::/64 - State: NoEntry -> ToAdd
Tue Feb 14 01:40:52 2023 user.info otbr-agent[29774]: 05:31:45.910 [I] NetDataPublshr: ExternalRoute 240e:3b0:3255:1bad::/64 (state:ToAd in netdata - total:0, preferred:0, desired:10
Tue Feb 14 01:40:52 2023 user.info otbr-agent[29774]: 05:31:45.920 [I] NetDataPublshr: ExternalRoute 240e:3b0:3255:1bad::/64 - State: ToAdd -> Adding
Tue Feb 14 01:40:52 2023 user.info otbr-agent[29774]: 05:31:45.921 [I] NetDataPublshr: ExternalRoute 240e:3b0:3255:1bad::/64 (state:Addi - update in 2430 msec
Tue Feb 14 01:40:52 2023 user.info otbr-agent[29774]: 05:31:45.934 [I] BorderRouter--: Start evaluating routing policy, scheduled in 3479 milliseconds
Tue Feb 14 01:40:52 2023 user.info otbr-agent[29774]: 05:31:46.507 [I] NetDataPublshr: ExternalRoute ::/0 - State: Adding -> Added
Tue Feb 14 01:40:52 2023 user.info otbr-agent[29774]: 05:31:46.508 [I] NetworkData---: Sent server data notification
Tue Feb 14 01:40:52 2023 user.info otbr-agent[29774]: 05:31:46.508 [I] NetworkData---: Received network data registration
Tue Feb 14 01:40:52 2023 user.info otbr-agent[29774]: 05:31:46.508 [I] NetworkData---: Sent network data registration acknowledgment
Tue Feb 14 01:40:52 2023 user.info otbr-agent[29774]: 05:31:46.508 [I] Notifier------: StateChanged (0x00000200) [NetData]
Tue Feb 14 01:40:52 2023 user.info otbr-agent[29774]: 05:31:46.509 [I] Mle-----------: Send Data Response (ff02:0:0:0:0:0:0:1)
Tue Feb 14 01:40:52 2023 user.info otbr-agent[29774]: 05:31:46.509 [I] BbrLeader-----: PBBR state: None
Tue Feb 14 01:40:52 2023 user.info otbr-agent[29774]: 05:31:46.509 [I] BbrLeader-----: Domain Prefix: ::/0, state: None
Tue Feb 14 01:40:52 2023 user.info otbr-agent[29774]: 05:31:46.509 [I] BorderRouter--: Start evaluating routing policy, scheduled in 3762 milliseconds
Tue Feb 14 01:40:52 2023 user.info otbr-agent[29774]: 05:31:46.509 [I] NetDataPublshr: DNS/SRP service (state:Added) in netdata - total:1, preferred:0, desired:2
Tue Feb 14 01:40:52 2023 user.info otbr-agent[29774]: 05:31:46.509 [I] NetDataPublshr: OnMeshPrefix fd11:22:0:0::/64 (state:Added) in netdata - total:1, preferred:0, desired:3
Tue Feb 14 01:40:52 2023 user.info otbr-agent[29774]: 05:31:46.510 [I] NetDataPublshr: ExternalRoute fd9f:4d2e:2419:b87d::/64 (state:Add in netdata - total:1, preferred:0, desired:10
Tue Feb 14 01:40:52 2023 user.info otbr-agent[29774]: 05:31:46.516 [I] NetDataPublshr: ExternalRoute fdda:b85f:3c43:7bbb::/64 (state:Add in netdata - total:1, preferred:0, desired:10
Tue Feb 14 01:40:52 2023 user.info otbr-agent[29774]: 05:31:46.518 [I] NetDataPublshr: ExternalRoute fdd4:1d8c:d98f:b204::/64 (state:Add in netdata - total:1, preferred:0, desired:10
Tue Feb 14 01:40:52 2023 user.info otbr-agent[29774]: 05:31:46.519 [I] NetDataPublshr: ExternalRoute ::/0 (state:Added) in netdata - total:1, preferred:0, desired:10
Tue Feb 14 01:40:52 2023 user.info otbr-agent[29774]: 05:31:46.520 [I] NetDataPublshr: ExternalRoute 240e:3b0:3255:1bad::/64 (state:Addi in netdata - total:0, preferred:0, desired:10
Tue Feb 14 01:40:52 2023 user.info otbr-agent[29774]: 05:31:46.566 [I] Platform------: Execute command `ipset flush otbr-ingress-allow-dst-swap` = 0
Tue Feb 14 01:40:52 2023 user.info otbr-agent[29774]: 05:31:46.603 [I] Platform------: Execute command `ipset flush otbr-ingress-deny-src-swap` = 0
Tue Feb 14 01:40:52 2023 user.info otbr-agent[29774]: 05:31:46.619 [I] Platform------: Execute command `ipset add otbr-ingress-deny-src-swap fd11:22:0:0::/64 -exist` = 0
Tue Feb 14 01:40:52 2023 user.info otbr-agent[29774]: 05:31:46.636 [I] Platform------: Execute command `ipset add otbr-ingress-deny-src-swap fdde:ad00:beef:0::/64 -exist` = 0
Tue Feb 14 01:40:52 2023 user.info otbr-agent[29774]: 05:31:46.652 [I] Platform------: Execute command `ipset add otbr-ingress-allow-dst-swap fd11:22:0:0::/64 -exist` = 0
Tue Feb 14 01:40:52 2023 user.info otbr-agent[29774]: 05:31:46.669 [I] Platform------: Execute command `ipset swap otbr-ingress-deny-src-swap otbr-ingress-deny-src` = 0
Tue Feb 14 01:40:52 2023 user.info otbr-agent[29774]: 05:31:46.686 [I] Platform------: Execute command `ipset swap otbr-ingress-allow-dst-swap otbr-ingress-allow-dst` = 0
Tue Feb 14 01:40:52 2023 user.info otbr-agent[29774]: 05:31:46.703 [I] MeshForwarder-: Sent IPv6 UDP msg, len:205, chksum:153e, ecn:no, to:0xffff, sec:yes, prio:net
Tue Feb 14 01:40:52 2023 user.info otbr-agent[29774]: 05:31:46.705 [I] MeshForwarder-:     src:[fe80:0:0:0:9c69:4748:292a:c3ed]:19788
Tue Feb 14 01:40:52 2023 user.info otbr-agent[29774]: 05:31:46.707 [I] MeshForwarder-:     dst:[ff02:0:0:0:0:0:0:1]:19788
Tue Feb 14 01:40:53 2023 user.info otbr-agent[29774]: 05:31:46.811 [I] MeshForwarder-: Received IPv6 UDP msg, len:107, chksum:d97f, ecn:no, from:4aaa76d6cb70ee5c, sec:no, prio:net, rss:-43.0

I don't know why this happened. I can't found the following rule in other test environment,

fd11:22::/64 from 240e:3b0:3255:1bad::/64 via fe80::bc9f:e1ff:fe29:9ed8 dev eth0 proto static metric 512 pref medium
fd11:22::/64 from 240e:3b0:3255:f6b5::/64 via fe80::bc9f:e1ff:fe29:9ed8 dev eth0 proto static metric 512 pref medium
fd11:22::/64 from fd9f:4d2e:2419:b87d::/64 via fe80::bc9f:e1ff:fe29:9ed8 dev eth0 proto static metric 512 pref medium
fd11:22::/64 from fdd4:1d8c:d98f:b204::/64 via fe80::bc9f:e1ff:fe29:9ed8 dev eth0 proto static metric 512 pref medium
fd11:22::/64 from fdda:b85f:3c43:7bbb::/64 via fe80::bc9f:e1ff:fe29:9ed8 dev eth0 proto static metric 512 pref medium

To Reproduce Information to reproduce the behavior, including:

1. Git commit id: ot-br-posix porting from GSDK4.2.0
2. IEEE 802.15.4 hardware platform EFR32MG21F768
3. Build steps

/usr/sbin/otbr-agent -d 6 -B eth0 -I wpan0 spinel+hdlc+uart:///dev/ttyCH343USB0?uart-baudrate=1000000&uart-flow-control trel://eth0

5. Network topology

Expected behavior A clear and concise description of what you expected to happen.

Console/log output If applicable, add console/log output to help explain your problem.

Additional context Add any other context about the problem here.

[lancersky2017]

Is it related to whether the router the OTBR connects to has access to the global IPv6 network?

[jwhui]

The problem is that you have routes to fd11:22::/64 going out the eth0 interface, rather than wpan0 interface. This can cause the error you see with ping command.

Are you manually configuring the fd11:22::/64 prefix anywhere? Typically, unique local addresses should have more randomness than that.

This could happen if you have another border router that is using the same fd11:22::/64 prefix, which it would advertise using ICMPv6 Router Advertisements and cause the route to be installed via eth0 interface.

[lancersky2017]

Yeah, I did manually configure the prefix from OTBR under test. And we are allowed to configure only one prefix from our WebUI.
Do we need to configure a different prefix if have another border router in the Thread network? All our border routers are configured with the prefix fd11:22::/64 by default.

[jwhui]

When using the latest OpenThread Border Router implementation, the Border Router should generate its own Unique-Local Prefix - see Thread Border Router codelab.

Configuring the same prefix across all your Thread networks will cause this issue.

[lancersky2017]

Hi @jwhui, Is it possible to configure prefix through different OTBR in a Thread network? And how to avoid configuring the same prefix?

[jwhui]

Just don't manually configure a prefix - Thread Border Routers will automatically generate their own random Unique Local Prefix. See Thread Border Router codelab.

[lancersky2017]

Sorry, I have to double-check. According to the document https://github.com/openthread/openthread/tree/main/src/cli#prefix-add-prefix-padcrosndprf.
It provides different configuration parameters. Do you mean that it is not recommended that we use the prefix add command to configure manually?

[jwhui]

Correct, do not use the prefix add command - there is no need to manually configure a prefix.