Regression in next-hop IPv4 address resolution (ARP)

[dpino]

IPv4 next-hop resolution is failing in some scenarios.

Consider the following config file:

softwire-config {
    binding-table {
        softwire { ipv4 193.5.1.100; psid 1; b4-ipv6 fc00:1:2:3:4:5:0:7e; br-address fc00::100; port-set { psid-length 6; }}
        softwire { ipv4 193.5.1.100; psid 2; b4-ipv6 fc00:1:2:3:4:5:0:7f; br-address fc00::100; port-set { psid-length 6; }}
        softwire { ipv4 193.5.1.100; psid 3; b4-ipv6 fc00:1:2:3:4:5:0:80; br-address fc00::100; port-set { psid-length 6; }}
        softwire { ipv4 193.5.1.100; psid 4; b4-ipv6 fc00:1:2:3:4:5:0:81; br-address fc00::100; port-set { psid-length 6; }}
        softwire { ipv4 193.5.1.100; psid 5; b4-ipv6 fc00:1:2:3:4:5:0:82; br-address fc00::100; port-set { psid-length 6; }}
        softwire { ipv4 193.5.1.100; psid 6; b4-ipv6 fc00:1:2:3:4:5:0:83; br-address fc00::100; port-set { psid-length 6; }}
        softwire { ipv4 193.5.1.100; psid 7; b4-ipv6 fc00:1:2:3:4:5:0:84; br-address fc00::100; port-set { psid-length 6; }}
        softwire { ipv4 193.5.1.100; psid 8; b4-ipv6 fc00:1:2:3:4:5:0:85; br-address fc00::100; port-set { psid-length 6; }}
        softwire { ipv4 193.5.1.100; psid 9; b4-ipv6 fc00:1:2:3:4:5:0:86; br-address fc00::100; port-set { psid-length 6; }}
        softwire { ipv4 193.5.1.100; psid 10; b4-ipv6 fc00:1:2:3:4:5:0:87; br-address fc00::100; port-set { psid-length 6; }}
    }
    external-interface {
        allow-incoming-icmp false;
        error-rate-limiting {
            packets 600000;
        }
        reassembly {
            max-fragments-per-packet 40;
        }
    }
    internal-interface {
        allow-incoming-icmp false;
        error-rate-limiting {
            packets 600000;
        }
        reassembly {
            max-fragments-per-packet 40;
        }
    }
    instance {
        device 82:00.0;
        queue {
            id 1;
            external-interface {
                ip 10.0.1.1;
                mac 02:aa:aa:aa:aa:aa;
                next-hop {
                    ip 10.0.1.2;
                }
            }
            internal-interface {
                ip fc00::100;
                mac 02:aa:aa:aa:aa:aa;
                next-hop {
                    ip fc00::101;                    
                }
            }
        }
    }
}

ARP resolution fails:

$ sudo ./snabb lwaftr run --conf lwaftr-small.conf --on-a-stick 82:00.0
lwaftr-small.conf: loading compiled configuration from lwaftr-small.o
lwaftr-small.conf: compiled configuration is up to date.
Warning: No assignable CPUs declared; leaving data-plane PID 1679 without assigned CPU.
No CPUs available; not binding to any NUMA node.
ARP: Resolving '10.0.1.2'
NDP: Resolving 'fc00::101'
ARP: Resolving '10.0.1.2'
NDP: Resolving 'fc00::101'
NDP: 'fc00::101' resolved (90:e2:ba:a9:84:1c)
ARP: Resolving '10.0.1.2'
ARP: Resolving '10.0.1.2'

However, it works if queue's id is 0:

$ sudo ./snabb lwaftr run --conf lwaftr-small.conf --on-a-stick 82:00.0
lwaftr-small.conf: loading compiled configuration from lwaftr-small.o
lwaftr-small.conf: failed to load compiled configuration: core/main.lua:26: incompatible version: 20480
lwaftr-small.conf: loading source configuration
lwaftr-small.conf: wrote compiled configuration lwaftr-small.o
Warning: No assignable CPUs declared; leaving data-plane PID 30344 without assigned CPU.
No CPUs available; not binding to any NUMA node.
ARP: Resolving '10.0.1.2'
NDP: Resolving 'fc00::101'
ARP: Resolving '10.0.1.2'
NDP: Resolving 'fc00::101'
ARP: '10.0.1.2' resolved (90:e2:ba:a9:84:1c)
NDP: 'fc00::101' resolved (90:e2:ba:a9:84:1c)

The regression was introduced somewhere between v2017.08.03 and v2017.08.04. After running a bisec the commit that introduced the regression was:

commit 5a279a3a785ee5b5d5b3ae52140f79f5461f8a3c
Author: Andy Wingo <[email protected]>
Date:   Tue Sep 26 08:59:39 2017 +0000

    Set queue ID as RSS queue ID

 src/program/lwaftr/setup.lua | 10 ++++++++++
 1 file changed, 10 insertions(+)

[dpino]

I tried to debug this issue and what I noticed that when queue 1, the ARP part is not receiving ARP packets. It receives an DHCP packet though (probably garbage generated in the network):

000000 ff ff ff ff ff ff 90 e2 ba a9 84 1c 08 00 45 00
000010 01 62 80 10 00 00 40 11 f9 7b 00 00 00 00 ff ff
000020 ff ff 00 44 00 43 01 4e 43 c0 01 01 06 00 66 5f
000030 91 71 00 0a 00 00 00 00 00 00 00 00 00 00 00 00
000040 00 00 00 00 00 00 90 e2 ba a9 84 1c 00 00 00 00
000050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
000060 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
000070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
000080 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
000090 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0000a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0000b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0000c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0000d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0000e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0000f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
000100 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
000110 00 00 00 00 00 00 63 82 53 63 35 01 01 74 01 01
000120 39 02 05 c0 3c 2e 64 68 63 70 63 64 2d 36 2e 31
000130 31 2e 35 3a 4c 69 6e 75 78 2d 34 2e 34 2e 38 30
000140 3a 78 38 36 5f 36 34 3a 47 65 6e 75 69 6e 65 49
000150 6e 74 65 6c 0c 06 73 6e 61 62 62 31 91 01 01 37
000160 0e 01 79 21 03 06 0c 0f 1a 1c 2a 33 3a 3b 77 ff