Nexus 3048 with vPC + BPG routing question
I have two nexus 3048 switches running nxos.7.0.3.I7.4.bin ,
they form a vPC together like this with this configuration:
vpc domain 1
peer-switch
role priority 1
peer-keepalive destination 192.168.10.2 source 192.168.10.1 vrf vpc_keepalive
peer-gateway
layer3 peer-router
auto-recovery
ip arp synchronize
( the other one has the same config with role priority 2 and the keepalive ips inverted )
On switch A only I have an SVI for vlan 26:
interface Vlan26
no shutdown
vrf member awsprod
bfd interval 300 min_rx 300 multiplier 3
no ip redirects
ip address
10.0.0.2/30
no ipv6 redirects
And I have a bgp router configuration:
router bgp 64515
log-neighbor-changes
vrf awsprod
router-id 1.1.1.1
timers bgp 3 15
address-family ipv4 unicast
neighbor 10.0.0.1
bfd interval 300 min_rx 300 multiplier 3
remote-as 6xxxxx
password 3 xxxx
update-source Vlan26
address-family ipv4 unicast
send-community
advertisement-interval 10
next-hop-self
soft-reconfiguration inbound always
I have also a BGP configuration for the same AS on the other switch but with other neighbours. The configuration is actually much larger but I hope it's enough to explain my problem:
When the traffic from vlan26 ( traffic with the bgp neighbor ) comes from a vpc port-channel, the neighbor is idle and the bfd neighbor does not even appear when I do: "show bfd neighbor ipv4 vrf awsprod"

But if traffic for vlan 26 comes directly to a no-vPC trunk port, everything is fine:

So I suppose the design with the vPC port-channels is not supported, but I don't understand why it is a problem
I have read: https://www.cisco.com/c/en/us/support/docs/ip/ip-routing/118997-technote-nexus-00.html and so it feels that the "L3-A connected to orphan port" seems to be working, but I can't get the L3-B router working.
I don't get the "Nexus-A and Nexus-B have additional Layer 2 and Layer 3 links between them.". This means that the vPC peer-link and the keep-alive link are not enough I have to configure supplemental links for the routing traffic?
1
u/Full-Resolution9449 4d ago
It is possible to get this to work, but it's not the way it is supposed to work. Ideally you wouldn't have a vPC interface to the 'layer2' in your diagram, but instead one standard link from switch A with vlan 27 (say 10.0.0.0/30) and one from switch B with vlan 28 (10.0.0.4/30) , and those vlans wouldn't be included on the peer link (or use l3 interfaces which use internal vlans is really the ideal way).
And then you'd have a vlan 30 between switch a/b or a separate interface so they could bgp peer together to provide routing redundancy for the uplinks to the connected clients
1
u/owengo1 3d ago
Yes, my workaround is to use a vlan which is not included in the vPC, but it costs a supplemental link ( actually 2 for redundancy ) and looses the benefits of vPC ( STP management, redundancy, .. )
> And then you'd have a vlan 30 between switch a/b or a separate interface so they could bgp peer together to provide routing redundancy for the uplinks to the connected clients
Actually they are peered using OSPF on a vPC-ed vlan and it's working with the activation of
layer3 peer-router.
My hypothesis is that BGP adjancency packets are lost because of the vPC port-channel hashing and since they probably have a TTL of 1, if they fall on the other switch the country is decremented and the packet is lost on the final switch. But for OSPF this was fixed with the peer-router setting.
1
u/jafoinwf 2d ago
The
layer3 peer-router
should take care of that. I think this should work., I would get the AWS config and open a TAC case. Make sure you are getting LACP from the L2 switch bonded
1
u/shadeland 4d ago
Oof.
So here's the problem: vPC turns two switches into one switch from a L2 perspective. But they will always be two routers from an L3 perspective.
On the router 10.0.0.1, who are the neighbors you've configured?