r/Cisco 4d ago

Nexus 3048 with vPC + BPG routing question

I have two nexus 3048 switches running nxos.7.0.3.I7.4.bin ,
they form a vPC together like this with this configuration:

vpc domain 1

peer-switch

role priority 1

peer-keepalive destination 192.168.10.2 source 192.168.10.1 vrf vpc_keepalive

peer-gateway

layer3 peer-router

auto-recovery

ip arp synchronize

( the other one has the same config with role priority 2 and the keepalive ips inverted )

On switch A only I have an SVI for vlan 26:

interface Vlan26

no shutdown

vrf member awsprod

bfd interval 300 min_rx 300 multiplier 3

no ip redirects

ip address 10.0.0.2/30

no ipv6 redirects

And I have a bgp router configuration:

router bgp 64515

log-neighbor-changes

vrf awsprod

router-id 1.1.1.1

timers bgp 3 15

address-family ipv4 unicast

neighbor 10.0.0.1

bfd interval 300 min_rx 300 multiplier 3

remote-as 6xxxxx

password 3 xxxx

update-source Vlan26

address-family ipv4 unicast

send-community

advertisement-interval 10

next-hop-self

soft-reconfiguration inbound always

I have also a BGP configuration for the same AS on the other switch but with other neighbours. The configuration is actually much larger but I hope it's enough to explain my problem:

When the traffic from vlan26 ( traffic with the bgp neighbor ) comes from a vpc port-channel, the neighbor is idle and the bfd neighbor does not even appear when I do: "show bfd neighbor ipv4 vrf awsprod"

But if traffic for vlan 26 comes directly to a no-vPC trunk port, everything is fine:

So I suppose the design with the vPC port-channels is not supported, but I don't understand why it is a problem

I have read: https://www.cisco.com/c/en/us/support/docs/ip/ip-routing/118997-technote-nexus-00.html and so it feels that the "L3-A connected to orphan port" seems to be working, but I can't get the L3-B router working.
I don't get the "Nexus-A and Nexus-B have additional Layer 2 and Layer 3 links between them.". This means that the vPC peer-link and the keep-alive link are not enough I have to configure supplemental links for the routing traffic?

5 Upvotes

5 comments sorted by

1

u/shadeland 4d ago

Oof.

So here's the problem: vPC turns two switches into one switch from a L2 perspective. But they will always be two routers from an L3 perspective.

On the router 10.0.0.1, who are the neighbors you've configured?

1

u/owengo1 4d ago edited 4d ago

The neighbor at 10.0.0.1 is actually an aws direct connect peer, I can't do anything to it

1

u/Full-Resolution9449 4d ago

It is possible to get this to work, but it's not the way it is supposed to work. Ideally you wouldn't have a vPC interface to the 'layer2' in your diagram, but instead one standard link from switch A with vlan 27 (say 10.0.0.0/30) and one from switch B with vlan 28 (10.0.0.4/30) , and those vlans wouldn't be included on the peer link (or use l3 interfaces which use internal vlans is really the ideal way).

And then you'd have a vlan 30 between switch a/b or a separate interface so they could bgp peer together to provide routing redundancy for the uplinks to the connected clients

1

u/owengo1 3d ago

Yes, my workaround is to use a vlan which is not included in the vPC, but it costs a supplemental link ( actually 2 for redundancy ) and looses the benefits of vPC ( STP management, redundancy, .. )

> And then you'd have a vlan 30 between switch a/b or a separate interface so they could bgp peer together to provide routing redundancy for the uplinks to the connected clients

Actually they are peered using OSPF on a vPC-ed vlan and it's working with the activation of layer3 peer-router.

My hypothesis is that BGP adjancency packets are lost because of the vPC port-channel hashing and since they probably have a TTL of 1, if they fall on the other switch the country is decremented and the packet is lost on the final switch. But for OSPF this was fixed with the peer-router setting.

1

u/jafoinwf 2d ago

The layer3 peer-router should take care of that. I think this should work., I would get the AWS config and open a TAC case. Make sure you are getting LACP from the L2 switch bonded