nsx-t federation stretched t0 gateway active & standby with locations primary & secondary

NSX-T Federation provides networking and security across multiple locations.

With NSX Federation, you can manage multiple NSX-T Data Center environments with a single pane of glass view, create gateways and segments that span one or more locations, and configure and enforce firewall rules consistently across locations.

NSX segments (overlay networks) can span multiple locations.

Span of NSX segment is equal to the span of the attached NSX gateway. These overlay networks of NSX can be connected to Tier 0 Gateway or Tier 1 Gateway ( in case of multi tier routing topology)

Federation key concepts are:

Global Manager – which provides a centralized management for networking and security services of multiple locations.

Global Manager Active instance (made of 3 Global Manager VMs) is placed in one Location.

And the Standby Global Manager instance (made of 3 Global Manager VMs) is placed in another location.

In addition to Global Managers, there are NSX-T Local Managers which are used to:

  • Configure Transport Zones in a location
  • Configure Transport Node Profiles which are used to install NSX on servers
  • Install NSX on servers in a location
  • Deploy edges and configure NSX on those NSX edges

NSX Gateways in the case of Federation can have a span which covers one location or can be stretched where the gateway spans multiple locations.

Tier-0 gateways can have one of the following configurations:

  • Non-stretched tier-0 gateway.
  • Stretched active-active with primary and secondary locations.
  • Stretched active-active with all primary locations.
  • Stretched active-standby with primary and secondary locations

The different topologies possible are documented here

In the case of NSX Federation, NSX edges (which mainly handle north-south traffic in the software defined data center and also provide services like NAT, gateway firewall) have Remote Tunnel Endpont Interface RTEP. This interface handles cross location traffic between different data centers. Essentially the cross site tunnels are established by using this RTEP interface.

Edge TEP interface will have tunnels between itself and TEP interfaces on hosts/servers within the same location. With NSX Federation, there will be no Geneve tunnels between two servers which are in different locations.

In one of my previous blogs, I have covered a topology where stretched Tier 0 Gateway is in Active-Active availability mode with locations as Primary & Secondary.

This blog covers

  • routing for stretched Tier 0 Gateway which spans two locations.
  • Two locations are Mumbai and Bangalore
  • Availability mode on Tier 0 Gateway is Active & Standby
  • Stretched Tier 0 Gateway has Mumbai as Primary Location and Bangalore as Secondary Location
Logical Topology
BGP Setup

In this lab setup, I have prepared NSX Fabric in both locations. There is no standby Global Manager in this lab. For production setup, it is a must to deploy standby Global Manager.

There are site local subnets for host TEP, edge TEP, edge RTEP, edge uplinks

There is NSX local manager in both the locations.

We need to define IP pools on the local NSX manager for host TEP, edge TEP and edge RTEP interfaces.

Compute TEP Pool
Edge TEP Pool

Besides the default transport zones, edge uplink transport zone has been configured. This transport zone is used to configure uplinks on NSX edges.

Transport Zones in Mumbai

We need to configure uplink profiles for:

  • Hosts
  • Edges
Uplink Profile for servers in Mumbai

Uplink profile for edges in Mumbai

Each server has 4 pnics and there are two VDS’ per cluster.

  • One VDS for handling ESXi management, vmotion, NSX traffic
  • Second VDS for storage traffic.
4 pnic server

Next configure Transport Node Profile which will be attached to cluster. You can have Transport Node Profile on a cluster basis.

Transport Node Profile for servers in Mumbai

Using this transport node profile, servers are prepared for NSX.

Servers in Mumbai configured for NSX

Next edge VMs are deployed and configured for NSX.

NSX Edge Config

Two edges have been deployed and edge cluster is created out of those edges.

NSX Edges configured for NSX
Edge Cluster in Mumbai location

With this, NSX fabric in Mumbai has been configured.

The above steps are to be followed to prepare NSX fabric in Bangalore location as well.

Next deploy NSX-T Global Managers, add locations and configure RTEP interfaces on edges.

Locations added to Global Manager
RTEP config

RTEP config has to be done on edges of both locations.

Next, we will login to NSX-T Active Global Manager and configure segments to be used for stretched Tier 0 Gateway uplink interfaces.

Segments on Global Manager

We will configure stretched Tier 0 Gateway and specify

  • Mumbai as Primary Location
  • Bangalore as Secondary Location
  • Availability mode as Active-Standby
  • Specify edge cluster for each location
Stretched Tier 0 Gateway which spans Mumbai Location & Bangalore Location

Configure Layer 3 interfaces on stretched Tier 0 Gateway

For each interface specify:

  • Location
  • IP Address
  • Connected segment
  • Edge Node
Layer 3 interfaces on Tier 0 Gateway

Configure BGP

  • Specify local AS number for the stretched Tier 0 Gateway
BGP AS number on stretched Tier 0 Gateway

Configure BGP peers. For each peer specify

  • Peer IP address
  • Location
  • Remote BGP AS number
  • Source addresses for BGP peering
BGP neighbors on stretched Tier 0 Gateway

Enable redistribution on Tier 0 Gateway for each location.

Redistribute connected interface.

Enable redistribution on stretched Tier 0 Gateway
Redistribute connected

Next configure stretched T1, connect it to Tier 0 Gateway and advertise connected networks.

Here Tier 1 Gateway is DR only Tier 1 Gateway which is not connected to any edge cluster.

In this lab, there are no stateful services running on Tier 1 Gateway but they can be enabled based on specific requirement.

Tier 1 Gateway

Next configure overlay network / segment which is attached to Tier 1 Gateway.

Segment connected to Tier 1 Gateway

Next from vsphere client, use this overlay network on appropriate VMs.

VMs connected to Web Overlay Network


Traffic flows

Traffic flow from Bangalore router to VM in Bangalore
Traffic flow from VM in Bangalore to loopback of Bangalore router

We will validate the HA status of edge in Bangalore and ensure it is in Active state.

BGP routes on edges

Here the key point is that edge in Bangalore location does not advertise routes nor does it receive BGP routes via e-BGP sessions.

No inbound/outbound routes on edge in Bangalore location

Now we will check BGP route learning on active edge in Mumbai location which is Primary location for the stretched Tier 0 Gateway.

Trace from VM in Bangalore location to loopback of router in Bangalore location
Trace from loopback of Bangalore router to VM in Bangalore location
No prefixes received on router in Bangalore from edges in Bangalore
BGP paths on router in Bangalore location

BGP paths on router in Bangalore location are the IP addresses belonging to routers in Mumbai location.

One thought on “nsx-t federation stretched t0 gateway active & standby with locations primary & secondary

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s