NSX-T Federation Stretched T0 gateway Active-Active with locations Primary & Secondary – OSPF in physical network

I have covered a couple of NSX-T Federation blogs earlier.

In this blog, we will setup OSPF in the underlay network as opposed to BGP. And we will setup stretched T0 Gateway with Active-Active HA mode. We will be using additional redistribution control from NSX end to ensure traffic ingresses/egresses via physical gateway in one location.

Here, I am using OSPF backbone area 0 but in practical designs one should consider multiple OSPF areas for large enterprise networks.

Technology Overview for NSX-T Federation:

NSX-T Federation provides networking and security across multiple locations.

With NSX Federation, you can manage multiple NSX-T Data Center environments with a single pane of glass view, create gateways and segments that span one or more locations, and configure and enforce firewall rules consistently across locations.

NSX segments (overlay networks) can span multiple locations.

Span of NSX segment is equal to the span of the attached NSX gateway. These overlay networks of NSX can be connected to Tier 0 Gateway or Tier 1 Gateway ( in case of multi tier routing topology). Multi tier routing topology is recommended because it gives flexibility to use stateful services on Tier 1 Gateway if needed.

Federation key concepts are:

Global Manager – provides a centralized management for networking and security services of multiple locations.

Global Manager Active instance (made of 3 Global Manager VMs) is placed in one Location.

And the Standby Global Manager instance (made of 3 Global Manager VMs) is placed in another location.

In addition to Global Managers, there are NSX-T Local Managers which are used to:

  • Configure Transport Zones in a location
  • Configure Transport Node Profiles which are used to install NSX on servers
  • Install NSX on servers in a location
  • Deploy edges and configure NSX on those NSX edges

NSX Gateways in the case of Federation can have a span which covers one location or can be stretched where the gateway spans multiple locations.

Tier-0 gateways can have one of the following configurations:

  • Non-stretched tier-0 gateway.
  • Stretched active-active with primary and secondary locations.
  • Stretched active-active with all primary locations.
  • Stretched active-standby with primary and secondary locations

The different topologies possible are documented here

In the case of NSX Federation, NSX edges (which mainly handle north-south traffic in the software defined data center and also provide services like NAT, gateway firewall) have Remote Tunnel Endpont Interface RTEP. This interface handles cross location traffic between different data centers. Essentially the cross site tunnels are established by using this RTEP interface.

Edge TEP interface will have tunnels between itself and TEP interfaces on hosts/servers within the same location. With NSX Federation, there will be no Geneve tunnels between two servers which are in different locations.

How segment stretching works?

How segment stretching works?

Above figure shows two stretched networks. Within a location, specific edge offers segment stretching for a particular stretched network. As seen above, blue stretched segment is active on one edge in Mumbai and this blue stretched segment is active on one edge in Bangalore location. And the same logic is followed for segment stretching related to Orange stretched network. Above figure shows traffic flow between VMs connected to stretched network.

In one of my previous blogs, I have covered a topology where stretched Tier 0 Gateway is in Active-Active availability mode with locations as Primary & Secondary. But in that scenario BGP was used in underlay physical network. In this lab, we are replacing BGP in the physical network with OSPF.

This blog covers

  • routing for stretched Tier 0 Gateway which spans two locations.
  • Two locations are Mumbai and Bangalore
  • Availability mode on Tier 0 Gateway is Active-Active
  • Stretched Tier 0 Gateway has Mumbai as Primary Location and Bangalore as Secondary Location
  • e-BGP between NSX edges and upstream physical routers
  • OSPF backbone area 0 is configured on physical routers. All interfaces on the physical routers are placed in backbone area 0.
Logical Topology
OSPF Setup in physical network

Above figure shows OSPF setup on the physical routers. In this lab, all the interfaces of all physical routers are placed in OSPF Area 0.

BGP Setup

Above figure shows how BGP is setup in physical network and on NSX Tier 0 Gateway. e-BGP is in use between NSX edges and upstream physical routers. Physical routers will advertise default route towards NSX edges on a per BGP neighbor basis.

In this lab setup, I have prepared NSX Fabric in both locations. There is no standby Global Manager in this lab. For production setup, it is a must to deploy standby Global Manager.

There are site local subnets for host TEP, edge TEP, edge RTEP, edge uplinks

There is NSX local manager in both the locations.

We need to define IP pools on the local NSX manager for host TEP, edge TEP and edge RTEP interfaces.

Compute TEP Pool
Edge TEP Pool

Besides the default transport zones, edge uplink transport zone has been configured. This transport zone is used to configure uplinks on NSX edges.

Transport Zones in Mumbai

We need to configure uplink profiles for:

  • Hosts
  • Edges
Uplink Profile for servers in Mumbai

Uplink profile for edges in Mumbai

Each server has 4 pnics and there are two VDS’ per cluster.

  • One VDS for handling ESXi management, vmotion, NSX traffic
  • Second VDS for storage traffic.
4 pnic server

Next configure Transport Node Profile which will be attached to cluster. You can have Transport Node Profile on a cluster basis.

Transport Node Profile for servers in Mumbai

Using this transport node profile, servers are prepared for NSX.

Servers in Mumbai configured for NSX

Next edge VMs are deployed and configured for NSX.

NSX Edge Config

Two edges have been deployed and edge cluster is created out of those edges.

NSX Edges configured for NSX
Edge Cluster in Mumbai location

With this, NSX fabric in Mumbai has been configured.

The above steps are to be followed to prepare NSX fabric in Bangalore location as well.

Next deploy NSX-T Global Managers, add locations and configure RTEP interfaces on edges.

Locations added to Global Manager
RTEP config

RTEP config has to be done on edges of both locations.

Next, we will login to NSX-T Active Global Manager and configure segments to be used for stretched Tier 0 Gateway uplink interfaces.

Segments on Global Manager

We will configure stretched Tier 0 Gateway and specify

  • Mumbai as Primary Location
  • Bangalore as Secondary Location
  • Availability mode as Active-Active
  • Specify edge cluster for each location
Stretched Tier 0 Gateway which spans Mumbai Location & Bangalore Location

Configure Layer 3 interfaces on stretched Tier 0 Gateway

For each interface specify:

  • Location
  • IP Address
  • Connected segment
  • Edge Node
Layer 3 interfaces on Tier 0 Gateway

Configure BGP

  • Specify local AS number for the stretched Tier 0 Gateway
BGP AS number on stretched Tier 0 Gateway

Configure BGP peers. For each peer specify

  • Peer IP address
  • Location
  • Remote BGP AS number
  • Source addresses for BGP peering
BGP neighbors on stretched Tier 0 Gateway

Enable redistribution on Tier 0 Gateway for Mumbai location only. We will not enable route redistribution for secondary location Bangalore.

Redistribute connected interfaces as shown below. As a result, NSX prefixes will be advertised via primary location Mumbai only.

Enable redistribution on stretched Tier 0 Gateway from primary location
Redistribute connected on Global T0

Next configure stretched T1, connect it to Tier 0 Gateway and advertise connected networks.

Here Tier 1 Gateway is DR only Tier 1 Gateway which is not connected to any edge cluster.

In this lab, there are no stateful services running on Tier 1 Gateway but they can be enabled based on specific requirement.

Tier 1 Gateway

Next configure overlay network / segment which is attached to Tier 1 Gateway.

Segment connected to Tier 1 Gateway

Next from vsphere client, use this overlay network on appropriate VMs.

VMs connected to Web Overlay Network


Traffic flows

Egress traffic flow from VMs in Bangalore location
Ingress traffic flow towards VMs in Bangalore location

BGP routes on edges

Here the key point is that edges in Bangalore location do not advertise NSX routes because redistribution is disabled from Bangalore location.

No NSX prefixes received on physical router 1 in Bangalore location
OSPF neighborship and BGP peering on physical router in Bangalore location

If we check the route for NSX prefix on physical router in Bangalore location, next hop is towards physical routers in primary location Mumbai

NSX prefix being learnt as OSPF External Type 2 LSA on physical router in Bangalore is loopback interface of physical router 1 in Mumbai location is loopback interface of physical router 2 in Mumbai location.

Trace from VM in Bangalore location to loopback of physical router 2 in Bangalore location

VM in Bangalore with IP can reach below loopbacks: on physical router 1 in Mumbai location. on physical router 2 in Mumbai location. on physical router 1 in Bangalore location. on physical router 2 in Bangalore location.

Trace from loopback of Bangalore router to VM in Bangalore location

Physical router 2 in Bangalore location can reach VM in Bangalore location.

Disaster Recovery:

Upon complete failure of primary location, as part of disaster recovery:

a. Global manager in secondary location will be made as active. Data plane recovers automatically in case stretched T0 is Active/Active and Tier 1 Gateway is DR only gateway.

b. Ensure NSX routes are redistributed from secondary location.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s