– Active and Standby Global Managers are placed in two different locations.
– Configuration change is done on the Active Global Manager which is then synced to Standby Global Manager.
Fabric preparation which includes creation of transport zones, host transport nodes, edge transport nodes is done using Local Manager.
– Tier 0 Gateways and Tier 1 Gateways can be stretched across multiple locations and here the number of locations are not just limited to two locations, you could also have a third location.
Segments can be stretched across multiple locations
– Ability to have non stretched Tier 1 Gateways in individual locations.
Here span of the segment could be limited to one location.
– Span of Tier 1 Gateway has to be less than or equal to the span of Tier 0 Gateway.
|Vcenters, Clusters, Hosts and Edge Nodes|
In the lab setup:
1. There is vcenter in each location
2. NSX-T manager is also deployed in each location
3. There are two locations here – Bangalore and Delhi
4. NSX T Edge Clusters comprising of two edges are deployed in each location.
Two edges in Bangalore and two edges in Delhi.
4. Hosts in each location are prepared as Host Transport Nodes.
5. In my lab setup, there is only one Active Global Manager deployed.
In a production setup there will Standby Global Manager as well in other location.
6. Tier 0 Gateway has Active-Active availability mode.
7. Bangalore is Primary location and Delhi is configured as Secondary location with respect to NSX-T Federation.
Deployment of Global Manager is similar to deployment of Local Manager, you need to specify Location name and very importantly the role Global Manager while deploying Global NSX-T Manager.
Once, Global Manager is deployed then you are able to add locations
While adding a location, you need to specify
– Location Name
– SHA 256 thumbprint which is retrieved from Local Manager GUI.
|Adding location on Global Manager|
|Locations on Global Manager|
Above you see that two locations Bangalore and Delhi have been added to the Global Manager.
Fabric setup in Bangalore:
In each location, you need to prepare NSX-T Fabric which includes
– Configuring Transport Zones in each location
For now, the default nsx-overlay-transportzone works for Overlay traffic in each location.
– Preparing Compute Transport Nodes
– Configuring Edge Transport Nodes
– Configuring Edge Cluster
– Configure RTEPs on edge cluster.
Sharing the screenshots for these steps from Bangalore Location.
Please note that same steps are to be followed in the other location Delhi as part of fabric preparation.
IP Address Pools for Compute TEPs, Edge TEPs and RTEPs in Bangalore:
|Compute TEP Pool in Bangalore|
|Edge TEP Pool in Bangalore|
|RTEP Pool in Bangalore|
|Uplink Profiles for Host TNs and Edge TNs|
Uplink profiles are used while configuring NSX on edges and the hosts.
Above screenshot shows uplink profiles being created for edges and hosts in Bangalore location.
Since the edge is up linked to VDS (used for configuring NSX), separate VLANs are used for edge TEP and compute TEP respectively.
VLAN for compute TEP is 7
VLAN for edge TEP is 9
|Edge Transport Node Configuration for edge in Bangalore|
|Compute TN Configuration for host in Bangalore|
Above screenshots show edge transport node configuration & compute transport node configuration respectively.
|Edge Cluster in Bangalore|
Above shows edge cluster in Bangalore
|Configure RTEP on edge cluster of Bangalore|
Configure RTEPs on edge cluster of Bangalore for cross site traffic.
Before proceeding to configurations on Global Manager, ensure that same steps are followed for preparing fabric in Delhi location.
RTEPs are to be created on the edge cluster in Delhi too.
There is RTEP to RTEP communication for cross site traffic.
Configurations on Global Manager:
With the fabric set up properly in both the locations – Bangalore and Delhi, now configuration can be applied on Global Manager.
VLAN 5 and VLAN 51 are used for uplink connectivity from edge to the physical routers in Bangalore.
VLANs 55 and 56 are used for uplink connectivity from edge to the physical routers in Delhi.
|Segment for uplink on Tier 0 Gateway using VLAN 5 in Bangalore|
|Segment for uplink on Tier 0 Gateway using VLAN 51 in Bangalore|
|Segment for uplink on Tier 0 Gateway using VLAN 55 in Delhi|
|Segment for uplink on Tier 0 using VLAN 56 in Delhi|
Above segments are created on Global Manager.
These segments are to be used while creating layer 3 interfaces on the stretched Tier 0 Gateway from Global Manager.
Next create stretched Tier 0 Gateway from Global Manager
|Create Tier 0 Gateway|
While creating Tier 0 Gateway, Bangalore location is set as primary for federation.
And Delhi location is set as secondary location for federation.
This Tier 0 Gateway has Active-Active availability mode as there is no requirement of services like edge firewall / NAT on this Tier 0 Gateway.
Active-Active availability mode provides greater throughput for North-South traffic.
|Layer 3 interfaces on Stretched Tier 0 Gateway|
Create Layer 3 interfaces on the stretched Tier 0 Gateway as shown above.
Here B1-V1-E1 corresponds to location Bangalore, VLAN 5 which is first uplink VLAN there, edge node 1 in Bangalore.
D1-V2-E4 corresponds to Delhi, VLAN 56 which is second uplink VLAN there, edge node 4 in Delhi.
And similar logic is used to create rest of the layer 3 interfaces on Tier 0 Gateway.
At this stage, we can test reach-ability between physical routers and Tier 0 Gateway interfaces.
Physical routers are in BGP AS 65001
Stretched Tier 0 Gateway uses BGP AS 65000
In the BGP configs, BGP AS number is configured and BGP neighbors are specified.
|Create stretched Tier 1 Gateway from Global Manager|
From the Global Manager, create a stretched Tier 1 Gateway.
|Create stretched segment from Global Manager|
While creating overlay segment, specify the Tier 1 Gateway and gateway IP.
Traffic type is Overlay
Now attach workloads to this segment and test East-West Connectivity and North-South connectivity.
|Ping from VM in Bangalore to VM in Delhi|
|Ping from VM in Delhi to VM in Bangalore|
From the above, we verify the MAC addresses associated with the VMs and that there is reach-ability between the VMs in different locations.
|Verify RTEP to RTEP communication|
Above we see that RTEP to RTEP tunnel is established properly.
|Routing tables of Tier 1 DR and Tier 0 DR on ESXi 12 in Delhi|
VM in Delhi is on ESXi 12
Above shows the routing tables corresponding to Tier 1 DR and Tier 0 DR of ESXi 12
|Trace from VM in Delhi to loopback on physical router 2 of Delhi
|Traceflow from VM in Delhi location to loopback of physical router in Delhi|
|RTEP interface on Edge Node 2 in Bangalore location|
From the above output, we see that traffic lands on Edge Node 4 in Delhi, then goes through the tunnel between RTEPs and lands on Edge Node 2 in Bangalore location. Edge Node 2 then routes it towards the physical network.