Run a VMware NSX(-T) Edge Node 4.x with bridging functionality on VMware vSphere 6.5 or 6.7
Use Case
Recently on a project, where I needed to do a workload migration from a vSphere 6.7 with NSX-V platform to vSphere 7.x with NSX(-T) platform. NSX(-T) 4.1 came out just before our testing phase. With all of our good thoughts, we didn’t want to upgrade in the middle of our workload migration from NSX-T 3.x to 4.1. So we decided to upgrade when we saw there were no major issues reported, but then we totally forgot something… Our workloads are being migrated over NSX(-T) Edge Nodes with bridging functionality and need to be deployed on VMware vSphere 6.7 clusters.
We faced an issue with deploying an NSX(-T) Edge 4.x (OVA) with bridging functionality on a vSphere 6.7 cluster for virtual network bridging NSX-V virtual wire port group to an NSX-T Overlay Segment.
Looking into some blog posts on the internet we found that this issue was related to the virtual hardware level of the OVA-file. While NSX(-T) 3.x was still on virtual hardware level 13 (ESXi 6.5 or later), the NSX(-T) 4.x OVA-file is on virtual hardware level 17 (ESXi 7.x or later).
Problem Description
This means that you cannot deploy an NSX(-T) Edge 4.x OVA on a vSphere 6.7 or lower instance. There are several blog posts that describe the process of manipulating the OVA file, but in our case, this didn’t work out in our environment. After some digging and testing, we found another much future-proof solution.
Solution
The solution is to deploy the NSX(-T) Edge 3.x OVA-file and perform after the deployment an in-place upgrade with the NSX(-T) Edge 4.x NUB-file. We have tested this for 4.1.0.1 and the 4.1.0.2 versions and that works like a charm. Keep in mind VMware doesn’t support NSX(-T) 4.x on vSphere 6.5 or 6.7, but we did some aggressive end-to-end migrations from NSX-V to NSX(-T) without any problems. Eventually we had no other option anymore. It works, so we decided not to do any product updates anymore. The workload VMs will be migrated with VMware HCX in Replicated-Assisted-vMotion (RAV) mode. We don’t do any network extensions with HCX, because there will be a big bang in this scenario. The gateway switch from NSX-V to NSX(-T) will always be at the end of HCX network extensions. If you make use of NSX(-T) bridges you can perform the gateway switch after the bridge is built and configured for the NSX(-T) overlay segment.
Prerequisites
- Download the NSX Edge 3.x OVA-file “nsx-edge-3.2.2.0.0.20737193.ova” from VMware Customer Connect site.
- Download the NSX Edge 4.x NUB-file “VMware-NSX-edge-4.1.0.2.0.21761699.nub” from VMware Customer Connect site.
- You have software like WinSCP and PuTTY installed on your client.
- NSX(-T) Edge Profiles are created for bridge overlay and bridge wire traffic
Follow the below steps to start with the NSX(-T) Edge Node VM version 3.x deployment and right after that we upgrade to version 4.x:
Deploy NSX(-T) 3.x Edge VM for bridging
Deploy the NSX(-T) Edge VM, which functions as a bridge on the source vCenter cluster (which is on vSphere 6.5 or 6.7).
Log into the source vCenter’s vSphere Web Client, right mouse click on a cluster or resource pool.
Choose “Deploy OVF Template…”
1. Browse to the location where you host the OVA-file “nsx-edge-3.2.2.0.0.20737193.ova” and select it.
Click “NEXT”.
2. Enter the “Virtual machine name”. In the example, I name it “testbridge” and select the vSphere Folder location where you want to deploy the NSX(-T) bridge.
Click “NEXT”.
3. Select the vSphere Cluster or Resource Pool you want to deploy the NSX(-T) bridge in.
Click “NEXT”.
4. Review details. It warns you that it sets some advanced features on the VM.
Click “NEXT”.
5. Configuration. Select the size of the NSX(-T) Edge VM with bridging functionality.
For a comparison between sizes, please guide the following VMware Docs Site:
Click “NEXT”.
6. Select the type of storage you want the NSX(-T) Edge VM deployed on.
Click “NEXT”.
7. Select the virtual networks which need to be attached to the NSX(-T) Edge VM (bridge).
- Network 0 = Management network (vSphere Distributed Port Group) for the NSX(-T) Edge VM.
Keep in mind: this network needs to be routable between the old vCenter and the new vCenter. The NSX(-T) Managers need to contact this NSX(-T) Edge VM on this virtual network. - Network 1 = NSX(-T) Edge TEP network, this is normally a VLAN assigned to vSphere Distributed Port Group in VLAN Trunk Range.
- Network 2 = The workload source network can be an NSX-V virtual wire port group or a VLAN-backed vSphere Distributed Port Group.
- Network 3 = Dummy network (will and cannot be used in case of an NSX(-T) Bridge!)
- Network 4 = Dummy network (will and cannot be used in case of an NSX(-T) Bridge!)
Watch out for the Network order!!!
Click “NEXT”.
8. Customize template. Enter all the required fields like:
- Application > System Root User Password
- Application > CLI “admin” User Password
- Application > CLI “audit” User Password
- Network properties > Hostname
- Network properties > Management Network IPv4 Address
- Network properties > Management Network Netmask
- Network properties > Default IPv4 Gateway
- DNS > DNS Server list (space separated)
- DNS > DNS Search List (space separated)
- Services Configuration > NTP Server List (space separated)
- Services Configuration > Enable SSH (thick checkbox)
- Services Configuration > Allow root SSH logins (thick checkbox)
Click “NEXT”.
9. Ready to complete. Check the summary for the NSX(-T) Edge VM deployment.
Click “FINISH” to start the OVA deployment.
After the OVA is deployed, put the NSX(-T) Edge VM into the NSX-V DFW Exclusion List.
Power on the NSX(-T) Edge VM through the vSphere Web Client.
Wait till the VM is finished booting up and configuring all the advanced settings. You can verify this if the VM shows the correct information as hostname and IP address in the vSphere Web Client.
As you can see when you open a VMware Remote/Web Console session, the current version of the NSX(-T) Edge VM is 3.2.2:
You can close the VMware Remote/Web Console now.
Upload the NSX(-T) 4.x Edge VM upgrade file (NUB)
Open now the application WinSCP and connect via SCP to the NSX(-T) Edge VM with the “root” credentials. Only the “root” account has access to the folder on the NSX(-T) Edge VM.
Browse in the right window to: /var/vmware/nsx/file-store.
Copy the NUB-file “VMware-NSX-edge-4.1.0.2.0.21761699.nub” to the /var/vmware/nsx/file-store. When the NUB-file is fully copied, then you can close/disconnect the WinSCP session to the NSX(-T) Edge VM.
Start the in-place upgrade from NSX(-T) version 3.x to version 4.x
Open now the application PuTTY and connect to the NSX(-T) Edge VM with the “admin” credentials.
The admin account has enough permissions to perform the in-place upgrade.
Click “OPEN”.
To avoid expiring passwords for the admin, root, and audit accounts during the time the network is bridged. Run the following commands:
clear user admin password-expiration
clear user root password-expiration
clear user audit password-expiration
With get user admin/root/audit password-expiration you can verify if the commands are executed successfully.
The first step is to verify the upgrade bundle (NUB-file). Execute this command you can easily work with tabs to auto-fill the commands and names:
verify upgrade-bundle VMware-NSX-edge-4.1.0.2.0.21761699
The verification process takes a couple of minutes.
You can continue if the text “Successfully verified upgrade bundle”.
The in-place upgrade process is divided into two stages. Step 1 to 5 and Step 6 to 9.
To start with stage one, execute the following command:
start upgrade-bundle VMware-NSX-edge-4.1.0.2.0.21761699 playbook VMware-NSX-edge-4.1.0.2.0.21761699
After step 5 the NSX(-T) Edge VM will be rebooted automatically and the PuTTY (SSH) connection will get disconnected. Now wait for approximately 5-7 minutes till the NSX(-T) Edge VM is finished booting up. It is better to wait some time longer, than shorter. If you try it shorter it is possible you interrupt the upgrade process.
Don’t close the PuTTY main window. Click on “OK” at the PuTTY Fatal Error window and then right-mouse click on the PuTTY main window (title bar) and choose “Restart Session”.
Login as the “admin” user again and execute the following command to resume the in-place upgrade with stage two:
resume upgrade-bundle VMware-NSX-edge-4.1.0.2.0.21761699 playbook
When the output is “Playbook finished successfully”. Then your in-place upgrade is successfully completed. You can already see the output of the line: NSX CLI (Edge 4.1.0.2.0.21671699).
If you are familiar with deploying NSX(-T) bridges you can skip the rest of the blog, but if you are in the middle of the process you can continue to read. Don’t close the PuTTY (SSH) session 😉.
Join the NSX(-T) Edge VM to the Management Plane
The NSX(-T) Edge VM operates now as something standalone and it has no function yet.
For creating an NSX(-T) Bridge we need to join the just deployed NSX(-T) Edge VM to the Management Plane which runs on the NSX(-T) Manager.
Therefore we need to find the active NSX(-T) Manager node and the associated certificate thumbprint.
Open a new browser tab and go to the NSX(-T) Manager Web Console.
After login, go to System > Configuration > Appliances
Note the IP address under Virtual IP behind “Assigned to” and click on “VIEW DETAILS” of the NSX-T Manager which holds the Virtual IP.
In the pop-up click on the “copy icon” behind “CERT THUMBPRINT”.
Note the IP address from the active NSX(-T) Manager and also the certificate thumbprint, we need them in the join command.
If the PuTTY (SSH) session is timed out, reconnect your session to the NSX(-T) Edge VM.
Let’s run the join command:
join management-plane <ip-nsx-active-manager> thumbprint <cert-thumbprint> username admin
Before you run the command, look up the password of the admin user account of the NSX(-T) Manager. You need to fill that one in after hitting enter.
Make sure that the output is “Node successfully registered”.
In the NSX(-T) Manager Console go to System > Configuration > Fabric > Nodes, it should look like this:
Status is “Not Configured”.
Configure NSX(-T) Edge VM as a Bridge Transport Node
To change the status from “Not Configured” to “Success” we need to add N-VDS switches to the NSX(-T) Edge Transport Node.
Thick the box in front of the NSX(-T) Edge VM where the status is “Not Configured” and click on “Edit”.
Fill in all required fields:
NOTE: The fp-eth0 matches on VM with the Network 1 (TEP).
Click on “+ ADD SWITCH”, so we can configure the bridge wire.
Fill in all required fields:
NOTE: The fp-eth1 matches on VM with Network 2 (source network / virtual wire port group).
Click on “SAVE”.
After a couple of seconds, you see the status changes.
The status is changed from “Not Configured” to “Success”. Also, we see the N-VDS count of 2 (Bridge Overlay TEP and Bridge Wire).
For security reasons, you can now disable SSH for the NSX(-T) Edge VM again.
Thick the box in front of the NSX(-T) Edge VM, and click Actions > Change Node Settings.
Set the “Allow SSH Login” from Yes to No.
Click on “SAVE”.
Strong advice is to repeat all the above steps for a second NSX(-T) Edge VM with bridging functionality, so we could create a High-Availability (Active/Passive) pair in an Edge Cluster.
Create an NSX(-T) Edge Cluster for the Bridge
To make use of the NSX(-T) Edge Node VM for bridging functionality we need to create an Edge Cluster.
In the NSX(-T) Manager Console go to System > Configuration > Fabric > Nodes > Edge Clusters.
Click “+ ADD EDGE CLUSTER”.
Fill in the required fields.
Click “ADD”.
Create the NSX(-T) Overlay Segment which needs to be bridged
If not already exist, create the NSX(-T) Overlay Segment which needs to be bridged from the source environment.
In the NSX(-T) Manager Console go to Networking > Connectivity > Segments > NSX.
Click “ADD SEGMENT”.
Fill in the required fields:
Give the Segment a meaningful name, attach a Tier1 Gateway, transport zone is Overlay, enter a Subnet for this segment, and not necessarily but useful enter a description.
Watch out, probably you extend a current (old) virtual network, that has an active gateway already and is living on that current (old) virtual network. In my case, the gateway is living in NSX(-V) on the DLR, so I cannot advertise the gateway in NSX(-T) yet. Turn the Gateway Connectivity off!
Click “SAVE”.
Create an NSX(-T) Edge Bridge Profile
Before we can bridge both virtual networks, you need to create an NSX(-T) Edge Bridge Profile.
In the NSX(-T) Manager Console go to Networking > Connectivity > Segments > Profiles > Edge Bridge Profiles.
Click “ADD EDGE BRIDGE PROFILE”.
Give the Edge Bridge Profile a meaningful name, select the earlier created Edge Cluster, select which NSX(-T) Edge is the Primary Node and who is the Backup Node, and the Fail Over mode is Preemptive, and enter a description.
Click “SAVE” and “CLOSE SETTINGS”.
Configure an NSX(-T) Edge Bridge on an Overlay Segment
Use the Edge Bridge Profile that we created in the previous chapter. The Edge Bridge helps you to extend the overlay segment with the Logical Switch in your NSX-V environment.
The NSX-V prepared host on which you have deployed the NSX(-T) Edge bridge node servers as the transport node. This host has workload VMs that require connectivity with the overlay segment in your NSX(-T) environment.
Let’s edit the earlier created NSX(-T) Overlay Segment.
In the NSX(-T) Manager Console go to Networking > Connectivity > Segments. Search for the NSX(-T) Overlay Segment you want to bridge.
Click on the three dots and click on “Edit”.
Unfold the “Additional Settings” section within the Segment.
Click on the text “SET” behind Edge Bridges.
Click “ADD EDGE BRIDGE”.
Select the earlier created Edge Bridge Profile, Transport Zone, VLAN = 0, and optional is setting a Teaming Policy.
Click “ADD” and then “APPLY”.
Back into the edit section of the segment, check if the text “SET” is changed to “1” and then click on “SAVE” + CLOSE EDITING.
For this specific virtual network, we are done with the configuration within NSX(-T).
Configure the Logical Switch to Connect to the Edge Bridge
There are two methods of enabling the connectivity between the NSX-V virtual wire port group and the NSX(-T) Edge bridge.
Method 1: Enable Promiscuous Mode and Forged Transmit
Enable these two configuration settings on the distributed port group of the Logical Switch where the NSX Edge bridge node is connected. The drawback of enabling promiscuous mode is that all the VMs on the Logical Switch can access the packets even if a single VM receives the packet. Therefore, enabling promiscuous mode might impact network performance.
Method 2: Enable MAC Learning and Forged Transmit
MAC Learning is more efficient as compared to promiscuous mode. MAC Learning is a native feature in vSphere Distributed Switch. This feature is available starting in vSphere 6.7, and it is supported in vSphere Distributed Switch 6.6.0 or later. However, you can enable MAC Learning only with the vSphere API, and you must be familiar with scripting to enable this feature on the port group.
The project’s intention is that the NSX(-T) Edge bridges are short-lived VMs after our migration is completed the NSX(-T) Edge bridges will be decommissioned. The customer has over the 100 NSX-V virtual wire port groups, so with this in mind, we choose Method 1.
Open a browser and go to the source vCenter’s vSphere Web Client and login.
In the menu go to Networks > NSX(-V) vSphere Distributed Switch > Search for the source virtual wire port group > Configure > Edit.
In the edit wizard go to “Security” and adjust here the settings from:
Promiscuous mode > Accept
Forged transmits > Accept
Click on “OK” and this enables the connectivity of both virtual networks. For more information about this topic consult the guide on the VMware Docs site: https://docs.vmware.com/en/VMware-NSX/4.1/migration/GUID-206CF244-3171-4146-9C60-43A797B15043.html
Change the MAC Address of NSX(-T) Virtual Distributed Router
The last thing for a reliable connection is to change the MAC address of the NSX(-T) Virtual Distributed Router, so it does not use the same MAC address that is used by the Distributed Logical Router (DLR) of NSX-V. For this, you need how to API of NSX(-T) works. It’s one simple PUT statement.
Follow the guide on the VMware Docs site and use also the same values:
https://docs.vmware.com/en/VMware-NSX/4.1/migration/GUID-538774C2-DE66-4F24-B9B7-537CA2FA87E9.html This step was already executed by one of my colleagues, so I don’t have any screenshots.
Test connectivity over the NSX(-T) Bridge
Verify now the configuration to deploy two test VM’s. One on the current (old) vSphere cluster and attach the NSX-V virtual wire port group and one VM on the future (new) vSphere cluster where you attach the NSX(-T) Overlay Segment where the bridge is configured on. Do some ping tests to the gateway and to each other. To get more insights you can also try a tracert/traceroute. Hope this guide was helpful for you, especially the in-place upgrade from an NSX(-T) 3.x Edge VM to an NSX(-T) 4.x Edge VM.
