Disclaimer: This post was made possible by Steve Marcolla, Application Support Engineer at VMware AirWatch. His insight and suggestion into this migration are very much appreciated!

Updated on 04/29/20: I came across the VMware KB below which will be useful once you are on SEG V2.

• V2 SEG Admin Page Diagnostics
• “The Admin page is locally available on your SEG at https://localhost:44444/seg/admin.”

Back in June of 2018, I learned about the end of general support for the Classic Secure Email Gateway (SEG) through product announcements.

“The Classic Secure Email Gateway (SEG) installer will reach End of General Support on May 5, 2019. On December 24, 2018, the Classic SEG installer will be removed from the Resources portal. After May 5, 2019, VMware cannot guarantee full support for Classic SEG… All customers currently utilizing Classic SEG should migrate over to SEG V2 before the End of General Support date.”

Since then, I’ve been paying attention to the steps required and gotcha before my own migration. Access to email especially on the go is so critical that any change that may jeopardize continuous and reliable access must be avoided at all costs.

Fortunately, the issue reported via the link below didn’t happen to me when I upgraded my SEG from version 9.2 to 9.4 as I do have Visual C++ 2015 64 Bit installed.

After upgrading Classic SEG in an AirWatch 9.4, Email Flow Ceases

Just from the VMware AirWatch community forum alone, others have expressed pains and concerns when migrating to SEG V2:

• “To solve the connection test issue between the console and the SEG v2 server, make sure the WorldWideWeb Publishing Service is disabled and off on the SEG server. “
• “Needed to import the SEG Server SSL Certificate and uninstall the classic SEG installation from the server before it would work.”
• “Issue is solved the certificate inside AW database was with a password for some unknown reason.”
• “I’m running it successfully in my Dev and QA environments for the last couple months but I’m very hesitant to pull the trigger in Prod. The main reason is we have seen the same error messages that Florian reported and had to remove all traces of the classic SEG configs in order to run the SEG v2 installers. Disabling WWW Publishing and IIS did not do the trick for us. In addition, I’m not sure how reliable SEG clustering is with v2. Unfortunately, we cannot generate enough traffic in Dev and QA for a true test.”
• “Turns out it was the IE TLS settings all 3 of them needed to be enabled and there was some service firewall was blocking too.”
• “Per our experience, I would reboot after you disable IIS, and before you install the v2 SEG. The instructions say you do not have to but are wrong, otherwise, IIS keeps grabbing the traffic.”

And a quick check-in with VMware AirWatch technical support more or less confirmed my concern as well:

• “Yes, it’s necessary to uninstall classic before installing SEG V2.”
• “Un-installation part is not mentioned in the document but it is a good practice to remove before upgrading because both the version uses different API’s and background software’s.“

Here’s another comment from my Technical Account Specialist (TAS):

“One thing to keep in mind with SEG V2 is that it can coexist on the same server as Classic SEG as V2 uses JAVA and Classic uses IIS. Therefore after you migrate you can just stop IIS service and see if the email flows through the new setup. If you have issues you can just stop the JAVA services and restart IIS.”

A little research goes a long way and it’s definitely time well spent to ensure this migration goes without a hitch. I noticed, however, that some of the links I included initially were no longer available when I visited them again. If that’s the case with the links below, you should be able to find them again under the newer version (i.e. 1902 versus 1811).

• Best Practices When Migrating from SEG Classic to SEG v2
  • Some say you can either create a new or modifying the existing email configuration. From what I’ve gathered, however, it’s best to create a new one whether you are using the same or different SEG server URL.
• Introduction to Secure Email Gateway(V2)
  • While you can skip directly to the section SEG Migration (Classic), I suggest reading from the beginning to have a full understanding of what V2 brings to the table. 

So after combing through a mountain of information from various sources, here are the steps at a high level to migrate and test V2 safely on multiple SEG servers while minimizing the impact to the users (it may take up to 10-20 minutes for mail to re-sync):

Pre-requisites

• Create a snapshot of the server.
• Make sure not to perform any load balancer checks on the SEG servers which check for IIS or the old SEG service being enabled, if applicable.
• Disable server in the load balancer so new connections will get routed to the other SEG server, if applicable.
• Retrieve SSL certificate in pfx format from the SEG server (this is needed to upload to the new email configuration).

You can use any online SSL checker to confirm the presence of the certificate such as the one below:

AirWatch Console

• Ensure REST API is enabled in the AirWatch console
• Create new email configuration in the AirWatch console
• Download and copy the installer onto the server

SEG Server

• Un-install Java, if applicable (the installer will install the latest compatible version for SEG V2)
  • The existing version installed for SEG classic might not be compatible.
• Stop/Disable World Wide Web service
• Stop/Disable IIS service
• Stop/Disable EAS Integration Service
• Execute installer
• Reboot server
• Disable Auto Update in Java

If stopping IIS/WWW/EAS Integration Service is not feasible to ensure users stay connected using SEG classic for as long as possible, alternatively, you may follow these steps:

• Un-install Java, if applicable (the installer will install the latest compatible version for SEG V2)
  • The existing version installed for SEG classic might not be compatible.
• Execute installer
• Stop/Disable World Wide Web service
• Stop/Disable IIS service
• Stop/Disable EAS Integration Service
• Reboot server
  • VMware support initially suggested restarting SEG V2 service which causes it to take over port 443 and for Java to present the HTTPs endpoint. However, you will get prompted to restart so it’s best to do so
• Disable Auto Update in Java

Validation

• Validate connectivity between AirWatch console and SEG server
• Re-enable server in the load balancer and disable the other SEG server, if applicable
• Validate mail sync on mobile devices (COPE, COBO – iOS and Android)

Below, I will provide screenshots from the relevant steps for your reference.

Ensure REST API is enabled in the AirWatch console

Go to Groups & Settings -> All Settings -> System -> Advanced -> API -> REST API.

In addition, the segadmin account (or whatever account you used for SEG installation) will also need both read and write permission for REST API as it needs to write to MEM.MEMDeviceActivity table using Rest API calls. You won’t find this step in any of the VMware KB just yet but I confirmed it with Steve Marcolla, Application Support Engineer at VMware AirWatch. 

Create new email configuration in the AirWatch console

Go to Groups & Settings -> All Settings -> Email -> Configuration and click ADD

Select V2 for Gateway Platform follow by the applicable Exchange version.

For External URL and Port, in my case, it is the same external URL as the one from SEG classic. Fill it in with this format: https://your-SEG-URL:443. It will then fill in the rest automatically (i.e. /segconsole/management.ashx). You may also specify a different port as applicable.

For SEG Server SSL Certificate, “this is found in the bindings for port 443 in IIS, assuming you will use the same SSL Cert that you used for the SEG Classic. You will need to export it as pfx format and you might need to use the certificate provider’s desktop application for conversion (If you load the cert file into the application, you can export it with a private key by entering a password. This process can be different for different cert providers).” quoted as this came straightly from Steve Marcolla, Application Support Engineer at VMware AirWatch.

Per another link below, it’s recommended to upload the SSL certificate in the console as this will become the single point for all future updates. Otherwise, you will have to re-run the installer to update the SSL certificate on an individual SEG server.

Updating Certificates for Workspace ONE UEM Services

“Note: Generally customers are encouraged to upload the certificate through the Workspace ONE UEM Console. During the initial setup, customers will have a choice between the two options noted below. It is recommended that, once an option is selected, it is used for all future certificate updates.”

However, you may choose to upload the SSL certificate locally instead if you have a SaaS environment as uploading it through the console may breach your security protocol.

Either way, you must click either Upload Locally to upload during SEG V2 installation or UPLOAD to upload the SSL certificate now. Otherwise, you will get the error below after clicking NEXT. 

For Email Server URL and Port, this was previously configured during SEG classic setup and I will now put it here (i.e. https://mail.mycompany.com:443).

If Delta Sync is already in place, you do NOT need to Enable Clustering.

Download and copy the SEG installer onto the server

Per VMware support, the SEG installer may vary depending on the option(s) you have enabled/disabled (i.e. KCD). It’s also unique per email configuration as it has information about your Email Server URL and Port (assuming that might be different from one to another.) Click on the drop-down arrow next to the new email configuration and select Download SEG Installer. It will then re-direct you to the public website to download the SEG installer.

Installation

I stumbled upon this link which has a series of screenshots as well up to console version 9.3. They are not much different from mine as shown below.

This message reminds you to reboot after the installation completes. Click No for now to avoid interruption to users.

For API URL, it is https://asXXX.awmdm.com or your internal API URL. You can locate the MEM Config GUID from the email configuration within your AirWatch console.

If you happen to forget the credential for your Admin account (i.e. segadmin), you can reset it at any time in the console and then return to complete this upgrade. This admin account is used to make the API call to pull down the configuration from the console. It will not affect mail sync on remaining SEG server(s) as they should already have the configuration downloaded previously.

Oops! By mistake, I included https:// in front of the API server hostname which is not needed. Just remove it will let you continue to the next step.

Oh no! Another error. Upon reviewing the log, it turns out I had the wrong Email Server URL configured within the email configuration. Updating the configuration and downloading the installer again follow by the steps above fixed it.

We are now prompted to install the SSL certificate since we chose the Upload Locally option within the email configuration. This proves once more the installer is unique per email configuration. 

Don’t be alarmed by the warning message below. This service will start properly after reboot.

Before rebooting, let’s take a moment to disable the following services:

• AirWatch EAS Integration Service
• AirWatch Kerberos Auth Service
• IIS Admin Service
• World Wide Web Publishing Service

As you can see, a new service for SEG V2 is now present. Again, this service will start after reboot.

                                                             VS.

Validation

If migration is successful, visiting localhost should render the masked IP address of the SEG server instead of the IIS splash page per this link mentioned earlier.

Before SEG V2 migration:

After SEG V2 migration (you should see the same if you access the SEG URL externally):

You should also be able to access the SEG V2 Admin page internally (not externally) at https://localhost:44444/seg/admin. Clicking on the Admin GUI icon on the desktop will take you to the same page as well.

Clicking on the Diagnostic GUI will show you the screen similar to the below.

Within the AirWatch console, you can also perform a test connection to confirm all is well.

After we migrated the same in CN135 which has the latest console version (i.e. 1904 as of this writing), the step Connectivity from AirWatch to SEG fails. Working with Steve Marcolla confirms it’s related to the console version as he helped test the same under a different environment running one version behind. I also received confirmation that this should be resolved in a future release.

As soon as devices begin syncing with the server that has been migrated, you should see it listed under Last Gateway Server in EMAIL -> List View.

Troubleshooting / Backout Plan (Plan B)

If the test connection fails or mail sync on the mobile device doesn’t happen after a reasonable amount of time, try the steps below:

• Review the link SEG V2 – AirWatch to SEG Test Connection Failing for suggestion.
• If you can’t resolve the issue immediately, re-enable IIS, WWW, and EAS Integration service and user’s device will re-sync with SEG classic service. Alternatively, revert the server with the snapshot.
• If you can remediate the issue, rename the AirWatch folder to AirWatch-OLD (recommended per VMware support) and re-install SEG V2 accordingly.
• Per other admins via this forum, check and make sure your firewall (Windows or 3rd party) is not blocking traffic such as Java.

FAQ

Again, credit goes to Steve Marcolla, Application Support Engineer at VMware AirWatch for answering these questions.

• Must I upgrade all SEG servers to V2 at once, or can I upgrade one SEG to V2 and leave the other as SEG classic for a day or two?
  • If you are updating the existing mem config to v2, then you will need to update all the segs as the classics will eventually fail API call.
  • If you create a new mem config for v2 (even using the same endpoint) you can update 1 and leave the others as SEG classic.
• For External URL and Port during the email configuration, do I need to specify port such as 8443 shown in the information bubble? It’s not used in SEG Classic.
  • No, you just need to specify 443, I’m not sure why they used 8443 as an example.
• If we stop/disable IIS/WWW services, doesn’t it mean we won’t be able to validate based on this link?
  • Yeah once you disable IIS/WWWPS, you won’t get an IIS splash page when browsing to the SEG URL. But, you’d get the page that java presents, which will be a real minimal page – it will contain seg version and masked IP address among a couple other fields. You can try localhost from the seg to see that the page presents itself. Then you can try browsing to the seg address externally, to see if the page again shows as expected.
• Another engineer insisted that SEG classic should be uninstalled first before installing SEG V2. Can you confirm?
  • The SEG Classic does not necessarily need to be uninstalled first. When we run into trouble with the initial deployment, sometimes we will suggest uninstalling the SEG Classic, and renaming the AirWatch folder, in case there was some aspect of the Classic service causing an issue with the update (not properly letting go of the port, or some cached config data affecting the new service). It should not be a required step, but once in a while, it is best to get the freshest state of the server we can get to ensure the least amount of potentially impacting variables.
• When can/should we disable the SEG Classic configuration in the console?
  • Disable the SEG Classic once complete with the migration. First verify that each device is reporting the Seg V2 version in the email list view rather than the seg classic, under the field “last gateway request” (Custom View of Email ListView).
• What about the log files? Do they just accumulate or get overwritten?
• 
  • You will find a file called “Logback” in the SEG V2 folder. This has the configuration for each log. You can set the max file size as well as the max # of files for each (max index). The server will clean up automatically as each limit is reached.
  • By default, the max file size is set to 35Mb per protocol.
  • The maximum number of files is set to 21.
• I have F5 as our load balancer. How can I configure my monitor properly for SEG V2?
  • After consulting with VMware support, we can monitor root / or /health or /lb-health. All three would respond with SEGv2 info by browsing to the URLs below:
    • https://SEG-URL
    • https://SEG-URL/health
    •  https://SEG-URL/lb-health
• What about monitoring the connection from SEG V2 to MS Exchange?
  • Under SEG Classic, visiting https://SEG-URL/Microsoft-Server-Activesync/healthcheck.htm should render the message below
    • healthcheck.htm does not exist either in SEG or Exchange. It’s an image in the memory of the Exchange server that is created and responded if appropriate service (Active Sync, OWA, EWS) is up. If the respecting service on the Exchange server is down, then 200 Ok status is not returned while monitors query for this URL.
    • every time you refresh in your browser https://SEG-URL/Microsoft-Server-ActiveSync/healthcheck.htm, you are getting a response from a different exchange server depending on which Exchange server your request reached out.
  • Under SEG V2, however, you will get this instead.
  • After consulting with VMware support, you can use the ActiveSync endpoint and monitor for a 401 response if the connection is good, as it would represent exchange requesting authentication:
    https://SEG-URL/microsoft-server-activesync
  • If the SEG connection to exchange is not there, you could get a 404, 403, 500, etc. depending on the issue.
• High memory utilization by Java is observed after migrating to SEG V2.
  • SEG V2 actually allocates the maximum memory for Java as it utilizes Java versus IIS, but it uses the memory on dynamic bases how much ever it is required.
  • Support confirmed the same observation in the lab. Thus, there should be no concern at this time.