I encountered this issue once in QA and just recently again in my production environment. I want to document the solution in case anyone runs into the same issue in his/her environment as well.

Before I get started, allow me to share a bit of my environment setup.

We have two data centers spanning across the state line. Every AirWatch component (i.e. device services, admin console, secure email gateway, etc.) has been setup independently within each data center while sharing the same URLs for web console and mail proxy between both sites. However, only one site is active at any given time thanks to Global Server Load Balancing (GSLB) setup with Citrix Netscaler. Also to expedite fail over (and back) between the sites, only the network adapter is disabled on all AirWatch servers (all virtual) within whichever site that is to remain passive while leaving all AirWatch services running.

Our AirWatch database used to reside on SQL 2008 hosts and was then mirrored between the two data centers. As part of version 9.1.1 upgrade, the AirWatch database was migrated to SQL 2012 Always On Availability Group (AAG) which in my opinion is better than database mirroring since there is only one SQL server name/listener to access the database as far as AirWatch is concerned regardless which site hosts the active/writable database.

As part of the fail over process such as from primary site to secondary site, we would first disable the network adapter on all AirWatch servers within the primary site instead of stopping all AirWatch services. And since the same AirWatch services were already started on all virtual machines within the secondary site, we simply re-enable the network adapter on all AirWatch servers over there. There are a few other steps we take as part of the fail over process, but the steps above are crucial.

If you have more than one console server set up behind a load balancer, the AirWatch Device Scheduler should only run from one console server at a time. For some reason, I could not start this service after failing to the secondary site. Instead, I received the prompt below when doing so:

I then reviewed the “SchedulerService” log (usually located at C:\AirWatch\Logs\Services) and saw the below.

2017/06/23 02:43:36.129 AirWatchHostName 037ea65f-869b-4590-a07d-fc97f2156254 [0000000-0000000]   (7)   Error AW.Service.Scheduler.Service.Configuration.ServiceConfiguration *** EXCEPTION ***2017/06/23 02:43:36.129 AirWatchHostName 037ea65f-869b-4590-a07d-fc97f2156254 [0000000-0000000]   (7)   Error AW.Service.Scheduler.Service.Configuration.ServiceConfiguration *** EXCEPTION ***System.NullReferenceException: Object reference not set to an instance of an object.   at AW.Service.Scheduler.Service.Configuration.ServiceConfiguration.GetServiceConfigurationsFromDb()Diagnostics Context*******************PID: 7844Process Name: AW.Service.Scheduler.ServiceProcess Identity: Error obtaining Application Identity: Object reference not set to an instance of an object.******************* 2017/06/23 02:43:38.174 AirWatchHostName 037ea65f-869b-4590-a07d-fc97f2156254 [0000000-0000000]   (7)   Error AW.Service.Scheduler.Service.SchedulerService.VerifyNoOtherInstanceIsRunning Multiple services are not allowed with clustering disabled

After speaking with AirWatch support, this behavior was expected because during the fail over process, the AirWatch Device Scheduler service was not stopped at the primary site beforehand. Thus, the database did not have the status of this service updated properly. So when the same service from the AirWatch server at the secondary site tried to start, it failed and generated the error above in the log.

In this case, we were given two options:

Option 1: stop the AirWatch Device Scheduler service first before disabling the network adapter as part of the fail over process.

Option 2: run the SQL script below on the AirWatch database to enable clustering.

We opted for option 2 and was then able to start the AirWatch Device Scheduler service once again.

You can download the SQL script from the link below.  Enjoy!

SystemCodeOverride_EnableClustering.sql

/*********************************This script is to create the System code override(Global only) for system code EnableClustering (5061).
*********************************/
DECLARE @LocationGroupID INT =  7DECLARE @Enable NVARCHAR(100) = N’True’
DECLARE @SystemCodeGroupOverrideId INT = NULL  ,  @SystemCodeOverrideID INT = NULL, @SystemCodeID int =5061 ,  @Admin_userID int =4,  @RowCount int  , @Current_Date datetime = GETUTCDATE() SELECT @rowCount=1 FROM [dbo].[SystemCodeGroupOverRide] WHERE SYSTEMCODEGROUPID = 23 AND LOCATIONGROUPID = @LocationgroupID IF @@ROWCOUNT=0 BEGIN  INSERT INTO [dbo].[SystemCodeGroupOverRide]                ([systemcodegroupid], [locationgroupid], [locationid], [enabled], [overridden], [childpermission], [createdby], [createdon], [modifiedby], [modifiedon])               VALUES (23, @LocationgroupID, NULL, 1, 1, 3, @Admin_userID, @Current_Date, @Admin_userID, @Current_Date)                SELECT @SystemCodeGroupOverrideId = SCOPE_IDENTITY()
INSERT INTO dbo.SystemCodeGroupOverrideAudit ( SystemCodeGroupOverrideID , SystemCodeGroupID , LocationGroupID , LocationID , Enabled , Overridden , ChildPermission , Operation , ModifiedBy , OperationDatetime ) SELECT   @SystemCodeGroupOverrideId , SystemCodeGroupID , LocationGroupID , LocationID , Enabled , Overridden , ChildPermission , ‘I’ , ModifiedBy , ModifiedOn FROM dbo.SystemCodeGroupOverride WHERE SystemCodeGroupOverrideID = @SystemCodeGroupOverrideId; END
IF @SystemCodeGroupOverrideId IS NOT NULL  BEGIN             INSERT INTO [dbo].[SystemCodeOverride]                   ([qualifier1], [qualifier2], [qualifier3], [systemcodeid], [overridevalue], [createdby], [createdon], [modifiedby], [modifiedon], [systemcodegroupoverrideid])                   VALUES(NULL, NULL, NULL, @SystemCodeID, CAST(@Enable AS NVARCHAR(100)), @Admin_userID, @Current_Date, @Admin_userID, @Current_Date, @SystemCodeGroupOverrideId)    SELECT @SystemCodeOverrideID=SCOPE_IDENTITY()
INSERT INTO dbo.SystemCodeOverrideAudit ( SystemCodeOverrideID , Qualifier1 , Qualifier2 , Qualifier3 , SystemCodeID , OverrideValue , SystemCodeGroupOverrideID , Operation , ModifiedBy , OperationDatetime ) SELECT   SystemCodeOverrideID , Qualifier1 , Qualifier2 , Qualifier3 , SystemCodeID , OverrideValue , SystemCodeGroupOverrideID , ‘I’ , ModifiedBy , ModifiedOn FROM dbo.SystemCodeOverride WHERE SystemCodeOverrideID = @SystemCodeOverrideID; END
ELSE
BEGIN  SELECT @SystemCodeGroupOverrideId=SystemCodeGroupOverrideId FROM [dbo].[SystemCodeGroupOverRide] WHERE SystemCodeGroupID = 23 AND LocationGroupID = @LocationgroupID SELECT @rowCount=1 FROM [dbo].[SystemCodeOverride] WHERE SystemCodeID = @SystemCodeID AND systemcodegroupoverrideid = @SystemCodeGroupOverrideId if @@ROWCOUNT=0 BEGIN INSERT INTO [dbo].[SystemCodeOverride]                   ([qualifier1], [qualifier2], [qualifier3], [systemcodeid], [overridevalue], [createdby], [createdon], [modifiedby], [modifiedon], [SystemCodeGroupOverrideID])                   VALUES(NULL, NULL, NULL, @SystemCodeID, CAST(@Enable AS NVARCHAR(100)), @Admin_userID,@Current_Date,@Admin_userID, @Current_Date, @SystemCodeGroupOverrideId) SELECT @SystemCodeOverrideID=SCOPE_IDENTITY()
INSERT INTO dbo.SystemCodeOverrideAudit ( SystemCodeOverrideID , Qualifier1 , Qualifier2 , Qualifier3 , SystemCodeID , OverrideValue , SystemCodeGroupOverrideID , Operation , ModifiedBy , OperationDatetime ) SELECT   SystemCodeOverrideID , Qualifier1 , Qualifier2 , Qualifier3 , SystemCodeID , OverrideValue , SystemCodeGroupOverrideID , ‘I’ , ModifiedBy , ModifiedOn FROM dbo.SystemCodeOverride WHERE SystemCodeOverrideID = @SystemCodeOverrideID; END  END