This post was first published back in February 2018. It has been updated once again with relevant information and hyperlinks for better readability. 

As support for Good For Enterprise (GFE) was about to expire, several months ago we began furiously researching different replacement options, configuring the infrastructure and testing with both administrators and most importantly end users. While VMware Boxer is relatively straightforward to deploy given that we already have the necessary infrastructure in place, there are a few additional features that give Blackberry Work an edge. I will share what those features are in a separate blog post.

Regardless of which option we may choose eventually, we want to make sure that the transition from GFE is as seamless as possible from an end user’s point of view. One particular feature we want to maintain is not requiring end users to enter their Active Directory credential during application provisioning, and whenever their credentials are changed. To achieve this goal, we implement Kerberos Constrained Delegation (KCD). Nevertheless, KCD does not safeguard against expired AD passwords. If an AD password is expired, a user will not be able to log into any resources until s/he updates it.

Kerberos Constrained Delegation is among some of the most difficult tasks to implement. As a result, most vendors will strongly encourage professional services (i.e. additional cost) to ensure a successful deployment. I firmly believe that generally speaking no one knows about your environment any better than the ones who design, implement and maintain it on a daily basis. In any case, I implemented KCD with assistance from various subject matter experts from my team, fellow Blackberry UEM admins from community forum, Blackberry technical support and professional services. With this blog post, I hope you can learn something new and more importantly save yourself lots of time and money with similar implementation in your environment.

I’ve captured and shared screenshots from a Blackberry webinar which provides a quick overview of KCD via this link in case you are interested. Again, I will go through each of these steps in detail on this post.

Disclaimer: Follow these steps at your own risk as your environment can be very similar to or different from mine. 

Before I begin, I assume you already have Blackberry UEM 12.7.2 or later installed and configured. If you are running Blackberry UEM 12.7.0 or 12.7.1, stop reading now and do yourself a favor and upgrade to at least 12.7.2 (or preferably 12.8 due to this known issue). There have been numerous known issues such as this one when it comes to KCD implementation with previous versions. 12.7.2 has fixed all these known issues and you might as well take advantage of this opportunity to upgrade to the latest release.

Below are a few links I found when implementing KCD. While they all attempt to achieve the same objective, individually none of them is sufficient on its own. This blog post was made possible by reviewing all the information publicly available along with the contribution of Blackberry professional services.

• Implementing KCD for Blackberry Work in Blackberry UEM – First published on 10/24/17 and last modified on 11/01/18
• CSO – Sleuth Blackberry Work – Kerberos Constrained Delegation
• Kerberos Constrained Delegation With Good Control

Below is the latest update from Blackberry as of 11/01/18. In particular, the suggestion below will help reduce both network traffic and battery consumption which has been a common complaint from my users.

In summary, we will:

• Step 1: Map the Blackberry Control service account to a service account principal name (SPN)
• Step 2: Create a Kerberos Keytab file for the Blackberry Control service account
• Step 3: Enable enumeration of AD user objects group membership
• Step 4: Enable the Blackberry Control service account to act as part of the operating system on UEM host
• Step 5: Set the appropriate values in the Blackberry UEM console
• Step 6: Enable Kerberos for Exchange CAS
• Step 7: Create an IIS alternate service account to represent load balanced Exchange hosts
• Step 8: Configure delegation to the Blackberry Control service account
• Step 9: Configure the Blackberry Work app for KCD

At the end of this post, I will share some common measures to help with troubleshooting as well.

Before we proceed, there are a few things we need to confirm in our environment to ensure smooth KCD deployment.

1. We need a Kerberos account. If you like, you may re-use the same UEM admin account when you implement Blackberry UEM (i.e. buemadmin)
2. We need to identify the primary authentication (AD) Domain and Realm. In a single domain, it might be company.net.

Step 1

First, we will map the Blackberry Control service account to a service account principal name (SPN). This is done to enable the delegation tab for the service in ADUC (Active Directory Users and Computers) which will be necessary for adding in resources for delegation later.

Open a command prompt as administrator from a domain controller. Then run the command below to set the SPN for the Blackberry Control Service (GCS or Good Control Service for short).

C:\>setspn -s GCSvc/FQDN_of_your_UEM_host DOMAIN\Kerberos_Account

Example: C:\>setspn -s GCSvc/UEMserver.company.net company.net\buemadmin

For this particular step, keep the below in mind:

• Do NOT use –a. Instead, use –s to verify that no duplicate SPNs exist before adding the new SPN.
• The domain name must be in FQDN format (i.e company.net).
• You can use the same service account that is used for UEM implementation for Kerberos_Account
• Repeat the same command above for each UEM host in your environment.

If you wonder why it is GCSvc instead of BCSvc. Per Blackberry technical support, GCSvc is hard-coded in the application which has yet to be re-branded after the merge between Blackberry and Good Technology. That may change in future release.

Step 2

Next, we will create a Kerberos Keytab file for the Blackberry Control service account.

What is a Kerberos Keytab? Per Blackberry, it is generated by the Kerberos Distribution Center (KDC) which contains pairs of Kerberos principals and encrypted keys. The Blackberry Control server uses this file to log into Kerberos without the service account’s credential.

Again, open a command prompt as administrator from a domain controller. Then run the command below.

C:\>ktpass /out gdadmin.keytab /mapuser Kerberos_Account@REALM_IN_ALL_CAPS /princ Kerberos_Account@REALM_IN_ALL_CAPS /pass Kerberos_Account_Password ****** /ptype KRB5_NT_PRINCIPAL

This saves the gdadmin.keytab file to the directory on the server from which you run the command.

For this particular step, keep the below in mind:

• You can rename gdadmin to any filename as long as it ends with .keytab.
• The realm must in all CAPS.
• The password must be the password of your blackberry admin service account (i.e. buemadmin). Enclose the value in quotation marks if it contains special characters, such as ^.
• If you change the password for this service account, this keytab file will have to be recreated.

Now copy the gdadmin.keytab file created in the above over to the C:\good directory on each of your BlackBerry UEM server.

Step 3

Next, we will enable enumeration of AD user objects group membership.

To properly determine a user’s group membership and therefore effective permissions over a resource, the UEM Service Account must be a member of the Windows Authorization Access Group.

1. In Active Directory Users and Computers mmc console, select “Builtin” from the list on the left, then right-click “Windows Authorization Access Group” and select “Properties”.
2. In the “Windows Authorization Access Group Properties” popup, click the “Members” tab, then click “Add…”
3. In the “Select Users, Contact, Computers or Group popup”, enter the name of the Blackberry UEM service account, and click OK.

Step 4

Next, we will enable the Blackberry Control service account to act as part of the operating system on UEM host.

For the UEM Service to be able to authenticate users, the Blackberry Control service account needs to have Act as Part of the Operating System. This can be done individually in the Local Security Policy of every UEM server if you use Blackberry Mail, and BEMS server if you use Blackberry Docs or Presence that has been set up for Delegation in the previous sections, or universally by GPO.

1. Open the “Local Security Policy” pane in the Windows console.
2. Under “Local Policies”, select “User Rights Assignments”, then right-click “Act as part of the operating system” in the right panel and select “Properties”.
3. In the “Properties” popup, click on “Add User or Group…”, then enter the name of the Blackberry Control service account and click OK.

Remember, repeat the steps above on every UEM and BEMS server as applicable.

Step 5

Now, we will set the appropriate values in the Blackberry UEM console.

Please note: The account used to configure the settings below must have Security Administrator role in UEM console!

1. Click Settings | BlackBerry Dynamics | Properties
2. Click on each UEM server
3. In the Kerberos Constrained Delegation section, for each of the following properties, enter the value indicated.
4. Restart Blackberry UEM – Blackberry Control Service on UEM server.
5. Review the latest gc_server_UEM-host_GDxxx.log file within the gclog folder on the server to confirm setting is correct (i.e. look for keyword TGT.)

For this particular step, keep the below in mind:

• For Fully qualified name for the KDC field, you should put in a single domain controller even if you have a DNS object setup with round robin in your load balancer for multiple domain controllers. This is to avoid any issue related to the load balancer for KCD authentication.
• Service account name under which KCD service is running (i.e. buemadmin) must be in all CAPS.
• Realm – Active Directory (i.e. company.net) must be in all CAPS.
• In the event that there is an issue with such domain controller, simply modify it in this section. Then restart Blackberry Control service on the UEM server and review the latest gclog again to confirm it is working properly.

Remember earlier I mentioned some of the known issues with UEM version 12.7.0 and 12.7.1 such as this one? If for whatever reason you are unable to upgrade to 12.7.2 at this time, you may consider the two workarounds below:

• Request a script from Blackberry support that will enable KCD for BlackBerry Dynamics applications, or,
• Run HeidiSQL or whatever SQL method you like to inject the values in the database table manually.

Feel free to contact me if you need additional info on either of these workarounds, but hopefully, you can avoid doing either by upgrading your UEM setup to version 12.7.2 or later.

Step 6

We will now enable Kerberos for Exchange CAS.

It is important to note that Kerberos authentication must be set for all Exchange ActiveSync (EAS) and Exchange Web Services (EWS) virtual directories in Exchange. Once this KCD is enabled on the BlackBerry Work client, it cannot fall back to Basic Authentication (NTLM).

Also, any Blackberry Control outage will further prevent users from being authenticated when their Kerberos tickets expire as there is no fall back to Basic Authentication (i.e. NTLM) once KCD is enabled.

1. On each Exchange CAS machine which holds the ActiveSync virtual directory, open IIS and select the Microsoft-Server-ActiveSync (EAS) virtual directory.
2. In the far right panel, click Authentication under Actions. Then, assuming Windows Authentication is enabled, select Providers…
3. In the Providers window, click Add.
4. From the list of Available Providers, select Negotiate:Kerberos.
5. Click the Move Up button until Negotiate:Kerberos is at the top of the list of Enabled Providers.
6. Remove Negotiate from the list, if present.
7. Click OK to save this setting.
8. Repeat the same steps above in the EWS virtual directory.
9. Reset IIS.

For this particular step, keep the below in mind:

• If you already have Negotiate in the list of providers, you should remove it since Negotiate:Kerberos already handles negotiate with support for Kerberos.
• In Negotiate:Kerberos, there is no fall back to NTLM whereas in Negotiate if Kerberos authentication fails, it falls back to NTLM.
• And since NTLM cannot be used for impersonation/delegation, we need to make sure that Kerberos is the only authentication mechanism for KCD.
• Per Blackberry support, using Negotiate instead of Negotiate:Kerberos may result in an issue such as SEC_E_WRONG_PRINCIPAL errors, SEC_E_INVALID_TOKEN errors and Autodiscover failing.
• However, I actually had to use Negotiate instead of Negotiate:Kerberos in my environment as it breaks authentication for one of my other application that leverages EWS and doesn’t support kerberos for authentication.

Step 7

If your Exchange environment is similar to mine with multiple hosts behind a load balancer for high availability and disaster recovery purposes, you will need to create an IIS alternate service account to represent these load balanced hosts before continuing with the remaining steps.

Fortunately, I already completed these steps as part of the AirWatch Certificate Authentication for EAS with Secure E-mail Gateway setup which I will share the steps via a different post in the near future. For now, you may reference the links below based on your specific Exchange version (2010, 2013 or 2016) to set up the alternate service account.

• Setting up Kerberos with a Client Access Server Array Exchange 2010 SP1
• Configuring Kerberos Authentication for Load-Balanced Client Access Servers
• Using the RollAlternateserviceAccountCredential.ps1 Script in the Shell

In this section, you will generally perform the tasks below:

1. Create and configure the IIS Alternate Service Account (ASA)
2. Set the SPN for the FQDN used above by specifying the ASA name
3. Perform a reset on all CAS machines within the organization

We are almost done with the setup here! Let’s follow along to the finish line.

Step 8

We will now configure delegation to the Blackberry Control service account. Again, this could be the same account you used during UEM implementation.

Be sure you have domain admin privilege in Active Directory before proceeding with the steps below. Otherwise, you will get this error:

From AD Users and Computers, select View, select Advanced Features, then click Users.

Double-click the service account running the BlackBerry Control service (i.e. buemadmin)

Open the Delegation tab (should be available once SPN above is created for the service account earlier in step 1).

Enable Trust this user for delegation to specified service only.

Select Use any authentication protocol.

Click Add.

Click Users or Computers

Enter the alternate service account created earlier for load balanced Exchange hosts.

Click Add Services and select HTTP service related to each of the CAS servers in your organization.

Click Add… again and then input the service account name; in this case, buemadmin, then click Check Names.

From the list of Available Services, you should see GCSvc with the FQDN of your Blackberry Control server that was created in Step 1 under the section “map the Blackberry Control service account to a service account principal name (SPN)” above. Click “Select All” and click “OK”

The list of specified services displayed should now allow Kerberos Constrained Delegation for the BlackBerry Work application against ActiveSync virtual directories when using a HWLB with Exchange 2013. Click Apply, then click OK.

Step 9

We are down to the final step which is to configure the Blackberry Work app for KCD.

In the BlackBerry UEM console, under Apps, search for or scroll down to BlackBerry Work and click it.

On the ‘BlackBerry Dynamics’ tab, click on app configuration.

Go to the ‘Basic Configuration tab’ and enabled ‘Use Kerberos Constrained Delegation in place of login/password‘.

Or ‘Expect Kerberos Constrained Delegation and supress username/password entry for Exchange‘ under the older version.

Click on ‘save’ to save the change to the app configuration and click on ‘save’ again to save your changes.

Once this is set up successfully, the user will not be prompted for the password when setting up BlackBerry Work or whenever the user changes Active Directory password.

iOS (you should see “System Provided” next to Password field)

Android (you should see “System Provided” next to Password field)

Troubleshooting

While I can’t cover every possible issue you may encounter after implementation, below are some common troubleshooting measures you can take.

When KCD is not authenticating as expected, users will see the below during device provisioning:

iOS

Android

Generally speaking, you need to review the GC server log (i.e. gc_server_UEM-host_GDxxx.log). By default, it’s located at C:\Program Files\BlackBerry\UEM\Logs\gclogs.

The two most important parameters in the error messages are krbErrCode and krbErrText, which furnish a description of possible error conditions detected. Below is an example from such log:

• KerberosException: javax.security.auth.login.LoginException: Login must be done first;
• krbErrCode: 999;
• krbErrText: Other error (description not available)

For example, you may notice the below from the GC server log possibly when user’s AD password has expired:

com.good.gmc.containermgmt.ContainerMgmtProtocolHandler – Impersonating User user@domain.net

2018-03-26 08:58:50,970 ERROR ContainerMgmtServerThread1/233fb1d2-7c54-40f6-bde5-e2a15e91f1f4 com.good.gmc.containermgmt.ContainerMgmtProtocolHandler – RequestKerberosKCDTGS Error while requesting KerberosKCDTGS for serviceClass:HTTP server:mail.domain.com port:443 serviceName:http

com.good.gmc.security.kerberos.KerberosException: Failed to impersonate userPrincipal user@domain.net;

               krbErrCode: 18;

                krbErrText: Clients credentials have been revoked

This KB seems to support such theory.

Another issue I encountered during device provisioning is the below:

As a result, both the Server and Domain fields are not pre-filled at all.

I then came upon this KB #1 and KB #2. However, neither of them fits my setup. It turns out I forgot to assign the proper app configuration to the security group my account is part of.

Optional – Enable Persistent Authentication (authPersistNonNTLM) on Exchange CAS

As mentioned previously, you may consider enabling Persistent Authentication (authPersistNonNTLM) on your CAS to help reduce network traffic and battery consumption when using Blackberry Work on user’s device.

Start by launching IIS on your CAS and launch Configuration Editor under Management.

Next to Section, browse to system.webServer/security/authentication/windowsAuthentication. Next to authPersistNonNTLM, set it to True.

I hope this post helps with your KCD implementation, and feel free to share your experience with me here as it may very well benefit other UEM admins.

Thanks for reading!