Automating Keytab Rotation for Identity Bridging on VMware Unified Access Gateway

When using web reverse proxy and identity bridging edge services on VMware Unified Access Gateway, end-users present their identity through certificate or SAML assertion to access internal web applications. Unified Access Gateway converts the certificate or SAML assertion to a Kerberos ticket on behalf of the user and uses that to authenticate against the internal webserver; this process is called Kerberos Constrained Delegation (KCD).

Because Unified Access Gateway is not domain joined, you must add Active Directory (AD) domain Kerberos support to the Unified Access Gateway. The use of a keytab file enables this process, because it contains information about the identity of an AD account (userPrincipalName) and the encryption keys necessary to encrypt/decrypt Kerberos tickets. The keytab file is used by Unified Access Gateway to interact with AD on behalf of the user.

Having a keytab file is the same as having access to your AD credentials; if it falls into the wrong hands, your environment could be at risk. For that reason, it's essential to keep the keytab secure and have a process in place in case it gets compromised.

The Challenge

Quite often, I receive questions from customers regarding best practices when using keytab; what to do in case the keytab gets compromised, how often to change the keytab, how Unified Access Gateway protects the keytab, and so on. 

One of the recent questions was about how to rotate the keytabs integrated with Privileged Access Management (PAM) Tools. This customer uses CyberArk, has hundreds of websites integrated with identity bridging, and has multiple service accounts in use beyond all the other accounts integrated with other systems. 

The use of PAM tools makes total sense because managing service accounts can be painful, especially across multiple accounts for different services, tasks, and other applications. Keeping service accounts in sync is time-consuming and error-prone when done manually. Service account password management is another challenge; administrators can’t safely change a service account password if they don’t know where it is used without the risk of bringing down other applications.

Solving the Problem

The CyberArk and VMware teams worked together in a proof of concept to solve the customer requirement. We used CyberArk to manage the service account password changes and Unified Access Gateway REST API to update the appliance configuration, including the upload of the new keytab files. Finally, we used Microsoft PowerShell to automate several steps through scripting. 

Implementing an automated process that supports keytab rotation on the Unified Access Gateway is extremely important. It helps to keep the environment secure, avoid disruption of the services, and impact on the end-user experience. 

This blog guides you through the end-to-end process to perform the following: 

• Generation of new keytab files 

• Keytab rotation on Unified Access Gateway 

• Required updates on Internet Information Server 
   

The next steps use the following values: 

• AD Domain / Kerberos realm —  AIRWLAB.COM 

• Service Principal Name (SPN) —  HTTP/kcdtest.airwlab.com@AIRWLAB.COM   

• User for Kerberos delegation (service or SPN account) –  cybrtest@AIRWLAB.COM 
   

Expand the drop-down menus to view details for each step.

Step 1 - Generate a New Keytab File

The first step is to generate a new keytab using the ktpass command-line tool. Ktpass resets the password when using the /pass parameter for the indicated service account on /mapuser parameter, which creates new encrypted keys and stores as part of the new keytab file.

The following command resets the password to P@sswOrd1 for the service account cybertest@AIRWLAB.COM and generates the kcdtest1.keytab file.

ktpass /princ HTTP/kcdtest.airwlab.com@AIRWLAB.COM /mapuser cybrtest@AIRWLAB.COM /mapOp set /pass P@sswOrd1 /crypto all /ptype KRB5_NT_PRINCIPAL /out c:\keys\kcdtest1.keytab
      

In our proof of concept, CyberArk was responsible for setting the new service account password, and during the use of ktpass, instead of using /pass, /setpass was used to supply the password already defined by CyberArk and avoid the reset of the password again on AD. Even when resetting the password to the same one previously set in AD, the ktpass generates new keys and stores them as part of the new keytab file.

Step 2 - Upload Keytab File to Unified Access Gateway

The next step is to upload the new keytab file to Unified Access Gateway. You must perform a PUT API call to https://UAGHOST:9443/rest/v1/config/kerberos/keytab, and pass the principalName and keytab file into the body of the request in JSON format. 

Convert the keytab file to base64 and pass this value as part of the keyTab parameter. You cannot pass the keytab filename including the path. 

 

The following PowerShell script converts the keytab file into Base64 string. 

$FileName = "c:\keys\kcdtest1.keytab"
      
$Data = get-content $FileName 
      
$Bytes = [System.Text.Encoding]::Unicode.GetBytes($Data) 
      
$EncodedData = [Convert]::ToBase64String($Bytes) 

 

The content body looks like this, including the keyTab in Base64 format. 

{ 
  "keyTab": "BQIAAAA+AAIACkNPUlAuTE9DQUwABEhUVFAAE2ludHJhbmV0LmNvcnAubG9jYWwAAAABAAAAAAcAAQAI/iX0v7M0AtwAAAA+AAIACkNPUlAuTE9DQUwABEhUVFAAE2ludHJhbmV0LmNvcnAubG9jYWwAAAABAAAAAAcAFwAQtwn2inIWR2ntdol8oVD9ngAAAFYAAgAKQ09SUC5MT0NBTAAESFRUUAATaW50mAksKTAAAAEYAAgAKQ09SUC5MT0NBTAAESFRUUAATa"   "principalName": "HTTP/kcdtest.airwlab.com@AIRWLAB.COM" 
} 

 

The HTTP 200 OK success status response code indicates that the request has succeeded, and the keytab is successfully uploaded.     

Unified Access Gateway quickly restarts all the reverse proxy instances associated with the SPN set in the new keytab, and re-establishes a new connection with the KDC (Kerberos Distribution Center) using the new keys obtained from the uploaded keytab file. 

 

Unified Access Gateway provides several mechanisms to secure and restrict access to the keytab files, including: 

• Disabling SSH (This setting is recommended). 

• Restricting command-line access to the appliance for users with access rights to the vCenter console. 

• No REST API interface to access the physical file after the keytab is uploaded. 

• Restricting access to the management interface from an internal network in case SSH is required. 

Step 3 - Update Credentials on the IIS Application Pool and Reset IIS

With the service account password changed, you must update the IIS Application Pool identity credentials assigned to your web application and reset the IIS to apply the changes; otherwise, the application pool cannot start as it is still configured to use the old password associated with the service account.

In this step, users might notice a brief disruption when trying to access any of the web applications hosted on the webserver, caused by the IIS reset or the update on the application pool.
 

Import-Module WebAdministration
      
Set-Item IIS:\AppPools\DefaultAppPool\ -Name ProcessModel -value @{username=”airwlab\cybrtest”;password=”P@sswOrd1”;identityType=3}
      
Start-WebPool -Name DefaultAppPool
      
Get-Item IIS:\AppPools\DefaultAppPool\
      
iisreset

You have completed the steps; end-users can now access the web application, and we proved that this works.

Automating the Process

In a real-world scenario, you might have to apply these steps to hundreds or thousands of websites that you have behind the Unified Access Gateway. Manual processes are not scalable, and Microsoft recommends that each internal web application has its own delegated user, and therefore, different keytab files are required.

With that in mind, I created a PowerShell script sample to demonstrate the automation of this process. The script is available at VMware Code and only requires the installation of the module to start invoking the commands.

First, download the script on the Windows Server hosting your IIS Server, ensure UAG REST API is accessible from that machine, and execute the following command to import the modules. For simplicity, you must run this sample on the Windows Server that hosts IIS.

Import-Module .\uagkeytabrotate.psm1

This module contains a few functions that handle all the required steps discussed earlier in this post.

1. Create a new keytab file-based using the ktpass tool.

   New-KeyTabfile -spn "HTTP/kcdtest.airwlab.com@AIRWLAB.COM" -user "cybrtest" -domain "AIRWLAB" -newpassword "P@ssw0rd1" -keytabfile “c:\keys\kcdtest-key.keytab”
   

2. Connect to Unified Access Gateway to validate the connection and obtain authorization.

   Connect-UAG -username "admin" -password "VMware2!" -fqdn "uagtest.airwlab.com"
   

3. Upload the new keytab file and inform the principalName using the Import-KeyTab command.

   Import-KeyTab -keytabfile "c:\keys\kcdtest-key.keytab" -principalName "HTTP/kcdtest.airwlab.com@AIRWLAB.COM"
   

4. Invoke the Update-IIS command to update the IIS application pool identity with the new credentials provided through the -username and -password parameters, and finally reset IIS.

   Update-IIS -username "airwlab\cybrtest" -password "P@ssw0rd1"

Figure: Output log running the four steps based on the script sample available in VMware Code.