Troubleshooting macOS Management: Workspace ONE Operational Tutorial

Overview

Introduction

VMware provides this operational tutorial to help you with your VMware Workspace ONE® environment. In this tutorial, you learn how to troubleshoot macOS features in a VMware Workspace ONE® UEM environment. Procedures include parsing the Unified Log, validating console settings, and deploying profiles that aid troubleshooting.

This macOS troubleshooting guide provides general troubleshooting guidance, as well as solutions to specific problems for various macOS features in Workspace ONE UEM. The exercises in this tutorial are targeted to those with previous macOS management experience in the Workspace ONE UEM product.

Audience

This operational tutorial is intended for IT professionals and Workspace ONE administrators of existing production environments. Both current and new administrators can benefit from using this tutorial. Knowledge of the following technologies is helpful:

Getting Started with macOS Troubleshooting

Introduction

This section contains a checklist for common troubleshooting scenarios and helpful background information. For more detailed troubleshooting guidance, skip this section and go straight to the other sections in this tutorial, which walk through each troubleshooting scenario step-by-step.

Using the macOS Troubleshooting Cheat Sheet

The macOS troubleshooting tips contained within this tutorial cover potential issues that might affect macOS management. If you prefer a paper format to keep at your desk, click the More button on the top menu and download the linked PDF.

Client-Side Components

Workspace ONE uses multiple components to create the entire macOS management stack. Some of these components are supplied by Apple, whereas others are value-added functionality included with your Workspace ONE licensing. With regards to components installed and running on macOS, review the following table:

Client Component Use(s)
mdmclient (process) Native MDM client running in root (daemon) and user (agent). Responsible for enacting any functionality defined in the MDM specification: profiles, apps, enrollment, and so on. Workspace ONE communicates to these two processes via Apple Push Notification Services (APNS).
dasd Duet Activity Scheduler daemon. This process monitors system health and can sometimes interrupt/delay normal activity and service startup. In some cases, this can affect things such as out-of-box enrollment.
storedownloadd (macOS before Big Sur) App Store Download Daemon. Responsible for downloading apps from the app store (for example, volume-purchased apps) and bootstrap packages.
appstored (macOS Big Sur and later) App Store download daemon. Responsible for downloading apps from the app store (for example, volume-purchased apps) and bootstrap packages.
Workspace ONE Intelligent Hub for macOS Used for value-add functionality with profiles and configuration, employee experience, and internal apps. Workspace ONE communicates with Intelligent Hub using AirWatch Cloud Messaging (AWCM).
Workspace ONE Assist Client Allows for remote control, file management, and running remote shell commands from Workspace ONE. Communicates outbound to the Workspace ONE Remote Management server and AWCM.
Unified Logging Most processes within macOS no longer write to system.log. Rather, they write to a binary file which must be queried and exported as human-readable text using command line tools. Successful troubleshooting requires proficient knowledge of how to search the unified log (which is covered in this tutorial).

Server-Side Components

macOS management in Workspace ONE makes use of numerous server/cloud-based components. It is critical to ensure that macOS has network connectivity to each of these components, as specified in the following lists of network requirements:

Remember, many of these network requirements point to DNS names, which are part of global load balancing systems. It is highly recommended that you ensure firewall rules match the network requirements exactly. Do not substitute IP addresses where DNS names are specified because this can cause troubleshooting issues at a later stage as the load-balanced services move to different IP addresses.

Understanding the Workspace ONE UEM Solution Stack

macOS Solution Stack diagram to help troubleshoot macOS.

There are many communication methods and clients used to manage macOS devices. This section explores management services and clients in detail. It is important to understand these components before proceeding.

Management Services

In Workspace ONE, the entire possibility of macOS management takes place in multiple systems. Some systems include:

  • APNS (Apple Push Notification service) and AWCM (AirWatch Cloud Messaging) for realtime device notifications.
  • Workspace ONE UEM (which includes the console, device services, and API servers) for device management.
  • Workspace ONE Assist for remote control (for example, helpdesk support).
  • Workspace ONE Access and Hub Services for employee experience, single sign-on, and conditional access.
  • Workspace ONE Intelligence for reporting and automation (including partner ecosystems).
  • Carbon Black for endpoint security monitoring, quarantine, and remediation.
  • Horizon for enabling the "Mac App Gap" or Windows-based applications which need to be accessed and used by macOS users.

Management Clients

Clients communicate to Workspace ONE UEM on behalf of the device. The following components are the primary list of clients you must manage on a device as you adopt the entire solution stack:

  • mdmclient - The built-in device management client which is part of macOS.
  • Workspace ONE Intelligent Hub - The agent which communicates to Workspace ONE UEM and Hub Services to augment device management value-added functionality, the upcoming system extension for process blocking, the unified catalog, and employee experience features.
  • Assist Client - The client which facilitates remote troubleshooting sessions (remote control, remote file, and remote terminal).
  • Carbon Black Sensor - The client which provides next-generation anti-virus and anti-malware protection.
  • Horizon Client - The client which provides access to remote datacenter-hosted applications from macOS.

Validating Workspace ONE UEM Console Settings

When configuring and managing Workspace ONE, some common misconfigurations can happen accidentally. This chapter provides a quick reference on where to find these common misconfigurations and how to correct them if necessary.

1. Network Settings

Network connectivity is a common issue and should be verified. Mac management with Workspace ONE requires network connectivity to a number of endpoints at a number of vendors:  Apple, VMware, Akamai, and more (depending on your particular situation). Managing a table of these endpoints in a cheat sheet would be unruly and difficult to continually manage, so instead, we have included pointers to the full list of required DNS and port names:

2. System Statuses

Use the following links to quickly verify if there are any known, reported outages at Apple or VMware:

3. Common Configuration Issues

Managing macOS requires regular maintenance, just as expected with other platforms. The following list highlights the most common configuration issues that can arise when managing devices.  

  1. If you have trouble deploying newly purchased volume licenses to machines, or devices are not proceeding through Automated Enrollment, check the following:
    1. Has Apple released new Terms of Use that must be accepted by an administrator in Apple Business Manager (or Apple School Manager)?
      See Updated Terms and Conditions for Apple Business Manager.
    2. Has the sToken for the Apple Business Manager location expired?
      Navigate to Groups & Settings > Configurations > VPP Managed Distribution and check the Valid Until date and time. If expired, click the Renew button and follow the process.
  2. If devices are not checking in for commands in a reasonable amount of time:
    1. Check the APNS Certificate Expiration Date: Navigate to Groups & Settings > Configurations > APNS for MDM and check the Valid To date. If expired, you must log in to the Apple Push Certificates portal and Renew the existing certificate.

      Caution: If you replace the certificate instead of renewing it, Workspace ONE will lose connectivity with your devices and you will be required to re-enroll those devices.
       

    2. Check the device-side APNS status - open Terminal.app and enter the following command: /System/Library/PrivateFrameworks/ApplePushService.framework/apsctl status
    3. Check for AWCM connectivity.  
      1. Navigate to Groups & Settings > All Settings > System > Advanced > Site URLs. Scroll down to find the AWCM Server External Hostname and then append /awcm/status to the end.
        For example, https://awcm1380.awmdm.com/awcm/status. AWCM should return OK.    
      2. In the Device Details view, ensure that the specific device in question shows AWCM Connected.

 

Getting Started with macOS Log Collection

Collecting macOS Intelligent Hub Logs

Determining the root cause is a logical first step in troubleshooting. To diagnose, it is helpful to know where to look and which logs to examine.   

For reference, most file-based troubleshooting information is found in the following files within hub logs:

  • install.log - log file generated by the built-in macOS installer process.
  • hublogs.log - Console logs related to Intelligent Hub processes and mdmclient.
  • ManagedSoftwareUpdate.log - log generated by Intelligent Hub components during Internal Application management.

1. Request Hub Logs From the Workspace ONE Console

Request Hub logs from Workspace ONE admin console to troubleshoot mac OS.

Workspace ONE administrators can request the Intelligent Hub to gather historical logging and deliver those logs to the Workspace ONE UEM Console:

  1. On the Device details page, click More Actions.
  2. Click Request Device Log.

2. Request Hub Logs Using Hubcli

Request Hub logs using hubcli to troubleshoot mac os.

From within Terminal.app, enter the following command line:  sudo hubcli logs --send --duration 1h

This command tells hub to send diagnostic logs to the UEM console for the past 1 hour.

3. Collect User-Generated Logs

Troubleshoot macOS and Debug Session from Intelligent Hub.

End users can also generate and send logs to the Workspace ONE UEM console by using the Collect & Send Logs feature.  

Launch the Intelligent Hub:

  1. Click Help
  2. Optionally, click Debug Session > Start Session to start a debug session (verbose logging). Clicking the same path allows the user to End the debugging session prior to the 30 minute default session time.
  3. Click Collect & Send Logs.

4. Download Hub Logs in Device Details

Download Hub logs from Workspace ONE UEM admin console to troubleshoot macOS.

After requesting logs from the device, you can view the logs as follows:

  1. Click Devices.
  2. Click the device to enter the Details View.
  3. Select Attachments in the drop-down menu.
  4. Select Documents.
  5. Click the Hub_Complete_Report you want to download and view.

5. Monitor Internal App Logging in Real-Time

If you are troubleshooting an issue with Internal Apps for macOS, you can easily view the logging for that in real-time on your test device (or via remote command line through Workspace ONE Assist). The following command tails the last 25 lines in the ManagedSoftwareUpdate.log file:

tail -n 25 -F /Library/Application\ Support/AirWatch/Data/Munki/Managed\ Installs/Logs/ManagedSoftwareUpdate.log

Understanding macOS Unified Logging

When Apple released macOS Sierra 10.12 in 2016, it introduced a new unified logging concept on macOS. The unified log replaced traditional Unix-style text-based logs with a single logging mechanism for user and kernel processes. The unified log takes a more developer-focused approach, capturing greater amounts of data in a compressed format in a fashion that is consistent across each Apple OS (macOS, iOS, tvOS, watchOS). The compressed format is no longer human-readable, and instead requires the use of either Console.app or the log and logger command-line utilities to read and write log entries.

When troubleshooting, the log command is the most flexible, in that it allows you to gather multiple processes and subsystems simultaneously. Although much of the information required to run the log command can be found in the manual (man log), the following cheat sheet should help get you started quickly.

1. Useful Tips and Tricks to Search Logs

Following are some tips and tricks that can save you time:  

  • The log command requires "straight" quotes and not "curly" quotes. This issue can crop up during copy/paste operations.
  • You can gain additional information by adding the --info and --debug parameters to your log command line.
  • Add --archive system_logs.logarchive to search inside a SysDiagnose logarchive file.
    • Generate a sysdiagnose from the computer in question:   sudo sysdiagnose -f -n ~/Downloads
  • For more in-depth detail, you can install some of the developer debugging profiles from https://developer.apple.com/bug-reporting/profiles-and-logs/
  • Predicates support "and" (&&) and "or" (||) operators as long as you include parentheses to wrap each filter. For example, --predicate '(process == "mdmclient") && (eventMessage contains "SetAutoAdminPassword")'

Important: Many details in logging commands are hidden for privacy. Most likely, you will not need to enable this. However, if you intend to enable private data logging, you must send a custom settings profile with the following content (or manually fit this content into a mobileconfig file and install it):

<dict>
      <key>PayloadDisplayName</key>
      <string>Enable ManagedClient Private Logging</string>
      <key>PayloadEnabled</key>
      <true/>
      <key>PayloadIdentifier</key>
      <string>com.apple.system.logging.982757D5-775E-4330-A5EF-089B3B5F4249</string>
      <key>PayloadType</key>
      <string>com.apple.system.logging</string>
      <key>PayloadUUID</key>
      <string>982757D5-775E-4330-A5EF-089B3B5F4249</string>
      <key>PayloadVersion</key>
      <integer>1</integer>
      <key>System</key>
      <dict>
        <key>Enable-Private-Data</key>
        <true/>
      </dict>
</dict>

2. Predicate-Based Filtering for Log Command

Note the following text on Predicates, clipped from the manual for the log command. Predicates are the building blocks for filtering the Unified Log.  

PREDICATE-BASED FILTERING
     Using predicate-based filters via the --predicate option allows users to
     focus on messages based on the provided filter criteria. For detailed
     information on the use of predicate based filtering, refer to the
     Predicate Programming Guide:
           https://developer.apple.com/library/mac/documentation/Cocoa/Conceptual/Predicates/Articles/pSyntax.html

     The filter argument defines one or more pattern clauses following NSPred-
     icate rules. See log help predicates for the full list of supported
     keys. Supported keys include:

     eventType          The type of event: activityCreateEvent, activityTran-
                        sitionEvent, logEvent, signpostEvent, stateEvent,
                        timesyncEvent, traceEvent and userActionEvent.

     eventMessage       The pattern within the message text, or activity name
                        of a log/trace entry.

     messageType        For logEvent and traceEvent, the type of the message
                        itself: default, info, debug, error or fault.

     process            The name of the process the originated the event.

     processImagePath   The full path of the process that originated the
                        event.

     sender             The name of the library, framework, kernel extension,
                        or mach-o image, that originated the event.

     senderImagePath    The full path of the library, framework, kernel exten-
                        sion, or mach-o image, that originated the event.

     subsystem          The subsystem used to log an event.  Only works with
                        log messages generated with os_log(3) APIs.

     category           The category used to log an event.  Only works with
                        log messages generated with os_log(3) APIs.  When cat-
                        egory is used, the subsystem filter should also be
                        provided.

3. Log and Find Troubleshooting Session Time Markers

As you begin to troubleshoot an issue, you can start logging time markers directly into the unified log by using the logger command. You can then later search for these markers by using the log command. The following represents a way to bookmark the start and end time for troubleshooting activities:

  • Log a marker in Unified Logging for troubleshooting events: logger "[WS1-Support] Starting <activity>" and logger "[WS1-Support] Ending <activity>"
  • Search for all markers to determine troubleshooting time frames: log show --predicate 'process=="logger" AND eventMessage CONTAINS "[WS1-Support]"'

With the list of times for troubleshooting activities output by the log command, you can later filter for events constrained to those time frames by using the --start "YYYY-MM-DD HH:MM:SS" --end "YYYY-MM-DD HH:MM:SS" parameters on your log commands. Remember that the hours (HH) are in 24-hour format, and displayed in the machine's configured time zone.

4. Find and Use System Boot Times

At times, you may be troubleshooting unexpected system restarts and kernel panics. In this instance, you may not know exactly when the system restarted in order to work your way backwards through the logs. In this particular case, you can search the unified log for system start messages in order to find timestamps of where to end your logging.

  • Find the system boot times using grep: log show --start "YYYY-MM-DD HH:MM:SS" | grep "=== system boot"
  • Use the system boot time as the "end" parameter in any log show or log collect command: --end "YYYY-MM-DD HH:MM:SS"

As mentioned above, you can filter for events constrained to specific time frames by using the --start "YYYY-MM-DD HH:MM:SS" and  --end "YYYY-MM-DD HH:MM:SS" parameters on your log commands. Remember that the hours (HH) are in 24-hour format, and displayed in the machine's configured time zone.

5. Example Commands to Get You Started

  • Find any Logging Markers for the past 12 hours in a SysDiagnose: log show --archive ~/Downloads/sysdiagnose_2020.10.06_00-55-25-0400_Mac-OS-X_iMacPro1-1_19G73/system_logs.logarchive  --predicate 'process=="logger" AND eventMessage CONTAINS "[WS1-Support]"' --last 12h
  • Filter for Events in Specific Time Slots (Based On Your Markers): log show --archive ~/Downloads/sysdiagnose_2020.10.06_00-55-25-0400_Mac-OS-X_iMacPro1-1_19G73/system_logs.logarchive  --predicate 'process == "mdmclient" OR process == "apsd" OR process == "dasd" OR process == "storedownloadd" OR process == "appstored" OR process == "logger"' --start "2020-10-06 00:47:00" --end "2020-10-06 00:48:49"

Troubleshooting macOS Management

Troubleshooting macOS Enrollment

This section covers potential troubleshooting items related to macOS enrollment.  

1. Enrollment Restrictions

In Workspace ONE UEM, navigate to Groups & Settings > All Settings > Devices & Users > General > Enrollment > Restrictions and ensure that no device restrictions are in place that might be preventing enrollment.

2. Registered Devices Only

In Workspace ONE UEM, navigate to Groups & Settings > All Settings > Devices & Users > General > Enrollment > Authentication and check the Devices Enrollment Mode option. If set to Registered Devices Only, then the device must be registered in the console before it can be enrolled.

Note: macOS devices synced from Apple Business Manager (or Apple School Manager) are considered Registered and can be enrolled.

3. Device Previously Enrolled

Apple Business Manager devices that have already been enrolled cannot re-enroll without first deleting the device record in Workspace ONE UEM.  

4. Devices Enrolled to Wrong Organization Group

When macOS enrolls to Workspace ONE UEM, numerous factors control the specific organization group where the device is placed.   Before the Organization Group for the device is finalized, Workspace ONE checks multiple attributes for both the DEP profile and the User account enrolling to best determine where the device should go.  The following attributes are considered during Organization Group selection:

  • User's "Enrollment Organization Group" -- Found in the User Account properties under the General tab
  • User's "Managed By" Organization Group -- The level at which this user is integrated into the console
  • The "Device Organization Group" associated with the Automated Device Enrollment (or "DEP") Profile
  • "User Group Mapping" setting in the console Groups & Settings > All Settings > Devices & Users > General > Enrollment > Grouping
  • "Batch Import" for device Organization Group Override following Automated Device Enrollment (or "DEP").  Note: This method is not typically advised.

For devices enrolling with Automated Device Enrollment (or "DEP") via Apple Business Manager:

  1. ONLY if the Automated Device Enrollment (or "DEP") Profile specifies "Authentication = OFF" and staging is configured, the device will enroll to the Staging User's "Enrollment Organization Group" and NOT the Automated Device Enrollment (or "DEP") Profile's "Device Organization Group"
  2. If "User Group Mapping" is configured, the device will enroll to the organization group based on the preconfigured AD user group and organization group mapping.  In this scenario, the Automated Device Enrollment (or "DEP") Profile's "Device Organization Group" and the User's "Enrollment Organization Group" are not considered.
  3. Without User Group Mapping, if the Automated Device Enrollment (or "DEP") Profile's "Device Organization Group" and the User's "Enrollment Organization Group" are in the same branch (parent/child), the device enrolls to the lower (child) Organization Group.
  4. Without User Group Mapping, if the Automated Device Enrollment (or "DEP") Profile's "Device Organization Group" and the User's "Enrollment Organization Group" are siblings, then the device enrolls to the Automated Device Enrollment (or "DEP") Profile's "Device Organization Group".

For more information (and a flowchart), refer to VMware KB 83132 - Organization Group Assignment In Workspace ONE for Automated Device Enrollment (ADE) Devices.

5. Enroll Virtual Machines

When attempting to enroll a macOS virtual machine, you may notice that the enrollment fails and/or the enrollment profile generated appears to be for iOS. This is because the virtual machine must emulate physical hardware attributes in order for Workspace ONE to generate the proper enrollment profile.  

To get the necessary hardware attributes, you should run the following commands on the hardware you want to emulate:

  • Hardware Model:  ioreg -l | awk '/product-name/ { split($0, line, "\""); printf("%s\n", line[4]); }'
  • Serial Number:  ioreg -l | awk '/IOPlatformSerialNumber/ { split($0, line, "\""); printf("%s\n", line[4]); }'
  • Board-ID:  ioreg -lp IOService | awk '/board-id/ { split($0, line, "\""); printf("%s\n", line[4]); }'

Using VMware Fusion, you should enter or modify the following items in the VMX file for your virtual machine:

Note: If the VM does not boot, you might have duplicated one of the options (typically the smbios.reflectHost).

board-id.reflectHost = "FALSE"
board-id = "MAC-F22589C8"
hw.model.reflectHost = "FALSE"
hw.model = "MacBookPro6,2"
serialNumber.reflectHost = "FALSE"
serialNumber = "RM129481AGW"
smbios.reflectHost = "FALSE"

6. Date & Time Zone Incorrect

From within Terminal.app, you can enter the following commands to get date and time zone information quickly:

  • Date:  date
  • Time Zone:  sudo systemsetup -gettimezone

If the date/time/zone is incorrectly set, then you will potentially have problems with certificate checking, trust, and more.

7. Obtain Shell and Console During Setup Assistant

One method of troubleshooting during automated enrollment is to obtain console and shell access during the Setup Assistant. Although your options are limited due to the current user context, it is still useful to know these commands:

  • Press Command + Option + Control + T during Setup Assistant to open Terminal.
  • Press Command + Option + Control + C during Setup Assistant to open Console.

8. Ensure Correct Automated Enrollment Profile

If you are not seeing the expected behavior during automated enrollment, you can check the settings currently assigned to the device and optionally re-assign a new automated enrollment profile. See the following:

  • Check if enrolled via Automated Enrollment:  /usr/bin/profiles status -type enrollment
  • Check the current settings applied via Automated Enrollment:  sudo /usr/bin/profiles show --type enrollment
  • Download a new Automated Enrollment profile:  sudo /usr/bin/profiles renew -type enrollment

9. Ensure Correct Staging for Multiple Users

macOS is inherently a multi-user operating system. However, the mdmclient processes that provide the foundational management capabilities for MDM are not. When macOS is not staged properly for multiple users, you might notice that one user will get BOTH device and user profiles, whereas subsequent users are not delivered any user profiles from Workspace ONE. This behavior is a function of the mdmclient built-in to macOS and can be altered only by a specific set of configurations.

To enable multi-user functionality for macOS and Workspace ONE, the device should be enrolled using multi-user staging and then bound to Active Directory.  

For more details, review the staging configuration & enrollment process: Tech Zone Onboarding Options for macOS Tutorial.

10. Check Logging

If all else fails, start collecting log information using the command line. The following command extracts relevant detail from the Unified Log:

  • log stream --predicate 'process == "mdmclient" OR process == "apsd" OR process == "dasd"'

Additionally, if you gather a sysdiagnose file (as outlined in Understanding macOS Unified Logging), you can filter using the --archive and --last parameters to narrow down the issue.

Troubleshooting Post-Enrollment Onboarding Experience

Workspace ONE Intelligent Hub with post-enrollment onboarding screen.

VMware Workspace ONE Intelligent Hub for macOS version 21.04 introduced a new built-in post-enrollment onboarding screen. The onboarding screen is shown for all enrollment types after hub is installed and displays a list of all auto-deployed apps. Workspace ONE administrators can customize the welcome text to personalize the end-user's experience.

Enable and Configure Post-enrollment onboarding

Post-Enrollment Onboarding can only be enabled in production environments running Workspace ONE UEM 2105 or later.  Validate that the Post-Enrollment onboarding screen is enabled and configured at Groups & Settings > All Settings > Devices & Users > General > Enrollment > Optional Prompt.

Troubleshoot with Logging

In the Workspace ONE UEM Console, browse to the Device Details view, and click Troubleshooting > Event Log. Look for Agent Settings and Enrollment Settings events.

From within Intelligent Hub Logs (or via Unified Logging), search for the following

  • Subsystem: “com.vmware.hub.hubservices and Category: “postEnrolmentOnboardingFlow” and “enrollment
  • Subsystem: “com.vmware.hub.uem and Category: “AgentSettings

Alternatively, you can search these events in Terminal with the log command as follows: log show --info --debug --predicate '((subsystem == "com.vmware.hub.hubservices") && (category IN { "postEnrolmentOnboardingFlow", "enrollment" })) || ((subsystem == "com.vmware.hub.uem") && (category == "AgentSettings"))' --last 10m

Note: You can alter the --last  parameter to adjust for your troubleshooting timeframe. For more details, see Understanding macOS Unified Logging.

Troubleshooting macOS Profiles

Profiles provide the most basic building blocks for macOS management in Workspace ONE. Apple defines much of the profile content in the Developer Reference for Device Management. Apple also provides an MDM for IT Administrators guide that helps admins understand the base management capabilities in all the Apple operating systems.  

This section of this tutorial aims to help you troubleshoot profile-related issues.

1. No User Profiles (Only Device Profiles)

This is most likely not a profile issue, but rather an enrollment issue. Generally, this behavior indicates that a device was improperly staged—check the staging configuration & enrollment process: Tech Zone Onboarding Options for macOS Tutorial.

Note: Multi-user macOS management requires multi-user staging and Active Directory binding.

2. No User or Device Profiles

If there are no profiles (user or device) in your environment, verify the following.

  1. Check that the device gets push notifications by attempting to send one via the console.
  2. Check that the APNs certificate is still valid as described in Validating Workspace ONE UEM Console Settings.
  3. Check Network Connectivity to APNS and AWCM
  4. Check the Event Log by browsing to Devices > List View > Select the Device (switch to Device Details View) > More > Troubleshooting and review the Event Log and Command Queue.
    Reminder: The content of these views can be filtered.
  5. Check Logging (using Terminal or a SysDiagnose file) as follows:
    1. sysdiagnose:   sudo sysdiagnose -f -n ~/Downloads
    2. Log Streaming:   log stream --predicate 'process == "mdmclient" OR process == "apsd" OR process == "dasd"'

3. Troubleshoot the Privacy Preferences Profile:

To troubleshoot the Privacy Preferences (or TCC or PPPC) behavior on macOS, you must stream all events related to Privacy Preference Policy Control prompts:

  • log stream --debug --predicate 'subsystem == "com.apple.TCC" AND eventMessage BEGINSWITH "AttributionChain"'
  • log stream --debug --predicate 'subsystem == "com.apple.TCC" AND eventMessage BEGINSWITH "AttributionChain" OR eventMessage CONTAINS "synchronous to com.apple.tccd.system"'

You can also read the TCC Databases using Terminal.app. Before attempting to manually review these files, ensure that Terminal.app is granted full control by MDM or manually in System Preferences. After granting permissions, run the following commands:

  • echo ".dump" | sudo sqlite3 /Library/Application\ Support/com.apple.TCC/TCC.db
  • echo ".dump" | sudo sqlite3 ~/Library/Application\ Support/com.apple.TCC/TCC.db

Troubleshooting macOS Bootstrap Packages

This section covers common troubleshooting steps for macOS Bootstrap Packages.

1. Bootstrap Package Troubleshooting

A bootstrap package is a lightweight distribution package created by a Workspace ONE administrator. The intent of a bootstrap package is to silently deliver and install a small baseline set of agents to onboard macOS for a user. For example, an administrator can use the bootstrap package to install a distribution package containing the chef, puppet, or saltstack agents.    

Note: Bootstrap Packages install through the InstallEnterpriseApplication command during the Await Configuration phase of device enrollment. Because of this, there is no confirmation of a successful install, other than to audit the ApplicationList sample that the device returns later on.  

The following outlines some potential fixes for Bootstrap packages:

  1. Is the package fairly small in size?
    It is best practice is to keep the bootstrap package as small as possible so that it installs quickly.
  2. Is the bootstrap package a signed Distribution type package?
    1. The package must be signed with an Apple Developer ID Installer Certificate. Only the package needs to be signed, not the app because the Apple Gatekeeper does not check apps installed through MDM.
    2. The package must be a distribution package (product archive), not a flat component package.
  3. Is Workspace ONE configured to attempt installing the Intelligent Hub also?
    Navigate to Devices > Devices Settings > Apple > Apple macOS > Intelligent Hub Settings. Some older versions of macOS could not install two bootstrap packages simultaneously—In this situation, all bootstrap packages would fail to install on the device. If the Install Hub After Enrollment switch is set to Enabled, try setting it to Disabled.    

Volume Purchase App Troubleshooting

As organizations deliver volume-purchased apps from Apple Business Manager, some unexpected issues may arise. This article addresses some common issues affecting volume-purchased app delivery.

1. Newly Purchased Apps Not Syncing to Workspace ONE UEM

This error is typically the result of one of the following issues:

  • Delays in Apple Business Manager from when you purchase the app to when the licenses are allocated to the Location Token.
  • Apple has released new Terms & Conditions in Apple Business Manager. 
    • To resume syncing, log in to Apple Business Manager with an account granted the Administrator role, and accept the updated Terms & Conditions.
  • The location token downloaded from Apple Business Manager and uploaded to Workspace ONE UEM has expired. 
    • Download a new server token from Apple Business Manager (under Settings > Apps & Books) and upload it to Workspace ONE UEM.

2. Device-Based Licensed Apps Not Installing

If you have assigned an app to a device using device-based assignment, one of the following could be an issue:

  • If the user is getting a prompt to log in with an Apple ID, there is most likely an assignment that was made to the user before the device-based assignment was created. 
    • Unassign the app from the user completely before you attempt to reassign the app to the device.
  • Ensure that an adequate number of licenses are available to assign to the device.
  • Ensure that the device has access to vpp.itunes.apple.com.
  • The location token downloaded from Apple Business Manager and uploaded to Workspace ONE UEM has expired. 
    • Download a new location token for that same location from Apple Business Manager (under Settings > Apps & Books) and upload it to Workspace ONE UEM.

3. Volume-Purchased Apps Not Removing from macOS

When a VPP app for macOS is no longer scoped to the device or user, or the device is enterprise wiped, the app is not removed from macOS. This is by design in macOS versions prior to macOS 11 (Big Sur), as these earlier versions of macOS did not support any commands to remove the application.

For More Information on Troubleshooting

If you are still experiencing issues with volume-purchased applications, refer to the Volume Purchase Program (VPP) Troubleshooting Guide or contact VMware support directly.

Introduction to Troubleshooting Non-Store Apps

Non-store macOS applications are delivered from Workspace ONE UEM by the Workspace ONE Intelligent Hub. As such, there are some basic prerequisites for macOS software delivery to work. If the prerequisites are not met, there might be unexpected behavior due to the specific installer package that must be worked around.

1. Prerequisites

Non-store macOS apps require the following to work correctly:

Note: AWCM provides notifications to the Intelligent Hub to trigger real-time non-store app installation. In the absence of AWCM, the Intelligent Hub reverts to a scheduled interval to check for new app installation commands.

Determining When to Install Non-Store Apps

When determining whether to install a non-store macOS application, Workspace ONE uses the following information from the manifest PLIST (or the configuration in the console).

Note: The Install Check Script and Installs Arrays are the most flexible methods for determining installation status.

1. Install Check Script

Whether entered in the Scripts tab in the console or entered manually in the PLIST file, the script should exit with a zero (0) return code to trigger an install. The script should include a shebang statement (such as #!/bin/zsh) at the beginning.

2. Installs Array

This key-value pair in the PLIST file specifies identifying information about a binary or file which should be directly compared to determine if the correct version of an app is installed. The installs list can contain any number of items. These can be applications, Preference Panes, Frameworks, or other bundle-style items, Info.plists, or simple directories or files. For more information, see How Munki Decides What Needs To Be Installed.

3. Receipts Array

When a package is installed, the installer leaves a receipt and bill of materials file on the machine. Some packages parsed by the Workspace ONE Admin Assistant will include detail on what receipts will be dropped by the installer in the PLIST file. When a PLIST file lacks the install check script or installs array, the lack of specified receipt(s) triggers an install. For more information, see How Munki Decides What Needs To Be Installed.

 

Troubleshooting Non-Store App Installer Problems

This section lists some of the common problems you might encounter when installing a non-store macOS app.

1. Incorrect Application Name in the Application Catalog

Sometimes an installer package parsed by the VMware Workspace ONE Admin Assistant generates a PLIST file where the application name is incorrect. It is also possible that organizations refer to software by a common or colloquial name that is easily recognized by end users. In either case, administrators can change the name displayed in the Intelligent Hub application catalog before uploading the PLIST to Workspace ONE UEM.

1.1. Open the PLIST Generated by Admin Assistant

Using Admin Assistant to help troubleshoot macOS.
  1. In Finder, browse to the PLIST file for the app in question (usually in ~/Documents/Workspace ONE Admin Assistant/<app name>-<version> ).  
  2. Right-click the PLIST file and select Open With and then select an editor (such as BBEdit).

1.2. Modify PLIST File

Modify the string value for the name key-value pair.  

2. Installers with No Package Version

Sometimes an installer package parsed by the VMware Workspace ONE Admin Assistant generates a PLIST file where the version shows Please Edit Me. This problem must be fixed before uploading the PLIST to Workspace ONE UEM.

2.1. Open the PLIST Generated by Admin Assistant

Using Admin Assistant to help troubleshoot mac OS.
  1. In Finder, browse to the PLIST file for the app in question (usually in ~/Documents/Workspace ONE Admin Assistant/<app name>-<version> ).  
  2. Right-click the PLIST file and select Open With and then select an editor (such as BBEdit).

2.2. Modify Plist File

Modify the string value for the version key-value pair.  

3. Installation is Looping (Always "Installing")

In some instances, an application successfully installs but the Intelligent Hub continually reports the app as "Installing". If you look in the ManagedSoftwareUpdate.log file (see Gathering Logs and Validating macOS App Installation), you'll see the app is constantly marked for installation each time the Hub checks for installed software. This is typically the result of a metadata PLIST that doesn't contain the correct receipt or installs arrays. In this instance, you must make one of the following changes to the metadata PLIST generated by Workspace ONE Admin Assistant:

  1. Validate or Add an Installs Array as discussed in Determining When to Install Non-Store Apps.
  2. Add an Install Check Script (and ensure it returns a zero (0) value return code when an install should proceed).

Troubleshooting Workspace ONE Tunnel on macOS

If a Per-App Tunnel problem occurs on macOS, there are a number of places to troubleshoot. This section of the tutorial covers where to troubleshoot on macOS at a high level. Depending on the problem, there might be steps that should be performed on the Unified Access Gateway. However, troubleshooting the Unified Access Gateway is outside the scope of this tutorial. Workspace ONE UEM administrators should contact VMware Support for assistance when troubleshooting Per-App Tunnel, Workspace ONE Tunnel, or the Unified Access Gateway.

This section covers a high-level set of initial troubleshooting steps.

1. Open Workspace ONE Tunnel

Troubleshoot macOS and Workspace ONE Tunnel on mac os.
  1. Click the Launchpad on the Dock.
  2. Click VMware Tunnel.

2. Ensure Tunnel is Configured

Confirm Workspace ONE Tunnel macOS application is configured to troubleshoot macOS.
  1. Ensure that the Device Configured status shows Configured. This indicates that Workspace ONE Tunnel has received configuration data from Workspace ONE UEM. If the status is not configured, try one of the following:
    • Check the Device Traffic Rules and Save and Publish the rules again.
    • Check the last seen value for the device in the Workspace ONE UEM console. Is the device communicating with Workspace ONE UEM?
    • Validate that other MDM commands are being sent to the device. Create an assignment (smart) group containing the single device and attempt to send it a new profile payload.
  2. Ensure that the Internet status shows Connected. If Tunnel cannot connect to the Internet, it probably cannot connect to the Unified Access Gateway.
    • Validate that the device has a working Ethernet or Wi-Fi connection (IP address, subnet mask, gateway, and DNS addresses are present).
    • Validate DNS resolution: Open Terminal and enter nslookup uag.fully.qualified.domain to ensure that an IP address is resolved.
    • Validate Connectivity to UAG: Within Terminal, enter nc -vz uag.fully.qualified.domain uagport (such as nc -vz uag.company.com 443).
  3. Ensure that the Enterprise Network status shows ConnectedIf Workspace ONE Tunnel is disconnected from the Enterprise network, apps cannot use Per-App Tunnel. This might indicate an issue with Workspace ONE Tunnel connecting to the Unified Access Gateway or an issue with Device Traffic Rules.
    • The remainder of this section details how to troubleshoot Tunnel connectivity.

3. Validate Per-App VPN Profile

Validate per-app VPN profiles to troubleshoot mac OS.
  1. Click System Preferences.
  2. Double-click Profiles.
  3. Scroll through the left panel.
  4. Click the Per-App VPN profile that was created.
  5. Ensure that the VPN App Layer Service details are correct, especially the VPN Remote Address and the OnDemand Enabled value.
    • If the profile is missing or misconfigured, check the profile configuration and re-push the profile to the device from within the UEM Console Device Details view (on the Profiles tab).

4. Validate Advanced Tunnel Information

Validate tunnel information to help troubleshoot macOS.
  1. Open the Workspace ONE Tunnel client and click the VMware Tunnel menu.
  2. Click Whitelisted Applications.
  3. Verify that the list of allowlisted applications matches the settings configured in the Device Traffic Rules.
  4. From the VMware Tunnel menu (#1), click Diagnostics.
  5. Click Enable Debug to get verbose information.
  6. Review Diagnostics information.
  7. Click Disable Debug when troubleshooting is complete.

6. General VPN Network Extension Troubleshooting

Per Apple's Developer Website (requires login), you can use the following commands to gather additional data from the VPN (Network Extension):

  • sudo defaults write /Library/Preferences/com.apple.networkextension.control.plist  LogToFile -boolean true
  • sudo defaults write /Library/Preferences/com.apple.networkextension.control.plist  LogLevel -int 7

Reproduce the issue and then enter this command in Terminal.app:

  • /System/Library/Frameworks/SystemConfiguration.framework/Resources/get-mobility-info

You should find additional information in the resulting get-mobility-info output file.

You can later deactivate the logging by issuing the following commands:

  • sudo defaults delete /Library/Preferences/com.apple.networkextension.control.plist  LogToFile
  • sudo defaults delete /Library/Preferences/com.apple.networkextension.control.plist  LogLevel

Troubleshooting SSO Extensions

SSO Extensions are still a relatively new concept with not many vendors publishing SSO Extensions for macOS as of today. If you do start beta testing an extension, here's a quick list of possible troubleshooting steps to help determine issues.

1. Enable Debug Modes for Logging

  1. Enable the necessary debug modes by running the following in Terminal.app:  
    1. $ sudo log config --mode "level:debug,persist:debug" --subsystem com.apple.AppSSO
    2. $ sudo log config --mode "level:debug,persist:debug" --subsystem com.apple.Heimdal
    3. $ sudo log config --mode "level:debug,persist:debug" --subsystem org.h5l.gss
  2. Restart the SSO Extension subsystems:
    pkill -9 KerberosExtension AppSSOAgent KerberosMenuExtra
  3. Reproduce the issue and generate a sysdiagnose (sudo sysdiagnose)

NOTE: When finished troubleshooting and gathering debug logs, reset the logging to normal levels by running the following commands:

  • $ sudo log config --subsystem com.apple.AppSSO --reset
  • $ sudo log config --subsystem com.apple.Heimdal --reset
  • $ sudo log config --subsystem org.h5l.gss --reset

You can also enable debug logging via a Custom Settings (XML) profile as follows:

<dict>
	<key>PayloadIdentifier</key>
	<string>com.apple.system.logging.appsso.1</string>
	<key>PayloadType</key>
	<string>com.apple.system.logging</string>
	<key>PayloadUUID</key>
	<string>8C31282A-DD51-46C3-A20C-324CBC263EC6</string>
	<key>PayloadVersion</key>
	<integer>1</integer>
	<key>Subsystems</key>
	<dict>
		<key>com.apple.AppSSO</key>
		<dict>
			<key>DEFAULT-OPTIONS</key>
			<dict>
				<key>Default-Privacy-Setting</key>
				<string>Public</string>
				<key>Enable-Oversize-Messages</key>
				<true/>
				<key>Level</key>
				<dict>
					<key>Enable</key>
					<string>Debug</string>
					<key>Persist</key>
					<string>Debug</string>
				</dict>
			</dict>
		</dict>
		<key>com.apple.Heimdal</key>
		<dict>
			<key>DEFAULT-OPTIONS</key>
			<dict>
				<key>Default-Privacy-Setting</key>
				<string>Public</string>
				<key>Enable-Oversize-Messages</key>
				<true/>
				<key>Level</key>
				<dict>
					<key>Enable</key>
					<string>Debug</string>
					<key>Persist</key>
					<string>Debug</string>
				</dict>
			</dict>
		</dict>
		<key>org.h5l.gss</key>
		<dict>
			<key>DEFAULT-OPTIONS</key>
			<dict>
				<key>Default-Privacy-Setting</key>
				<string>Public</string>
				<key>Enable-Oversize-Messages</key>
				<true/>
				<key>Level</key>
				<dict>
					<key>Enable</key>
					<string>Debug</string>
					<key>Persist</key>
					<string>Debug</string>
				</dict>
			</dict>
		</dict>
	</dict>
</dict>

2. Logging Commands

As you troubleshoot the SSO extensions, the following command line will stream all Events related to the Kerberos SSO Extension and additional Apple-built SSO Extension binaries. If you are using an SSO extension from another identity provider (such as Okta or Azure Active Directory), you must also add the appropriate predicate parameters in the following command:

  • log stream --debug --predicate '(subsystem == "com.apple.Heimdal") OR (subsystem == "com.apple.AppSSO") OR (subsystem == "org.h5l.gss") OR (subsystem == "com.apple.network")'

3. Other Command Line Tools

  1. Show all shared web credentials associated with any apps using AASA or SSO Extensions:
    • /System/Library/PrivateFrameworks/SharedWebCredentials.framework/Support/swcutil show
  2. Determine if SSO Extension is loaded:  
    • pluginkit -vvv -m -i <SSO Extension App ID>

 

4. Relevant Apple Documentation

The following Apple documentation may prove useful in troubleshooting SSO Extensions as well:

Troubleshooting Workspace ONE Assist on macOS

VMware provides Workspace ONE Assist to help you remotely support your macOS fleet. To troubleshoot, check the following.

1. Check Prerequisites

Check that the following pre-requisites have been made:

  1. Has the user provided screen recording permissions to the client-side assist app?
  2. Has the Assist client application been downloaded and installed on the Mac which needs remote assistance?
  3. Does the device show as AWCM-connected in the Device Details view?

2. Analyze the Logging

From within Terminal.app, run the following command to find out what's going on:

  • log stream --predicate '(process == "tccd" AND eventMessage CONTAINS "com.aetherpal") OR process == "awcmd" OR process == "assistd" OR process == "AssistCore"'

Troubleshooting Intelligent Hub Processes

The Workspace ONE Intelligent Hub for macOS provides a good deal of functionality to augment the built-in mdmclient functionality. This section illustrates a number of methods you can use to troubleshoot Intelligent Hub for macOS.  

Important: The primary method to gather Hub-related logging is to Request Hub Logs from the device.

1. General Log Streaming

Stream events related to Hub, awcm, Software distribution, and so on (Internal Apps, NOT VPP):

  • log stream --predicate 'process == "awcmd" OR process == "apsd" OR subsystem BEGINSWITH "com.vmware.hub"'

2. Hub Process Blocking (Currently in Beta - Version 2010)

Stream events related to Hub Process Blocking (System Extension):

  • log stream --debug --predicate 'subsystem == "com.vmware.hub.security" OR process == "com.airwatch.mac.agent.EndpointSecurity" OR process == "hubd" OR process == "IntelligentHubAgent" or process == "endpointsecurityd"'

3. Sensors (Currently in Tech Preview - Version 2010)

If you are troubleshooting sensors with Hub 2010, the following commands are helpful:

  1. View the list of sensors downloaded to Hub:
    • sudo hubcli sensors --list
    • sudo hubcli sensors --list json
  2. Note: Sensors require a client-side feature flag:
    • sudo plutil -insert MacOsSensorsCliListValueFeatureFlag -bool YES /Library/Application\ Support/AirWatch/Data/com.vmware.hub.flags.plist
  3. Trigger a sensor run on-demand:
    • sudo hubcli sensors --trigger <sensor_name>

NOTE:  Sensors and scripts require the VMware Workspace ONE Workflow Engine for macOS, which is a separate installer from the Intelligent Hub.

Troubleshooting FileVault Encryption

FileVault encryption has been rapidly changing over the past few years. With Apple's introduction of the Secure Enclave and various system on chip components specifically aimed at securing data, the background processes and requirements to make FileVault work have added additional complexity. This chapter aims to outline the basic features underlying FileVault and how to troubleshoot it.

Per Apple's Platform Security Guide, macOS computers offer FileVault, a built-in encryption capability, to secure all data at rest. FileVault uses the AES-XTS data encryption algorithm to protect full volumes on internal and removable storage devices. There are two methods FileVault secures data:  using a volume key (Apple Silicon hardware) or using the Secure Enclave and AES Engine (Macs with T2 Chip). In either method, the user must enter their credentials in order to boot macOS.

NOTE: This chapter specifically aims to aid troubleshooting Filevault for macOS Big Sur using Personal Recovery Keys (PRK). There are currently no plans to expand this section to include support for troubleshooting FileVault on CoreStorage volumes. Additionally, at this time there are no plans to expand this section for Institutional Recovery Keys (IRK) as I see more administrators moving away from IRK in favor of PRK.

1. What is SecureToken?

Per Apple's Platform Security Guide:

"Apple File System (APFS) in macOS 10.13 or later changes how FileVault encryption keys are generated. In previous versions of macOS on CoreStorage volumes, the keys used in the FileVault encryption process were created when a user or organization turned on FileVault on a Mac. In macOS on APFS volumes, the keys are generated either during user creation, setting the first user’s password, or during the first login by a user of the Mac. This implementation of the encryption keys, when they’re generated, and how they’re stored are all part of a feature known as Secure Token. Specifically, a secure token is a wrapped version of a key encryption key (KEK) protected by a user’s password. In macOS 11, setting the initial password for the very first user on the Mac results in that user being granted a secure token."

Basically, think of SecureToken as an encryption key tied to a user. Some type of user-related "event" must happen (user creation, user password creation, or first login by a user) which creates the key that comprises the SecureToken.    SecureToken creation must occur before

  1. FileVault can encrypt a disk, and
  2. Before a device can generate a Bootstrap Token. 

Additionally, it is known that subsequent users added to the system must be created by a user with a SecureToken in order to have their own SecureToken generated. The alternative to user creation by a SecureToken-enabled user is the use of a Bootstrap Token to generate a user's SecureToken.

Interacting with SecureToken:

  • Check if a User is SecureToken Enabled: sysadminctl -secureTokenStatus <username>
  • List Accounts which can unlock FileVault encrypted disk: diskutil apfs listcryptousers /
  • Map UUIDs of CryptoUsers with Usernames: sudo fdesetup list

2. What is Bootstrap Token?

Per Apple's Platform Security Guide:

"macOS 10.15 introduced a new feature—Bootstrap Token—to help with granting a secure token to both mobile accounts and the optional device enrollment-created administrator account (“managed administrator”). In macOS 11, the Bootstrap Token can grant a secure token to any user logging in to a Mac computer, including local user accounts. Using the Bootstrap Token feature of macOS 10.15 or later requires:

  • Mac enrollment in MDM using Apple School Manager or Apple Business Manager, which makes the Mac supervised
  • MDM vendor support

In macOS 10.15.4 or later, a Bootstrap Token is generated and escrowed to MDM on the first login by any user who is Secure Token–enabled if the MDM solution supports the feature. A bootstrap token can also be generated and escrowed to MDM using the profiles command-line tool, if needed.

In macOS 11, the bootstrap token may also be used for more than just granting secure token to user accounts. On a Mac with Apple silicon, the bootstrap token, if available, can be used to authorize the installation of both kernel extensions and software updates when managed using MDM."

NOTE: Workspace ONE UEM introduced bootstrap token support in version 2008 for Catalina, and 2011 for Big Sur.

3. How Everything Fits Together

There's no way around it - understanding FileVault can be tough for administrators new to macOS management.  Hopefully this summary makes using secure and bootstrap tokens easier:

  1. The First User to log on (for example, "First User") to macOS gets a SecureToken.
  2. SecureToken enables a user account to unlock a FileVault encrypted disk.
  3. Users created by the First User are granted their own SecureToken. In other words, SecureToken is passed from one trusted account to another.
  4. MDM informs macOS of its Bootstrap Token capability, and macOS sends a Bootstrap Token derived from the First User's SecureToken after the First User logs in.
  5. MDM can use the Bootstrap Token to grant a SecureToken to user accounts NOT created by the First User.

4. Troubleshooting Bootstrap Token

Bootstrap Token is a straightforward process for troubleshooting. If the user is a standard user (non-admin), you need to use su <hiddenITadminaccount> to change to a user that can run the following commands from within Terminal.app.

  • Check if Workspace ONE already sets the MDM Option for BootstrapTokenAllowed: In the device details view, check the Troubleshooting tab and search for bootstrap.   You should see an event where we set the MDM Option and another event where Bootstrap Token is escrowed.
    • This MDM Option should trigger macOS to generate a Bootstrap Token when possible.
  • Check if Bootstrap Token is supported from the device: sudo profiles status -type bootstraptoken
    • The first line indicates that UEM supports Bootstrap Token and the second line indicates if it has already been escrowed or not.
    • If Bootstrap Token supported on server is NO, but UEM is a version supporting Bootstrap Token, then the device did not receive the MDM Option.
    • If Bootstrap Token escrowed to server is NO but supported on server is YES,  check that there is at least one account with SecureToken: diskutil apfs listcryptousers / and optionally sudo fdesetup list  
    • NOTE: Bootstrap Token cannot be escrowed to server if none of the user accounts present on the system have SecureToken.
  • Manually Generate and Escrow Bootstrap Token: sudo profiles install -type bootstraptoken
    • The command is interactive and requires the Admin username and password.
    • Re-validate on the UEM console that the Bootstrap Token is escrowed.
  • Validate the BootStrap Token for macOS: sudo profiles validate -type bootstraptoken

5. Troubleshooting FileVault Recovery Key Escrow

macOS ONLY escrows the recovery key when instructed to do so, and ONLY for personal recovery keys. The way this process worked differs depending on the version of macOS. This guide covers the escrow process for macOS 10.13 and later.

FileVault Recovery Key escrow is initiated by the com.apple.security.FDERecoveryKeyEscrow payload in a profile.  

NOTE: If a previous-style payload (com.apple.security.FDERecoveryRedirect) is delivered to macOS 10.13 and later, it is ignored. Also, if FileVault was already enabled and escrowed with the old payload, no warning or error will be shown.  

Troubleshooting Steps:

  1. Confirm the FDERecoveryKeyEscrow payload landed before FileVault was enabled.
    1. Remember, if FileVault was already enabled and escrowed with the old payload, no warning or error will be shown.
  2. Ensure there is no more than one FileVault payload delivered to the device.
    1. Was the device upgraded to macOS 10.13 or later and potentially already had a previous-style payload? It may have been removed after upgrade, so look at the UEM Console to determine if a previous-style payload was sent pre-upgrade.
  3. Look to see if the /var/db/FileVaultPRK.dat file was created.
    1. If the FileVaultPRK.dat file exists, but the Personal Recovery Key is not shown in the console, check the device details troubleshooting tab for three events: Security Information Requested,  Security Information Confirmed, and Security Information.
    2. The Personal Recovery Key is not escrowed until the device receives a SecurityInfo command and responds to MDM with the encrypted Personal Recovery Key. You can trigger a Security query from the More menu in device details. Remember, the device doesn't forcibly send the recovery key; the MDM server must request it via the SecurityInfo command.
  4. Look to see if there is anything in the fdestatus.plist:   sudo nano /var/db/ConfigurationProfiles/fdesetup.plist
    1. Look for a RecoveryKey key-value pair. You can check if the PRK is valid for the currently encrypted disk:   sudo fdesetup validaterecovery and enter the PRK (in the format AAAA-AAAA-AAAA-AAAA-AAAA-AAAA).
  5. Perform a sudo log collect --last 1h (substituting 1h for the appropriate time frame where your testing began) to output a system_logs.logarchive file which can be opened and reviewed in Console.app.

If Workspace ONE Intelligent Hub is NOT Installed:

Recovery Key Escrow still occurs without Intelligent Hub installed, as the key escrow process is a product of the built-in mdmclient and fdesetup processes. The FileVaultPRK.dat file remains as long as the version of the FileVault payload that triggered encryption remains on the device. If a change is made to the FileVault profile, macOS removes the FileVaultPRK.dat file even if the disk continues to be encrypted by the same Personal Recovery Key. You can attempt to validate the personal recovery key by performing the following commands:

  1. Check if a PRK is listed: sudo nano /var/db/ConfigurationProfiles/fdesetup.plist
  2. Check if the PRK is valid for the currently encrypted disk:   sudo fdesetup validaterecovery and enter the PRK (in the format AAAA-AAAA-AAAA-AAAA-AAAA-AAAA).

If Workspace ONE Intelligent Hub is Installed:

The Workspace ONE Intelligent Hub monitors for the presence of the FileVaultPRK.dat file when a FileVault profile is assigned. If the file is determined to be missing, Workspace ONE Intelligent Hub version 19.04 and later will prompt the user for their password and use that password to rotate the key via fdesetup. On the next SecurityInfo commmand, macOS should report the new Personal Recovery Key back to MDM for escrow. Additionally, macOS removes this file each time a change is made to the FileVault profile. As such, Workspace ONE Intelligent Hub will prompt the user for their password to ensure the Personal Recovery Key is re-escrowed.

NOTE: If an admin needs to immediately start this process to re-escrow the PRK, re-install the Intelligent Hub and the event that monitors for a missing FileVaultPRK.dat file will immediately trigger.

You can validate the PRK has been successfully rotated via Unified Logging using the command: log show  -predicate 'process = "hubd"' | grep com.vmware.hub.events

log show  -predicate 'process = "hubd"' | grep com.vmware.hub.events

<snip>    hubd: [com.vmware.hub.events:Recovery Key] Started
<snip>    hubd: [com.vmware.hub.events:Recovery Key] Processing : Proceeding with Recovery Key change. Sending notification.
<snip>    hubd: [com.vmware.hub.events:Recovery Key] Successfully Processed : Recovery Key Rotated.

6. Unified Logging - Anatomy of FileVault Enablement via MDM

If you enable the Managed Client Debug Logging, this is the approximate set of events you can expect to see (snipped for easier viewing). You can gather events in Terminal.app by entering sudo log collect --last 1h (where 1h is 1 hour). Replace the last time frame with however long was required to perform the testing. You can also dump the events using log show --predicate "process = 'mdmclient' OR process ='fdesetup'" --last 30m > logs.txt in order to get a text file for keyword searching in the text editor of your choice.

NOTE: The events shown below come from Unified Logging in macOS, not the "hub logs" collected from the UEM console. These events originate at the device which has received the disk encryption (FileVault) payload.

<snip> mdmclient	<snip> PROGRESS: Installing profile “FileVault/V_1”

<snip> mdmclient	<snip> <<<<< Completed PlugIn: InstallPayload [FileVault2Plugin] <<<<<

<snip> mdmclient	CPProfileManager.installProfile matched bundle - com.apple.security.FDERecoveryKeyEscrow

<snip> mdmclient	Installed configuration profile: FileVault/V_1 <snip> for <Computer> (Source: MDM)

<snip> fdesetup		fdesetup: output = /var/db/ConfigurationProfiles/fdesetup.plist

<snip> fdesetup		fdesetup: WriteDeferralConfigurationDictionary adding <userName> to loginwindow logout dict

<snip>	fdesetup	fdesetup:usersForVolume apfs returning: 
(
        {
        username = <userName>;
        usertypedescription = "OS User";
        uuid = "46EAA4D9-2637-403E-B4DD-797512C87E6B";
    },
        {
        usertypedescription = "Bootstrap Token";
        uuid = "2457711A-523C-4604-B75A-F48A571D5036";
    }
)

<snip>	fdesetup	checkSecureTokenUsersForEmptyPasswords record has secure token

<snip>	mdmclient	[0:MDMDaemon:<0x32e7>] ### XPC request: FDEPRKEscrow:SavePRK ### from: <fdesetup (pid: 1212; uid: 0)>

<snip>	fdesetup	fdesetup:startEncryption verifyCryptoUserPassdataForVolume returned success (0)

<snip>	fdesetup	fdesetup:startEncryption Encryption starting....  Additional users count = 0.

<snip>  mdmclient: [com.apple.ManagedClient:MDMDaemon] Processing server request: SecurityInfo for: <Device>

<snip>  mdmclient: [com.apple.ManagedClient:MDMDaemon] Read PRK escrow file length: <length>

Troubleshooting DEP Admin Password Rotation

As of Workspace ONE UEM 1912, VMware implemented support for rotating the password for the administrator account created during automated enrollment. These auto-rotated passwords adhere to the CIS MS-ISAC Best Practices. The bulk majority of this process is driven via Apple APIs for the mdmclient. The aim of this section is to show how the Admin Password Auto-Rotation process works and where to look if it doesn't seem to work as expected.

DEP Admin Password Rotation Process

The password rotation process at a high level works as follows:

  1. Workspace ONE UEM Console generates a randomized password and saves it in the Automated Enrollment (DEP) Profile when assigned to each device record at Apple. The password is also saved to the device record.
  2. When the Mac proceeds through the SetupAssistant, macOS creates the local administrator account using the profile-provided username and randomized password.
  3. Workspace ONE receives the auto-created admin user account on macOS in the DeviceInformation query, which includes the GUID for the user account.
  4. When the password is accessed in Workspace ONE, a scheduled job is created that automatically issues SetAutoAdminPasswordCommand using a newly generated password and the admin account's GUID after a period of approximately 8 hours.
  5. If necessary, Workspace ONE administrators can also manually force a password rotation via API: POST /api/mdm/devices/<DeviceID>/commands?command=RotateDEPAdminPassword

Troubleshoot Password Rotation via UEM Console

From within the Workspace ONE UEM Console, browse to the device details view and click on the Troubleshooting tab. In the Event Log, search for events such as the following:

  • ManagedAdminPassword Accessed -- Indicates an Administrator has accessed the admin password. This starts a timer for the password rotation.
  • ScheduledAdminPassword Rotation Requested -- Indicates Workspace ONE UEM has attempted to change the admin password.

Workspace ONE UEM attempts to rotate the password approximately 8 hours after the initial password access.  

Troubleshoot Password Rotation via Unified Logging

Troubleshoot macOS with unified logging.

Using Unified Logging, you can filter for events specific to the Auto Admin Password rotation via this command:

log show --predicate '(process == "mdmclient") && (eventMessage contains "SetAutoAdminPassword")'

There may be instances where the command may not be immediately processed, which can lengthen the amount of time between initial password access and password rotation. These will show in the logs as a NotNow response to MDM, and should include a reason for the delayed change.  

Once the device has successfully changed the admin account password, you will see an Acknowledged(SetAutoAdminPassword) entry in Unified Logging.

Confirming VMware Carbon Black Sensor Install on macOS

In this activity, validate that the Carbon Black Cloud Sensor for macOS has installed successfully.

1. Validate UI for Sensor Install

Open Confer app in macOS to confirm installation of Carbon Black Cloud sensor for macOS
  1. Open Finder and click Applications.
  2. Ensure that the VMware Carbon Black Cloud folder is present and contains the Sensor app and bundles.
  3. You might also see the Confer menulet in the menu bar.

2. Review Logging

Review log to confirm if the Carbon Black Cloud sensor for macOS was installed
  1. Open Terminal.App and enter the following command:   tail -50 -F /Library/Application\ Support/AirWatch/Data/Munki/Managed\ Installs/Logs/ManagedSoftwareUpdate.log
  2. Review the log for a note stating that the Install of CbDefense was successful.

If the sensor appears to not be installing, or is installing repeatedly, you may need to adjust the metadata PLIST to include an installs array (as covered in Make Modifications to the PLIST).

3. Review RepCLI Output

  1. Open Terminal.app and enter the following command:   cd /Applications/VMware\ Carbon\ Black\ Cloud/repcli.bundle/Contents/MacOS
  2. Enter the following command (and the administrative password if prompted):  sudo ./repcli status
  3. Observe the values for System Extension Status, Sensor State, and Cloud Registration Status.

If the System Extensions are not loading, ensure that you have staged the correct profile payloads as covered in macOS Prerequisites for Deploying Carbon Black Cloud Sensor.

If the Kernel Extensions are not loading in macOS Big Sur, you might need to rebuild the kernel cache as shown in the next step.

4. Rebuild Kernel Cache (If Necessary)

If you have installed the Carbon Black Cloud Sensor for macOS in KEXT mode and the KEXTs are not loading, you can try to load them by Rebuilding the Kernel Extension Cache.  

  1. Click Devices.
  2. Click List View.
  3. Select the device that needs the kernel cache rebuilt.
  4. Click More Actions.
  5. Click Custom Command.
  6. Paste the command XML from the following example, making sure to add the full list of KextPaths into the array, or remove the key and the array of values.
  7. Click Send.

Note: If you send the KextPaths key, you must include the Carbon Black KEXT path, as well as any other paths you want to include in the Kernel Cache Rebuild. If you do not specify the KextPaths key, macOS attempts to rebuild the cache with any known kernel extensions (for example, from Apps that have been launched and attempted to load a KEXT).

CUSTOM COMMAND TO REBUILD THE KERNEL CACHE WITH A SPECIFIC LIST OF KEXTS:


<dict>
	<key>RebuildKernelCache</key>
	<true/>
	<key>KextPaths</key>
	<array>
		<string>/Library/Extensions/CbDefenseSensor.kext</string>
		<string>/Library/Extensions/SomeOtherExtension.kext</string>
	</array>
	<key>RequestType</key>
	<string>RestartDevice</string>
</dict>
CUSTOM COMMAND TO REBUILD THE KERNEL CACHE WITH CURRENTLY KNOWN KEXTS:


<dict>
	<key>RebuildKernelCache</key>
	<true/>
	<key>RequestType</key>
	<string>RestartDevice</string>
</dict>

Troubleshooting OS Updates

Apple allows OS Update installation by the user and by automation (through MDM and via command line using softwareupdate).   This section illustrates how to troubleshoot OS updates that may not be applying or may be applying in a way that generates negative end-user impact.

1. General Log Streaming

When troubleshooting OS Updates, there may be multiple subsystems at work.   As such, you may want to start with the following general command line:

log show --info --debug --predicate 'subsystem BEGINSWITH "com.vmware.hub" OR process == "mdmclient" or process == "softwareupdate"' --last 30m

This command will get you started down the path as follows:

  • mdmclient logging will show you if any commands were received that triggered the update via MDM
  • softwareupdate logging will show you if the softwareupdate process was triggered (by user or automation)
  • com.vmware.hub logging shows you what activities the hub is doing (and if hub is triggering the softwareupdate command)

Additionally, if you're troubleshooting an issue where updates are not applying, check the OS Installer isn't restricted from running.   For more information on process blocking, see Troubleshooting App and Process Blocking.

Summary and Additional Resources

Conclusion

This operational tutorial provided general troubleshooting guidance and solutions to specific problems for various macOS features in Workspace ONE UEM, including macOS log collections and management.

Additional Resources

For more information about Workspace ONE, explore the VMware Workspace ONE Activity Path. The activity path provides step-by-step guidance to help you level-up in your Workspace ONE knowledge. You will find everything from beginner to advanced curated assets in the form of articles, videos, and labs.

Additionally, you can check out the VMware Workspace ONE and VMware Horizon Reference Architecture which provides a framework and guidance for architecting an integrated digital workspace using VMware Workspace ONE and VMware Horizon. 

For more information on macOS, see Understanding macOS Management

Change Log

The following updates were made to this guide:

Date Change
2021-06
  • Added "Devices Enrolled to Wrong Organization Group” section within “Troubleshooting macOS Enrollment"
  • Added “Troubleshooting Post-Enrollment Onboarding Experience” section
  • Rename “Confirming Sensor Install on macOS” section to “Confirming VMware Carbon Black Sensor Install on macOS” for clarity
2021-03
  • Fixed error in “Understanding Unified Logging” section and added information on “&&” (and) and “||” (or) filtering
  • Added updates in “Troubleshooting SSO Extensions” section
  • Added updates to “Troubleshooting Per-App Tunnel on macOS” section
  • Added updates to “Troubleshooting Intelligent Hub Processes” section
  • Added “Troubleshooting FileVault Encryption” (including information on SecureToken and Bootstrap Token)
  • Added “Troubleshooting DEP Admin Password Rotation” section
2020-10 Initial Posting

About the Author

This tutorial was written by:

  • Robert Terakedis, Senior Technical Marketing Manager, End-User-Computing Technical Marketing, VMware

Feedback

The purpose of this tutorial is to assist you. Your feedback is valuable. To comment on this tutorial, contact VMware End-User-Computing Technical Marketing at euc_tech_content_feedback@vmware.com.

Associated Content

home-carousel-icon From the action bar MORE button.

Filter Tags

Workspace ONE Workspace ONE UEM Document Operational Tutorial Intermediate macOS Manage