Troubleshooting macOS Management: Workspace ONE Operational Tutorial

Workspace ONE uses multiple components to create the entire macOS management stack. Some of these components are supplied by Apple, whereas others are value-added functionality included with your Workspace ONE licensing. With regards to components installed and running on macOS, review the following table:

macOS management in Workspace ONE makes use of numerous server/cloud-based components. It is critical to ensure that macOS has network connectivity to each of these components, as specified in the following lists of network requirements:

• Using Apple Products on Enterprise Networks
• If your Apple devices are not getting Apple push notifications
• Workspace ONE ports and DNS names 

Remember, many of these network requirements point to DNS names, which are part of global load balancing systems. It is highly recommended that you ensure firewall rules match the network requirements exactly. Do not substitute IP addresses where DNS names are specified because this can cause troubleshooting issues at a later stage as the load-balanced services move to different IP addresses.

In Workspace ONE, the entire possibility of macOS management takes place in multiple systems. Some systems include:

• APNS (Apple Push Notification service) and AWCM (AirWatch Cloud Messaging) for realtime device notifications.
• Workspace ONE UEM (which includes the console, device services, and API servers) for device management.
• Workspace ONE Assist for remote control (for example, helpdesk support).
• Workspace ONE Access and Hub Services for employee experience, single sign-on, and conditional access.
• Workspace ONE Intelligence for reporting and automation (including partner ecosystems).
• Carbon Black for endpoint security monitoring, quarantine, and remediation.
• Horizon for enabling the "Mac App Gap" or Windows-based applications which need to be accessed and used by macOS users.

Clients communicate to Workspace ONE UEM on behalf of the device. The following components are the primary list of clients you must manage on a device as you adopt the entire solution stack:

• mdmclient - The built-in device management client which is part of macOS.
• Workspace ONE Intelligent Hub - The agent which communicates to Workspace ONE UEM and Hub Services to augment device management value-added functionality, the upcoming system extension for process blocking, the unified catalog, and employee experience features.
• Assist Client - The client which facilitates remote troubleshooting sessions (remote control, remote file, and remote terminal).
• Carbon Black Sensor - The client which provides next-generation anti-virus and anti-malware protection.
• Horizon Client - The client which provides access to remote datacenter-hosted applications from macOS.

Network connectivity is a common issue and should be verified. Mac management with Workspace ONE requires network connectivity to a number of endpoints at a number of vendors:  Apple, VMware, Akamai, and more (depending on your particular situation). Managing a table of these endpoints in a cheat sheet would be unruly and difficult to continually manage, so instead, we have included pointers to the full list of required DNS and port names:

• Managing Apple Devices on Enterprise Networks 
• If your device does not have connectivity to APNS 
• Workspace ONE DNS and Port Requirements

Use the following links to quickly verify if there are any known, reported outages at Apple or VMware:

• System Status  
• Developer System Status  
• Workspace ONE Status 
• Workspace ONE Intelligence Status  
• Workspace ONE Access  

Managing macOS requires regular maintenance, just as expected with other platforms. The following list highlights the most common configuration issues that can arise when managing devices.  

1. If you have trouble deploying newly purchased volume licenses to machines, or devices are not proceeding through Automated Enrollment, check the following:
   1. Has Apple released new Terms of Use that must be accepted by an administrator in Apple Business Manager (or Apple School Manager)?
      See Updated Terms and Conditions for Apple Business Manager.
   2. Has the sToken for the Apple Business Manager location expired?
      Navigate to Groups & Settings > Configurations > VPP Managed Distribution and check the Valid Until date and time. If expired, click the Renew button and follow the process.
2. If devices are not checking in for commands in a reasonable amount of time:
   1. Check the APNS Certificate Expiration Date: Navigate to Groups & Settings > Configurations > APNS for MDM and check the Valid To date. If expired, you must log in to the Apple Push Certificates portal and Renew the existing certificate.

      Caution: If you replace the certificate instead of renewing it, Workspace ONE will lose connectivity with your devices and you will be required to re-enroll those devices.
       

   2. Check the device-side APNS status - open Terminal.app and enter the following command: /System/Library/PrivateFrameworks/ApplePushService.framework/apsctl status
   3. Check for AWCM connectivity.  
      1. Navigate to Groups & Settings > All Settings > System > Advanced > Site URLs. Scroll down to find the AWCM Server External Hostname and then append /awcm/status to the end.
         For example, https://awcm1380.awmdm.com/awcm/status. AWCM should return OK.    
      2. In the Device Details view, ensure that the specific device in question shows AWCM Connected.

Workspace ONE administrators can request the Intelligent Hub to gather historical logging and deliver those logs to the Workspace ONE UEM Console:

1. On the Device details page, click More Actions.
2. Click Request Device Log.

From within Terminal.app, enter the following command line:  sudo hubcli logs --send --duration 1h

This command tells hub to send diagnostic logs to the UEM console for the past 1 hour.

End users can also generate and send logs to the Workspace ONE UEM console by using the Collect & Send Logs feature.  

Launch the Intelligent Hub:

1. Click Help
2. Optionally, click Debug Session > Start Session to start a debug session (verbose logging). Clicking the same path allows the user to End the debugging session prior to the 30 minute default session time.
3. Click Collect & Send Logs.

After requesting logs from the device, you can view the logs as follows:

1. Click Devices.
2. Click the device to enter the Details View.
3. Select Attachments in the drop-down menu.
4. Select Documents.
5. Click the Hub_Complete_Report you want to download and view.

If you are troubleshooting an issue with Internal Apps for macOS, you can easily view the logging for that in real-time on your test device (or via remote command line through Workspace ONE Assist). The following command tails the last 25 lines in the ManagedSoftwareUpdate.log file:

tail -n 25 -F /Library/Application\ Support/AirWatch/Data/Munki/Managed\ Installs/Logs/ManagedSoftwareUpdate.log

Following are some tips and tricks that can save you time:  

• The log command requires "straight" quotes and not "curly" quotes. This issue can crop up during copy/paste operations.
• You can gain additional information by adding the --info and --debug parameters to your log command line.
• Add --archive system_logs.logarchive to search inside a SysDiagnose logarchive file.
  • Generate a sysdiagnose from the computer in question:   sudo sysdiagnose -f -n ~/Downloads
• For more in-depth detail, you can install some of the developer debugging profiles from https://developer.apple.com/bug-reporting/profiles-and-logs/
• Predicates support "and" (&&) and "or" (||) operators as long as you include parentheses to wrap each filter. For example, --predicate '(process == "mdmclient") && (eventMessage contains "SetAutoAdminPassword")'

Important: Many details in logging commands are hidden for privacy. Most likely, you will not need to enable this. However, if you intend to enable private data logging, you must send a custom settings profile with the following content (or manually fit this content into a mobileconfig file and install it):

Note the following text on Predicates, clipped from the manual for the log command. Predicates are the building blocks for filtering the Unified Log.  

As you begin to troubleshoot an issue, you can start logging time markers directly into the unified log by using the logger command. You can then later search for these markers by using the log command. The following represents a way to bookmark the start and end time for troubleshooting activities:

• Log a marker in Unified Logging for troubleshooting events: logger "[WS1-Support] Starting <activity>" and logger "[WS1-Support] Ending <activity>"
• Search for all markers to determine troubleshooting time frames: log show --predicate 'process=="logger" AND eventMessage CONTAINS "[WS1-Support]"'

With the list of times for troubleshooting activities output by the log command, you can later filter for events constrained to those time frames by using the --start "YYYY-MM-DD HH:MM:SS" --end "YYYY-MM-DD HH:MM:SS" parameters on your log commands. Remember that the hours (HH) are in 24-hour format, and displayed in the machine's configured time zone.

At times, you may be troubleshooting unexpected system restarts and kernel panics. In this instance, you may not know exactly when the system restarted in order to work your way backwards through the logs. In this particular case, you can search the unified log for system start messages in order to find timestamps of where to end your logging.

• Find the system boot times using grep: log show --start "YYYY-MM-DD HH:MM:SS" | grep "=== system boot"
• Use the system boot time as the "end" parameter in any log show or log collect command: --end "YYYY-MM-DD HH:MM:SS"

As mentioned above, you can filter for events constrained to specific time frames by using the --start "YYYY-MM-DD HH:MM:SS" and  --end "YYYY-MM-DD HH:MM:SS" parameters on your log commands. Remember that the hours (HH) are in 24-hour format, and displayed in the machine's configured time zone.

• Find any Logging Markers for the past 12 hours in a SysDiagnose: log show --archive ~/Downloads/sysdiagnose_2020.10.06_00-55-25-0400_Mac-OS-X_iMacPro1-1_19G73/system_logs.logarchive  --predicate 'process=="logger" AND eventMessage CONTAINS "[WS1-Support]"' --last 12h
• Filter for Events in Specific Time Slots (Based On Your Markers): log show --archive ~/Downloads/sysdiagnose_2020.10.06_00-55-25-0400_Mac-OS-X_iMacPro1-1_19G73/system_logs.logarchive  --predicate 'process == "mdmclient" OR process == "apsd" OR process == "dasd" OR process == "storedownloadd" OR process == "appstored" OR process == "logger"' --start "2020-10-06 00:47:00" --end "2020-10-06 00:48:49"

In Workspace ONE UEM, navigate to Groups & Settings > All Settings > Devices & Users > General > Enrollment > Restrictions and ensure that no device restrictions are in place that might be preventing enrollment.

In Workspace ONE UEM, navigate to Groups & Settings > All Settings > Devices & Users > General > Enrollment > Authentication and check the Devices Enrollment Mode option. If set to Registered Devices Only, then the device must be registered in the console before it can be enrolled.

Note: macOS devices synced from Apple Business Manager (or Apple School Manager) are considered Registered and can be enrolled.

Apple Business Manager devices that have already been enrolled cannot re-enroll without first deleting the device record in Workspace ONE UEM.  

When macOS enrolls to Workspace ONE UEM, numerous factors control the specific organization group where the device is placed.   Before the Organization Group for the device is finalized, Workspace ONE checks multiple attributes for both the DEP profile and the User account enrolling to best determine where the device should go.  The following attributes are considered during Organization Group selection:

• User's "Enrollment Organization Group" -- Found in the User Account properties under the General tab
• User's "Managed By" Organization Group -- The level at which this user is integrated into the console
• The "Device Organization Group" associated with the Automated Device Enrollment (or "DEP") Profile
• "User Group Mapping" setting in the console Groups & Settings > All Settings > Devices & Users > General > Enrollment > Grouping
• "Batch Import" for device Organization Group Override following Automated Device Enrollment (or "DEP").  Note: This method is not typically advised.

For devices enrolling with Automated Device Enrollment (or "DEP") via Apple Business Manager:

1. ONLY if the Automated Device Enrollment (or "DEP") Profile specifies "Authentication = OFF" and staging is configured, the device will enroll to the Staging User's "Enrollment Organization Group" and NOT the Automated Device Enrollment (or "DEP") Profile's "Device Organization Group"
2. If "User Group Mapping" is configured, the device will enroll to the organization group based on the preconfigured AD user group and organization group mapping.  In this scenario, the Automated Device Enrollment (or "DEP") Profile's "Device Organization Group" and the User's "Enrollment Organization Group" are not considered.
3. Without User Group Mapping, if the Automated Device Enrollment (or "DEP") Profile's "Device Organization Group" and the User's "Enrollment Organization Group" are in the same branch (parent/child), the device enrolls to the lower (child) Organization Group.
4. Without User Group Mapping, if the Automated Device Enrollment (or "DEP") Profile's "Device Organization Group" and the User's "Enrollment Organization Group" are siblings, then the device enrolls to the Automated Device Enrollment (or "DEP") Profile's "Device Organization Group".

For more information (and a flowchart), refer to VMware KB 83132 - Organization Group Assignment In Workspace ONE for Automated Device Enrollment (ADE) Devices.

When attempting to enroll a macOS virtual machine, you may notice that the enrollment fails and/or the enrollment profile generated appears to be for iOS. This is because the virtual machine must emulate physical hardware attributes in order for Workspace ONE to generate the proper enrollment profile.  

To get the necessary hardware attributes, you should run the following commands on the hardware you want to emulate:

• Hardware Model:  ioreg -l | awk '/product-name/ { split($0, line, "\""); printf("%s\n", line[4]); }'
• Serial Number:  ioreg -l | awk '/IOPlatformSerialNumber/ { split($0, line, "\""); printf("%s\n", line[4]); }'
• Board-ID:  ioreg -lp IOService | awk '/board-id/ { split($0, line, "\""); printf("%s\n", line[4]); }'

Using VMware Fusion, you should enter or modify the following items in the VMX file for your virtual machine:

Note: If the VM does not boot, you might have duplicated one of the options (typically the smbios.reflectHost).

From within Terminal.app, you can enter the following commands to get date and time zone information quickly:

• Date:  date
• Time Zone:  sudo systemsetup -gettimezone

If the date/time/zone is incorrectly set, then you will potentially have problems with certificate checking, trust, and more.

One method of troubleshooting during automated enrollment is to obtain console and shell access during the Setup Assistant. Although your options are limited due to the current user context, it is still useful to know these commands:

• Press Command + Option + Control + T during Setup Assistant to open Terminal.
• Press Command + Option + Control + C during Setup Assistant to open Console.

If you are not seeing the expected behavior during automated enrollment, you can check the settings currently assigned to the device and optionally re-assign a new automated enrollment profile. See the following:

• Check if enrolled via Automated Enrollment:  /usr/bin/profiles status -type enrollment
• Check the current settings applied via Automated Enrollment:  sudo /usr/bin/profiles show --type enrollment
• Download a new Automated Enrollment profile:  sudo /usr/bin/profiles renew -type enrollment

macOS is inherently a multi-user operating system. However, the mdmclient processes that provide the foundational management capabilities for MDM are not. When macOS is not staged properly for multiple users, you might notice that one user will get BOTH device and user profiles, whereas subsequent users are not delivered any user profiles from Workspace ONE. This behavior is a function of the mdmclient built-in to macOS and can be altered only by a specific set of configurations.

To enable multi-user functionality for macOS and Workspace ONE, the device should be enrolled using multi-user staging and then bound to Active Directory.  

For more details, review the staging configuration & enrollment process: Tech Zone Onboarding Options for macOS Tutorial.

If all else fails, start collecting log information using the command line. The following command extracts relevant detail from the Unified Log:

• log stream --predicate 'process == "mdmclient" OR process == "apsd" OR process == "dasd"'

Additionally, if you gather a sysdiagnose file (as outlined in Understanding macOS Unified Logging), you can filter using the --archive and --last parameters to narrow down the issue.

Post-Enrollment Onboarding can only be enabled in production environments running Workspace ONE UEM 2105 or later.  Validate that the Post-Enrollment onboarding screen is enabled and configured at Groups & Settings > All Settings > Devices & Users > General > Enrollment > Optional Prompt.

In the Workspace ONE UEM Console, browse to the Device Details view, and click Troubleshooting > Event Log. Look for Agent Settings and Enrollment Settings events.

From within Intelligent Hub Logs (or via Unified Logging), search for the following

• Subsystem: “com.vmware.hub.hubservices” and Category: “postEnrolmentOnboardingFlow” and “enrollment”
• Subsystem: “com.vmware.hub.uem” and Category: “AgentSettings”

Alternatively, you can search these events in Terminal with the log command as follows: log show --info --debug --predicate '((subsystem == "com.vmware.hub.hubservices") && (category IN { "postEnrolmentOnboardingFlow", "enrollment" })) || ((subsystem == "com.vmware.hub.uem") && (category == "AgentSettings"))' --last 10m

Note: You can alter the --last  parameter to adjust for your troubleshooting timeframe. For more details, see Understanding macOS Unified Logging.

This is most likely not a profile issue, but rather an enrollment issue. Generally, this behavior indicates that a device was improperly staged—check the staging configuration & enrollment process: Tech Zone Onboarding Options for macOS Tutorial.

Note: Multi-user macOS management requires multi-user staging and Active Directory binding.

If there are no profiles (user or device) in your environment, verify the following.

1. Check that the device gets push notifications by attempting to send one via the console.
2. Check that the APNs certificate is still valid as described in Validating Workspace ONE UEM Console Settings.
3. Check Network Connectivity to APNS and AWCM
   • Use Apple Products on Enterprise Networks 
   • Workspace ONE Ports Requirement 
4. Check the Event Log by browsing to Devices > List View > Select the Device (switch to Device Details View) > More > Troubleshooting and review the Event Log and Command Queue.
   Reminder: The content of these views can be filtered.
5. Check Logging (using Terminal or a SysDiagnose file) as follows:
   1. sysdiagnose:   sudo sysdiagnose -f -n ~/Downloads
   2. Log Streaming:   log stream --predicate 'process == "mdmclient" OR process == "apsd" OR process == "dasd"'

To troubleshoot the Privacy Preferences (or TCC or PPPC) behavior on macOS, you must stream all events related to Privacy Preference Policy Control prompts:

• log stream --debug --predicate 'subsystem == "com.apple.TCC" AND eventMessage BEGINSWITH "AttributionChain"'
• log stream --debug --predicate 'subsystem == "com.apple.TCC" AND eventMessage BEGINSWITH "AttributionChain" OR eventMessage CONTAINS "synchronous to com.apple.tccd.system"'

You can also read the TCC Databases using Terminal.app. Before attempting to manually review these files, ensure that Terminal.app is granted full control by MDM or manually in System Preferences. After granting permissions, run the following commands:

• echo ".dump" | sudo sqlite3 /Library/Application\ Support/com.apple.TCC/TCC.db
• echo ".dump" | sudo sqlite3 ~/Library/Application\ Support/com.apple.TCC/TCC.db

A bootstrap package is a lightweight distribution package created by a Workspace ONE administrator. The intent of a bootstrap package is to silently deliver and install a small baseline set of agents to onboard macOS for a user. For example, an administrator can use the bootstrap package to install a distribution package containing the chef, puppet, or saltstack agents.    

Note: Bootstrap Packages install through the InstallEnterpriseApplication command during the Await Configuration phase of device enrollment. Because of this, there is no confirmation of a successful install, other than to audit the ApplicationList sample that the device returns later on.  

The following outlines some potential fixes for Bootstrap packages:

1. Is the package fairly small in size?
   It is best practice is to keep the bootstrap package as small as possible so that it installs quickly.
2. Is the bootstrap package a signed Distribution type package?
   1. The package must be signed with an Apple Developer ID Installer Certificate. Only the package needs to be signed, not the app because the Apple Gatekeeper does not check apps installed through MDM.
   2. The package must be a distribution package (product archive), not a flat component package.
3. Is Workspace ONE configured to attempt installing the Intelligent Hub also?
   Navigate to Devices > Devices Settings > Apple > Apple macOS > Intelligent Hub Settings. Some older versions of macOS could not install two bootstrap packages simultaneously—In this situation, all bootstrap packages would fail to install on the device. If the Install Hub After Enrollment switch is set to Enabled, try setting it to Disabled.    

This error is typically the result of one of the following issues:

• Delays in Apple Business Manager from when you purchase the app to when the licenses are allocated to the Location Token.
• Apple has released new Terms & Conditions in Apple Business Manager. 
  • To resume syncing, log in to Apple Business Manager with an account granted the Administrator role, and accept the updated Terms & Conditions.
• The location token downloaded from Apple Business Manager and uploaded to Workspace ONE UEM has expired. 
  • Download a new server token from Apple Business Manager (under Settings > Apps & Books) and upload it to Workspace ONE UEM.

If you have assigned an app to a device using device-based assignment, one of the following could be an issue:

• If the user is getting a prompt to log in with an Apple ID, there is most likely an assignment that was made to the user before the device-based assignment was created. 
  • Unassign the app from the user completely before you attempt to reassign the app to the device.
• Ensure that an adequate number of licenses are available to assign to the device.
• Ensure that the device has access to vpp.itunes.apple.com.
• The location token downloaded from Apple Business Manager and uploaded to Workspace ONE UEM has expired. 
  • Download a new location token for that same location from Apple Business Manager (under Settings > Apps & Books) and upload it to Workspace ONE UEM.

When a VPP app for macOS is no longer scoped to the device or user, or the device is enterprise wiped, the app is not removed from macOS. This is by design in macOS versions prior to macOS 11 (Big Sur), as these earlier versions of macOS did not support any commands to remove the application.

If you are still experiencing issues with volume-purchased applications, refer to the Volume Purchase Program (VPP) Troubleshooting Guide or contact VMware support directly.

Non-store macOS apps require the following to work correctly:

• Supported version of macOS Intelligent Hub installed on target devices
• Non-Store Software Management features enabled
• AWCM (AirWatch Cloud Messaging) services installed and working

Note: AWCM provides notifications to the Intelligent Hub to trigger real-time non-store app installation. In the absence of AWCM, the Intelligent Hub reverts to a scheduled interval to check for new app installation commands.

Whether entered in the Scripts tab in the console or entered manually in the PLIST file, the script should exit with a zero (0) return code to trigger an install. The script should include a shebang statement (such as #!/bin/zsh) at the beginning.

This key-value pair in the PLIST file specifies identifying information about a binary or file which should be directly compared to determine if the correct version of an app is installed. The installs list can contain any number of items. These can be applications, Preference Panes, Frameworks, or other bundle-style items, Info.plists, or simple directories or files. For more information, see How Munki Decides What Needs To Be Installed.

When a package is installed, the installer leaves a receipt and bill of materials file on the machine. Some packages parsed by the Workspace ONE Admin Assistant will include detail on what receipts will be dropped by the installer in the PLIST file. When a PLIST file lacks the install check script or installs array, the lack of specified receipt(s) triggers an install. For more information, see How Munki Decides What Needs To Be Installed.

Sometimes an installer package parsed by the VMware Workspace ONE Admin Assistant generates a PLIST file where the application name is incorrect. It is also possible that organizations refer to software by a common or colloquial name that is easily recognized by end users. In either case, administrators can change the name displayed in the Intelligent Hub application catalog before uploading the PLIST to Workspace ONE UEM.

Sometimes an installer package parsed by the VMware Workspace ONE Admin Assistant generates a PLIST file where the version shows Please Edit Me. This problem must be fixed before uploading the PLIST to Workspace ONE UEM.

In some instances, an application successfully installs but the Intelligent Hub continually reports the app as "Installing". If you look in the ManagedSoftwareUpdate.log file (see Gathering Logs and Validating macOS App Installation), you'll see the app is constantly marked for installation each time the Hub checks for installed software. This is typically the result of a metadata PLIST that doesn't contain the correct receipt or installs arrays. In this instance, you must make one of the following changes to the metadata PLIST generated by Workspace ONE Admin Assistant:

1. Validate or Add an Installs Array as discussed in Determining When to Install Non-Store Apps.
2. Add an Install Check Script (and ensure it returns a zero (0) value return code when an install should proceed).

1. Click the Launchpad on the Dock.
2. Click VMware Tunnel.

1. Ensure that the Device Configured status shows Configured. This indicates that Workspace ONE Tunnel has received configuration data from Workspace ONE UEM. If the status is not configured, try one of the following:
   • Check the Device Traffic Rules and Save and Publish the rules again.
   • Check the last seen value for the device in the Workspace ONE UEM console. Is the device communicating with Workspace ONE UEM?
   • Validate that other MDM commands are being sent to the device. Create an assignment (smart) group containing the single device and attempt to send it a new profile payload.
2. Ensure that the Internet status shows Connected. If Tunnel cannot connect to the Internet, it probably cannot connect to the Unified Access Gateway.
   • Validate that the device has a working Ethernet or Wi-Fi connection (IP address, subnet mask, gateway, and DNS addresses are present).
   • Validate DNS resolution: Open Terminal and enter nslookup uag.fully.qualified.domain to ensure that an IP address is resolved.
   • Validate Connectivity to UAG: Within Terminal, enter nc -vz uag.fully.qualified.domain uagport (such as nc -vz uag.company.com 443).
3. Ensure that the Enterprise Network status shows Connected. If Workspace ONE Tunnel is disconnected from the Enterprise network, apps cannot use Per-App Tunnel. This might indicate an issue with Workspace ONE Tunnel connecting to the Unified Access Gateway or an issue with Device Traffic Rules.
   • The remainder of this section details how to troubleshoot Tunnel connectivity.

1. Click System Preferences.
2. Double-click Profiles.
3. Scroll through the left panel.
4. Click the Per-App VPN profile that was created.
5. Ensure that the VPN App Layer Service details are correct, especially the VPN Remote Address and the OnDemand Enabled value.
   • If the profile is missing or misconfigured, check the profile configuration and re-push the profile to the device from within the UEM Console Device Details view (on the Profiles tab).

1. Open the Workspace ONE Tunnel client and click the VMware Tunnel menu.
2. Click Whitelisted Applications.
3. Verify that the list of allowlisted applications matches the settings configured in the Device Traffic Rules.
4. From the VMware Tunnel menu (#1), click Diagnostics.
5. Click Enable Debug to get verbose information.
6. Review Diagnostics information.
7. Click Disable Debug when troubleshooting is complete.

1. Press CMD+SpaceBar (⌘+Space) and enter console into the Finder window.
2. Select the Console application.
3. Enter process:macOSAppProxyProvider into the search bar and press Return on the keyboard.
4. Without clearing the contents of the search bar, add an additional filter parameter by adding process:VMware Tunnel into the search bar and press Return on the keyboard.
5. Click the Action menu and confirm that Include Info Messages and Include Debug Messages are selected.
6. Review the logging produced within the Console application.

Tip: If the console filters do not provide any meaningful data, you can optionally attempt to view information and debug messages from entire subsystems. Some filters that may help include:

• process:macOSAppProxyProvider
• any:com.vmware.macos-tunnel

Also, if troubleshooting Kerberos over the Per-App Tunnel, you can include the following console filters:

• subsystem:com.apple.appsso
• subsystem:com.apple.appssokerberosextension

The following Terminal command might provide meaningful output:   log stream --debug --predicate '(subsystem == "com.apple.Heimdal") OR (subsystem == "com.apple.AppSSO") OR (subsystem == "org.h5l.gss") OR (subsystem == "com.apple.network") OR (process == "VMware Tunnel") '

Per Apple's Developer Website (requires login), you can use the following commands to gather additional data from the VPN (Network Extension):

• sudo defaults write /Library/Preferences/com.apple.networkextension.control.plist  LogToFile -boolean true
• sudo defaults write /Library/Preferences/com.apple.networkextension.control.plist  LogLevel -int 7

Reproduce the issue and then enter this command in Terminal.app:

• /System/Library/Frameworks/SystemConfiguration.framework/Resources/get-mobility-info

You should find additional information in the resulting get-mobility-info output file.

You can later deactivate the logging by issuing the following commands:

• sudo defaults delete /Library/Preferences/com.apple.networkextension.control.plist  LogToFile
• sudo defaults delete /Library/Preferences/com.apple.networkextension.control.plist  LogLevel

1. Enable the necessary debug modes by running the following in Terminal.app:  
   1. $ sudo log config --mode "level:debug,persist:debug" --subsystem com.apple.AppSSO
   2. $ sudo log config --mode "level:debug,persist:debug" --subsystem com.apple.Heimdal
   3. $ sudo log config --mode "level:debug,persist:debug" --subsystem org.h5l.gss
2. Restart the SSO Extension subsystems:
   pkill -9 KerberosExtension AppSSOAgent KerberosMenuExtra
3. Reproduce the issue and generate a sysdiagnose (sudo sysdiagnose)

NOTE: When finished troubleshooting and gathering debug logs, reset the logging to normal levels by running the following commands:

• $ sudo log config --subsystem com.apple.AppSSO --reset
• $ sudo log config --subsystem com.apple.Heimdal --reset
• $ sudo log config --subsystem org.h5l.gss --reset

You can also enable debug logging via a Custom Settings (XML) profile as follows:

As you troubleshoot the SSO extensions, the following command line will stream all Events related to the Kerberos SSO Extension and additional Apple-built SSO Extension binaries. If you are using an SSO extension from another identity provider (such as Okta or Azure Active Directory), you must also add the appropriate predicate parameters in the following command:

• log stream --debug --predicate '(subsystem == "com.apple.Heimdal") OR (subsystem == "com.apple.AppSSO") OR (subsystem == "org.h5l.gss") OR (subsystem == "com.apple.network")'

1. Show all shared web credentials associated with any apps using AASA or SSO Extensions:
   • /System/Library/PrivateFrameworks/SharedWebCredentials.framework/Support/swcutil show
2. Determine if SSO Extension is loaded:  
   • pluginkit -vvv -m -i <SSO Extension App ID>

The following Apple documentation may prove useful in troubleshooting SSO Extensions as well:

• Verify DNS Consistency for AD Binding:   https://support.apple.com/en-us/HT201885

Check that the following pre-requisites have been made:

1. Has the user provided screen recording permissions to the client-side assist app?
2. Has the Assist client application been downloaded and installed on the Mac which needs remote assistance?
3. Does the device show as AWCM-connected in the Device Details view?

From within Terminal.app, run the following command to find out what's going on:

• log stream --predicate '(process == "tccd" AND eventMessage CONTAINS "com.aetherpal") OR process == "awcmd" OR process == "assistd" OR process == "AssistCore"'

Stream events related to Hub, awcm, Software distribution, and so on (Internal Apps, NOT VPP):

• log stream --predicate 'process == "awcmd" OR process == "apsd" OR subsystem BEGINSWITH "com.vmware.hub"'

Stream events related to Hub Process Blocking (System Extension):

• log stream --debug --predicate 'subsystem == "com.vmware.hub.security" OR process == "com.airwatch.mac.agent.EndpointSecurity" OR process == "hubd" OR process == "IntelligentHubAgent" or process == "endpointsecurityd"'

If you are troubleshooting sensors with Hub 2010, the following commands are helpful:

1. View the list of sensors downloaded to Hub:
   • sudo hubcli sensors --list
   • sudo hubcli sensors --list json
2. Note: Sensors require a client-side feature flag:
   • sudo plutil -insert MacOsSensorsCliListValueFeatureFlag -bool YES /Library/Application\ Support/AirWatch/Data/com.vmware.hub.flags.plist
3. Trigger a sensor run on-demand:
   • sudo hubcli sensors --trigger <sensor_name>

NOTE:  Sensors and scripts require the VMware Workspace ONE Workflow Engine for macOS, which is a separate installer from the Intelligent Hub.

Per Apple's Platform Security Guide:

"Apple File System (APFS) in macOS 10.13 or later changes how FileVault encryption keys are generated. In previous versions of macOS on CoreStorage volumes, the keys used in the FileVault encryption process were created when a user or organization turned on FileVault on a Mac. In macOS on APFS volumes, the keys are generated either during user creation, setting the first user’s password, or during the first login by a user of the Mac. This implementation of the encryption keys, when they’re generated, and how they’re stored are all part of a feature known as Secure Token. Specifically, a secure token is a wrapped version of a key encryption key (KEK) protected by a user’s password. In macOS 11, setting the initial password for the very first user on the Mac results in that user being granted a secure token."

Basically, think of SecureToken as an encryption key tied to a user. Some type of user-related "event" must happen (user creation, user password creation, or first login by a user) which creates the key that comprises the SecureToken.    SecureToken creation must occur before

1. FileVault can encrypt a disk, and
2. Before a device can generate a Bootstrap Token. 

Additionally, it is known that subsequent users added to the system must be created by a user with a SecureToken in order to have their own SecureToken generated. The alternative to user creation by a SecureToken-enabled user is the use of a Bootstrap Token to generate a user's SecureToken.

Interacting with SecureToken:

• Check if a User is SecureToken Enabled: sysadminctl -secureTokenStatus <username>
• List Accounts which can unlock FileVault encrypted disk: diskutil apfs listcryptousers /
• Map UUIDs of CryptoUsers with Usernames: sudo fdesetup list

Per Apple's Platform Security Guide:

"macOS 10.15 introduced a new feature—Bootstrap Token—to help with granting a secure token to both mobile accounts and the optional device enrollment-created administrator account (“managed administrator”). In macOS 11, the Bootstrap Token can grant a secure token to any user logging in to a Mac computer, including local user accounts. Using the Bootstrap Token feature of macOS 10.15 or later requires:

• Mac enrollment in MDM using Apple School Manager or Apple Business Manager, which makes the Mac supervised
• MDM vendor support

In macOS 10.15.4 or later, a Bootstrap Token is generated and escrowed to MDM on the first login by any user who is Secure Token–enabled if the MDM solution supports the feature. A bootstrap token can also be generated and escrowed to MDM using the profiles command-line tool, if needed.

In macOS 11, the bootstrap token may also be used for more than just granting secure token to user accounts. On a Mac with Apple silicon, the bootstrap token, if available, can be used to authorize the installation of both kernel extensions and software updates when managed using MDM."

NOTE: Workspace ONE UEM introduced bootstrap token support in version 2008 for Catalina, and 2011 for Big Sur.

There's no way around it - understanding FileVault can be tough for administrators new to macOS management.  Hopefully this summary makes using secure and bootstrap tokens easier:

1. The First User to log on (for example, "First User") to macOS gets a SecureToken.
2. SecureToken enables a user account to unlock a FileVault encrypted disk.
3. Users created by the First User are granted their own SecureToken. In other words, SecureToken is passed from one trusted account to another.
4. MDM informs macOS of its Bootstrap Token capability, and macOS sends a Bootstrap Token derived from the First User's SecureToken after the First User logs in.
5. MDM can use the Bootstrap Token to grant a SecureToken to user accounts NOT created by the First User.

Bootstrap Token is a straightforward process for troubleshooting. If the user is a standard user (non-admin), you need to use su <hiddenITadminaccount> to change to a user that can run the following commands from within Terminal.app.

• Check if Workspace ONE already sets the MDM Option for BootstrapTokenAllowed: In the device details view, check the Troubleshooting tab and search for bootstrap.   You should see an event where we set the MDM Option and another event where Bootstrap Token is escrowed.
  • This MDM Option should trigger macOS to generate a Bootstrap Token when possible.
• Check if Bootstrap Token is supported from the device: sudo profiles status -type bootstraptoken
  • The first line indicates that UEM supports Bootstrap Token and the second line indicates if it has already been escrowed or not.
  • If Bootstrap Token supported on server is NO, but UEM is a version supporting Bootstrap Token, then the device did not receive the MDM Option.
  • If Bootstrap Token escrowed to server is NO but supported on server is YES,  check that there is at least one account with SecureToken: diskutil apfs listcryptousers / and optionally sudo fdesetup list  
  • NOTE: Bootstrap Token cannot be escrowed to server if none of the user accounts present on the system have SecureToken.
• Manually Generate and Escrow Bootstrap Token: sudo profiles install -type bootstraptoken
  • The command is interactive and requires the Admin username and password.
  • Re-validate on the UEM console that the Bootstrap Token is escrowed.
• Validate the BootStrap Token for macOS: sudo profiles validate -type bootstraptoken

macOS ONLY escrows the recovery key when instructed to do so, and ONLY for personal recovery keys. The way this process worked differs depending on the version of macOS. This guide covers the escrow process for macOS 10.13 and later.

FileVault Recovery Key escrow is initiated by the com.apple.security.FDERecoveryKeyEscrow payload in a profile.  

NOTE: If a previous-style payload (com.apple.security.FDERecoveryRedirect) is delivered to macOS 10.13 and later, it is ignored. Also, if FileVault was already enabled and escrowed with the old payload, no warning or error will be shown.  

Troubleshooting Steps:

1. Confirm the FDERecoveryKeyEscrow payload landed before FileVault was enabled.
   1. Remember, if FileVault was already enabled and escrowed with the old payload, no warning or error will be shown.
2. Ensure there is no more than one FileVault payload delivered to the device.
   1. Was the device upgraded to macOS 10.13 or later and potentially already had a previous-style payload? It may have been removed after upgrade, so look at the UEM Console to determine if a previous-style payload was sent pre-upgrade.
3. Look to see if the /var/db/FileVaultPRK.dat file was created.
   1. If the FileVaultPRK.dat file exists, but the Personal Recovery Key is not shown in the console, check the device details troubleshooting tab for three events: Security Information Requested,  Security Information Confirmed, and Security Information.
   2. The Personal Recovery Key is not escrowed until the device receives a SecurityInfo command and responds to MDM with the encrypted Personal Recovery Key. You can trigger a Security query from the More menu in device details. Remember, the device doesn't forcibly send the recovery key; the MDM server must request it via the SecurityInfo command.
4. Look to see if there is anything in the fdestatus.plist:   sudo nano /var/db/ConfigurationProfiles/fdesetup.plist
   1. Look for a RecoveryKey key-value pair. You can check if the PRK is valid for the currently encrypted disk:   sudo fdesetup validaterecovery and enter the PRK (in the format AAAA-AAAA-AAAA-AAAA-AAAA-AAAA).
5. Perform a sudo log collect --last 1h (substituting 1h for the appropriate time frame where your testing began) to output a system_logs.logarchive file which can be opened and reviewed in Console.app.

If Workspace ONE Intelligent Hub is NOT Installed:

Recovery Key Escrow still occurs without Intelligent Hub installed, as the key escrow process is a product of the built-in mdmclient and fdesetup processes. The FileVaultPRK.dat file remains as long as the version of the FileVault payload that triggered encryption remains on the device. If a change is made to the FileVault profile, macOS removes the FileVaultPRK.dat file even if the disk continues to be encrypted by the same Personal Recovery Key. You can attempt to validate the personal recovery key by performing the following commands:

1. Check if a PRK is listed: sudo nano /var/db/ConfigurationProfiles/fdesetup.plist
2. Check if the PRK is valid for the currently encrypted disk:   sudo fdesetup validaterecovery and enter the PRK (in the format AAAA-AAAA-AAAA-AAAA-AAAA-AAAA).

If Workspace ONE Intelligent Hub is Installed:

The Workspace ONE Intelligent Hub monitors for the presence of the FileVaultPRK.dat file when a FileVault profile is assigned. If the file is determined to be missing, Workspace ONE Intelligent Hub version 19.04 and later will prompt the user for their password and use that password to rotate the key via fdesetup. On the next SecurityInfo commmand, macOS should report the new Personal Recovery Key back to MDM for escrow. Additionally, macOS removes this file each time a change is made to the FileVault profile. As such, Workspace ONE Intelligent Hub will prompt the user for their password to ensure the Personal Recovery Key is re-escrowed.

NOTE: If an admin needs to immediately start this process to re-escrow the PRK, re-install the Intelligent Hub and the event that monitors for a missing FileVaultPRK.dat file will immediately trigger.

You can validate the PRK has been successfully rotated via Unified Logging using the command: log show  -predicate 'process = "hubd"' | grep com.vmware.hub.events

If you enable the Managed Client Debug Logging, this is the approximate set of events you can expect to see (snipped for easier viewing). You can gather events in Terminal.app by entering sudo log collect --last 1h (where 1h is 1 hour). Replace the last time frame with however long was required to perform the testing. You can also dump the events using log show --predicate "process = 'mdmclient' OR process ='fdesetup'" --last 30m > logs.txt in order to get a text file for keyword searching in the text editor of your choice.

NOTE: The events shown below come from Unified Logging in macOS, not the "hub logs" collected from the UEM console. These events originate at the device which has received the disk encryption (FileVault) payload.

The password rotation process at a high level works as follows:

1. Workspace ONE UEM Console generates a randomized password and saves it in the Automated Enrollment (DEP) Profile when assigned to each device record at Apple. The password is also saved to the device record.
2. When the Mac proceeds through the SetupAssistant, macOS creates the local administrator account using the profile-provided username and randomized password.
3. Workspace ONE receives the auto-created admin user account on macOS in the DeviceInformation query, which includes the GUID for the user account.
4. When the password is accessed in Workspace ONE, a scheduled job is created that automatically issues SetAutoAdminPasswordCommand using a newly generated password and the admin account's GUID after a period of approximately 8 hours.
5. If necessary, Workspace ONE administrators can also manually force a password rotation via API: POST /api/mdm/devices/<DeviceID>/commands?command=RotateDEPAdminPassword

From within the Workspace ONE UEM Console, browse to the device details view and click on the Troubleshooting tab. In the Event Log, search for events such as the following:

• ManagedAdminPassword Accessed -- Indicates an Administrator has accessed the admin password. This starts a timer for the password rotation.
• ScheduledAdminPassword Rotation Requested -- Indicates Workspace ONE UEM has attempted to change the admin password.

Workspace ONE UEM attempts to rotate the password approximately 8 hours after the initial password access.  

Using Unified Logging, you can filter for events specific to the Auto Admin Password rotation via this command:

log show --predicate '(process == "mdmclient") && (eventMessage contains "SetAutoAdminPassword")'

There may be instances where the command may not be immediately processed, which can lengthen the amount of time between initial password access and password rotation. These will show in the logs as a NotNow response to MDM, and should include a reason for the delayed change.  

Once the device has successfully changed the admin account password, you will see an Acknowledged(SetAutoAdminPassword) entry in Unified Logging.

1. Open Finder and click Applications.
2. Ensure that the VMware Carbon Black Cloud folder is present and contains the Sensor app and bundles.
3. You might also see the Confer menulet in the menu bar.

1. Open Terminal.App and enter the following command:   tail -50 -F /Library/Application\ Support/AirWatch/Data/Munki/Managed\ Installs/Logs/ManagedSoftwareUpdate.log
2. Review the log for a note stating that the Install of CbDefense was successful.

If the sensor appears to not be installing, or is installing repeatedly, you may need to adjust the metadata PLIST to include an installs array (as covered in Make Modifications to the PLIST).

1. Open Terminal.app and enter the following command:   cd /Applications/VMware\ Carbon\ Black\ Cloud/repcli.bundle/Contents/MacOS
2. Enter the following command (and the administrative password if prompted):  sudo ./repcli status
3. Observe the values for System Extension Status, Sensor State, and Cloud Registration Status.

If the System Extensions are not loading, ensure that you have staged the correct profile payloads as covered in macOS Prerequisites for Deploying Carbon Black Cloud Sensor.

If the Kernel Extensions are not loading in macOS Big Sur, you might need to rebuild the kernel cache as shown in the next step.

If you have installed the Carbon Black Cloud Sensor for macOS in KEXT mode and the KEXTs are not loading, you can try to load them by Rebuilding the Kernel Extension Cache.  

1. Click Devices.
2. Click List View.
3. Select the device that needs the kernel cache rebuilt.
4. Click More Actions.
5. Click Custom Command.
6. Paste the command XML from the following example, making sure to add the full list of KextPaths into the array, or remove the key and the array of values.
7. Click Send.

Note: If you send the KextPaths key, you must include the Carbon Black KEXT path, as well as any other paths you want to include in the Kernel Cache Rebuild. If you do not specify the KextPaths key, macOS attempts to rebuild the cache with any known kernel extensions (for example, from Apps that have been launched and attempted to load a KEXT).

When troubleshooting OS Updates, there may be multiple subsystems at work.   As such, you may want to start with the following general command line:

log show --info --debug --predicate 'subsystem BEGINSWITH "com.vmware.hub" OR process == "mdmclient" or process == "softwareupdate"' --last 30m

This command will get you started down the path as follows:

• mdmclient logging will show you if any commands were received that triggered the update via MDM
• softwareupdate logging will show you if the softwareupdate process was triggered (by user or automation)
• com.vmware.hub logging shows you what activities the hub is doing (and if hub is triggering the softwareupdate command)

Additionally, if you're troubleshooting an issue where updates are not applying, check the OS Installer isn't restricted from running.   For more information on process blocking, see Troubleshooting App and Process Blocking.