App Volumes Architecture
This chapter is one of a series that make up the , a framework that provides guidance on the architecture, design considerations, and deployment of Workspace ONE and Horizon solutions. This chapter provides information about architecting VMware App Volumes. A companion chapter, , covers common configuration tasks.
The VMware App Volumes™ just-in-time application model separates IT-managed applications and application suites into administrator-defined application containers. App Volumes also introduces an entirely different container used for persisting user changes between sessions.
Figure 1: App Volumes Just-in-Time Application Model
This version of the App Volumes reference architecture was built using App Volumes 4. App Volumes 4 architecture is similar to that of the earlier App Volumes 2.x versions, but there are some notable differences in components, lifecycle management, and terminology. The App Volumes 4 Feature Review is an interactive demo that will help you quickly familiarize yourself with the new concepts. If you are planning to upgrade from App Volumes 2.x to 4, see VMware App Volumes 4 Installation and Upgrade Considerations to learn about the various upgrade paths available, including App Volumes 2.18 and App Volumes 4 co-existence.
App Volumes serves two functions. The first is delivery of software programs that are not in the golden image VM image for VDI and RDSH. App Volumes groups one or more programs into packages, based on the requirements of each use case. A package is a virtual disk containing one or more programs that are captured together.
The packages are added to applications. Applications are used to assign packages to Active Directory (AD) entities such as user, group, organizational unit (OU), or machine. The packages can be mounted each time the user logs in to a desktop, or at machine startup. For VDI use cases, packages can be mounted at login. With RDSH use cases, because packages are assigned to the machine account, the packages are mounted when the App Volumes service starts.
App Volumes also provides user Writable Volumes, which can be used in specific use cases. Writable Volumes provide a mechanism to capture user profile data, user-installed applications that are not or cannot be delivered by packages, or both. This reduces the likelihood that persistent desktops would be required for a use case. User profile data and user-installed applications follow the user as they connect to different virtual desktops.
Table 1: Implementation Strategy for App Volumes
App Volumes was deployed and integrated into the VMware Horizon® on-premises environment.
This design was created for an environment capable of scaling to 8,000 concurrent user connections.
This strategy allows the design, deployment, and integration to be validated and documented.
Note: If you are new to App Volumes, VMware recommends the following resources to help you familiarize yourself with this product:
- VMware App Volumes Product Page
- App Volumes Activity Path
- VMware Horizon - Getting Started with App and Desktop Virtualization (hands-on lab)
The App Volumes Agent is installed in the guest operating system of nonpersistent VMs. The agent communicates with the App Volumes Manager instances to determine package and Writable Volumes entitlements. Packages and Writable Volumes virtual disks are attached to the guest operating system in the VM, making applications and personalized settings available to end users.
Figure 2: App Volumes Logical Components
The components and features of App Volumes are described in the following table:
Table 2: App Volumes Components and Concepts
App Volumes Manager
App Volumes Agent
VMware vCenter Server®
The following figure shows the high-level logical architecture of the App Volumes components, scaled out with multiple App Volumes Manager servers using a third-party load balancer:
Figure 3: App Volumes Logical Architecture
Key Design Considerations
- Always use at least two App Volumes Manager servers, preferably configured behind a load balancer.
Note: This setup requires a shared SQL Server.
- An App Volumes instance is bounded by the SQL database.
- Any kernel mode applications should reside in the base image and not in a package.
- Use storage groups (if you are not using VMware vSAN™) to aggregate load and IOPS.
Note: Packages are very read intensive.
- Storage groups may still be applicable to vSAN customers for replicating packages. See
- Place packages on storage that is optimized for read (100 percent read).
- Place Writable Volumes on storage optimized for random IOPS (50/50 read/write).
- Assign as few packages as possible per user or device. See the VMware Knowledge Base article for the recommended number of packages per VM.
Note: This KB article was written for App Volumes 2.x AppStacks. Although most of the content is applicable to App Volumes 4, the maximum number of package attachments tested has increased.
- App Volumes 4 defaults to an optimized Machine Managers configuration. Use the default configuration and make changes only when necessary.
Figure 4: Default Machine Managers Configuration
Note: With previous versions of App Volumes, configuring the Mount ESXi option or, mount on host was recommended to reduce the load on vCenter Server and improve App Volumes performance. App Volumes 4 and later provides new optimizations in the communication with vCenter Server. Most implementations will no longer benefit from enabling the Mount ESXi option.
You can enable the Mount Local storage option in App Volumes to check local storage first and then check central storage. Packages are mounted faster if stored locally to the ESXi (vSphere) host. Place VMDKs on local storage and, as a safeguard, place duplicates of these VMDKs on central storage in case the vSphere host fails. Then the VMs can reboot on other hosts that have access to the centrally stored VMDKs.
Network Ports for App Volumes
App Volumes in a Horizon Environment
One key concept in a VMware Horizon environment design is the use of pods and blocks, which gives us a repeatable and scalable approach. See the Pod and Block section of for more information on pod and block design.
Consider the Horizon block design and scale when architecting App Volumes.
Table 3: Strategy for Deploying App Volumes in Horizon Pods
An App Volumes Manager instance was deployed in each pod in each site.
The App Volumes machine manager was configured for communication with the vCenter Server in each resource block.
Standardizing on the pod and block approach simplifies the architecture and streamlines administration.
In a production Horizon environment, it is important to adhere to the following best practices:
Scalability and Availability
As with all server workloads, it is strongly recommended that enterprises host App Volumes Manager servers as vSphere virtual machines. vSphere availability features such as cluster HA, VMware vSphere® Replication™, and VMware Site Recovery Manager™ can all complement App Volumes deployments and should be considered for a production deployment.
In production environments, avoid deploying only a single App Volumes Manager server. It is far better to deploy an enterprise-grade load balancer to manage multiple App Volumes Manager servers connected to a central, resilient SQL Server database instance.
As with all production workloads that run on vSphere, underlying host, cluster, network, and storage configurations should adhere to VMware best practices with regard to availability. See the for more information.
App Volumes Managers
App Volumes Managers are the primary point of management and configuration, and they broker volumes to agents. For a production environment, deploy at least two App Volumes Manager servers. App Volumes Manager is stateless, all of the data required by App Volumes is located in a SQL database. Deploying at least two App Volumes Manager servers ensures the availability of App Volumes services and distributes the user load.
Although two App Volumes Managers might support the 8,000 concurrent users design, additional managers are necessary to accommodate periods of heavy concurrent usage, such as for logon storms.
Table 4: Strategy for Scaling App Volumes
Four App Volumes Manager servers were deployed with a load balancer.
This strategy satisfies the requirements for load and provides redundancy.
Multiple vCenter Server Considerations
Configuring multiple vCenter Servers is a way to achieve scale for a large Horizon pod, for multiple data centers, or for multiple sites.
With machine managers, you can use different credentials for each vCenter Server, but vSphere host names and datastore names must be unique across all vCenter Server environments. After you have enabled multi-vCenter-Server support in your environment, it is not recommended to revert back to a single vCenter Server configuration.
Note: In a multiple-vCenter-Server environment, a package is tied to storage that is available to each vCenter Server. It is possible that a package visible in App Volumes Manager could be assigned to a VM that does not have access to the storage. To avoid this issue, use storage groups to replicate packages across vCenter Servers. For instructions, see .
Multiple AD Domain Considerations
An administrator can add multiple Active Directory domains through the Configuration > Active Directories tab in App Volumes Manager. An account with a minimum of read-only permissions for each domain is required. You must add each domain that will be accessed for App Volumes by any computer, group, or user object. In addition, non-domain-joined entities may be allowed by enabling this setting.
Figure 5: Enabling Non-Domain-Joined Entities
Host configurations have significant impact on performance at scale. Consider all ESXi best practices during each phase of scale-out. To support optimal performance of packages and Writable Volumes, give special consideration to the following host storage elements:
- Host storage policies
- Storage network configuration
- HBA or network adapter (NFS) configuration
- Multipath configuration
- Queue-depth configuration
For best results, follow the recommendations of the relevant storage partner when configuring hosts and clusters.
Use a minimum of at least two App Volumes Managers in production, and configure each App Volumes Agent to point to a load balancer. For high performance and availability, we recommend that a load balancer be located in front of the App Volumes Managers to balance connections between each server.
The main concern with App Volumes Managers is handling login storms. During the login process, user-based packages and Writable Volumes must be attached to the guest OS in the VMs. The greater the number of concurrent attachment operations, the more time it might take to get all users logged in.
For App Volumes 4, the exact number of users each App Volumes Manager can handle will vary, depending on the load and the specifics of each environment. See the VMware Knowledge Base article for tested limits of users per App Volumes Manager server and login rates.
VMware recommends that you test the load and then size the number of App Volumes Manager servers appropriately. To size this design, we assumed each App Volumes Manager was able to handle 2,000 users.
The following figure shows how virtual desktops and RDS Hosts (published applications) can point to a load balancer that distributes the load to two App Volumes Managers.
Figure 6: App Volumes Managers Load Balancing
Table 5: Strategy for App Volumes Scalability and Availability
A load balancer was placed in front of the App Volumes Manager servers.
The VMware NSX Advanced Load Balancer (formerly Avi Networks) was used.
The load balancer properly distributes load and keeps the services available in the event of an issue with one of the managers.
No additional configuration is required on the App Volumes Manager servers.
Third-party load balancers can also be used, and you should refer to the guidance given by the load balancer vendor. Typical settings are given in the table below.
Table 6: Load Balancing Settings for App Volumes Managers
App Volumes 4 uses a Microsoft SQL Server database to store configuration settings, assignments, and metadata. This database is a critical aspect of the design, and it must be accessible to all App Volumes Manager servers.
An App Volumes instance is defined by the SQL database. Multiple App Volumes Manager servers may be connected to a single SQL database.
For nonproduction App Volumes environments, you can use the Microsoft SQL Server Express database option, which is included in the App Volumes Manager installer. Do not use SQL Server Express for large-scale deployments or for production implementations.
App Volumes works well with both SQL Server failover cluster instances (FCI) and SQL Server Always On availability groups. Consult with your SQL DBA or architect to decide which option better fits your environment.
Table 7: Implementation Strategy for the SQL Server Database
A SQL database was placed on a highly available Microsoft SQL Server. This database server was installed on a Windows Server Failover Cluster, and an Always On availability group was used to provide high availability.
An Always On availability group achieves automatic failover. Both App Volumes Manager servers point to the availability group listener for the SQL Server.
A successful implementation of App Volumes requires several carefully considered design decisions with regards to disk volume size, storage IOPS, and storage replication.
Package and Writable Volume Template Placement
When new packages and Writable Volumes are deployed, predefined templates are used as the copy source. Administrators should place these templates on a centralized shared storage platform. As with all production shared storage objects, the template storage should be highly available, resilient, and recoverable. See to get started.
Package sizing and Writable Volume sizing are critical for success in a production environment. Package volumes should be large enough to allow programs to be installed and should also allow for software updates. Packages should always have at least 20 percent free disk space available so administrators can easily update programs without having to resize the package volumes.
Writable Volumes should also be sufficiently sized to accommodate all users’ data. Storage platforms that allow for volume resizing are helpful if the total number of Writable Volume users is not known at the time of initial App Volumes deployment.
Because packages and Writable Volumes use VMware vSphere® VMFS, the thin-provisioned, clustered Virtual Machine File System from VMware, storage space is not immediately consumed. Follow VMware best practices when managing thin-provisioned storage environments. Free-space monitoring is essential in large production environments.
Writable Volumes Delay Creation Option
Two policy options can complicate free-space management for Writable Volumes:
- The option to create Writable Volumes on the user’s next login means that storage processes and capacity allocation are impacted by user login behavior.
- The option to restrict Writable Volume access (and thus initial creation) to a certain desktop or group of desktops can also mean that user login behavior dictates when a Writable Volume template is copied.
In a large App Volumes environment, it is not usually a good practice to allow user behavior to dictate storage operations and capacity allocation. For this reason, VMware recommends that you create Writable Volumes at the time of entitlement, rather than deferring creation.
App Volumes uses a construct called storage groups. A storage group is a collection of datastores that are used to serve packages or distribute Writable Volumes.
The two types of storage groups are:
- Package storage groups – Used for replication.
- Writable Volume storage groups – Used for distribution.
In App Volumes 4, the packages within a storage group can be replicated among its peers to ensure all packages are available. Having a common datastore presented to all hosts in all vCenter Servers allows packages to be replicated across vCenter Servers and datastores.
Two automation options for package storage groups are available:
- Automatic replication – Any package placed on any datastore in the storage group is replicated across all datastores in the group.
- Automatic import – After replication, the package is imported into App Volumes Manager and is available for assignment from all datastores in the storage group.
When using package storage groups, the App Volumes Manager manages the connection to the relevant package, based on location and number of attachments across all the datastores in the group.
Once created, packages are read-only. As more and more users are entitled to and begin using a given package, the number of concurrent read operations increases. With enough users reading from a single package, performance can be negatively impacted. Performance can be improved by creating one or more copies of the package on additional datastores, and spreading user access across them.
Packages can be automatically replicated to multiple datastores in a storage group. This replication creates multiple copies of packages. Access is spread across the datastores, ensuring good performance as App Volumes scales to serve more end users. See the VMware Knowledge Base article for the recommended number of concurrent attachments per package.
Storage groups can also be used to replicate packages from one site to another in multi-site App Volumes configurations. By using a non-attachable datastore available to hosts in each site, packages created at one site can be replicated to remote sites to serve local users.
A datastore configured as not attachable is ignored by the App Volumes Manager while mounting volumes, and the storage can be used solely for replication of packages. This means you can use a datastore on slow or inexpensive storage for replication, and use high-speed, low-latency storage for storing mountable volumes.
This not-attachable datastore can also be used as a staging area for package creation before deploying to production storage groups. This topic is covered in more detail in the Replication of Packages section of .
Note: Marking a datastore as not-attachable or read-only is only applicable for vSphere environments. The check boxes to configure this would not be available in an App Volumes VHD configuration.
Writable Volume storage groups are used to distribute volumes across datastores to ensure good performance as Writable Volumes are added. See the VMware Knowledge Base article for the recommended number of Writable Volumes per datastore.
Table 8: Implementation Strategy for Storage Groups
Storage groups were set up to replicate packages between datastores.
An NFS datastore was used as a common datastore between the different vSphere clusters.
This strategy allows the packages to be automatically replicated between VMFS datastores, between vSAN datastores and between vSphere clusters.
This section provides guidance about creating, sizing, scaling, provisioning, configuring, and updating packages.
By default, a single 20-GB package template is deployed in an App Volumes environment. This template is thin-provisioned and is provided in both a VMDK and VHD format. This template can be copied and customized, depending on how large the package needs to be for a given deployment scenario.
For more information, see the VMware Knowledge Base article . Note: Although this KB article was written for AppStacks, it is applicable to App Volumes 4 packages as well. With App Volumes 4.x, up to 25 packages are recommended.
If you have packages from a previous App Volumes 4 release, they will continue to work. However, additional features or fixes included in later versions are not applied to packages created with earlier versions.
Packages at Scale
The number of packages that can be attached to a given VM is technically limited by the maximum number of possible drive attachments in Windows and vSphere. For example, ESXi has a limit of 59 VMDKs + 1 OS disk. See the VMware Knowledge Base article for guidance. In practice, the number of packages attached to a VM will likely be considerably lower than the maximum values.
Attaching packages involves the following processes:
- The disk mount (mounting the package VMDK to the VM)
- The virtualization process applied to the content in the package (merging files and registry entries with the guest OS)
The time required to complete the virtualization process varies greatly, depending on the programs contained in a given package. The more packages that need to be attached, the longer this operation might take to complete.
Recommended Practices for Packages in Production Environments
The size of the default package template is 20 GB. The default Writable Volume template is 10 GB. In some environments, it might make sense to add larger or smaller templates. For information on creating multiple, custom-sized templates, see the VMware Knowledge Base article . Note: This KB article is applicable to both App Volumes 4 packages and AppStacks. With App Volumes 4.x, up to 25 packages are recommended.
App Volumes 4 includes enhancements to the agent compared with 2.x agents. The result is improved logon times and better performance with many packages attached. Although you can likely attach more App Volumes 4 packages than App Volumes 2.x AppStacks, we recommend keeping the total number of packages assigned to a given user or computer relatively small. This can be accomplished by adding multiple programs to each package. Group the apps in such a way as to simplify distribution.
The following is a simple example for grouping App Volumes programs into packages:
- Create a package containing core programs (apps that most or all users should receive). This package can be assigned to a large group or OU.
- Create a package for departmental programs (apps limited to a department). This package can be assigned at a group or departmental level.
For traditional storage (VMFS, NFS, and so on):
- Do not place packages and VMs on the same datastore.
- Use storage groups for packages when packages are assigned to a large population of users or desktops. This helps to distribute the aggregated I/O load across multiple datastores, while keeping the assignments consistent and easy to manage.
- Packages and VMs can be placed on a single datastore.
- Storage groups for packages are not applicable in a vSAN implementation.
Packaging Applications Recommended Practices
Consider the following best practices when creating and packaging applications:
- The following characters cannot be used when naming packages: & “ ‘ < >
- For the packaging VM, use a clean golden image that resembles as closely as possible the target environment where the package is to be deployed. For example, the packaging VM and target should be at the same OS patch and service pack level and, if programs are included in the golden image, they should also be in the packaging VM.
- The packaging VM should have the App Volumes Agent installed but should not have either the Horizon Agent or the Dynamic Environment Manager FlexEngine agent installed.
- Do not use a packaging machine where you have previously installed and then uninstalled any of the programs that you will capture. Uninstalling a program might not clean up all remnants of the software, and the subsequent App Volumes package capture might not be complete.
- Always take a snapshot of your packaging VM before packaging or attaching any packages to it. If any packages have been assigned to the VM, or if the VM has been used previously for packaging, revert that VM to the clean snapshot before creating a new package.
Updating and Assigning Updated Packages Recommended Practices
App Volumes 4 introduced assignment types called marker and package to improve the administrative process of updating application packages. Using the CURRENT marker enables distribution of the current package to your end-user population, whereas using the package assignment type enables distribution of test versions to a subset of users for validation.
Once a new package has been tested and approved, you can simply change the CURRENT marker to point to the new package. As end users log off their desktop sessions, the old version of the package is detached. When they log on again, the new version is attached.
The following illustration shows the 7-Zip application, which contains three packages with different versions of the software.
Figure 7: Portion of the Application Tab Detailing the Three Packages on the Right
7-Zip 16.04 has the CURRENT marker, so it is distributed to the general population of end users. 7-Zip 19.0 is an updated package that contains a newer version of the 7-Zip program.
Figure 8: Portion of the Application Tab Detailing the Two Assignments on the Right
7-Zip 19.0 is using a package assignment type to directly assign that specific package to a group of test users for validation.
To initiate an update to programs in an existing package, use the App Volumes Manager console to invoke the update process. This process clones the original package for you to work with and apply updates. End users continue to work from the original package to prevent user downtime. The new package with the updated programs is distributed by simply moving the CURRENT marker once it has been approved.
Consider the following best practices when updating and assigning updated packages:
- When creating and updating packages, use the Stage drop-down list to select an appropriate value. This makes it easy for you and other App Volumes admins to manage the lifecycle of the package.
- Use the marker assignment type to simplify updates for your general population of users.
- Use the Unset CURRENT option to prevent delivery of a package without modifying assignments.
- Use the package assignment type for one-off, explicit assignments of a specific version.
- Note: If both package and marker assignments are made, the package assignment is used.
Although not required, App Volumes is often implemented in a Horizon environment. Consider the following when integrating App Volumes and Horizon.
- Do not attempt to include the Horizon Agent in an App Volumes package. The Horizon Agent should be installed in the golden image VM.
- Do not use a Horizon VM (guest OS with Horizon Agent installed) as a clean packaging VM. You must uninstall the Horizon Agent if it is present. Dependencies previously installed by the Horizon Agent, such as Microsoft side-by-side (SxS) shared libraries, are not reinstalled, and therefore are not captured by the App Volumes packaging process.
- See for information on agent installation order.
Performance Testing for Packages
Test packages immediately after packaging to determine their overall performance. Using a performance analytics tool, such as VMware vRealize® Operations Manager™, gather virtual machine, host, network, and storage performance information for use when packages are operated on a larger scale. Do not neglect user feedback, which can be extremely useful for assessing the overall performance of an application.
Because App Volumes provides an application container and brokerage service, storage performance is very important in a production environment. Packages are read-only. Depending on utilization patterns, the underlying shared storage platform might have significant read I/O activity. Consider using flash and hybrid-flash storage technologies for packages.
This evaluation can be time-consuming for the administrator, but it is necessary for any desktop- transformation technology or initiative.
ThinApp Integration with Packages
Network latency is often the limiting factor for scalability and performance when deploying ThinApp packages in streaming mode. But ThinApp provides exceptional application-isolation capabilities. With App Volumes, administrators can present ThinApp packages as dynamically attached applications that are located on storage rather than as bits that must traverse the data center over the network.
Using App Volumes to deliver ThinApp packages removes network latency due to Windows OS and environmental conditions. It also allows for the best of both worlds—real-time delivery of isolated and troublesome applications alongside other applications delivered on packages.
With App Volumes in a virtual desktop infrastructure, enterprises can take advantage of local deployment mode for ThinApp packages. ThinApp virtual applications can be provisioned inside a package using all the storage options available for use with packages. This architecture permits thousands of virtual desktops to share a common ThinApp package through packages without the need to stream or copy the package locally.
Microsoft Office Application Packages
Core Microsoft Office applications can either be packaged with App Volumes or included in the base golden image used to create the desktop pool. Factors to consider include your preference in application packaging, and whether all users will get access to core Microsoft Office applications. See to understand some of the considerations.
Non-core Microsoft Office applications, such as Visio and Project, can be packaged separately with App Volumes and then assigned to only the users that require those applications. Versions of core and non-core office that are delivered to the same VM, should be the same version.
Note: Ensure that core and non-core Microsoft Office applications are of similar version and patch levels.
Office Plug-Ins and Add-Ons
The most straightforward method is to include Microsoft Office plug-ins or add-ons in the same package as the Microsoft Office installation.
However, if necessary, you can include plug-ins or add-ons in packages that are separate from the packages that contain the Microsoft applications to which they apply. Before packaging the plug-in or add-on, install the primary application natively in the OS of the packaging VM.
Note: Ensure the plug-in or add-on is at the same version as the Microsoft Office application in the package. This includes any patches or updates.
RDSH Integration with Packages
App Volumes supports package integration with Microsoft RDSH-published desktops and published applications. Packages are assigned to RDSH servers rather than directly to users. Packages are attached to the RDSH server when the machine is powered on, and the App Volumes service starts. Users are then entitled to the RDSH-published desktops or applications through the Horizon entitlement process.
Note: Writable Volumes are not supported with RDSH assignments.
Consider associating packages at the OU level in Active Directory, rather than to individual computer objects. This practice reduces the number of package entitlements and ensures packages are always available as new hosts are created and existing hosts are refreshed.
Entitling packages to an OU where Horizon instant-clone RDSH server farms are provisioned ensures that all hosts are configured exactly alike, which supports dynamic growth of farms with minimal administrative effort.
Create dedicated packages for RDSH servers. Do not reuse a package that was originally created for a desktop OS.
When creating the package, install programs on a packaging machine that has the same operating system as that used on the deployed RDSH servers. Before installing software, switch the RDSH server to RD-Install mode. For more information, see .
Application Suitability for Packages
Most Windows applications work well with App Volumes, including those with services and drivers, and require little to no additional interaction. If you need an application to continue to run after the user logs out, it is best to natively install this application on the desktop or desktop image.
The following sections briefly describe situations and application types where App Volumes might need special attention to work properly or where the application would work best when installed in the golden image, rather than in a package.
Applications that should be available to the OS in the event that a package or Writable Volume is not present should remain in the golden image and not in an App Volumes virtual disk. These types of applications include antivirus, Windows updates, and OS and product activations, among others. Applications that should be available to the OS when no user is logged in should also be placed in the golden image.
Similarly, applications that integrate tightly with the OS should not be virtualized in a package. If these apps are removed from the OS in real time, they can cause issues with the OS. Again, if the application needs to be present when the user is logged out, it must be in the golden image and not in a package. Applications that start at boot time or need to perform an action before a user is completely logged in, such as firewalls, antivirus, and Microsoft Internet Explorer, fall into this category.
Applications that use the user profile as part of the application installation should not be virtualized in an App Stack. App Volumes does not capture the user profile space C:\users\<username>. If, as part of its installation process, an application places components into this space, those components will not be recorded as part of the packaging process. If this happens, undesired consequences or failure of the application might result when the application is captured in a package.
In the rare event that an issue with an application does present itself, it is important to have a thorough understanding of how the application functions. Understanding the processes that are spawned, file and registry interactions, and where files are created and stored is useful troubleshooting information.
App Volumes is a delivery mechanism for applications. It is important to understand that App Volumes does an intelligent recording of an installation during the packaging process and then delivers that installation. If the installation is not accurate or is configured incorrectly, the delivery of that application will also be incorrect (“garbage in, garbage out”). It is important to verify and test the installation process to ensure a consistent and reliable App Volumes delivery.
The App Volumes Agent is a mini-filter driver. Microsoft applies altitudes to filter drivers. The concept is that the larger the number, the “higher” the altitude. Mini-filter drivers can see only the other filter drivers that are at a higher altitude. The actions at a lower altitude are not seen by filter drivers operating at a higher altitude.
The lower-altitude mini-filter drivers are the first to interact with a request from the OS or other applications. Generally speaking, the requests are then given to the next mini-filter driver in the stack (next highest number) after the first driver finishes processing the request. However, this is not always the case because some mini-filter drivers might not release the request and instead “close” it out to the OS or application.
In the case where a request is closed, the subsequent mini-filter drivers will never see the request at all. If this happens with an application running at a lower altitude than App Volumes, the App Volumes mini-filter driver will never get a chance to process the request, and so will not be able to virtualize the I/O as expected.
This is the primary reason that certain applications that use a mini-filter driver should be turned off or removed from the OS while you install applications with App Volumes. There might be additional scenarios where App Volumes Agent should be turned off, allowing other applications to install correctly in the base OS.
The following guidelines will also help you determine whether an application requires special handling during the virtualization process or whether virtualization in a package is even possible:
- Additional application virtualization technologies – Other application virtualization technologies (Microsoft App-V, ThinApp, and others) should be turned off during packaging because the filter drivers could potentially conflict and cause inconsistent results in the packaging process.
- Mixing of 32- and 64-bit OS types – The OS type (32- or 64-bit) of the machine that the package is attached to should match the OS type that applications were packaged on. Mixing of application types in App Volumes environments follows the same rules as Windows application types—that is, if a 32-bit application is certified to run in a 64-bit environment, then App Volumes supports that configuration also.
- Exceptional applications – Some software apps just do not work when installed on an App Volumes package. There is no list of such applications, but an administrator might discover an issue where an application simply does not work with App Volumes.
In summary, most applications work well with App Volumes, with little to no additional interaction needed. However, you can save time and effort by identifying potential problems early, by looking at the application type and use case before deciding to create a package.
Writable Volumes can be used to persist a variety of data as users roam between nonpersistent desktop sessions. As is described in Outlook OST and Windows Search Index files are automatically redirected to Writable Volumes, improving search times for companies using these technologies. See for information on creating and managing writable volumes.
Writable Volumes are often complemented by VMware Dynamic Environment Manager™ to provide a comprehensive profile management solution. For technical details on using App Volumes with Dynamic Environment Manager, see the VMware blog post .
Writable Volume Templates
The UIA (user-installed applications)-only template provides persistence for user-installed applications. After a Writable Volume with the UIA-only template is created and assigned to a user, that user can install and configure applications as they normally would. The installation is automatically redirected to the Writable Volume and persisted between desktop sessions.
Note: Due to the complexities of file and registry conflicts, we recommend that you use either packages or Writable Volumes with a UIA-based template for a given VM. For example, if a user installs a patch or update to an application, even if a package or base is updated, the writable volume will still retain older files that will be available to the user.
Key Differences Between Packages and Writable Volumes
There are some key differences and considerations for using Packages and Writable Volumes.
- Package VMDKs are mounted as read-only and can be shared among all desktop VMs within the data center.
- Writable Volumes are dedicated to individual users and are mounted as the user authenticates to the desktop. Writable Volumes are user-centric and roam with the user for nonpersistent desktops.
Note: When files and registry entries exist in multiple places, the files and registry entries in Writable Volumes will always take precedence over files and registry keys in packages and in the base image.
Table 9: Implementation Strategy for App Volumes Writable Volumes
Writable Volumes were created for and assigned to end users who required the ability to install their own applications.
The UIA-only Writable Volume template was used.
Writable Volumes provide added flexibility for end users who are permitted to install software outside of the IT-delivered set of applications.
The UIA-only template ensures application installation and configuration data is stored, while profile data is managed using other technologies.
If a Writable Volume becomes corrupt, applications can be reinstalled without the risk of data loss.
Performance Testing for Writable Volumes
Writable Volumes are read-write. Storage utilization patterns are largely influenced by user behavior with regard to desktop logins and logouts, user-installed applications, and changes to local user profiles. Group each set of similar users into use cases, and evaluate performance based on peak average use.
Additional Writable Volumes Operations
VMware recommends designing multi-site implementations using a multi-instance model with separate instances per site. In this model, each instance works independently, with its own set of App Volumes Managers and its own database instance. When deploying separate instances in different sites, separate, local SQL Servers should be used.
Using separate instances per site is simple to implement and allows for easy scaling if you have more than two sites.
During an outage, the remaining instances in the remaining sites can provide access to packages with no intervention required. For detailed information on the failover steps required and the order in which they need to be performed, refer to
Figure 9: App Volumes Multi-instance model
This strategy makes use of the following components:
- App Volumes Managers – At least two App Volumes Manager servers are used in each site to form an instance, for local redundancy and scalability.
- Load balancers – Each site has its own namespace for the local App Volumes Manager servers. This is generally a local load balancer virtual IP that targets the individual managers.
Note: The App Volumes Agent, which is installed in virtual desktops and RDSH servers, must be configured to use the appropriate local namespace.
- Separate databases – A separate database is used for each instance; each site should have separate SQL servers. That is, you have a separate Windows Server Failover Clustering (WSFC) cluster and an SQL Server Always On availability group listener for each site, in order to achieve automatic failover within a site.
- vCenter Server machine managers – The App Volumes Manager instance and servers at each site have machine managers registered only for the vCenter Servers from their own site.
Table 10: Strategy for Deploying App Volumes in Multiple Sites
App Volumes was set up in the second site used for Horizon (on-premises).
A separate database and App Volumes instance deployment option was used.
An NFS datastore was used as a common datastore among the storage groups to facilitate cross-site package replication.
This strategy provides App Volumes capabilities in the second site.
The separate-databases option is the most resilient, provides true redundancy, and can also scale to more than two sites.
With packages replicated between sites, the packages are available for use at both locations.
Replication of Packages
containing a shared, non-attachable datastore can be used to automatically replicate packages from one instance of App Volumes to another. This shared datastore must be visible to at least one vSphere host from each App Volumes instance. As this shared datastore is configured to be non-attachable, it will not be used to deliver attachments to user machines, so does not need to be overly performant, and often a low-cost NFS share is used to provide this.
Figure 10: Replicating VMDK Packages between Instances with a Shared Datastore
Using NFS Datastores on VMware Cloud on AWS
When deployed on VMware Cloud on AWS, App Volumes can use AWS managed external NFS datastores to assist in the replication of packages from one App Volumes instance to another. AWS managed external NFS datastores are provided by .
These NFS datastores can be attached to VMware Cloud on AWS vSphere clusters and used as shared datastore between different vSphere clusters to facilitate App Volumes package replication between App Volumes instances.
For more information on using Amazon FSx for NetApp ONTAP see:
In some environments, network design might prevent the use of storage group replication between sites. See for more information about manually copying AppStacks and packages from one instance of App Volumes to another.
App Volumes version 2111 introduced the ability to join App Volumes instances together to control them from a single source instance. Used in combination with storage groups replicating the packages, this allows you to set a source and target relationship between App Volumes instances so that packages are distributed across different App Volumes instances and locations. With App Volumes version 2203, the ability to synchronize metadata, markers, and assignments between instances was also added.
A multi-instance deployment of App Volumes provides the capability to:
- Use one App Volumes instance as the source for replicated packages
- Synchronize application and package metadata across all connected App Volumes instances
- Synchronize application and package deletes
- Synchronize application markers across instances (optional)
- Synchronize assignments across instances (optional)
- Assisted replication and import removes the need for Import Background Job
Table 11: Multi-Instance Components
App Volumes Manager Instance
An App Volumes install bound by an SQL database
Multiple App Volumes Manager servers can access a single SQL database
App Volumes instance that synchronizes replicated applications and packages to other App Volumes instances
App Volumes instance with replicated applications and packages that are synchronized by another (source) App Volumes instance
A target instance or a source instance with reference to a current App Volumes Manager instance
An App Volumes feature which enables the copying of applications and packages ( .vmdk and .vhd) between storage locations
An App Volumes feature which ensures information about the replicated applications and packages remain the same across App Volumes Manager instances
When configuring replication between multiple App Volumes instances, you should designate one instance as the source, and the other instances as targets. This gives a star topology, with one instance acting as the source, where the administrative changes will be made. This avoids the risk of having conflicting information from different App Volumes instances.
Figure 11: Star Topology with one Source Instance and Multiple Target Instances
Determine which App Volumes instance you will use as the administrative source. That instance will be where you will create packages and assignments. From that instance, add in the other instances as targets. For more information, see .
To properly set up the source and target instances relationships, configure the following:
- When adding a target instance to a source instance, select the option to enable Application Package Import. You can also select whether to synchronize markers and assignments; with production target instances, you usually want to replicate both markers and assignments.
- On all instances, configure the shared datastore storage location (that is used for replicating between instances) to Not Attachable. This prevents this datastore being used for when mounting AppStacks and packages. See for instructions.
- On the target instances, mark the shared datastore as Read-only. This prevents inadvertent changes on target instances from being replicated back to the source instance, keeping it as the single source of truth for packages.
- Set up packages replication using a shared datastore and storage groups as described in .
- Storage groups on all instances should be configured to:
- Automatically Import AppStacks and Packages = Off
- Automatically Replicate AppStacks and Packages = On
With certain types of target instances, such as test or development, you might not want to synchronize the assignments or markers. When you add a target instance, you can select whether that metadata is synchronized to that target instance.
Figure 12: Selective metadata synchronization to test and dev instances
Writable Volumes Replication
Writeable Volumes can be replicated between App Volumes instances using storage replication of the datastore that contains the writable volumes. The storage arrays must be capable of replicating the LUN () that is used for the writable volumes datastore.
Writable Volumes contain user-generated data, so care should be taken to only have one copy active for a particular user at a time. If more than one copy is available, a user could make changes to one copy of their writable copy, and then not find those changes present in the other active copies.
In this model, each site works independently, with its own set of App Volumes Managers and its own database instance. During an outage, the remaining site can provide access to packages with no intervention required.
- The packages have previously been copied between sites using non-attachable datastores that are members of both sites’ storage groups.
- The entitlements to the packages are replicated from the source App Volumes instance to the target instances.
In use cases where Writable Volumes are being used, there are a few additional steps:
- Attach and mount the storage array replicated datastore that contains the Writable Volumes, against the vSphere hosts in the failover site.
- Perform a rescan of that datastore. If the datastore was the default Writable Volume location, App Volumes Manager automatically picks up the user entitlements after the old assignment information has been cleaned up.
- (Optional) If the datastore is not the default Writable Volume location, perform an Import Writable Volumes operation from the App Volumes Manager at the second site.
All assignments to Writable Volumes are successfully added, but to the new valid location.
Recommended Practices for Production Environments
The following recommendations pertain to golden image optimization, setting up virtual desktops, and security best practices.
Golden Image Optimization
Golden images should be optimized for VDI or RDSH to ensure the best performance possible in a virtualized environment. Consider using the instructions in when building your golden images. The helps optimize Windows desktop and server operating systems for use with VMware Horizon.
The OS Optimization Tool includes a customizable template to enable or turn off Windows system services and features across multiple systems, per VMware recommendations and best practices. Because most Windows system services are enabled by default, the OS Optimization Tool can be used to easily turn off unnecessary services and features to improve performance.
Desktop Pool Recommendations
When setting up client endpoint devices, consider the following best practices:
- When reverting a desktop VM that is running the App Volumes Agent to a previous snapshot, make sure that the VM is gracefully shut down, to avoid synchronization issues. This is primarily relevant to the packaging desktop and golden image VMs for Horizon linked- and instant-clone desktop pools.
- If you are using a Horizon desktop pool, the App Volumes Agent should be installed on the golden image VM for linked- and instant-clone pools, or on the VM template for full-clone pools, for ease of distribution.
- If using a Horizon linked-clone pool, make sure the Delete or Refresh machine on logoff policy in Desktop Pool Settings is set to Refresh Immediately. This policy ensures that the VMs stay consistent across logins.
App Volumes Security Recommendations
To support a large production environment, there are some important security configurations that administrators should put in place:
- Open only essential, required firewall ports on App Volumes Manager and SQL. Consider using an advanced firewall solution, such as VMware NSX®, to dynamically assign virtual machine firewall policies based on server role.
- Replace the default self-signed TLS/SSL certificate with a certificate for App Volumes Manager signed by a reliable certificate authority. See .
- Verify that App Volumes Manager can accept the vCenter Server certificate. App Volumes Manager communicates with vCenter Server over TLS/SSL. The App Volumes Manager server must trust the vCenter Server certificate.
- Note: It is possible to configure App Volumes Manager to accept an unverifiable (self-signed) certificate. Navigate to Configuration > Machine Managers in the management console. Each machine manager (vCenter Server) has a Certificate option that shows the current status of the certificate and allows an administrator to explicitly accept an unverifiable certificate.
- Consider using ThinApp to package applications, to take advantage of the security benefits of isolation modes, when required. Each ThinApp package can be isolated from the host system, and any changes, deletions, or additions made by the application to the file system or registry are recorded in the ThinApp sandbox instead of in the desktop operating system.
- For Writable Volumes, determine which end users require ongoing administrator privileges. Writable Volumes with user-installed applications require that each desktop user be assigned local computer administrator privileges to allow the installation and configuration of applications.
- Some use cases could benefit from temporary, request-based elevated privileges to allow incidental application installation for a specific user or user group. Carefully consider the security risks associated with granting users these elevated privileges.
- Create and use an AD user service account specifically for App Volumes. This is good security hygiene and a forensics best practice. It is never a good idea to use a blanket or general-purpose AD user account for multiple purposes within AD.
- Consider creating an administrative role in vCenter Server to apply to the App Volumes service account.
Installation and Initial Configuration
After installation is complete, you must perform the following tasks to start using App Volumes:
- Complete the App Volumes Initial Configuration Wizard (https://avmanager).
- Install the App Volumes Agent on one or more clients and point the agent to the App Volumes Manager address (load-balanced address).
- Select a clean packaging system and create an application package. See Working with Applications and Working with Packages in the for instructions.
- Assign the package to a test user and verify it is connecting properly.
- Assign a Writable Volume to a test user and verify it is connecting properly.
For better performance, it is recommended to use a local vCenter account instead of an active directory account when configuring a vCenter Server as a Machine Manager.
- For details on how to create a local user account, see .
- Information on the permissions required for a vCenter account are detailed in the App Volumes Manager console, at the bottom of the Configuration, Machine Managers section.
- Overview chapters provide understanding of business drivers, use cases, and service definitions.
- Architecture chapters explore the products you want to include in your platform, including Workspace ONE UEM, Workspace ONE Access, Workspace ONE Assist, Workspace ONE Intelligence, Horizon, App Volumes, Dynamic Environment Manager, and Unified Access Gateway.
- Integration chapters cover the integration of components and services you need to create the platform capable of delivering what you want.
- Configuration chapters provide reference for specific tasks as you build your platform, such as installation, deployment, and configuration processes for Horizon, App Volumes, Dynamic Environment Management, and more.