Working on a Citrix upgrade project a few years ago I was continually getting a blue screen when a Citrix PVS image booted. The symptom was obvious during the boot process. It was as if there were two NICs or the image was trying to load twice. PVS write-cache was also incorrectly being redirected to the server.
I checked and rechecked all the obvious suspects, such as ghosted NICs and other devices, old DHCP information, antivirus, etc. Everything looked good. It really did not make a lot of sense. I then stripped all the apps and tools out of the build and it worked! So what was I doing wrong?
At the time we were using the SoftPerfect RAM Disk product in our images to provide a fast RAM Disk for temporary data (scratch space) to boost the IO performance for some of the Mining and Engineering applications. Now we’ve switched to the Arsenal Image Mounter (AIM) product, but the issue remains the same.
Both of these products add a SCSI Adapter to the image. So if you install them BEFORE the PVS Target Device Agent, they will take the first available SCSI adapter instance ID, which is 0000. That’s fine and was working well with PVS 7.6 LTSR. But as I will demonstrate below, from 7.11 the Citrix Developers made a change so that the PVS Target Agent forces bits of it’s Citrix Virtual Hard Disk Adapter driver into instance ID 0000. It has no respect for anything already using that ID. It just overwrites some of whatever is there with it’s own values. SCSI Adapter instance ID 0000 becomes an ugly two headed monster of a SCSI Adapter, essentially serving two different SCSI devices. Further to that, instance ID 0000 and 0001 were both referencing the Citrix Virtual Hard Disk Adapter driver. Hence the issue I saw and the reboot loop.
Here’s a screen shot of the registry on a clean build with only the SoftPerfect RAM Disk installed.
And here’s a screen shot of what it looks like after the PVS Target Agent is installed.
WOW! What had changed to do this? I downloaded several Provisioning Services ISOs and extracted the driver so I could understand when the change was introduced. I tracked the change back to 7.11. The driver CVhdMp.inf file from 7.11 contains the following invalid information, which is still present in 2203 LTSR CU1 (7.33.5.22), and no doubt the current release.
[DeviceInstall32] AddDevice = ROOT\SCSIADAPTER\0000,,CVhdMp.RootDevice [CVhdMp.RootDevice] HardwareIds = root\CVhdMp
Here’s a screen shot of the full INF file showing the difference between the versions, and that it remains unchanged in 2203 LTSR CU1.
In this case the [DeviceInstall32] section of the INF file instructs the driver installation process to create a SCSI ADAPTER with an instance of 0000 and Hardware ID of “root\CVhdMp”. But what if there’s a device already using the ROOT\SCSIADAPTER\0000 instance path? No problems, as the PnP (Plug and Play) manager is clever enough to create a new device instance using the next available ID, which is ROOT\SCSIADAPTER\0001. This is successful and works for the PVS Target Agent. Awesome! So why is there a still problem with the PVS Target Agent?
Well, the driver install ALSO sets the Hardware ID for the “ROOT\SCSIADAPTER\0000” device instance to “root\CVhdMp”. This is where the issue lies. It changes the Hardware ID of any existing device using instance ID 0000 to “root\CVhdMp”.
- In the case of the SoftPerfect Virtual Bus SCSI Adapter, it is changed from “ROOT\SPVD” to “root\CVhdMp”
- In the case of the Arsenal Image Mounter SCSI Adapter, it is changed from “root\phdskmnt” to “root\CVhdMp”
You can actually see the DrvInst.exe (Driver Installation Module) doing this via the output from the “C:\Windows\Inf\setupapi.dev.log” file, and also monitoring the install process using Process Monitor.
The following screen shot is the output of the “C:\Windows\Inf\setupapi.dev.log” file capturing the full installation of the Citrix Virtual Hard Disk Adapter via the CVhdMp.inf. You’ll see that although it’s installing correctly into “ROOT\SCSIADAPTER\0001”, it still configures “ROOT\SCSIADAPTER\0000”.
Here’s a screen shot of the Process Monitor output showing DrvInst.exe making a change to the HardwareID under “ROOT\SCSIADAPTER\0000”. The timestamp is aligned with what we see in the setupapi.dev.log. Note that some of the timestamps in the setupapi.dev.log are 7 hours ahead due to my remote connection when capturing the data for this post.
There is not too much documentation on the [DeviceInstall32] section of the INF file. From what I understand this section was from the old Windows Driver Model (WDM). If added or left in the INF file with the more modern Operating Systems that should be leveraging the newer Windows Driver Framework (WDF) and Universal Driver Model (UDM) models, the outcome for the Customer may not be as expected from the pnputil.exe, dpinst.exe and drvinst.exe driver installation utilities.
In my opinion these lines must be removed, or commented out at the very least, for correct operation. I cannot modify the CVhdMp.inf to test my theory, as the hash for this file is stored in the cvhdmp.cat catalog file to protect it from being tampered with. So only Citrix can fix this.
In 99.9% of Citrix PVS deployments you would probably never have multiple SCSI Adapters. So I’d hit one of those rare issues that no one else has probably ever seen. But it just shows how lazy development can lead to issues for Customers.
By this time I was a little gob smacked. How do we solve the problem and remain within the realms of support? And how do I ensure these images are stable for the business?
I could have just moved the install of the SoftPerfect RAM Disk to after the Citrix PVS Target Device, which would have been the easy way. But I didn’t want to change the way my Task Sequence (automation) works just to resolve this problem. In the end I found that by manually changing the Hardware ID of the “SoftPerfect Virtual Bus” driver (ROOT\SCSIADAPTER\0000) back to “root\SPVD” before creating the PVS image resolved the problem. This meant that the…
- SoftPerfect Virtual Bus driver was running on instance ID 0000.
- Citrix Virtual Hard Disk Adapter driver was ONLY running on instance ID 0001.
So how do you automate this? I found no existing tools as part of the driver installation process that would achieve this. The only way to change it was via a direct registry edit under the “HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\ROOT\SCSIADAPTER\0000” key.
There is a trick here. The “HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum” registry key structure ONLY gives the SYSTEM account rights to modify the key, subkeys and values. I did not want to take control of the structure just to fix this. So I wrote a Citrix PVS Target Agent post install script that creates a Scheduled Task on the fly which runs as the SYSTEM account and triggers at task creation (immediately). It makes the registry changes and then the task expires and deletes itself. Problem solved!
That worked flawlessly. But when changing from the SoftPerfect RAM Disk to the Arsenal Image Mounter (AIM) I revisited this process and got a little smarter. I found that you can “reserve” instance ID 0000 just by creating the empty “HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\ROOT\SCSIADAPTER\0000” key. So I simply do this at the start of my Task Sequence, which then allows any other SCSI bus Adapters, such as the SoftPerfect Virtual Bus driver or Arsenal Image Mounter driver to install as normal and take instance ID 0001. Then, before the PVS Target Agent install, I delete the “HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\ROOT\SCSIADAPTER\0000” key, allowing the PVS Target Agent to install successfully as instance ID 0000 where it will not interfere with other devices.
This article has been in draft for some time, but I wanted to finish documenting it as an example of how deep diving on an issue and understanding the underlying mechanisms are great learning opportunities, and empowers you with the ability to work you way through and solve these issues without the need to engage the Vendors. You learn so much more this way instead of just logging a support case with them. More often than not they waste your time going through the basics, and may never actually take ownership and prioritise to fix the issue for you.
Here is the Reserve SCSI Adapter Instance ID (58 downloads) script, which is run twice during the build process. Refer to the documentation in the script header.
<#
This script will create a dummy device instance path of ROOT\SCSIADAPTER\0000 so that the new SCSI Adapter is created
on the next available instance path, reserving instance 0000 for the Citrix Virtual Hardware Disk Adapter (PVS). This
is required as the Citrix Virtual Hardware Disk Adapter is a "SCSI Controller Bully" and will forcibly use an ID of
0000, overwriting/merging into any existing device. This not only breaks that device, but also the Citrix PVS image.
The ActionToTake parameter should either be add or remove.
- To add the instance path run...
.\Reserve-SCSI-Adapter-Instance-ID.ps1 -ActionToTake:Add
- To remove the instance path run...
.\Reserve-SCSI-Adapter-Instance-ID.ps1 -ActionToTake:Remove
Script name: Reserve-SCSI-Adapter-Instance-ID.ps1
Release 1.1
Written by Jeremy Saunders (jeremy@jhouseconsulting.com) 29th September 2018
Modified by Jeremy Saunders (jeremy@jhouseconsulting.com) 14th February 2024
#>
#-------------------------------------------------------------
param(
[switch]$CreateTask=$True,
[Parameter(Mandatory = $true)]
[ValidateSet("Add", "Remove")]
[string]$ActionToTake
)
# Set Powershell Compatibility Mode
Set-StrictMode -Version 2.0
# Enable verbose, warning and error mode
$VerbosePreference = 'Continue'
$WarningPreference = 'Continue'
$ErrorPreference = 'Continue'
$StartDTM = (Get-Date)
#-------------------------------------------------------------
# Set the working folder to the users TEMP folder
$LogFolder = [System.IO.Path]::GetTempPath()
# Get the script name
$ScriptName = [System.IO.Path]::GetFilenameWithoutExtension($MyInvocation.MyCommand.Path.ToString())
# Start the transcript
try {
Start-Transcript "$LogFolder$ScriptName.log"
}
catch {
Write-Verbose "$(Get-Date -format "dd/MM/yyyy HH:mm:ss"): This host does not support transcription"
}
#------------------------------------
# Get the current script path and name
$ScriptPath = {Split-Path $MyInvocation.ScriptName}
$ScriptPath = $(&$ScriptPath)
$ScriptNamewithExtention = [System.IO.Path]::GetFilename($MyInvocation.MyCommand.Path.ToString())
#-------------------------------------------------------------
# Both XMLTime functions achieve the same output.
function XMLTime1{
# This function was posted by Thomas Lee (tfl@psp.co.uk) 24th September 2010
# - http://pshscripts.blogspot.com.au/2010/09/new-taskps1.html
Param ($T)
$csecond = $t.Second.ToString()
$cminute = $t.minute.ToString()
$chour = $t.hour.ToString()
$cday = $t.day.ToString()
$cmonth = $t.month.ToString()
$cyear = $t.year.ToString()
$date = $cyear + "-"
if ($cmonth.Length -eq 1) { $date += "0" + $cmonth + "-"}
else { $date += $cmonth + "-"}
if ($cday.length -eq 1) { $date += "0" + $cday + "T"}
else { $date += $cday + "T"}
if ($chour.length -eq 1) { $date += "0" + $chour + ":"}
else { $date += $chour + ":"}
if ($cminute.length -eq 1){ $date += "0" + $cminute + ":"}
else { $date += $cminute + ":"}
if ($csecond.length -eq 1){ $date += "0" + $csecond}
else { $date += $csecond}
return $date
}
function XMLTime2 ([datetime] $d){ $d.Touniversaltime().tostring("u") -replace " ","T"}
#-------------------------------------------------------------
If ($CreateTask) {
# The name of the scheduled task
$TaskName = "Reserve-SCSI-Adapter-Instance-ID"
# The task description
$TaskDescription = ""
# The Task Action command
#$TaskCommand = "${env:SystemRoot}\system32\WindowsPowerShell\v1.0\powershell.exe"
$TaskCommand = @(Get-Command powershell.exe)[0].Definition
# The script to be executed
$TaskScript = "$env:SystemRoot\Temp\$ScriptNamewithExtention"
# The Task Action command argument
$TaskArguments = '-Executionpolicy bypass -Command "& ' + " '" + $TaskScript + "'" +' -CreateTask:$False -ActionToTake:' + "'" + $ActionToTake + "'" + '"'
# The Task working directory
$TaskWorkingDirectory = "$env:SystemRoot\Temp"
copy-item -path "$ScriptPath\$ScriptNamewithExtention" -Destination "$env:SystemRoot\Temp\" -Recurse -Force -Verbose
# Create the TaskService object.
Try {
[Object] $service = new-object -com("Schedule.Service")
If (!($service.Connected)){
Try {
$service.Connect()
# Get a folder to create a task definition in
# This is actually the %SystemRoot%\System32\Tasks folder.
$rootFolder = $service.GetFolder("\")
# Delete the task if already present
$ScheduledTasks = $rootFolder.GetTasks(0)
$Task = $ScheduledTasks | Where-Object{$_.Name -eq "$TaskName"}
If ($Task -ne $Null){
Try {
$rootFolder.DeleteTask($Task.Name,0)
# 'Success'
}
Catch [System.Exception]{
# 'Exception Returned'
}
} Else {
# "Task Not Found"
}
# Create the new task
$taskDefinition = $service.NewTask(0)
# Create a registration trigger with a trigger type of (7) TASK_TRIGGER_REGISTRATION
$triggers = $taskDefinition.Triggers
$trigger = $triggers.Create(7)
# Set the task to expire so it can be deleted automatically
#$time = ([system.datetime]::now).addminutes(1)
#$endTime = XMLTime1($time)
#$trigger.EndBoundary = $endTime
$trigger.EndBoundary = XMLTime2 ((Get-date).addminutes(1))
# Create the action for the task to execute.
$Action = $taskDefinition.Actions.Create(0)
$Action.Path = $TaskCommand
$Action.Arguments = $TaskArguments
If ($TaskWorkingDirectory -eq "") {
$Action.WorkingDirectory = $TaskWorkingDirectory
}
# Set the settings for the task
$settings = $taskDefinition.Settings
# Set the Task Compatibility to V2 (Windows 7/2008R2)
$Settings.Compatibility = 3
# Delete the task immediately (PT0M) after the trigger expires
# as per the EndBoundary
$settings.DeleteExpiredTaskAfter = "PT0M"
# Set the privilege level
# Principal.RunLevel -- 0 is least privilege, 1 is highest privilege #
#$Principal = $taskDefinition.Principal
#$Principal.RunLevel = 1
# Register (create) the task.
# Note that the task is created as an XML file under the %SystemRoot%\System32\Tasks folder
$regInfo = $taskDefinition.RegistrationInfo
$regInfo.Description = $taskDescription
$regInfo.Author = $Env:Username
# Note that the task is created as an XML file under the %SystemRoot%\System32\Tasks folder
# 6 == Task Create or Update
# 5 == A Local System, Local Service, or Network Service account is being used as a security context to run the task.
$rootFolder.RegisterTaskDefinition($taskName, $taskDefinition, 6, "System", $null , 5) | out-null
write-verbose "Scheduled Task Created Successfully" -verbose
}
Catch [System.Exception]{
write-warning "Scheduled Task Creation Failed" -verbose
# " EXCEPTION:" $_
}
}
}
Catch [System.Exception]{
write-warning "Scheduled Task Creation Failed" -verbose
# " EXCEPTION:" $_
}
}
If (!$CreateTask) {
$Path = "HKLM:\SYSTEM\CurrentControlSet\Enum\ROOT\SCSIADAPTER\0000"
$KeyExists = $False
$ErrorActionPreference = "stop"
try {
Get-Item -Path "$Path" | Out-Null
$KeyExists = $true
}
catch {
#
}
$ErrorActionPreference = "Continue"
If ($KeyExists -eq $False -AND $ActionToTake -eq "Add") {
write-verbose "Adding new path: `"$path`"" -vebose
New-Item -Path "$path" -Force | Out-Null
}
If ($KeyExists -eq $True -AND $ActionToTake -eq "Remove") {
If ((Get-ItemProperty -Path "$Path" | Select-Object -ExpandProperty "Service") -eq $null) {
write-verbose "Removing path: `"$path`"" -vebose
Remove-Item -Path "$path" -Force | Out-Null
} Else {
write-verbose "Unable to remove as it's a valid device: `"$path`"" -verbose
}
}
Remove-item -Path "$env:SystemRoot\Temp\$ScriptNamewithExtention" -force -confirm:$false
}
#------------------------------------
Write-Verbose "Stop logging" -Verbose
$EndDTM = (Get-Date)
Write-Verbose "Elapsed Time: $(($EndDTM-$StartDTM).TotalSeconds) Seconds" -Verbose
Write-Verbose "Elapsed Time: $(($EndDTM-$StartDTM).TotalMinutes) Minutes" -Verbose
# Stop the transcript
try {
Stop-Transcript
}
catch {
Write-Verbose "$(Get-Date -format "dd/MM/yyyy HH:mm:ss"): This host does not support transcription"
}
Enjoy!
Jeremy Saunders