There is a known bug where PowerShell does not correctly manage a garbage collection whilst executing a pipeline or loop of an object.
Simply using [System.GC]::Collect() within the pipeline or loop does not work as expected. Memory continually grows until the pipeline or loop has completed. This becomes a serious problem if you’re script is processing large objects. You can potentially exhaust memory resources and your script will fail with out of memory errors. This has been driving me nuts for years, as many of my Active Directory Health Check, Audit and Remediation Scripts process large objects in large environments.
There is a good overview of the bug here: No garbage collection while PowerShell pipeline is executing. Whilst this post claims that it seems to have been resolved in PowerShell 5, this doesn’t appear to be the case from my testing.
Here’s what was happening to powershell.exe after processing a couple of hundred users using my script to create a Kerberos Token Size Report…
It continually grows…and grows…and grows.
Just by chance, and trying to add some smarts to my scripts for memory usage reporting, I found a way around this. Simply use…
[System.GC]::GetTotalMemory(‘forcefullcollection’) | out-null
or
[System.GC]::GetTotalMemory($true) | out-null
Where $true or ‘forceFullCollection’ is used to indicate that this method can wait for garbage collection to occur before returning.
Even though [System.GC]::Collect() does not force a garbage collection whilst executing in a pipeline or loop of an object, [System.GC]::GetTotalMemory() with either $true or ‘forceFullCollection’ does indeed successfully force a garbage collection. Wow! Go figure! My testing so far has found this to be a reliable method to use across different versions of PowerShell.
So now here is what happens to powershell.exe after processing a couple of hundred users using the same script…
A massive difference in memory usage, which doesn’t really grow too much further even after processing tens of thousands of users in the object.
I’ve actually taken this one step further and have started to add a function to my scripts to dynamically report on memory allocated to the managed components. It only reports the bytes allocated by the PowerShell scripting engine and does not account for the memory used by the PowerShell engine itself as can be seen in Task Manager. But I really like this function:
- The PowerShell implementation dynamically gets the memory consumed by the current script runtime
- Get Memory Consumption
This is one of the many vagaries of PowerShell that’s been annoying me for a very long time.
I hope this helps anyone else struggling with the same challenges.
Happy scripting 🙂
Jeremy Saunders