When comparing the installation time for CUs for SharePoint Server 2019 and SharePoint Server Subscription Edition there is a significant difference.
E.g. on my test machines installing a SharePoint Server 2019 fix takes roughly 20 minutes compared to nearly an hour for a similar SharePoint Server Subscription Edition fix.
The reason for this extended install time is the fact that for SharePoint Server 2019 fixes the installer stops all relevant SharePoint and IIS Windows Services before invoking the fix installation while the services fail to be stopped for SharePoint Server Subscription Edition fixes. The root cause here is that the method used to stop the services no longer works with SharePoint Server Subscription. This leads to performance problems when the installer tries to update assemblies in the global assembly cache due files being in use.
Solution
To improve the performance the issue the relevant Windows services need to be stopped before the fix is applied and restarted afterwards.
Here is the list of services to be stopped before the fix is applied. The sequence is important to ensure that the services remain stopped and are not automatically restarted:
| Server Role | Service |
|---|---|
| All | SharePoint Timer Service (SPTimerV4) |
| All | SharePoint Tracing Service (SPTraceV4) |
| All | SharePoint Administration (SPAdminV4) |
| All | World Wide Web (W3SVC) |
| Search | SharePoint Server Search (OSearch16) |
| Search | SharePoint Search Host Controller Service (SPSearchHostController) |
| Distributed Cache | SharePoint Caching Service (SPCache) |
The reverse sequence needs to be applied when restarting the services after the SharePoint CU has been installed.
Important: Extra care needs to be taken for the Distributed Cache when using zero-downtime-patching as a graceful shutdown of the Distributed Cache host is required to guarantee that the cache data is replicated to other Distributed Cache servers still running in the farm before the service is stopped.
Solution Script
The above listed steps of stopping/restarting the services can be automated using a script. For your convenience I have created a sample script which performs the required operations:
https://github.com/stefangossner/Install-SPSE_Fix/blob/main/Install-SPSE_Fix.ps1
The script has two parameters:
- CULocation (mandatory): Here you pass in the location of the CU you are planning to install. E.g. C:\temp\uber-subscription-kb5002560-fullfile-x64-glb.exe
- ShouldGracefulStopDCache (optional): pass in $true if a graceful shutdown of the distributed cache service on the current server should be attempted,
E.g: to apply February 2024 SharePoint CU which was previously downloaded to the temp directory on a server hosting an instance of the Distributed Cache in a farm with multiple Distributed Cache Hosts where zero downtime patching is being performed:
Install-SPSE_Fix.ps1 -CULocation C:\temp\uber-subscription-kb5002560-fullfile-x64-glb.exe -ShouldGracefulStopDCache $true
Without zero downtime patching the following command should be used:
Install-SPSE_Fix.ps1 -CULocation C:\temp\uber-subscription-kb5002560-fullfile-x64-glb.exe
Permalink
Many thanks for sharing, Stefan! I have some “déjà vue” with SP2013 patching when I see the approach. 🙂
Permalink
Indeed – although the root cause there was different. In SP2013 the time delay occurred due to the large amount of packages we installed – and between each package we stopped and restarted the services which took a long time. The famous “Russ Maxwell script” disabled the services preventing them from starting and stopping leading to performance gains.
Here the problem is that the services are not stopped in the first place.
Permalink
We usually restart the server for extra precaution after the CU patch. So I guess we would still start the services in reverse order before restarting the server then. This is a good find and will quicken the patching process.
Permalink
Hi Stefan!
I need your advice on how to reduce the running time of Configuration Wizard during CU installation. I have a very large Sharepoint 2019 farm (4 servers per role), 4 SQL clusters and 1200 content databases. As a result, the Configuration Wizard on the main server (on which the CA is installed) runs for about 10 hours + about 15-20 minutes on all other servers. The total time, taking into account the installation of the files themselves, takes 15-16 hours.
Naturally, before installing the CU, I stop the search service via Suspend-SPEnterpriseSearchServiceApplication.
Is there any way to speed up the Configuration Wizard in this farm configuration?
Permalink
Hi Evgeny,
I’m pretty sure the majority of time is spent on the upgrading the 1200 content databases.
But this is something you don’t have to do in the config wizard!
My suggestion would be – before starting the config wizard, start 10+ powershell windows and run upgrade-spcontentdatabase in parallel for mulitple databases.
You can even create a Powershell script which launches multiple powershell instances in parallel to automate this.
After the content databases are upgraded you can then launch the config wizard and it will only have to upgrade the config db and the service application databases and perform the other local machine tasks.
Cheers,
Stefan
Permalink
Stefan, thanks a lot!
Yes, majority of time is spent on the upgrading the 1200 content databases – it’s information appers in the Diagnostioc Logs.
I’ll try running multiple PS-sessions next time.
Permalink
Hi Stefan, thanks for sharing. As usual you’re a life saver. I want to ask that, this situation will be fixed with future updates?
Permalink
Hi Hasan,
a fix request is currently in the backlog but we do not have information when it will be picked up.
Cheers,
Stefan
Permalink
Hi Stefan,
When using the new script on our 2 AP/3 WFE SPSE farm it has greatly improved the install time for our app servers, non-user access, crawl servers but still seeing over an hour or hour and 1/2 to install same on WFE servers. We stop user traffic via the Load balancer however we have noticed several suspended node runner processes where the PIDs correspond to the Restart Manager event log errors .
‘Unknown’ (pid 16848) cannot be restarted – Application SID does not match Conductor SID..
If stopping the OSearch16 service isn’t killing those noderunner processes is there another way to do that before installing updates?
Permalink
Hi Allison,
i just suspend the Search Service Application, run the installation and after the complete patching process am starting it again.
$ssa = Get-SPEnterpriseSearchServiceApplication -Identity “Your Search Service Application”
Suspend-SPEnterpriseSearchServiceApplication -Identity $ssa
Running the patching process and after that
Resume-SPEnterpriseSearchServiceApplication -Identity $ssa
Helped for me. It took just 30 minutes on my Single Server (with all the search components as well)
Permalink
Hi Stephan,
like your script. better than patchit.
However: I guess I found a bug
if ($srvOSearch16.Status -eq “Running” -and $restartOSearch16)
guess you mean “-ne”
Permalink
Hi,
Is this going to be fixed in future patches?
Permalink
Hi Dilan,
a fix is on the backlog but there is no information about when it might be implemented available at this time.
Cheers,
Stefan
Permalink
Thanx’s for the great script! The Scripts install the CU on our SP Farm Server in approx 1hr. That is 2jr faster.
Permalink
Hello,
it does go faster, but on some servers there must be still something stopping the install as it stalls for quite a while.
For example on a WFE, we saw the same RestartManager (eventid 10010) as Alison wrote about above. In our case the pid was for the powershell window that launched the install -script:
Application ‘C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe’ (pid 11660) cannot be restarted – Application SID does not match Conductor SID..
In eventviewer we can see these type of events for almost an hour, then eventuelly it succeeds.
Hope we can find the cause for these issues are patching many servers is very time consuming now.
Thanks,
David