After installing May 2023 CU for SharePoint Server 2019 and Subscription Edition we received a couple of calls around problems with custom solution deployment.
A fix for this issue was released with July 2023 CU
Symptoms
- updating or retracting farm solutions takes a very long time
- IIS Application Pools for SharePoint web applications remain in stopped state after the update/retract operation
When the problem occurs you will find the following entries in your ULS log:
OWSTIMER.EXE ... Topology 8uda Medium Solution Deployment : Trying to stop Application pool SharePoint Central Administration v4 OWSTIMER.EXE ... General 8ude Medium Solution Deployment : Stopping application pool SharePoint Central Administration v4 OWSTIMER.EXE ... Topology 00000 Medium Stopping ApplicationPool SharePoint Central Administration v4. OWSTIMER.EXE ... Topology 00000 Medium Successfully stopped ApplicationPool SharePoint Central Administration v4. OWSTIMER.EXE ... General 5tenh Medium Solution Deployment : Error stopping application pool SharePoint Central Administration v4. Will retry. OWSTIMER.EXE ... General 8ude Medium Solution Deployment : Stopping application pool SharePoint Central Administration v4 OWSTIMER.EXE ... Topology 00000 Medium Stopping ApplicationPool SharePoint Central Administration v4. OWSTIMER.EXE ... Topology 00000 Unexpected Unable to stop ApplicationPool SharePoint Central Administration v4: System.Runtime.InteropServices.COMException (0x80070425): The service cannot accept control messages at this time... (Exception from HRESULT: 0x80070425) at ... OWSTIMER.EXE ... General 5tenh Medium Solution Deployment : Error stopping application pool SharePoint Central Administration v4. Will retry. OWSTIMER.EXE ... General 8ude Medium Solution Deployment : Stopping application pool SharePoint Central Administration v4 OWSTIMER.EXE ... Topology 00000 Medium Stopping ApplicationPool SharePoint Central Administration v4. OWSTIMER.EXE ... Topology 00000 Unexpected Unable to stop ApplicationPool SharePoint Central Administration v4: System.Runtime.InteropServices.COMException (0x80070426): The service has not been started. (Exception from HRESULT: 0x80070426) at ... OWSTIMER.EXE ... General 5tenh Medium Solution Deployment : Error stopping application pool SharePoint Central Administration v4. Will retry.
You will see similar entries for all other SharePoint application pools you have configured in your farm.
When will this happen?
The problem will occur if the SharePoint Farm Service Account is in the local administrator group of the SharePoint server machine.
Solution
To fix the issue remove the SharePoint Farm Service Account (the account assigned to central admin application pool and SharePoint Timer Service [owstimer.exe]) from the local administrator group of each SharePoint server in your farm and reboot the machine or restart all relevant services.
The SharePoint Farm Service Account should not have local administrator rights on the SharePoint server machines.
Permalink
Amazing. Yet you need it to be in the local admin group for patch updates or the user profile sync service encounters issues.
Permalink
Thanks for the update Stefan
Permalink
This solution concerns me. The “Account permissions and security settings in SharePoint Servers” article (https://learn.microsoft.com/en-us/sharepoint/install/account-permissions-and-security-settings-in-sharepoint-server-2016#sharepoint-farm-service-account) does not specifically say this account should not have local administrator rights. I agree that environments that enforce minimal permissions shouldn’t grant these rights, but I will admit that since it is not specifically called out in this article, that I do use this account in situations that require local admin. For example, I have custom timer jobs that perform functions using PowerShell that require elevated local admin permissions.
Is there another solution you can offer or must I re-architect my use of this account.
It might be helpful to update the Account Permissions article so that administrators don’t break their environment since you have pointed out this dependency.
Thanks for all you do Stefan.
Permalink
Hi Tom,
I understand your scenario.
The issue is a side effect of a fix that was released in May.
The behavior is not intentional. It was not discovered during testing as there is no requirement to add the farm service account to the local administrator group and that scenario was not tested.
A fix for this which will again allow to add the farm service account into the local administrators group is currently being evaluated and is planned to be released in July CU. Be aware that this is a planning date and the fix might also be released later.
Cheers,
Stefan
Permalink
Stefan,
Is the Farm account needed in the Local Admin group to run the Product Version Job which is run at the conclusion of the deployment.
https://blog.stefan-gossner.com/2016/08/09/sharepoint-patching-and-get-spproduct-local/
Permalink
Hi Girish,
no this is not required.
I actually never add the farm service account to the local administrator group on any of my machines.
Cheers,
Stefan
Permalink
Thanks Stefan!
I did have my app pool service account in the local admin group. Removed it and rebooted, but still taking a long time to deploy farm solutions. Those app pools still remained stopped after the deployment job finishes.
My farm account is still in the local admin users, and wondering if that might still be causing delays.
Permalink
Hi Tom,
the problem is the farm service account (account for central admin app pool and owstimer) – not the app pool account.
So your steps did not fix the issue
Cheers,
Stefan
Permalink
Just wanted to follow up. Removing the farm account from the local admin group did fix my long running deployments and also the app pool not restarting issue. Thanks Stefan!! Cheers!
Permalink
Thanks for the confirmation! 🙂
Permalink
Thanks for this information. To be clear, this condition only occurs when a custom solution is updated or removed, correct? Is there any other condition that will cause this error? Until a fix is provided for this, would simply rebooting the servers after updating/removing a custom solution resolve the issue?
Permalink
Hi John,
there might be more scenarios – these are the ones we have seen so far.
The recommended solution right now for the issue is to remove the SharePoint Farm service account from the local administrators group of the machine.
If that is not possible – e.g. because you created custom timerjobs which required local admin permission – then I would suggest to stop all application pools manually in IIS before performing the operation.
In this scenario the operation will be fast. Afterwards manually restart the application pools.
Cheers,
Stefan
Permalink
Does this issue also apply to SharePoint 2016?
Permalink
No.
Permalink
So what is this? third time in 5 or 6 months there are serious bugs in the CU patches. Our security dpt demands swift patching procedures, but I will have to start lagging a month with patches. MS maybe wants to sell SP online a tad bit too much and do this intentionally ;P
Permalink
Hi Pierre,
I hope you will understand that all code changes carry the risk of unexpected side effects.
As this scenario (having the farm account in the local administrator group) is not a required or recommended setup this issue was not uncovered during testing.
Cheers,
Stefan
Permalink
I found that you can stop application pools before the solution deployment if you use PowerShell to deploy solutions and restart the application pools at the end. This also improves the deployment speed as solution deployment spends a lot of time stopping and starting the application pools while it doesn’t need them running.
Permalink
Hello Stefan,
maybe in this context…
After installing the May 2023 Secrity Update, not the full CU, we experienced the effect in our installation that metadata column values were no longer visible to normal users in some site collections. This was because the permission entries in the TaxonomyHiddenList were removed.
Is there something to be known about this?
Many greetings
Matthias
Permalink
Hi Matthias,
I haven’t heard about such an issue. I would recommend to open a support case with Microsoft support to get this analyzed.
Cheers,
Stefan
Permalink
Stefan –
I’ve noticed a different issue on my 2019 On-Prem environment.
From the moment I installed the May 2023 patches, my system stopped creating the Trace logs. I’ve gone in to the Diagnostic logging and re-selected various categories and changed the throttling, decreasing the severity, and still nothing. It’s just not producing any logs, has not created a new one since the patching.
To be very specific, when I open the ULS Viewer, and select ‘ULS’ as the ‘open from’ option to get Real Time entries, nothing appears.
Has anyone else reported this, and do you have any suggestions to regain this functionality?
Thanks,
Keith Vollero
Permalink
Hi Keith,
I haven’t heard about such an issue. But the traces are written by the SharePoint Tracing Service. Please double check if this service is running on your server in services.msc.
Cheers,
Stefan
Permalink
Brilliantly simple. Solved the problem.
Thank you!