Trending Issue: Problem when updating Farm Solutions after installing May 2023 CU

After installing May 2023 CU for SharePoint Server 2019 and Subscription Edition we received a couple of calls around problems with custom solution deployment.

A fix for this issue was released with July 2023 CU

Symptoms

  • updating or retracting farm solutions takes a very long time
  • IIS Application Pools for SharePoint web applications remain in stopped state after the update/retract operation

When the problem occurs you will find the following entries in your ULS log:

OWSTIMER.EXE ...   Topology   8uda    Medium      Solution Deployment : Trying to stop Application pool SharePoint Central Administration v4
OWSTIMER.EXE ...   General    8ude    Medium      Solution Deployment : Stopping application pool SharePoint Central Administration v4
OWSTIMER.EXE ...   Topology   00000   Medium      Stopping ApplicationPool SharePoint Central Administration v4.
OWSTIMER.EXE ...   Topology   00000   Medium      Successfully stopped ApplicationPool SharePoint Central Administration v4.
OWSTIMER.EXE ...   General    5tenh   Medium      Solution Deployment : Error stopping application pool SharePoint Central Administration v4. Will retry.
OWSTIMER.EXE ...   General    8ude    Medium      Solution Deployment : Stopping application pool SharePoint Central Administration v4
OWSTIMER.EXE ...   Topology   00000   Medium      Stopping ApplicationPool SharePoint Central Administration v4.
OWSTIMER.EXE ...   Topology   00000   Unexpected  Unable to stop ApplicationPool SharePoint Central Administration v4: System.Runtime.InteropServices.COMException (0x80070425): The service cannot accept control messages at this time... (Exception from HRESULT: 0x80070425)     at ...
OWSTIMER.EXE ...   General    5tenh   Medium      Solution Deployment : Error stopping application pool SharePoint Central Administration v4. Will retry.
OWSTIMER.EXE ...   General    8ude    Medium      Solution Deployment : Stopping application pool SharePoint Central Administration v4
OWSTIMER.EXE ...   Topology   00000   Medium      Stopping ApplicationPool SharePoint Central Administration v4.
OWSTIMER.EXE ...   Topology   00000   Unexpected  Unable to stop ApplicationPool SharePoint Central Administration v4: System.Runtime.InteropServices.COMException (0x80070426): The service has not been started. (Exception from HRESULT: 0x80070426)     at ...
OWSTIMER.EXE ...   General    5tenh   Medium      Solution Deployment : Error stopping application pool SharePoint Central Administration v4. Will retry.

You will see similar entries for all other SharePoint application pools you have configured in your farm.

When will this happen?

The problem will occur if the SharePoint Farm Service Account is in the local administrator group of the SharePoint server machine.

Solution

To fix the issue remove the SharePoint Farm Service Account (the account assigned to central admin application pool and SharePoint Timer Service [owstimer.exe]) from the local administrator group of each SharePoint server in your farm and reboot the machine or restart all relevant services.
The SharePoint Farm Service Account should not have local administrator rights on the SharePoint server machines.

22 Comments


  1. Amazing. Yet you need it to be in the local admin group for patch updates or the user profile sync service encounters issues.

    Reply

  2. This solution concerns me. The “Account permissions and security settings in SharePoint Servers” article (https://learn.microsoft.com/en-us/sharepoint/install/account-permissions-and-security-settings-in-sharepoint-server-2016#sharepoint-farm-service-account) does not specifically say this account should not have local administrator rights. I agree that environments that enforce minimal permissions shouldn’t grant these rights, but I will admit that since it is not specifically called out in this article, that I do use this account in situations that require local admin. For example, I have custom timer jobs that perform functions using PowerShell that require elevated local admin permissions.

    Is there another solution you can offer or must I re-architect my use of this account.

    It might be helpful to update the Account Permissions article so that administrators don’t break their environment since you have pointed out this dependency.

    Thanks for all you do Stefan.

    Reply

    1. Hi Tom,
      I understand your scenario.
      The issue is a side effect of a fix that was released in May.
      The behavior is not intentional. It was not discovered during testing as there is no requirement to add the farm service account to the local administrator group and that scenario was not tested.
      A fix for this which will again allow to add the farm service account into the local administrators group is currently being evaluated and is planned to be released in July CU. Be aware that this is a planning date and the fix might also be released later.
      Cheers,
      Stefan

      Reply

    1. Hi Girish,
      no this is not required.
      I actually never add the farm service account to the local administrator group on any of my machines.
      Cheers,
      Stefan

      Reply

  3. Thanks Stefan!

    I did have my app pool service account in the local admin group. Removed it and rebooted, but still taking a long time to deploy farm solutions. Those app pools still remained stopped after the deployment job finishes.

    My farm account is still in the local admin users, and wondering if that might still be causing delays.

    Reply

    1. Hi Tom,
      the problem is the farm service account (account for central admin app pool and owstimer) – not the app pool account.
      So your steps did not fix the issue
      Cheers,
      Stefan

      Reply

      1. Just wanted to follow up. Removing the farm account from the local admin group did fix my long running deployments and also the app pool not restarting issue. Thanks Stefan!! Cheers!

        Reply

        1. Thanks for the confirmation! 🙂

          Reply

  4. Thanks for this information. To be clear, this condition only occurs when a custom solution is updated or removed, correct? Is there any other condition that will cause this error? Until a fix is provided for this, would simply rebooting the servers after updating/removing a custom solution resolve the issue?

    Reply

    1. Hi John,
      there might be more scenarios – these are the ones we have seen so far.
      The recommended solution right now for the issue is to remove the SharePoint Farm service account from the local administrators group of the machine.
      If that is not possible – e.g. because you created custom timerjobs which required local admin permission – then I would suggest to stop all application pools manually in IIS before performing the operation.
      In this scenario the operation will be fast. Afterwards manually restart the application pools.
      Cheers,
      Stefan

      Reply

  5. Does this issue also apply to SharePoint 2016?

    Reply

  6. So what is this? third time in 5 or 6 months there are serious bugs in the CU patches. Our security dpt demands swift patching procedures, but I will have to start lagging a month with patches. MS maybe wants to sell SP online a tad bit too much and do this intentionally ;P

    Reply

    1. Hi Pierre,
      I hope you will understand that all code changes carry the risk of unexpected side effects.
      As this scenario (having the farm account in the local administrator group) is not a required or recommended setup this issue was not uncovered during testing.
      Cheers,
      Stefan

      Reply

  7. I found that you can stop application pools before the solution deployment if you use PowerShell to deploy solutions and restart the application pools at the end. This also improves the deployment speed as solution deployment spends a lot of time stopping and starting the application pools while it doesn’t need them running.

    Reply

  8. Hello Stefan,
    maybe in this context…
    After installing the May 2023 Secrity Update, not the full CU, we experienced the effect in our installation that metadata column values were no longer visible to normal users in some site collections. This was because the permission entries in the TaxonomyHiddenList were removed.
    Is there something to be known about this?
    Many greetings
    Matthias

    Reply

    1. Hi Matthias,
      I haven’t heard about such an issue. I would recommend to open a support case with Microsoft support to get this analyzed.
      Cheers,
      Stefan

      Reply

  9. Stefan –

    I’ve noticed a different issue on my 2019 On-Prem environment.
    From the moment I installed the May 2023 patches, my system stopped creating the Trace logs. I’ve gone in to the Diagnostic logging and re-selected various categories and changed the throttling, decreasing the severity, and still nothing. It’s just not producing any logs, has not created a new one since the patching.

    To be very specific, when I open the ULS Viewer, and select ‘ULS’ as the ‘open from’ option to get Real Time entries, nothing appears.

    Has anyone else reported this, and do you have any suggestions to regain this functionality?

    Thanks,
    Keith Vollero

    Reply

    1. Hi Keith,
      I haven’t heard about such an issue. But the traces are written by the SharePoint Tracing Service. Please double check if this service is running on your server in services.msc.
      Cheers,
      Stefan

      Reply

      1. Brilliantly simple. Solved the problem.
        Thank you!

        Reply

Leave a Reply to Matt Cancel reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.