Trending Issue – 500 server error and application pool crashes after September 2023 CU for SharePoint

- - 38 Comments

With September 2023 CU the AMSI security feature for SharePoint Server 2016, 2019 and Subscription Edition has been automatically enabled for a customers.

After the activation of this feature a couple of customers have reported application pool crashes and 500 server errors.

Background info

The Windows Antimalware Scan Interface (AMSI) is a security feature introduced with Windows 10 and Windows Server 2016. It allows applications to pass information to an installed AMSI compatible antimalware solutions to check against common threads.

The SharePoint AMSI integration allows AMSI-capable antimalware solutions to scan HTTP and HTTPS requests that are sent to SharePoint Server. When this feature is enabled, libraries of the configured antimalware solutions are loaded into each SharePoint worker process (w3wp.exe) to perform the scanning.

Be aware that there is a second issue which can cause very similar symptoms which is discussed separately.

What is causing the stability issues?

Our investigation shows, that the issues customers reported (application pool crashes, 500 server errors, …) are caused by unhandled exceptions in the antimalware solution liberaries loaded into the SharePoint worker process when performing the analysis of the HTTP and HTTPS requests. As these exceptions are not handled by the antimalware solution they are causing these stability issues in the worker processes.

How can this be resolved?

As a workaround(!) the SharePoint AMSI integration can be disabled for each affected web application using the following PowerShell command:

Disable-SPFeature -Identity 4cf046f3-38c7-495f-a7da-a1292d32e8e9 -Url http://url-to-webapp

Important: This command disables an important security feature which was intentionally added and activated in SharePoint. This step should not be the final solution.

It is recommended that affected customers contact the antimalware solution provider to allow them to investigate and resolve the underlying compatibilty issues of the antimalware software with SharePoint and reactivate the SharePoint AMSI feature after the issue is resolved.

Microsoft is currently investigating possible solutions to prevent malfunctioning antimalware solutions from destabilizing the SharePoint worker processes.

38 Comments


    1. Hi Daniela,
      the entry should be there if the operating system is Windows Server 2016 or higher.
      Do you mean in your comment that it is inserted with Windows Server 2012 R2?
      Cheers,
      Stefan

      Reply

      1. Hi Stefan,
        it was inserted with CU 09/23 and causes our HTTP Error 500. We have Windows Server 2019 and Sharepoint 2016. When I deleted it, everything works fine.

        Grüße
        Daniela

        Reply

        1. Hi Daniela,

          in Windows Server 2019 this module is expected and should be there – it contains the implementation of the SharePoint AMSI integration.
          You were actually running into the issue I outlined here in this article and the correct workaround is to disable the feature.
          In parallel you should contact the antimalware software provider to investigate the issue.

          Cheers,
          Stefan

          Reply

    2. web.config entry I have deleted: (missing in my comment, after posting )

      add name=”SPRequestFilterModule”

      Reply

    1. Interesting – seems something went wrong in your case when the Module was registered in IIS – it omitted the path to the actual dll.

      Reply

      1. Hi Ronald,
        I just double checked – might be that in your case the following entry is missing in the applicationHost.config:

        <add name=”SPRequestFilterModule” image=”C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\16\isapi\sprequestfilteringmodule.dll” preCondition=”integratedMode,bitness64″ />

        Cheers,
        Stefan

        Reply

  3. When I enable the feature the web application starts throwing 500 errors. Failed request logging captures the following:

    \?\C:\inetpub\wwwroot\wss\VirtualDirectories\4167\web.confg (847) : Cannot add duplicate collection entry of type ‘add’ with unique key attribute ‘name’ set to ”SPRequestFilterModule’

    We are using Microsoft Defender on desktop machines. I note that you suggest contacting the vendor. Are there any online resources for this that you are aware of in relation to MS Defender compatibility problems?

    Reply

    1. Hi Clarky,
      that is interesting as it points to a completely different problem.
      The problem occurs when parsing the web.config file and loading the modules and not when invoking the antimalware scan.

      This error indicates that the same module has already been registerd before for this application – either in the same web.config or a parent web.config or possibly in the applicationHost.config.
      Cheers,
      Stefan

      Reply

    2. most likely some SPrequest modules got registered at server level in IIS and need to be removed from there.

      Reply

      1. The registration on server level is expected.
        I would avoid removing/changing module registrations without advise through support as such manipulations will create problems later when code expects these modules to be there.

        Cheers,
        Stefan

        Reply

  4. Anybody is facing issues with search after updating this CU?
    In multi server farm solution only one index partition is working, rest is marked as degradated and out-of sync status when checked via powershell

    Reply

    1. Unfortunately this CU breaks index replica synchronization.

      Additionally the noderunnerindex process will spam your ULS with tons of error messages and generate high CPU load.

      It’s not related to the topic of this post. It’s a different issue resulting from additional security hardening included in September 2023 CU for SPS SE.

      Reply

      1. Hi Tomasz,
        this is a known issue which is currently investigated by our product group.
        Cheers,
        Stefan

        Reply

    2. Hi Leszek,
      this is a known issue which is currently investigated by our product group.
      Cheers,
      Stefan

      Reply

      1. Hi Stefan,
        Are you aware if latest October CU have a fix for search issues?

        Reply

        1. October CU doesn’t have a fix for that.

          Reply

      2. Hello Stefan,
        Any updates on the search index replica issue?

        Thanks

        Reply

          1. Yes it is. Thank you for the update!


    3. Hi Leszek,

      What edition of SharePoint do you have?
      I wonder if the search issue also occurs on SP 2019.

      Cheers,
      Lucian

      Reply

      1. Only Subscription Edition is affected by the Search Issue.

        Reply

  5. We are seeing this issue on just our SPSE farms and not 2016 nor 2019 farms. We have submitted an MS support ticket and awaiting a resolution to this issue. Just as a note we have discovered that if you are running indexes on just one search server – indexes are not affected – when you have more than one search server and crawling, the indexes become degraded and only an index reset will straighten them out until you go to crawl again.

    Reply

  6. Hi Stefan,
    what happened that there is no October CU for SP 2016?
    Thanks and best regards
    Philip

    Reply

    1. Hi Philip,
      SP2016 is in extended support – means only security fixes will be released.
      If no security vulnerabilities are found that needs fixing then there will be no fix.
      Cheers,
      Stefan

      Reply

  7. Hi Stefan,

    is this failer resolved by CU October?

    Best Regards

    Reply

    1. Hi MZar,
      as you can see in the blog post this error is not caused by SharePoint but by Antimalware solutions integrating with SharePoint through the Windows AMSI feature.
      So a fix would have to come from the Antimalware solution provider. What the SharePoint product group is looking into is to prevent an application pool crash caused by exceptions in the the antimalware software.
      This investigation for such a solution is still ongoing.

      Even if this code change would be implemented it would still mean that the AMSI security feature is not functional as the problem in the antimalware solution would still happen.

      Cheers,
      Stefan

      Reply

  8. In my case problem apparently was in fact that I proactively enabled AMSI in one of my web applications. After 23H2 exactly that web application failed with HTTP 500. Rolled farm back from the backup. Next time I deactivated AMSI prior upgrade and got a success. After upgrade AMSI was enabled in all web applications.
    Problem is twofold:
    – wizard apparently didn’t properly check AMSI configuration prior enabling it forcefully (aforementioned two entries?)
    – I disabled AMSI and wizard didn’t take that into an account – it was enabled anyway. Is that a new norm between feature upgrades e.g., will it honor my choice only after feature is ‘released’ and ignore otherwise?

    … and web.config file is still getting plagued with duplicate entries after every upgrade. Like, how many times we must repeat this in it (for me – around 13 times at the moment):

    We can’t increase MaxWebConfigFileSizeInKB forever.

    Reply

    1. Hi Atis,
      about the duplicate entries in the web.config: please open a ticket with Microsoft Support to get this investigated.
      Cheers,
      Stefan

      Reply

  9. Hello,
    firstly thank you for this information it helped me when the IIS just not respoded on our environments.
    We have SharePoint Subscription Edition on Windows Server 2019 Datacenter.
    After installing September CU 2023 for SPSE we started faced another problems, primarly with Search and full ULS logs which now have 1,2 – 1,6GB and previously had like 0,5-0,7GB per 30 minutes.

    Most errors are these. Can you look at it and try to help me please. Is needed to register these libraries?
    I tried many times use SPUI config or PSConfig in SPPS, but no results.

    RequestSender[SPccf96e812c44:I.1.0]: Service invocation exception received, retrying.: Microsoft.Ceres.CoreServices.Services.Container.ServiceInvocationException: Parameter not serializable in remote invocation: Method: RequestGeneration —> System.Runtime.Serialization.SerializationException: Remote invocation server side serialization failure (service=Microsoft.Ceres.SearchCore.Services.JournalShipper.IJournalStreamControl method=RequestGeneration location=URI[s=net.tcp,h=czstrspseapp82,p=808,r=229A1A_IndexComponent2_Services_InvokerService]) —> System.ServiceModel.FaultException1[Microsoft.Ceres.CoreServices.Services.Container.SerializationFailedException]: SafeSerialization[Context=result type of RequestGeneration in Microsoft.Ceres.SearchCore.Services.JournalShipper.IJournalStreamControl]: type System.Collections.Generic.Dictionary2[[System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.UInt16, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]] not allowed. Server stack trace:
    at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc)
    at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
    at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
    at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message) Exception rethrown
    at [0]:
    at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
    at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
    at Microsoft.Ceres.CoreServices.Remoting.IInvokerService.ExecuteSerialize2(Byte[] invocation, String typeName, String methodName)
    at Microsoft.Ceres.CoreServices.Remoting.WcfRemotingServiceProxy.ExecuteSerialize(IInvokerService invoker, InvocationStream invocationStream, Type type, String methodName)
    at Microsoft.Ceres.CoreServices.Remoting.WcfServiceProxy1.InvokeDelegate[TReturn,TArgument](GenericDelegate2 genericDelegate, TArgument param, Type serviceType, String serviceMethodName, Uri serviceLocation) –
    — End of inner exception stack trace —
    at Microsoft.Ceres.CoreServices.Remoting.WcfServiceProxy1.InvokeDelegate[TReturn,TArgument](GenericDelegate2 genericDelegate, TArgument param, Type serviceType, String serviceMethodName, Uri serviceLocation)
    at Microsoft.Ceres.CoreServices.Remoting.WcfRemotingServiceProxy.ExecuteLoop(Invocation invocation) –
    — End of inner exception stack trace —
    at Microsoft.Ceres.CoreServices.Remoting.WcfRemotingServiceProxy.ExecuteLoop(Invocation invocation)
    at Microsoft.Ceres.CoreServices.Remoting.ServiceProxy.Invoke(IMessage msg)
    at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
    at Microsoft.Ceres.SearchCore.Services.JournalShipper.IJournalStreamControl.RequestGeneration(Int32 receiverId, UniqueGeneration completedGeneration)
    at Microsoft.Ceres.SearchCore.JournalReceiver.Receiver.RequestSender.RequestNextGeneration(UniqueGeneration lastGeneration)

    Microsoft.Ceres.CoreServices.Remoting.WcfServiceProxy1[[Microsoft.Ceres.CoreServices.Remoting.IInvokerService, Microsoft.Ceres.CoreServices.Remoting, Version=16.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c]] : Remote invocation server side serialization failure (service=Microsoft.Ceres.SearchCore.Services.JournalShipper.IJournalStreamControl method=RequestGeneration location=URI[s=net.tcp,h=czstrspseapp82,p=808,r=229A1A_IndexComponent2_Services_InvokerService]) System.ServiceModel.FaultException1[Microsoft.Ceres.CoreServices.Services.Container.SerializationFailedException]: SafeSerialization[Context=result type of RequestGeneration in Microsoft.Ceres.SearchCore.Services.JournalShipper.IJournalStreamControl]: type System.Collections.Generic.Dictionary2[[System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.UInt16, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]] not allowed. (Fault Detail is equal to Microsoft.Ceres.CoreServices.Services.Container.SerializationFailedException: SafeSerialization[Context=result type of RequestGeneration in Microsoft.Ceres.SearchCore.Services.JournalShipper.IJournalStreamControl]: type System.Collections.Generic.Dictionary2[[System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.UInt16, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]] not allowed.).

    SafeSerialization[Context=result type of RequestGeneration in Microsoft.Ceres.SearchCore.Services.JournalShipper.IJournalStreamControl]: type System.Collections.Generic.Dictionary2[[System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.UInt16, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]] not allowed.
    SafeSerialization[Context=result type of RequestGeneration in Microsoft.Ceres.SearchCore.Services.JournalShipper.IJournalStreamControl]: allow-listed type 'System.IntPtr' is unsafe. Reason: 'Can be used to access arbitrary unmanaged code when deserialized'.
    SafeSerialization[Context=result type of RequestGeneration in Microsoft.Ceres.SearchCore.Services.JournalShipper.IJournalStreamControl]: allow-listed type 'System.UIntPtr' is unsafe. Reason: 'Can be used to access arbitrary unmanaged code when deserialized'.
    SafeSerialization[Context=result type of RequestGeneration in Microsoft.Ceres.SearchCore.Services.JournalShipper.IJournalStreamControl]: allow-listed type 'System.EventHandler    1[[System.Runtime.Serialization.SafeSerializationEventArgs, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]]’ inherits from unsafe type ‘System.MulticastDelegate’. Reason: An arbitrary function pointer can be used to execute unexpected code..
    SafeSerialization[Context=result type of RequestGeneration in Microsoft.Ceres.SearchCore.Services.JournalShipper.IJournalStreamControl]: allow-listed type ‘System.Action1[[Microsoft.Ceres.Evaluation.Services.Distributed.GraphSegmentation, Microsoft.Ceres.Evaluation.Services, Version=16.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c]]' inherits from unsafe type 'System.MulticastDelegate'. Reason: An arbitrary function pointer can be used to execute unexpected code..
    SafeSerialization[Context=result type of RequestGeneration in Microsoft.Ceres.SearchCore.Services.JournalShipper.IJournalStreamControl]: type System.Collections.Generic.Dictionary    2[[System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.UInt16, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]] not allowed.
    Can’t serialize result of RequestGeneration() in Microsoft.Ceres.SearchCore.Services.JournalShipper.IJournalStreamControl: Blocked type: SafeSerialization[Context=result type of RequestGeneration in Microsoft.Ceres.SearchCore.Services.JournalShipper.IJournalStreamControl]: type System.Collections.Generic.Dictionary2[[System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.UInt16, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]] not allowed..
    Exception in remote service invocation (service=InvokerService, method=http://schemas.microsoft.com/ceres/runtime/2010/09/invokerservice/IInvokerService/ExecuteSerialize2, user=DOMAIN\service_tes): System.ServiceModel.FaultException    1[Microsoft.Ceres.CoreServices.Services.Container.SerializationFailedException]: SafeSerialization[Context=result type of RequestGeneration in Microsoft.Ceres.SearchCore.Services.JournalShipper.IJournalStreamControl]: type System.Collections.Generic.Dictionary2[[System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.UInt16, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]] not allowed. (Fault Detail is equal to Microsoft.Ceres.CoreServices.Services.Container.SerializationFailedException: SafeSerialization[Context=result type of RequestGeneration in Microsoft.Ceres.SearchCore.Services.JournalShipper.IJournalStreamControl]: type System.Collections.Generic.Dictionary2[[System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.UInt16, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]] not allowed.).

    Reply

    1. I am also experiencing similar errors in ULS. Did you find any solution?

      Reply

      1. MS Support tells us: Product engineering team has created a fix to address the issue which is currently being tested rigorously. The targeted date for release of this fix would be in December. This would be shipped in the December 2023 Patch.

        Reply

          1. I’m seeing similar errors in ULS logs, but I’m using SPSE with December 12, 2023 patch applied. What could cause it?

            01/03/2024 16:02:16.78 NodeRunnerIndex1-77ae689f-e6a9- (0x2AEC) 0x26C8 Search General 4vcbe Unexpected SafeSerialization[Context=parameter type of QueryExecutionFailed in Microsoft.Ceres.SearchCore.Services.Query.IIndexNodeQueryClient]: allow-listed type ‘System.Action`1[[Microsoft.Ceres.Evaluation.Services.Distributed.GraphSegmentation, Microsoft.Ceres.Evaluation.Services, Version=16.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c]]’ inherits from unsafe type ‘System.MulticastDelegate’. Reason: An arbitrary function pointer can be used to execute unexpected code.. fe0ffea0-105e-20f4-403f-c7fb15380cd8 01/03/2024 16:02:16.78 NodeRunnerIndex1-77ae689f-e6a9- (0x2AEC) 0x2E1C Search General 4vcbl Unexpected SafeSerialization[Context=parameter type of QueryExecutionFailed in Microsoft.Ceres.SearchCore.Services.Query.IIndexNodeQueryClient]: type Microsoft.Ceres.SearchCore.FastServer.FastServerPermanentErrorException not allowed. 0010fea0-1015-20f4-403f-c6922a2aa8b5


          2. Luckily SafeSerialization error resolved by restarting the server, but got a bigger problem – after december patch none of my farms understand legacy office documents (.doc in this case). You get Error initializing IFilter for extension ‘.doc’ (Error code is 0x80030109) error after touching an old document or do a full crawl. Please, fix.


          3. and SafeSerialization error is back…
            Please, fix.


  10. In the SP 2019 environment of one of our customers, we had huge performance issues after the AMSI feature was enabled. Our servers are running Windows Server 2016 with the Trellix (formerly McAfee) anti-virus solution. We noticed that the mcshield.exe process was constantly using one CPU core only, while normally it used to be mostly idle. Looks like Trellix does support AMSI and SharePoint detects this and starts using it, but the impact on performance is far too high.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.