Understanding Machine Key Rotation using the new automatic SharePoint timerjob

- - 56 Comments

With the September 2025 CU, SharePoint introduced an automatic machine key rotation timer job to enhance security by periodically updating the ASP.NET machine keys used for view state encryption and validation. This post explains how the job works, how keys are updated in memory, and how you can monitor and troubleshoot the process.

How the Automatic Machine Key Rotation Job works

The automatic machine key rotation job is designed to improve security while minimizing downtime and performance impact. Here’s the detailed process:

  1. Timer Job Execution
    • Runs weekly by default (configurable via Central Administration)
    • Generates a new set of machine keys (validationKey and decryptionKey) for the entire farm
  2. Centralized Storage in Configuration Database
    • Instead of writing keys to each web.config, the job updates the configuration database.
    • This ensures centralized management and avoids triggering application domain restarts.
  3. Propagation to Configuration Cache
    • Each SharePoint server retrieves the updated keys from the configuration database and stores them in its local configuration cache.
  4. In-Memory Update in Worker Processes
    • The IIS worker processes (w3wp.exe) monitor the configuration cache.
    • When new keys are detected, they update their in-memory configuration without requiring an IIS reset or app pool recycle.

Why Don’t We See Changes in web.config?

This is by design. The Machine Key Rotation Job does not modify the web.config file. The keys are updated in the configuration database and later retrieved and processed directly in the w3wp.exe worker process, which prevents performance problems through application domain recycles.
If you check the last modified date of the web.config file after the job runs, it will remain unchanged.

What Happens After Encryption Keys Are Updated?

Once the new keys are active in memory, all new requests use them for view state encryption and validation.

If a user had an open page before the rotation, the browser may still hold view state encrypted with the old key. Submitting that page can cause invalid view state errors.
Solution: Refresh the page to load a new view state encrypted with the current keys.

Manual Rotation vs. Automatic Rotation

  • Automatic Job: Updates keys in memory only. No web.config changes.
  • Manual Rotation:
    • Use Set-SPMachineKey to generate new keys and update web.config files.
    • If required use Update-SPMachineKey to replicate existing keys across the web.config files on all servers in the farm (does not generate new keys).

Troubleshooting

Using PowerShell

To verify the current machine keys for all web applications in the configuration cache and in the web.config, use the following PowerShells script I created:

Important:

  • After a machine key rotation was done using the Machine Key Rotation job it is expected that the machine keys emitted by both scripts are different, as the timer job DOES NOT update the web.config files.
  • After a machine key rotation was done using the Set-SPMachineKey PowerShell CmdLet it is expected that the machine keys emitted by both scripts are identical for the web application where the new machine keys were generated as the CmdLet DOES update the web.config file.
  • Update-SPMachineKey is irrelevant after the Machine Key Rotation job has been executed at least once in the farm as it will update the web.config machine keys which are no longer being used.

The Machine Key Rotation Job Timer job can found in the Central Administration Website and can also be displayed using PowerShell:

get-sptimerjob | ? {$_.Name -like "job-config-machine-key-rotation"} | fl

The Configuration Refresh Timer job which replicates the content to all servers in the farm is a hidden system timer job which is NOT visible in the Central Administration website – but you can find it using PowerShell:

get-sptimerjob | ? {$_.Name -like "job-config-refresh"} | fl

ULS Log Monitoring

When machine keys are rotated using the Machine Key Rotation job, SharePoint logs specific events in the ULS logs.
Below is the sequence of events you should find in the ULS log for a successful machine key rotation.

Information:
To troubleshoot machine key rotation issues, it’s essential to collect a merged log from all servers. Since different operations occur on different servers, a combined log allows you to view the entire sequence in a single file.A merged log can be generated using the following PowerShell command:

merge-splogfile -path "C:\temp\FarmMergedLog.log" -StartTime "MM/DD/YYYY HH:MM" -EndTime "MM/DD/YYYY HH:MM"

See here for more possible options for this command:

To analyze and filter ULS logs use UlsViewer

Information:
The GUID following MachineKeyConfiguration_ in some of the messages below represents the ID of the web application for which a new Machine Key was generated.
You can retrieve the Web Application ID by running the following PowerShell command:

(get-spwebapplication http://url-to-web-app).Id
Machine Key Rotation Timer Job

This timer job runs once a week by default. It generates new machine keys (for encryption and validation) and stores them in the configuration database.
The job executes on a single server in the farm, which must have the AllowServiceJobs property set to True for its TimerService instance.
You can check this by running the following PowerShell command:

(get-spfarm).TimerService.Instances | select Server,AllowServiceJobs
  • Indication that the Machine Key Rotation Job was started 
    Event ID: xmnv
Level: Medium
Process: OWSTIMER.EXE
Category: Logging Correlation Data
Message: Name=Timer Job job-config-machine-key-rotation
  • Indication for successful generation of new machine keys for a web application 
    Event ID: 4l4ri
Level: High
Process: OWSTIMER.EXE
Category: Runtime
Message: MachineKeyConfiguration_10110dbf-8776-4721-b4b2-abf8895bb5bd password changed. Needs update.
  • Indication for successfull update of machine keys for a web application in the configuration database 
    Event ID: 4l4rh
Level: High
Process: OWSTIMER.EXE
Category: Runtime
Message: Successfully persisted MachineKeyConfiguration_10110dbf-8776-4721-b4b2-abf8895bb5bd credential.
Configuration Refresh Timer Job

This timer job copies configuration data from the configuration database to the local configuration cache on each server in the farm.
It runs every 15 seconds on ALL servers in the farm.

If this timerjob does NOT run on all machines in the farm it might be that the TimerService instance is disabled on some servers in the farm.

You can check if all TimerService Instances are only by running the following PowerShell command:

(get-spfarm).TimerService.Instances

If you find a server where the TimerService instance is not Online you need to set it Online using this command:

(get-spfarm).TimerService.Instances |? {$_.Status -ne "Online"} |% {$_.Status = "Online"; $_.Update()}
  • Indication that the Configuration Refresh Timer Job was started 
    Event ID: bjgst
Level: Medium
Process: OWSTIMER.EXE
Category: Config Cache
Message: SPConfigurationRefreshJobDefinition.Execute: Refreshing timer config cache with refreshCacheFlags=[Default, SkipSetFlagOnFailure], newestVersion=[15186],
  • Indication that the machine keys for a web application have been replicated to the local filesystem configuration cache 
    Event ID: a3c26
Level: Medium
Process: OWSTIMER.EXE
Category: Config Cache
Message: Entering Monitored Scope (PutObjectInFileSystem: Storing object [SPSecureDBCredential Name=MachineKeyConfiguration_10110dbf-8776-4721-b4b2-abf8895bb5bd] with id ...
Machine Key update in the IIS worker process.

At the start of each HTTP request, the system checks whether the machine keys for the current web application have changed in the configuration cache. If changes are detected, the keys are updated in memory within the IIS worker process (w3wp.exe).

  • Indication that Machine keys in the worker process were replaced with the new machine keys from the configuration cache 
    Event ID: 7cvqt
Level: Medium
Process: W3WP.EXE
Category: Runtime
Message: Successfully updated machine key.

If any of these events is missing, investigate further by reviewing potential error messages.

56 Comments


  1. A nice article. Thank you!

    Please fix:
    – web application Id example is missing an open bracket;
    – “Indication that the Configuration Refresh Timer Job was started” appears twice.

    Reply

    1. Thanks Atis!
      Fixed both errors. 🙂

      Reply

  2. This is awesome. Thank you so much for putting everything “Machine keys” at one place.

    Reply

  3. Hi Stefan,

    We jumped over the september 2025 CUs because of all the problems reported. We started to update our first server with the october 2025 CUs. It went well. This server is a sole one that has all the roles in his farm (we’ll update our larger farms eventually).

    I searched the events that you described to see if all was ok with machine key rotation on this server.

    I can find the events about (so that’s fine) :
    Indication that the Machine Key Rotation Job was started
    Indication for successful generation of new machine keys for a web application (4 times, we have 4 web apps)
    Indication for successfull update of machine keys for a web application in the configuration database (4 times, we have 4 web apps)

    I cannot find these ones :
    Indication that the Configuration Refresh Timer Job was started
    Indication that Machine keys in the worker process were replaced with the new machine keys from the configuration cache

    I think that it’s probably because i have only one server in the farm ? Could you confirm this ?

    Thanks !

    Jeff

    Reply

      1. Thanks Stefan for the quick reply.

        I tested the two scripts on 6 servers (standalone farms, one server in each farm) and they work well on 4 of them but for two, the script DumpWebConfigMachineKeys.ps1 works fine, but the script DumpConfigDBMachineKeys.ps1 (config cache) returns the error :
        Object reference not set to an instance of an object.

        For the script DumpWebConfigMachineKeys.ps1, i had to adjust the web root paths (because we don’t use the default C:\Inetpub) for it to work but it does.

        For the script DumpConfigDBMachineKeys.ps1, would there be a way to troubleshoot ?

        Thanks again !

        Jeff

        Reply

        1. Hi Jeff,
          interesting – sounds to me as if on these machines the machine keys have not arrived.
          This is most likely the line:

          string bothKeys = cred.Password;

          if cred is null you will get the above error.

          I have updated the script. Please download it from scratch and run it in a new PowerShell windows.
          Should now tell you if my assumption is correct.

          Cheers,
          Stefan

          Reply

          1. I downloaded and ran the new version of the script and you are correct, it returns for each web app :
            No machine keys found for web application in config cache

            you assumption is spot on.

            Jeff


          2. Hi Jeff,
            this means that the config-refresh timerjob does not run on this machine.
            This can happen if the TimerService Instance on this machine is Offline.
            Please check using this command:
            (get-spfarm).TimerService.Instances

            I have updated the article above to includes this check and also the command to fix it if required.
            Cheers,
            Stefan


  4. Hi Stefan,

    On both servers where i get the error :
    No machine keys found for web application in config cache

    If i check for the TimerService instance, it is online (on both servers) :
    PS D:\Applications\Scripts> (get-spfarm).TimerService.Instances

    TypeName Status Id
    ——– —— —
    Microsoft SharePoint Foundati… Online 52871029-52c1-4447-a40c-7a014e83c919

    Jeff

    Reply

    1. Hi Jeff,
      please double check service.msc that the SharePoint Timer Service is running.
      If yes, I would recommend to open a support case with Microsoft to get this investigated.
      Cheers,
      Stefan

      Reply

  5. The SharePoint Timer Service is ok on both servers.

    On one of the two servers, i manually executed the timer job and afterward, the DumpConfigDBMachineKeys.ps1 script shows me the keys for each web app.

    For the other server, since we updated it yesterday, i’ll wait till next wednesday to check if the timer Job was run and if not, i’ll open a case with Microsoft as you suggested.

    Jeff

    Reply

    1. Hi Jeff,
      the machine key rotation job will always only run on one server in the farm. And not necessarily the server where you started it from. It is the config refresh job which replicates the details from the config db to the local machines config cache. The script check only the config cache on the file system – not the info in the config db itself. The config refresh job runs every 15 seconds. So if it is not there now, I do not see how it will be there next Wednesday.
      Cheers,
      Stefan

      Reply

      1. My 6 servers each have their own farm (one server in his own farm with all the roles). So 6 separate farms with one server in each.

        So the Timer Job should at least run once on each server and for now, it says that it never ran (“Non applicable” for the last execution time).

        It should run automatically till the end of the week.

        Jeff

        Reply

  6. What is surprising is that when i check the “Rotate machine keys in config DB” Timer Job on each server that was updated yesterday or last week with the october 2025 CUs, it says “Non applicable” for the last execution time for every server, as if the Timer Job never ran once.

    But on some servers (4 out of 6), i have machine keys displayed by your scripts.

    I do have to say that in july/august, when we installed the critical update, we also rotated the machine keys manually in the web.config of the web apps (maybe we forgot to do it on some servers).

    Jeff

    Reply

    1. Hi Jeff,
      manual key rotation before September CU used a different method and stored the info differently.
      Cheers,
      Stefan

      Reply

  7. Hi Stefan,

    It appears that I am running into issues with automatic Machine Key Rotation after installing October 2025 CU in a SharePoint 2016 environment.
    I plan to create a Microsoft support ticket. But decided to add a comment here as well – maybe you or others have faced similar issues.

    About the environment:
    – SharePoint 2016
    – Previous update level: August 2025 CU (September 2025 CU was skipped due to its issues)
    – 5 SP servers (2 of them are WFEs), 1 SQL server
    – The environment is mostly configured as per good practices. There are few areas for improvements, but the main things should be configured correctly – including the ones that would have been required to install September 2025 CU.

    No issues were observed directly after October 2025 CU was installed.
    But after the weekend (I assume – after the weekly Timer Job “Machine Key Rotation Job” was run) view state errors started to occur in UI when users where modifying document properties.

    In Windows Event Logs the following was showing up:

    Exception message: Unable to validate data.
    at System.Web.Configuration.MachineKeySection.EncryptOrDecryptData(Boolean fEncrypt, Byte[] buf, Byte[] modifier, Int32 start, Int32 length, Boolean useValidationSymAlgo, Boolean useLegacyMode, IVType ivType, Boolean signData)
    at System.Web.UI.Page.DecryptString(String s, Purpose purpose)

    at System.Web.Handlers.AssemblyResourceLoader.System.Web.IHttpHandler.ProcessRequest(HttpContext context)

    As it is a Production environment, then after initial investigation I decided to rotate Machine Keys manually (Set-SPMachineKey + IISRESET) in order to potentially have a temporary fix as soon as possible.
    It helped – the environment is working as expected now.
    But I am afraid that it may stop working again after next weekend when the Timer Job “Machine Key Rotation Job” runs. I could disable the Timer Job and it may help for now, but we are seeking for a permanent solution.

    Notes about the investigation so far:
    1) Another SharePoint 2016 environment (Test – 1 SP, 1 SQL) was also updated to October 2025 CU (previous update – August 2025 CU). It is configured as per good practices. No issues are observed in this environment. I am mentioning it as I will reference this environment below as well.
    2) Info: For all TimerService Instances “AllowServiceJobs” is “True” and “Status” is “Online”.
    3) About logs related to the Timer Job “Machine Key Rotation Job” – all expected entries are present in the Test environment. But in Production there is only the Timer Job start entry as expected. Further for the Timer Job and correlation token there are only several errors similar to the below. Note that the Timer Job completed successfully (when checking it in Central Admin).
    –
    SPLoggingLock held lock for 2391 milliseconds. Call stack:
    at Microsoft.SharePoint.Utilities.SPLoggingLockTag.Dispose()
    at Microsoft.SharePoint.Administration.SPPersistedObjectCollectionCache.InvalidateCache(SPConfigurationDatabase configDb, Guid objectId, Type objectType, SPPersistedObjectChangeEvent changeType, Boolean updateFileCache)
    at Microsoft.SharePoint.Administration.SPConfigurationDatabase.Microsoft.SharePoint.Administration.ISPPersistedStoreProvider.InvalidateCollectionCache(Guid objectId, Type objectType, SPPersistedObjectChangeEvent changeType)
    at Microsoft.SharePoint.Administration.SPPersistedObject.BaseUpdateCore(Boolean legacyUpdate)
    at Microsoft.SharePoint.Administration.SPEncryptedString.UpdateSecureStringValue(SecureString value)
    at Microsoft.SharePoint.Administration.SPSecureDBCredential.Update()
    at Microsoft.SharePoint.Administration.MachineKeyPersistedConfiguration.Persist(String validationKey, String decryptionKey, Guid webAppID, Boolean validateKeys)
    at Microsoft.SharePoint.Administration.SPMachineKeyRotationJobDefinition.Execute(Guid targetInstanceId)
    at Microsoft.SharePoint.Administration.SPTimerJobInvokeInternal.Invoke(SPJobDefinition jd, Guid targetInstanceId, Boolean isTimerService, Int32& result)
    at Microsoft.SharePoint.Administration.SPTimerJobInvoke.Invoke(TimerJobExecuteData& data, Int32& result)
    –
    SPLoggingLock waited 3359 milliseconds to acquire lock. Call stack:
    at Microsoft.SharePoint.Utilities.SPLoggingLockTag..ctor(UInt32 tagID, Object obj)
    at Microsoft.SharePoint.Administration.SPPersistedObjectCollectionCache.InvalidateCache(SPConfigurationDatabase configDb, Guid objectId, Type objectType, SPPersistedObjectChangeEvent changeType, Boolean updateFileCache)
    at Microsoft.SharePoint.Administration.SPConfigurationDatabase.Microsoft.SharePoint.Administration.ISPPersistedStoreProvider.InvalidateCollectionCache(Guid objectId, Type objectType, SPPersistedObjectChangeEvent changeType)
    at Microsoft.SharePoint.Administration.SPPersistedObject.BaseUpdateCore(Boolean legacyUpdate)
    at Microsoft.SharePoint.Administration.SPSecureDBCredential.Update()
    at Microsoft.SharePoint.Administration.MachineKeyPersistedConfiguration.Persist(String validationKey, String decryptionKey, Guid webAppID, Boolean validateKeys)
    at Microsoft.SharePoint.Administration.SPMachineKeyRotationJobDefinition.Execute(Guid targetInstanceId)
    at Microsoft.SharePoint.Administration.SPTimerJobInvokeInternal.Invoke(SPJobDefinition jd, Guid targetInstanceId, Boolean isTimerService, Int32& result)
    at Microsoft.SharePoint.Administration.SPTimerJobInvoke.Invoke(TimerJobExecuteData& data, Int32& result)
    –
    4) The hidden “Config Refresh” (“job-config-refresh”) Timer Job is present (accessible via PowerShell), but the property “LastRunTime” is displayed as “01.01.0001 00.00.00”. It is so in both Test and Production environments. Also, in none of the environments I see traces of the Timer Job running in SharePoint Logs. So, I am not sure if has been running. But note that the Test environment does not show any issues.
    5) The log entry “Successfully updated machine key” by “w3wp.exe” is present in the Test environment (that does not have issues). But it is not present in the Production environment (that has issues).
    6) Machine Keys in the Production environment (the problematic one) were present in the config cache (checked via your script) while the issue was happening. I am mentioning it because some persons in the comments have mentioned that Machine Keys didn’t exist for them in config cache.
    7) As described above, I used manual Machine Key rotation (Set-SPMachineKey + IISRESET) to temporarily fix the issue. I was expecting (based on your article) that after this activity Machine Keys in “web.config” files and config cache would be the same. However, they are different. I am not sure if this is as expected. But at least the issues are no longer present after this temporary workaround.
    8) I consider clearing SharePoint Configuration Cache (C:\ProgramData\Microsoft\SharePoint\Config\GUID) and running the automatic Machine Key rotation Timer Job again. Perhaps it works afterwards. However, as it is a world-wide Production environment, I could plan such test only during a weekend.

    If you have any comments or suggestions, then please let me know.

    Regards,
    Martins Grube

    Reply

    1. Hi Martins,
      please send me the case number after you have opened the ticket with Microsoft using “Contact the blog author” at the top right of this page.
      If I have sufficient bandwidth I will take ownership. Of course cannot promise it at this time. 🙂
      Cheers,
      Stefan

      Reply

    2. Thank you Stefan Gossner for the summary above—the mechanism is clearly described and a huge help with troubleshooting.

      I observed a similar problem in a four-server SharePoint 2019 farm after updating from CU July 21/2025 directly to CU October/2025.
      The two timer jobs run history is not visible in the job history in Central Administration (yes, I know, one of them is hidden) and when queried via PowerShell, they show a last run time of 01.01.0001. However, the ULS log entries above checked show that the jobs did run according to their schedule.
      The result after the first Sunday after patching was a farm with ViewState errors for users, and each of the four farm servers had a different MachineKey in their config cache.
      Only a manual restart of the “Machine Key Rotation” timer job, clearing the config cache on all four servers, followed by IISRESET on each server fixed the problem – after a few minutes, all servers had the same MachineKey in their config cache again.
      It looks as if the rotated MachineKeys were not correctly transferred to the config caches of the individual farm servers. Interestingly, the problem did not occur on the identical test system, only on the customer’s production system. Has anyone else observed this behavior?

      Reply

      1. Another Sunday has passed, and once again, automatic machine key rotation has left things in an inconsistent state. This time, only one of four servers was affected—fortunately, it was just a search server that clients don’t browse. Deleting the config cache on the affected server followed by an IIS reset solved the problem. I will take a closer look at the ULS logs in the coming days. In any case, it is unfortunate that the automatic machine key rotation cannot be relied upon at present.

        Reply

    3. Just an update on the above case:
      1) As a temporary workaround, we have disabled the Timer Job “Machine Key Rotation Job”. Therefore, we are not running into the ViewState problems after every weekend.
      2) This Friday (November 21) evening we are planning to clear SharePoint Configuration Cache and restart the SharePoint environment – to have a clean state. We haven’t done it yet as it is a Production environment used across different time zones and also during weekends. Perhaps this helps. After it we will enable the Timer Job “Machine Key Rotation Job” and run it. If it succeeds, then we would continue monitoring for several weekends to ensure that the issue does not repeat.
      3) We have a Microsoft support ticket open, but there is no real progress yet (organizing the first call).

      Reply

    4. Update on my above case – I tried clearing SharePoint Configuration Cache (C:\ProgramData\Microsoft\SharePoint\Config\GUID), restarting the environment and running the automatic Machine Key rotation Timer Job again.

      Unfortunately it didn’t help – the Timer Job “Machine Key Rotation Job” succeeded in general, but in SharePoint Logs it generated errors during the execution (the same as before – described in the above post). And view state related errors started to occur in SharePoint UI.

      As a temporary workaround, I again did the following:
      1) Rotate Machine Keys manually (Set-SPMachineKey + IISRESET) – to make sure that the environment becomes functional again
      2) Disabled the Timer Job “Machine Key Rotation Job” – so the automatic Machine Key rotation does not happen on the nights to Sundays and we don’t run into the issue again

      The Microsoft ticket is updated with this info as well – waiting for feedback.

      Reply

        1. Hi Stefan,

          About errors in the ULS logs – in order not to make this comment long, I am adding reference to my original comment: https://blog.stefan-gossner.com/2025/10/24/understanding-machine-key-rotation-using-the-new-automatic-sharepoint-timerjob/#comment-46056. It contains the errors. The same errors happened on a repeated attempt to run the Timer Job “Machine Key Rotation Job”.

          About the Machine Key scripts – I didn’t execute them this time. But I did execute them when we got the issue originally. Machine Keys were present and I didn’t notice any issues with them. Here is a note I had made originally about it:

          Machine Keys in the Production environment (the problematic one) were present in the config cache (checked via your script) while the issue was happening. I am mentioning it because some persons in the comments have mentioned that Machine Keys didn’t exist for them in config cache.

          Reply

          1. Thanks Martins!
            Thats indeed an interesting issue. The error occurs in the System.Web.Configuration.MachineKeySection.EncryptOrDecryptData class which belongs to IIS.
            So the support case will most likely require a collaboration between the SharePoint and IIS support team.
            Cheers,
            Stefan


          2. Hi Martins!

            Did you get any solution for this via your Microsoft case?

            We are getting exactly the same problems in a SP2016 farm. The Rotation job runs as expected, but the Refresh jobs won’t run (they have a LastRunTime of “0001-01-01 00:00:00”).

            Thanks!


          3. Hi Will,
            the config refresh job is a system job. It always shows this LastRunTime. So this is not an indication that the job did not run.
            Please check the ULS log for the bjgst and a3c26 events mentioned in the article above.
            Cheers,
            Stefan


          4. Hi Will,

            My issue is not yet resolved – the Microsoft case is in progress.

            The current plan is the following:
            1) Install the newly released January 2026 CU in the environment (preliminary planned on January 30).
            2) Try to repeat the issue. Perhaps January 2026 CU has fixed it.

            3) If the issue repeats, then capture a dump when the below end-user error is happening – send the dump to Microsoft for investigation:

            Exception message: Unable to validate data.
            at System.Web.Configuration.MachineKeySection.EncryptOrDecryptData(Boolean fEncrypt, Byte[] buf, Byte[] modifier, Int32 start, Int32 length, Boolean useValidationSymAlgo, Boolean useLegacyMode, IVType ivType, Boolean signData)
            at System.Web.UI.Page.DecryptString(String s, Purpose purpose)

            at System.Web.Handlers.AssemblyResourceLoader.System.Web.IHttpHandler.ProcessRequest(HttpContext context)

            As the issue causes major problems for end users, then I can repeat it only in a planned and dedicated maintenance window (it is a Production environment and used constantly).

            As a temporary workaround, I have disabled the Timer Job “Machine Key Rotation Job”. Therefore, the issue does not happen every week (after the Timer Job would have run). The drawback is that Machine Keys are not being rotated.

            About the system Timer Job “Config Refresh” (“job-config-refresh”) you have discussed above with Stefan – interestingly I don’t have traces of it running in any of our environments (no “bjgst” and “a3c26” events) – even in those where the Machine Key rotation functionality overall seems to work as expected with no errors (and all other SharePoint Log entries from Stefan’s article are present).

            I plan to update here in comments if, when and how the issue gets resolved.


    5. Hello everyone!
      Is there any update about this problem in new (January 2026) SharePoint updates?

      Reply

  8. Hi Stefan,

    Would it be possible for you to produce a slightly different version of your script DumpConfigDBMachineKeys.ps1 that writes the machine keys in a file instead of in the console ?

    It would be interesting to create a scheduled task that dumps the machine keys every week to make sure that they are indeed rotated.

    Just a suggestion.

    Thanks !

    Jeff

    Reply

  9. Hi all,
    I am administering SP 2016 farm in single server topology.
    Firstly, I can not find any evidence of running TJ “job-config-refresh”. I am quite sure the timer service is ok and running, but in ULS there is no evidence, nor in CA or PowerShell (there is LastRunTime = N/A, or 01-Jan-01 00:00:00).
    I guess it is because this TJ is running under SharePoint Timer Service identity (account) which is Farm Account, and I am looking for evidences under another identity (FarmAdmin Account (under which was farm installed and is administered)).
    This is true also for TJ “job-config-machine-key-rotation”: it shows in CA LastRunTime = N/A, or 01-Jan-01 00:00:00 in PowerShell. But in ULS I can find both of messages: MachineKeyConfiguration_ password changed. Needs update… Successfully persisted MachineKeyConfiguration__ credential… that indicate machine keys are generated and updated in config db, for all of our web apps, and during all previous sunday mornings at 12:00:00 AM approximately.
    Can someone explain how these may/should be interpreted?
    With best wishes to everyone involved and many thanks to Stefan who is making great job supporting the community.

    Reply

    1. Little correction of myself: not all previous sundays, but all sundays after October 2025 CU installed.
      And small addition, on first sunday there were slightly different 3 messages (repeated for every web app) in ULS:

      Creating the SPPersistedObject. Updating object for the first time. SPEncryptedString Name=MachineKeyConfiguration_:MachineKeyConfiguration_

      Creating the SPPersistedObject. Updating object for the first time. SPSecureDBCredential Name=MachineKeyConfiguration_

      Successfully persisted MachineKeyConfiguration_ credential.

      Reply

    2. Hi eLKey,
      first of all, to verify if the machine keys have changed you can use my script which shows you the current keys:
      https://github.com/stefangossner/PowerShellScripts/blob/main/DumpConfigDBMachineKeys.ps1

      If you run this before executing the timerjob and (>30 seconds) after the timer job has run you should see that all machine keys have changed.

      If you see that these did not change, we need to look into the other messages in more detail.
      Cheers,
      Stefan

      Reply

      1. Sorry Stefan for my delayed response, I was out-of-office for some time. Today I dumped machine keys with both of your scripts and I can confirm that configdb machine keys were changed and webconfig machine keys are “static” (remain the same), when comparing results with previous week output. But when I look to both TJs details in PowerShell I see that they were never ran, both TJs show LastRunTime 01-Jan-01 00:00:00. I still suppose that this is identity problem, aka I am using not the correct account to display both TJs properties.

        Reply

        1. Hi eLKey,
          for the config refresh system job this is expected.
          And if the machine keys got changed and replicated to all servers you can see that they were indeed executed.
          Cheers,
          Stefan

          Reply

  10. Hi Stefan,

    Machine key rotation is fine on all our SharePoint farms, except our lab environment.

    The Timer job executes fine (no error).

    But when i use your script (DumpConfigDBMachineKeys.ps1), i get an error for each web app saying :
    Error reading configuration for web application xxxx : Exception decrypting. Decryption failed.

    In the ULS logs, i get these infos/errors :

    anog9 Medium SPDatabase.GetSecuredPassword(): trying to fetch password from dead drop location for SOFTWARE\Microsoft\Shared Tools\Web Server Extensions\16.0\Secure\CredentialsMachineKeyConfiguration_4e933407-adc8-4e6d-b70c-be9a392ae805
    11/18/2025 09:01:09.00 w3wp.exe (0x0B6C) 0x3508 SharePoint Foundation Runtime

    4l39g Exception System.Exception: Exception decrypting. Decryption failed. at Microsoft.SharePoint.Utilities.SPProtectData.Decrypt(Byte[] cipherText, Byte[] optionalEntropy) at Microsoft.SharePoint.Administration.SPSecureDBCredential.Reload() at Microsoft.SharePoint.Administration.SPSecureDBCredential.GetSecuredPassword() at Microsoft.SharePoint.Administration.MachineKeyPersistedConfiguration.EnsureViewstateKeyUpgrade(SPWebApplication webApp) StackTrace: at onetnative.dll: (sig=5b788f5d-7ce3-43aa-9a45-d22f3ed2dfb4|2|onetnative.pdb, offset=36C9D) at onetnative.dll: (offset=1E322)

    Could it be a permission problem ? I tried with the farm admin account and other accounts that are members of the Farm Administrators with no avail.

    The other script, DumpWebConfigMachineKeys.ps1 works fine. so accessing the machine keys in the web.config files work.

    The problem seems just with the Database part.

    Do you have an idea ?

    Thanks,

    Jeff

    Reply

    1. Hi Jeff,
      sorry – I haven’t seen this.
      Not sure why this happens.
      If you need assistance, please open a ticket with Microsoft support to get this investigated.
      Cheers,
      Stefan

      Reply

  11. Ok, will do, thanks !

    Jeff

    Reply

  12. Hi Stefan,

    Thanks again for all your time and help.

    When you say :
    Important:

    After a machine key rotation was done using the Machine Key Rotation job it is expected that the machine keys emitted by both scripts are different, as the timer job DOES NOT update the web.config files.
    After a machine key rotation was done using the Set-SPMachineKey PowerShell CmdLet it is expected that the machine keys emitted by both scripts are identical for the web application where the new machine keys were generated as the CmdLet DOES update the web.config file.

    -Update-SPMachineKey is irrelevant after the Machine Key Rotation job has been executed at least once in the farm as it will update the web.config machine keys which are no longer being used.

    If we only use the Timer Job (no manual Machine Key rotation with PS), if i understand correctly, the DB config Machine Keys will be updated once a week and will be different from last week, but the Machine Keys in the web.config files will stay the same ? And it would be ok, since the Machine Keys from the web.config files are no longer used ?

    Thanks,

    Jeff

    Reply

    1. Hi Jeff,
      yes – thats correct.
      The web.config machine keys are not used anymore. Independent if the change is done with timer job or with PowerShell.
      Cheers,
      Stefan

      Reply

      1. Thanks for the confirmation !

        Jeff

        Reply

  13. Just a comment, I have two clients that have the same issue (View State) and cases open with Microsoft with no resolution. One client it is failing on both their stage and production. Since those have multiple from ends. We will test the work around.

    “As a temporary workaround, I again did the following:
    1) Rotate Machine Keys manually (Set-SPMachineKey + IISRESET) – to make sure that the environment becomes functional again
    2) Disabled the Timer Job “Machine Key Rotation Job” – so the automatic Machine Key rotation does not happen on the nights to Sundays and we don’t run into the issue again”

    Reply

    1. Update: We installed the Nov CU in their Test Environment, Same issue. I will run the scripts to check the keys and do the manual work around to see if it addresses the issue. This happen in both the test and production environments, so I have 3 tenants with the issue.

      Reply

  14. Update: Disabling the timer job and manually Updating resulted in the web config files matching but the DB keys not changing. Clearing the Config Cache, I was able to see no DB keys be returned then the original keys as it was cached on the server again. So, all the DB Keys match, all the web configs match, but both are different. At this point it looks like the DB keys are not being updated. Sending the information to MS Support.

    Reply

    1. Hi Damein,
      sounds as if you are using SP2016 or SP2019.
      There is currently a known problem with these two versions – Set-SPMachineKey does not update the machine keys in the config DB.
      Means it is currently useless in SP2016 and SP2019 till a fix is available.
      The only way to change the machine keys for SP2016 and SP2019 at this time is the machine key rotation timer job.
      Cheers,
      Stefan

      Reply

      1. Hi Stefan,

        Adding a comment for info as Damien and I may be facing the same or similar issues.
        My problematic environment is SP 2016. The whole issue is described in this comment thread: https://blog.stefan-gossner.com/2025/10/24/understanding-machine-key-rotation-using-the-new-automatic-sharepoint-timerjob/#comment-46056

        I am adding this comment because you are mentioning the known issue with “Set-SPMachineKey”.
        For me Set-SPMachineKey + IISRESET is what fixes the environment after the Timer Job “Machine Key Rotation Job” breaks it.

        If Set-SPMachineKey is not functioning fully as it is expected to, then I am wondering how it is helping. Maybe it is only IISRESET on all servers that helps (not Set-SPMachineKey). I have not tested it – when we first got the issue, then I proceeded with Set-SPMachineKey + IISRESET when I understood that it is Machine Key related problem. And it helped. Maybe IISRESET alone would have helped.
        If we will do further tests involving running the Timer Job “Machine Key Rotation Job” and then getting the environment back to work, then I could try only IISRESET and also update here.

        Reply

  15. Hello Stefan,
    at first thank you for your fantastic and always helpful work on this blog!

    Unfortunately we are also facing some problems with the machine-key-rotation-job on each of our multi-server-farms. In our applications we have some UI-errors which might be a result from a failed machine-key-rotation-job.

    During the first part (generating the key and updating in the config-db) we got the first three messages you described.
    But in the “Timer Job job-config-machine-key-rotation” we are also running in the following errors, even the farm account has access:
    1. Failed to acquire mutex ‘{351cb96a-be0d-4ed4-94fe-61a2444ade32}’: System.UnauthorizedAccessException: Access to the path ‘{351cb96a-be0d-4ed4-94fe-61a2444ade32}’ is denied.
    at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
    at System.Threading.Mutex.OpenExistingWorker(String name, MutexRights rights, Mutex& result)
    at System.Threading.Mutex.OpenExisting(String name, MutexRights rights)
    at Microsoft.SharePoint.Administration.SPAdministrationMutex.Acquire()
    2. SPDatabaseServiceInstance.OnDeserialization(): could not set encrypted password in the secured password location. System.UnauthorizedAccessException: Access to the path ‘{351cb96a-be0d-4ed4-94fe-61a2444ade32}’ is denied.
    at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
    at System.Threading.Mutex.OpenExistingWorker(String name, MutexRights rights, Mutex& result)
    at System.Threading.Mutex.OpenExisting(String name, MutexRights rights)
    at Microsoft.SharePoint.Administration.SPAdministrationMutex.Acquire()
    at Microsoft.SharePoint.Administration.SPSecureDBCredential.SetSecuredPassword(String password, Int64 passwordversion)
    at Microsoft.SharePoint.Administration.SPSecureDBCredential.EnsureCredentialDeployment(Boolean force)

    We also could not find some messages indicating that the configuration-refresh-timerjob and the update of the IIS worker process were successful. Even the timerjob instances are running on each server.

    Do you have an idea?
    Best regards
    Dennis

    Reply

    1. Hi Dennis,
      I haven’t seen this error – from the message this is not a simple issue and most likely requires a dump analysis to get more details about the exception the the specific condition that led to it.
      My recommendation would be to open a support case with Microsoft to ensure this can be analyzed.
      Cheers,
      Stefan

      Reply

    1. Hi Jaskirat,
      you mean about the Set-SPMachineKey CmdLet not being able to set new machine keys?
      The workaround is currently to execute the machine key rotation job which will successfully assign new random machine keys.
      In the rare case that you need to assign specific machine keys: there is currently no workaround. As always: direct SQL updates would lead to an unsupported farm.
      Cheers,
      Stefan

      Reply

  17. Thank you Stefan

    Once the timer job has run, is there any way to revert back to not use the DB config values, and reverr to web config ?

    We have also had an issue where the timer job ran before ps config was run on a Tuesday. Not sure how that happened

    Reply

  18. Hi Stefan!

    We have two strange scenarios after testing in our test and acceptance farms:

    1) After manually triggering the Machine Key timer job, we get all ULS events EXCEPT “bjgst”. Is that a problem? Everything seems to work and the keys are indeed updated when we check with etc DumpConfigDB script.

    2) After a scheduled trigger of the Machine Key timer job, we also miss A3c26 in ULS, which is worrying. (More correctly: we miss A3c26 entries that contain “SPSecureDBCredential”.) Note that they do appear in scenario #1 above.

    We are a bit hesitant to go into production with the above issues. What to do?

    Thanks!

    Reply

    1. Hi Will,
      which SharePoint version are you using?
      The list of events above is from SharePoint Server Subscription Edition – I did not verify if the events in SP2016 and SP2019 are 100% identical.
      Cheers,
      Stefan

      Reply

      1. Sorry, I forgot to mention this behavior is on SP2016. That could then explain a difference in what we see?

        Maybe it’s OK that these events are missing… what is the “final” proof that the rotation process has worked well?

        An idea: maybe we should scrutinize everything in the ULS category “Config Cache”? Could be a fruitless and a lot of work though…

        Thanks!

        Reply

        1. Hi Will,
          to be honest I did not have time to collect the details on events for SP2016 and SP2019. Need to see when and if I find time to collect the relevant details to include the relevant information in this article.

          I’m certain you are aware that SP2016 will go out of support in 6 months and only SPSE will be supported going forward. If not already done you should plan for the migration to SPSE or the cloud as soon as possible.

          Cheers,
          Stefan

          Reply

          1. Hi Stefan!

            No worries. If you have time, it’d be appreciated but I understand that SP2016/2019 aren’t high on the priority list anymore. 🙂

            And yes, we are migration planning for SPSE, and that’s basically why we don’t have time for any hiccups in the SP2016 farm… 😉

            Thanks!

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.