SharePoint Server 2016 patching using side-by-side functionality explained

The nature of zero downtime patching is that – during patching – some servers in the farm are already on a higher patch level while other servers are still on the previous patch level. In this situation different servers in the farm will potentially return different javascript files to the client. This can lead to problems – especially as these javascript files can be cached in browser caches, proxy servers and potentially CDNs.
To guarantee that all servers in the SharePoint farm serve the same version of the Javascript files and that – after finishing the patching – the most current version of the javascript files are used right away SharePoint Server 2016 offers the side-by-side functionality.
The Side-by-side functionality in SharePoint 2016 did not get a lot of attention in current documentation so I would like to explain how it works and how SharePoint administrators can enable it.
The only place where I’m aware of that Side-by-Side functionality is mentioned in public documentation is in the following article – but the information provided is very basic:

So how does the SharePoint Server 2016 side-by-side functionality work?

If side-by-side functionality is enabled (and in previous patch levels of SharePoint 2016 even if it is not enabled) the final step of the SharePoint Configuration Wizard will copy all Javascript and CSS files which reside inside the …\16\TEMPLATE\LAYOUTS directory into a directory which has the version number of the SharePoint component with the highest patch level on the system. (You might remember that different components on a SharePoint server can have different patch levels.)
Some of you might have noticed this directory and might have been curious about the purpose of this directory:
16.0.4456.1002 directory inside the layouts folder
Inside this directory you will find a copy of all the javascript and CSS files which are also in the layouts directory.
content of 16.0.4456.1002 folder in layouts directory
After enabling of SharePoint Server 2016 side-by-side functionality SharePoint will generate Urls to the javascript inside this specific directory:
dev-tools-4456
The benefit is clear: if SharePoint references a specific version of the javascript files with the version number included in the Url, then caching of old versions is no longer an issue. As soon as the SharePoint farm configured to use the new version number the referenced Urls will be updated and the correct version of the javascript file will be downloaded.
After installing a new CU and running PSConfig you will notice a second version directory inside the LAYOUTS directory:
layouts-4471
Be aware that the SharePoint configuration wizard will remove all side-by-side directories except the currently used one before creating the new side-by-side directory for the latest version. And the algorithm used here to remove the directories deletes all directory which slightly look like a side-by-side directory. And that means all directories which have the pattern of a version number. So a directory 1.2.3.4 would also be removed. Be careful not to use this pattern when creating custom directories inside the layouts directory!

So how does this help us with Zero-Downtime-Patching?

The side-by-side functionality does not automatically determine which javascript version to use. This is something a SharePoint administrator has to configure.
So the relevant steps related to Zero-downtime-patching would be as follows:

  1. Ensure that the side-by-side functionality has been enabled previously and that a version folder exists inside the LAYOUTS directory
  2. Configure the site-by-side functionality to use the specific version directory inside the layouts folder (in our example 16.0.4456.1002) by specifying the appropriate side-by-side token.
  3. Follow the normal instructions to perform zero-downtime-patching as outlined in the following articles:
  4. After all servers are patched and the configuration wizard was executed on all machines configure the site-by-side functionality to use the new version directory inside the layouts folder (in our example 16.0.4471.1000) by specifying the appropriate side-by-side token.

These steps will guarantee that before and during patching the javascript files from the old directory (in our example 16.0.4456.1002) will be served. Only after the patching is complete and an administrator manually configures SharePoint to use the new version number (in our example 16.0.4471.1000) SharePoint will switch to the latest javascript files.
The result is that all servers in the farm always serve the same Javascript files independent from their specific patch level.

Ok, enough theory – how can I make use of the side-by-side functionality?

To enable the side-by-side functionality you need to set the EnableSideBySide property of the WebService to true (e.g.) by using Powershell:
sp-powershell-enable-side-by-side
To ensure that a current side-by-side directory exists you can call the Copy-SPSideBySideFiles powershell command which will create the side-by-side directory for your current patch level:
sp-powershell-copy-side-by-side-files
If you would like to get a logfile about the performed copy operation you can specify the optional -LogFile parameter:
sp-powershell-copy-side-by-side-files-logfile
To ensure that the side-by-side files will be used in future http requests to the server you finally have to configure the SideBySideToken. The value of this token has to match the name of the side-to-side directory that should be used:
sp-powershell-set-side-by-side-token
Thats it! From now on links to the relevant javascript files in the layouts directory will point to the relevant file in the side by side directory of version 16.0.4456.1002.
After patching to a new CU (e.g. with build 16.0.4471.1000) and running PSConfig on all machines you only have to update the side-by-side token and all generated URLs will automatically be redirected to the new side-by-side directory:
sp-powershell-set-side-by-side-token-4471
 
 

27 Comments


  1. so this:
    ((Get-SPProduct) | % {$product = $_; $product.PatchableUnitDisplayNames | % {$product.GetPatchableUnitInfoByDisplayName($_) | % {$_.LatestPatch}}} | sort Version -Descending | select -First 1).Version.ToString()
    is the new (Get-SPFarm).BuildVersion?

    Reply

    1. Hi Piotr,
      not really. If you install just a security fix in (e.g.) December and not all CU files then you would end up with the same version number as if you intalled the December CU.
      So a single number is never reliable.
      Cheers,
      Stefan

      Reply

  2. My question is about what happens when you update an existing custom solution that puts scripts into layouts/customfolder. The ones being used will now be in the version subdirectory. How does a developer ensure the updated scripts will be used? Deploy with -versions All? Target the new directory in the solution every time a CU is installed? This makes custom solutions that utilize that directory tough to maintain. Right now the version token only addresses the hiver root, not a version based subfolder during deployment.

    Reply

  3. Hi Jason,
    to be honest I’m not sure if custom javascript files would also be redirected as I don’t have any customizations on my boxes right now to test.
    If you test this it would be great if you could provide feedback here if the redirect also applies to custom javascript files.
    If they are all you would have to do is to run copy-sidebysidefiles after deploying and the updates will be copied to the latest side-by-side directory as well.
    Cheers,
    Stefan

    Reply

    1. They are redirected into the new location which makes development a bit confusing. We will do some testing on our side and see if we can figure out a way to solve that issue. Thanks for the great articles, not sure what we would do without your blog 😉

      Reply

      1. And by solve I mean doing development that involves JS in the layouts directory. We are going to have to figure out a way to get them deployed to the right place. Maybe a post deployment command built into our projects which does the side by side copy after each deployment or go back to the days of putting custom scripts into the IIS directories.

        Reply

  4. Hi Jason,
    in such a case I would assume that it will be required to call copy-sidebysidefiles to ensure that the updated javascript files are copied into the version directory.
    Alternatively you could just enable the side-by-side functionality while updating the farm. After the farm has been fully updated you could disable the side-by-side functionality which will ensure that all servers again use the latest version of the javascript files in the layouts directory.
    Cheers,
    Stefan

    Reply

  5. Thanks we will try a few options and see what works best for us. We have lots of custom code that has been written over the years so we need something stable and reproducible across environments.

    Reply

  6. @Stefan
    @Jason
    A couple things as they relate to deploying custom solutions. The Copy-SPSideBySideFiles script only copies css, js, and htm files. Which is a problem for us because we have other file types in our layouts directory that aren’t copied over. I can deploy directly to the CU specific folder by changing the deployment location and folder name of the layouts folder properties in the solution, but this is mistake prone as you’ll have to do this to every solution and make sure you have the right folder every time you deploy a solution.
    I ended up running a diff of the two folders and it appears that the CU process updates both folders (…\TEMPLATE\LAYOUTS\16.0.xxxx.xxxx\ and …\TEMPLATE\LAYOUTS\) with the same changes, which I wasn’t aware of. So, instead of changing the Web App SideBySide token to point to the new folder, just leave it alone, or set it to an empty string if you’ve already changed it. This will load the files from your layouts folder instead of the new CU specific. You don’t have to change your deployment process and can still update with zero downtime. The only caveat is that some users may see cached versions of updated files because the file path/name is not changing.
    Dan

    Reply

    1. Hi Dan,
      its not exactly as you mentioned. The patching is only done in the layouts folder. PSConfig later copies the current JS, CSS and HTM files into the new created “version” folder. So of course the version folder will have those files after the update.
      Without solution update you should not have to care about this as SharePoint ensures that the JS files are taken from the version folder and other stuff is taken from the layouts folder. Only if you are updating your solutions outside of a CU update (I know many customers use the same maintenance window used to install CUs to update their solutions) you would have to take special care to ensure that the updated files get copied to the version folder. Running copy-sidebysidefiles willl do that.
      Cheers,
      Stefan

      Reply

      1. Ah! That makes more sense. I was not aware that SP knew to use the JS files from the version folder and other stuff from the layouts folder. I figured it was one or the other. In that case running copy-sidebysidefiles post deployment should be fine. Thanks for your help.
        Dan

        Reply

  7. This is pretty important information: “Be careful not to use this pattern when creating custom directories inside the layouts directory”.
    Could you take care of putting this in the official documentation somewhere / a KB article? I am not aware of any third parties using just version numbers as folder names, but if updating might just blindly delete a folder with e.g. business critical custom applications this should be documented clearly.

    Reply

  8. Hi Stefan,
    Great article.
    I have two questions though:
    In your script you say:
    $webapp = get-spwebapplication http://myserver
    $webapp.WebService.EnableSideBySide = $true;
    Can I use the URL of the Central Administration ?
    Also, shouldn’t I run a ” $webapp.WebService.Update() ” after enabling SideBySide ? (and the script will become:
    $webapp = get-spwebapplication http://myserver
    $webapp.WebService.EnableSideBySide = $true;
    $webapp.WebService.Update()
    Thanks a lot,
    Andy.

    Reply

    1. Hi Andy,
      you are right, the $webapp.WebService.Update() is missing in the first Powershell script. 🙂
      Regarding Central Admin: No you cannot use the central admin URL as the central admin uses a different WebService instance than regular web applications.
      You need to use the Url for one of your web applications.
      Cheers,
      Stefan

      Reply

      1. Or you could use [Microsoft.SharePoint.Administration.SPWebService]::ContentService instead of $webApp.WebService. This way you don’t need no URL. Alternatively Get-SPWebApplication | select -First 1 will also give you an SPWebApplication and you don’t need the URL.

        Reply

        1. Hi Piotr,
          for reasons I don’t understand right now that does not seem to work reliably in all scenarios.
          We recently released a fix for the side-by-side functionality where the internal code was changed from checking the static ContentService attribute of the SPWebService to the web service of a given web application.
          So I would stick to that.
          Cheers,
          Stefan

          Reply

        1. Hi Andy,
          SharePoint has a timerjob which does the database maintenance for the sharepoint config databases. You should not manually intercept here.
          Cheers,
          Stefan

          Reply

          1. Thanks Stefan,
            Does it mean this can be done for contentDBs and not for the configDB ?
            Could you share the name of the timer job ?
            Thanks,
            Andy


          2. Hi Andy,
            there are different jobs for different databases. Check the “Health Analysis Jobs …” timerjobs in central admin.
            Cheers,
            Stefan


  9. Hi Stefan,
    Thanks for the great article on this topic. Just one question, do we run the script on one server or all SP servers in the farm?
    Thanks

    Reply

    1. Hi Ali,
      depends on which script you mean.
      Copy-SPSideBySideFiles has to be run on each machine – the other scripts are for the whole farm and only have to be executed once per farm.
      Cheers,
      Stefan

      Reply

      1. Many thanks Stefan for the quick reply – that answers our question.
        Kind regards,
        Ali

        Reply

  10. Awesome article Stefan, thanks! I have a few questions, I would be really grateful for some insights:
    1) I found some SP2016 course material that says you have to run the “$webapp.WebService.EnableSideBySide = $true;” script on every server in the farm before you start. Is this true?
    2) I’m confused about the side-by-side directory referencing… With CSS & JS files being copied to the version directories in \template\layouts\\, does this mean that these files are *never* being loaded from plain old “\template\layouts\” anymore? (There seems to be no “disable side-by-side” operation at the end of the patch process, which would “reset” the references back to the original folder…? This would be logical to me. 🙂 )
    3) The official ZDP instructions on TechNet [https://technet.microsoft.com/en-us/library/mt743024(v=office.16).aspx] differ from your instructions. They say that you have to configure $webapp.WebService.SideBySideToken = “BuildNumberInQuotes” after patching, you say before…? (I can understand that assuming that the concept is that you never disable side-by-side, so you would always reference a version directory…?)
    Thanks for any help!

    Reply

    1. Hi Will,
      1) That is not correct. This line changes a property in web services in the farm. There is only a single one independent how many servers you have. This only has to be done once on one of the servers.
      2) correct. If side-by-side is enabled it will not use the files from the directory below.
      3) When you first enable it you need to set it to the old patch level before patching and to the new one after patching. With future patching it is already set to the previous patch level and you can skip as it would set it to the same value. So you only need to set it after the next patching is complete.
      Cheers,
      Stefan

      Reply

  11. Thanks for the article! What process/job deletes the old version folder? How long does it take to remove the folder? Ours has not been deleted yet so wondering if something is awry..

    Reply

    1. Hi Scott,
      as I mentioned: “the SharePoint configuration wizard will remove all side-by-side directories except the currently used one before creating the new side-by-side directory for the latest version. And the algorithm used here to remove the directories deletes all directory which slightly look like a side-by-side directory. And that means all directories which have the pattern of a version number. So a directory 1.2.3.4 would also be removed. Be careful not to use this pattern when creating custom directories inside the layouts directory!”

      Sounds to me as if you did not run the config wizard while the side-by-side functionality was enabled. If this is not the case you can of course manually delete remaining directories as only the one specified in the SideBySideToken will be used.

      Cheers,
      Stefan

      Reply

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.