SharePoint Server 2016/2019 patching using side-by-side functionality explained

The nature of zero downtime patching is that – during patching – some servers in the farm are already on a higher patch level while other servers are still on the previous patch level. In this situation different servers in the farm will potentially return different javascript files to the client. This can lead to problems – especially as these javascript files can be cached in browser caches, proxy servers and potentially CDNs.
To guarantee that all servers in the SharePoint farm serve the same version of the Javascript files and that – after finishing the patching – the most current version of the javascript files are used right away SharePoint Server 2016 offers the side-by-side functionality.
The Side-by-side functionality in SharePoint 2016 did not get a lot of attention in current documentation so I would like to explain how it works and how SharePoint administrators can enable it.
The only place where I’m aware of that Side-by-Side functionality is mentioned in public documentation is in the following article – but the information provided is very basic:

So how does the SharePoint Server 2016 side-by-side functionality work?

If side-by-side functionality is enabled (and in previous patch levels of SharePoint 2016 even if it is not enabled) the final step of the SharePoint Configuration Wizard will copy all Javascript and CSS files which reside inside the …\16\TEMPLATE\LAYOUTS directory into a directory which has the version number of the SharePoint component with the highest patch level on the system. (You might remember that different components on a SharePoint server can have different patch levels.)
Some of you might have noticed this directory and might have been curious about the purpose of this directory:
16.0.4456.1002 directory inside the layouts folder
Inside this directory you will find a copy of all the javascript and CSS files which are also in the layouts directory.
content of 16.0.4456.1002 folder in layouts directory
After enabling of SharePoint Server 2016 side-by-side functionality SharePoint will generate Urls to the javascript inside this specific directory:
dev-tools-4456
The benefit is clear: if SharePoint references a specific version of the javascript files with the version number included in the Url, then caching of old versions is no longer an issue. As soon as the SharePoint farm configured to use the new version number the referenced Urls will be updated and the correct version of the javascript file will be downloaded.
After installing a new CU and running PSConfig you will notice a second version directory inside the LAYOUTS directory:
layouts-4471
Be aware that the SharePoint configuration wizard will remove all side-by-side directories except the currently used one before creating the new side-by-side directory for the latest version. And the algorithm used here to remove the directories deletes all directory which slightly look like a side-by-side directory. And that means all directories which have the pattern of a version number. So a directory 1.2.3.4 would also be removed. Be careful not to use this pattern when creating custom directories inside the layouts directory!

So how does this help us with Zero-Downtime-Patching?

The side-by-side functionality does not automatically determine which javascript version to use. This is something a SharePoint administrator has to configure.
So the relevant steps related to Zero-downtime-patching would be as follows:

  1. Ensure that the side-by-side functionality has been enabled previously and that a version folder exists inside the LAYOUTS directory
  2. Configure the site-by-side functionality to use the specific version directory inside the layouts folder (in our example 16.0.4456.1002) by specifying the appropriate side-by-side token.
  3. Follow the normal instructions to perform zero-downtime-patching as outlined in the following articles:
  4. After all servers are patched and the configuration wizard was executed on all machines configure the site-by-side functionality to use the new version directory inside the layouts folder (in our example 16.0.4471.1000) by specifying the appropriate side-by-side token.

These steps will guarantee that before and during patching the javascript files from the old directory (in our example 16.0.4456.1002) will be served. Only after the patching is complete and an administrator manually configures SharePoint to use the new version number (in our example 16.0.4471.1000) SharePoint will switch to the latest javascript files.
The result is that all servers in the farm always serve the same Javascript files independent from their specific patch level.

Ok, enough theory – how can I make use of the side-by-side functionality?

To enable the side-by-side functionality you need to set the EnableSideBySide property of the WebService to true (e.g.) by using Powershell:
sp-powershell-enable-side-by-side
To ensure that a current side-by-side directory exists you can call the Copy-SPSideBySideFiles powershell command which will create the side-by-side directory for your current patch level:
sp-powershell-copy-side-by-side-files
If you would like to get a logfile about the performed copy operation you can specify the optional -LogFile parameter:
sp-powershell-copy-side-by-side-files-logfile
To ensure that the side-by-side files will be used in future http requests to the server you finally have to configure the SideBySideToken. The value of this token has to match the name of the side-to-side directory that should be used:
sp-powershell-set-side-by-side-token
Thats it! From now on links to the relevant javascript files in the layouts directory will point to the relevant file in the side by side directory of version 16.0.4456.1002.
After patching to a new CU (e.g. with build 16.0.4471.1000) and running PSConfig on all machines you only have to update the side-by-side token and all generated URLs will automatically be redirected to the new side-by-side directory:
sp-powershell-set-side-by-side-token-4471
 
 

51 Comments


  1. so this:
    ((Get-SPProduct) | % {$product = $_; $product.PatchableUnitDisplayNames | % {$product.GetPatchableUnitInfoByDisplayName($_) | % {$_.LatestPatch}}} | sort Version -Descending | select -First 1).Version.ToString()
    is the new (Get-SPFarm).BuildVersion?

    Reply

    1. Hi Piotr,
      not really. If you install just a security fix in (e.g.) December and not all CU files then you would end up with the same version number as if you intalled the December CU.
      So a single number is never reliable.
      Cheers,
      Stefan

      Reply

  2. My question is about what happens when you update an existing custom solution that puts scripts into layouts/customfolder. The ones being used will now be in the version subdirectory. How does a developer ensure the updated scripts will be used? Deploy with -versions All? Target the new directory in the solution every time a CU is installed? This makes custom solutions that utilize that directory tough to maintain. Right now the version token only addresses the hiver root, not a version based subfolder during deployment.

    Reply

  3. Hi Jason,
    to be honest I’m not sure if custom javascript files would also be redirected as I don’t have any customizations on my boxes right now to test.
    If you test this it would be great if you could provide feedback here if the redirect also applies to custom javascript files.
    If they are all you would have to do is to run copy-sidebysidefiles after deploying and the updates will be copied to the latest side-by-side directory as well.
    Cheers,
    Stefan

    Reply

    1. They are redirected into the new location which makes development a bit confusing. We will do some testing on our side and see if we can figure out a way to solve that issue. Thanks for the great articles, not sure what we would do without your blog 😉

      Reply

      1. And by solve I mean doing development that involves JS in the layouts directory. We are going to have to figure out a way to get them deployed to the right place. Maybe a post deployment command built into our projects which does the side by side copy after each deployment or go back to the days of putting custom scripts into the IIS directories.

        Reply

  4. Hi Jason,
    in such a case I would assume that it will be required to call copy-sidebysidefiles to ensure that the updated javascript files are copied into the version directory.
    Alternatively you could just enable the side-by-side functionality while updating the farm. After the farm has been fully updated you could disable the side-by-side functionality which will ensure that all servers again use the latest version of the javascript files in the layouts directory.
    Cheers,
    Stefan

    Reply

  5. Thanks we will try a few options and see what works best for us. We have lots of custom code that has been written over the years so we need something stable and reproducible across environments.

    Reply

  6. @Stefan
    @Jason
    A couple things as they relate to deploying custom solutions. The Copy-SPSideBySideFiles script only copies css, js, and htm files. Which is a problem for us because we have other file types in our layouts directory that aren’t copied over. I can deploy directly to the CU specific folder by changing the deployment location and folder name of the layouts folder properties in the solution, but this is mistake prone as you’ll have to do this to every solution and make sure you have the right folder every time you deploy a solution.
    I ended up running a diff of the two folders and it appears that the CU process updates both folders (…\TEMPLATE\LAYOUTS\16.0.xxxx.xxxx\ and …\TEMPLATE\LAYOUTS\) with the same changes, which I wasn’t aware of. So, instead of changing the Web App SideBySide token to point to the new folder, just leave it alone, or set it to an empty string if you’ve already changed it. This will load the files from your layouts folder instead of the new CU specific. You don’t have to change your deployment process and can still update with zero downtime. The only caveat is that some users may see cached versions of updated files because the file path/name is not changing.
    Dan

    Reply

    1. Hi Dan,
      its not exactly as you mentioned. The patching is only done in the layouts folder. PSConfig later copies the current JS, CSS and HTM files into the new created “version” folder. So of course the version folder will have those files after the update.
      Without solution update you should not have to care about this as SharePoint ensures that the JS files are taken from the version folder and other stuff is taken from the layouts folder. Only if you are updating your solutions outside of a CU update (I know many customers use the same maintenance window used to install CUs to update their solutions) you would have to take special care to ensure that the updated files get copied to the version folder. Running copy-sidebysidefiles willl do that.
      Cheers,
      Stefan

      Reply

      1. Ah! That makes more sense. I was not aware that SP knew to use the JS files from the version folder and other stuff from the layouts folder. I figured it was one or the other. In that case running copy-sidebysidefiles post deployment should be fine. Thanks for your help.
        Dan

        Reply

  7. This is pretty important information: “Be careful not to use this pattern when creating custom directories inside the layouts directory”.
    Could you take care of putting this in the official documentation somewhere / a KB article? I am not aware of any third parties using just version numbers as folder names, but if updating might just blindly delete a folder with e.g. business critical custom applications this should be documented clearly.

    Reply

  8. Hi Stefan,
    Great article.
    I have two questions though:
    In your script you say:
    $webapp = get-spwebapplication http://myserver
    $webapp.WebService.EnableSideBySide = $true;
    Can I use the URL of the Central Administration ?
    Also, shouldn’t I run a ” $webapp.WebService.Update() ” after enabling SideBySide ? (and the script will become:
    $webapp = get-spwebapplication http://myserver
    $webapp.WebService.EnableSideBySide = $true;
    $webapp.WebService.Update()
    Thanks a lot,
    Andy.

    Reply

    1. Hi Andy,
      you are right, the $webapp.WebService.Update() is missing in the first Powershell script. 🙂
      Regarding Central Admin: No you cannot use the central admin URL as the central admin uses a different WebService instance than regular web applications.
      You need to use the Url for one of your web applications.
      Cheers,
      Stefan

      Reply

      1. Or you could use [Microsoft.SharePoint.Administration.SPWebService]::ContentService instead of $webApp.WebService. This way you don’t need no URL. Alternatively Get-SPWebApplication | select -First 1 will also give you an SPWebApplication and you don’t need the URL.

        Reply

        1. Hi Piotr,
          for reasons I don’t understand right now that does not seem to work reliably in all scenarios.
          We recently released a fix for the side-by-side functionality where the internal code was changed from checking the static ContentService attribute of the SPWebService to the web service of a given web application.
          So I would stick to that.
          Cheers,
          Stefan

          Reply

        1. Hi Andy,
          SharePoint has a timerjob which does the database maintenance for the sharepoint config databases. You should not manually intercept here.
          Cheers,
          Stefan

          Reply

          1. Thanks Stefan,
            Does it mean this can be done for contentDBs and not for the configDB ?
            Could you share the name of the timer job ?
            Thanks,
            Andy


          2. Hi Andy,
            there are different jobs for different databases. Check the “Health Analysis Jobs …” timerjobs in central admin.
            Cheers,
            Stefan


  9. Hi Stefan,
    Thanks for the great article on this topic. Just one question, do we run the script on one server or all SP servers in the farm?
    Thanks

    Reply

    1. Hi Ali,
      depends on which script you mean.
      Copy-SPSideBySideFiles has to be run on each machine – the other scripts are for the whole farm and only have to be executed once per farm.
      Cheers,
      Stefan

      Reply

      1. Many thanks Stefan for the quick reply – that answers our question.
        Kind regards,
        Ali

        Reply

  10. Awesome article Stefan, thanks! I have a few questions, I would be really grateful for some insights:
    1) I found some SP2016 course material that says you have to run the “$webapp.WebService.EnableSideBySide = $true;” script on every server in the farm before you start. Is this true?
    2) I’m confused about the side-by-side directory referencing… With CSS & JS files being copied to the version directories in \template\layouts\\, does this mean that these files are *never* being loaded from plain old “\template\layouts\” anymore? (There seems to be no “disable side-by-side” operation at the end of the patch process, which would “reset” the references back to the original folder…? This would be logical to me. 🙂 )
    3) The official ZDP instructions on TechNet [https://technet.microsoft.com/en-us/library/mt743024(v=office.16).aspx] differ from your instructions. They say that you have to configure $webapp.WebService.SideBySideToken = “BuildNumberInQuotes” after patching, you say before…? (I can understand that assuming that the concept is that you never disable side-by-side, so you would always reference a version directory…?)
    Thanks for any help!

    Reply

    1. Hi Will,
      1) That is not correct. This line changes a property in web services in the farm. There is only a single one independent how many servers you have. This only has to be done once on one of the servers.
      2) correct. If side-by-side is enabled it will not use the files from the directory below.
      3) When you first enable it you need to set it to the old patch level before patching and to the new one after patching. With future patching it is already set to the previous patch level and you can skip as it would set it to the same value. So you only need to set it after the next patching is complete.
      Cheers,
      Stefan

      Reply

  11. Thanks for the article! What process/job deletes the old version folder? How long does it take to remove the folder? Ours has not been deleted yet so wondering if something is awry..

    Reply

    1. Hi Scott,
      as I mentioned: “the SharePoint configuration wizard will remove all side-by-side directories except the currently used one before creating the new side-by-side directory for the latest version. And the algorithm used here to remove the directories deletes all directory which slightly look like a side-by-side directory. And that means all directories which have the pattern of a version number. So a directory 1.2.3.4 would also be removed. Be careful not to use this pattern when creating custom directories inside the layouts directory!”

      Sounds to me as if you did not run the config wizard while the side-by-side functionality was enabled. If this is not the case you can of course manually delete remaining directories as only the one specified in the SideBySideToken will be used.

      Cheers,
      Stefan

      Reply

  12. Hi Stefan,
    What does Copy-SPSideBySideFiles do at the database level? According to the command’s log file (LogFile param used), it looks like it’s just a robocopy operation, but the activity shows up in the SQL Server and ULS. It used to take about 20 minutes to finish running the command, but now it’s taking more than 4 hours and I’m trying to figure out what the cause is…. Thanks.

    Reply

  13. Is there a way to see what my current sidebyside token version is set to? And if I don’t set it after patching will the URL default to the current version in the layout folder?

    Reply

    1. Hi Joye,
      you should be able to see the current token using these powershell commands:

      $webapp = get-spwebapplication http://myserver
      $webapp.WebService.SideBySideToken

      SharePoint will always serve the content from the directory configured using the sidebysidetoken.
      If an old sidebysidetoken is set, SharePoint will continue to serve content from here.
      If no token is set it will serve the content from the layouts directory.

      Cheers,
      Stefan

      Reply

    1. No this is not necessary as long as you updated the side by side token with the correct new value after patching is complete.

      Reply

  14. Stefan,

    Does Copy-SPSideBySideFiles command need to run on all servers within farm using ZDP in SharePoint 2016/2019?

    Reply

    1. Yes. This has to be run once when you decide to use ZDP on each server.

      Reply

      1. can this command run parallelly at the same time on all servers within farm or run on one server at a time within farm?

        Reply

  15. Hi Stefan, Thank you for this wonderful article. I have two questions for you if you can help me to understand.

    Question 1:
    I see two contradicting statements in this article so if you can simplify it for me.
    Statement 1: “the final step of the SharePoint Configuration Wizard will copy all Javascript and CSS files which reside inside the …\16\TEMPLATE\LAYOUTS directory into a directory which has the version number of the SharePoint component with the highest patch level on the system.”

    Statement 2: “Be aware that the SharePoint configuration wizard will remove all side-by-side directories except the currently used one before creating the new side-by-side directory for the latest version.”

    So does that mean PSConfig in final steps will remove old side-by-side directory and will create new Side-by-Side directories and copy files from \Layouts to new directories?

    If this is true then does it mean Copy-SidebySide actually copies files from \Layouts\ to new directories ? (because we won’t have old Side-by-side directories after final step of PSConfig).

    In that scenario, any customization has to be done in \layouts location (not directly in new Side-By-Side directories)? and then we can ran Copy-Sidebysidefiles commands after deployment.

    Question 2:
    What if Side-By-Side is NOT enabled in SharePoint environment and SideBySide Token is returning no value (no build number), so does that mean that my environment is using files from \Layouts location (not from new directories) .

    In that scenario, if i leave this settings as it is and perform patching then will it impact any functionality ? I understand that usability of this functionality is to let users have one experience of the UI until the upgrade is completed, and then we’re ready to switch over to the new interface.

    So ideally without Side-By-Side functionality or feature , SharePoint patching won’t be called out as ZERO downtime patching just because User Interface will be different 🙂

    last but not least, If i am not bothered about Zero downtime patching then can i simply skip this step and leave this side-by-side settings as it is ( not enabled).

    Reply

    1. Hi Dushyant,
      1) you will have two directories: the currently configured one and the new one for the current patch.
      2) yes. impact will be that different servers in the farm serve different javascript files till the last server is patched which in some situations can cause problems.
      without side-by-side the server is not down – but it might not function correctly in some scenarios. So if you are really interested in patching without disruption you should also use side-by-side patching.
      if you are accepting a downtime you don’t need this – but be aware that some javascript files from old patch level might be cached in browser cache. using side-by-side patching a new url is used after each patch level guarateeing that no old cached scripts are used by browsers.
      Cheers,
      Stefan

      Reply

  16. In our case $w.WebService.EnableSideBySide returns “false” . We don’t have it enabled and no plans to use it in near future. But I noticed C:\ is filling up with the folder like these “16.0.10361.12114” of each version number due to PSConfig performned monthly. Do we have to delete them manually, if not using this feature ? are not they should not be there at the first place when not using this feature ?

    Reply

    1. Hi Umr,

      what is the exact directory path for these folders?

      Cheers,
      Stefan

      Reply

      1. Hello Stefan,

        Folder with the verion numbers are keep piling up at C:\Program Files\Common Files\microsoft shared\Web Server Extensions\16\TEMPLATE\LAYOUTS. and filling up C:. Aren’t they should not be there when we are no using sidebyside feature ?

        Reply

          1. Hi Umr,
            this has changed with March 2017 CU: https://support.microsoft.com/en-us/kb/3178672

            Quote: “Previously, files are copied side-by-side if the EnableSideBySide setting is true. Now, files are copied side-by-side regardless of the EnableSideBySide setting.”

            Cheers,
            Stefan


  17. I set the side-by-side token as our build version every month successfully – but this month when setting the side-by-side property as the build version it failed and brought our site down. I noticed the side-by-side folder version was different than the build version. Looking at the product and installation status in CA I see the highest version number as the side-by-side token and not the build version on the manage servers page. Is there a better way to set the side-by-side token aside from using build version?

    Reply

    1. Hi Stephen,
      the side-by-side-token always will have the highest included build in the fix – independet if the last change was for a javascript file or for an executable.
      The version is determined from the xml file coming with each msp file. So assuming that it is the same as the build version of the farm is not working.
      See my comments in the following blog post on this topic:
      https://blog.stefan-gossner.com/2016/08/23/sharepoint-does-not-have-a-build-version-full-stop/
      Cheers,
      Stefan

      Reply

      1. Thank you for the quick response (I may have posted twice unsure if one took). At the beginning we verify side-by-side is enabled and at the end set the side-by-side token to the new version number. Is there another way to get the value dynamically? Our patching is automated so we were using the build version -the article looks like it is entered manually.

        Reply

        1. Hi Stephen,
          you could check the directories with side-by-side name pattern and pick the oen with the highest sort order. Alternatively you can retrieve the file version of exe file for the patch – it should match the desired version.
          Cheers,
          Stefan

          Reply

          1. Thank you, that’ll be a nice update for next month. I’ll get the file version and compare to the layouts directory.


  18. Hi Stefan,

    I am confused on which server to execute the “side-by-side functionality”. Essentially, I have a Share Point 2016 server farm with 2 WFE (HA) and 2 APP (HA). I want to know if I have to execute the below code on both WFE and both APP or on a particular server only. If it is only on a particular server, than which one?

    $webapp = Get-SPWebApplication
    $webapp.WebService.EnableSideBySide = $true
    $webapp.WebService.update()

    Thank You,
    Shuvajit Roy.

    Reply

    1. Hi Shuvajit,
      as this command changes only settings in the SharePoint config db it does not matter on which of the SharePoint servers in your farm you are executing them.
      Cheers,
      Stefan

      Reply

  19. Hi Stefan,

    Thanks for your great advice on patching over the years… I’m sure I speak for many SharePoint people that we have all benefitted from your efforts.

    This Side by Side stuff is difficult to pin down. You seem to be one of the few people talking about it as an authority and many of your commenters here seem to contradict each other on how it all works. I am hoping you can clarify and/or confirm my own findings.

    We are trying to get ZDP going in SP 2019 and understand the benefits to caching/conflict management that SbS provides. We have quite a massive SharePoint installation and thus need a fair amount of consideration for any sort of patching etc. We also have quite extensive custom built frameworks that sit on top of SP and require elements to be deployed to the hive layouts directory.

    With that in mind I started playing with it and noted the following:
    1) After enabling SbS, running a first Copy-SPSideBySide, and then setting the token to that current version, I noted that the folder is created under the layouts directory as expected and contains a copy of ALL of our custom files despite previous comments referring to only certain file types being copied.
    2) After testing with a few files I realized that most files are still being referenced in the root layouts folder and that SbS is primarily about JS files. And further, after checking the source of my homepage, it became apparent that SP is simply altering the references to the virtual directory, so all the SP based script references are updated dynamically but our custom references are not, which negates the concern from #1.
    3) Understanding #2 means that we can do the following steps during patching for our environment and we should have no issues either with caching/conflicts during patching or with custom code having to be handled in a special way:
    a) Enable SbS
    b) Set Token (or at least confirm it is already set) to old version and Copy to pick up any changes made in our
    custom code in the intervening periods between patches
    c) Do patching
    d) Set token to new version and Copy newly upgraded root layouts to version folder for next round of patching
    e) Disable SbS

    I realise if we were deploying new code at the same time as patching, we might have to think a bit further on this, but our context is much too large for that so we should be able to simply update our code in between patches as we always have, directly into the layouts folder and thus the only changes to process are during patching.

    Can you please confirm this is correct to your knowledge? and if not, provide a bit more clarity around this process?

    Thanks again,
    Gabe

    Reply

    1. Hi Gabe,
      there should be no need to disable Side-by-side between patches.
      The idea is to enable it once and then update the token after each patch was installed.
      If you installed custom solutions between patches you might need to run the copy-spsidebyside after the installation to ensure that the new installed files get copied into the version directory.
      Cheers,
      Stefan

      Reply

Leave a Reply to Stephen Jackshaw Cancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.