Part 1 – The Basics
Part 2 – The Basics continued
Part 3 – Configuration
Part 4 – Communication
Part 5 – Quick Deployment
Part 6 – Logging
Part 7 – Change Token Basics
Part 8 – Job and Change Token
Part 9 – Job and Timerjob
Incremental deployment allows to deploy all content changes in the source database since the last successful deployment which are in the configured scope of the content deployment job.
To implement this functionality content deployment makes use of so called “change tokens”.
Understanding Change Tokens
Introduction
A very common issue with content deployment is that incremental content deployment fails with the following error message:
The changeToken refers to a time before the start of the current change log
But what does this actually mean?
What we have seen in the earlier article series about the content deployment and migration API to perform an incremental deployment we have to pass in a change token that will be used as a reference to identify which objects need to be exported.
Using a change token, an application can determine “what content has changed” within a given scope since the change token has been generated.
To do this the information included in the change token is compared against Change log.
About the Change Log
The Change log is stored in the EventCache table of the content database. This table stores the information about all content changes in the content database. Each entry in the Change log contains among other things the following information:
- A sequential number (Id) identifying the row in the event cache table
- The timestamp (EventTime) when the change has happened
- The type of the object (ObjectType) that has changed
- Identification (one or multiple IDs) of the object that was changed
- URL to the object that was changed (if the object has a URL)
- The type of modification (EventType) such as add, delete, … (see table below)
The following article contains some more details about these entries:
http://msdn.microsoft.com/en-us/library/bb861918.aspx
The change log stores information for a timeframe that can be configured in the Web application general settings:
About the Change Token
Each time the change log is queried for changes, it returns the changes together with a Change Token which corresponds to a database entry up to which the querying application has been synchronized. This change token is of the format Int1; Int2; Guid; Time; Int3.
Example: 1;1;df6828cf-c303-453e-9eb6-e733f85611ac;633782543234470000;1510
Where:
- Int1: Contains version information for the change token. Currently this value is always 1.
- Int2: The scope the change token has been requested for. For example, if you ask for changes in the site collection this value will have the scope “Site”. If you request changes for a sub site, it will contain the scope “Web”. See the following table for possible values. Related to content deployment and the underlying API, this value is always 1 as content deployment always generates a change token with a scope of the whole site collection.
- Guid: Guid of the object the change token belongs to. If the change token was requested for a site collection this will be the Id of the SPSite object.
- Time: Timestamp (in ticks) of the change the token belongs to.
- Int3: change number in the event cache table the change belongs to.
The following scopes are possible for a change token:
|
Change Token Scope |
Int value |
|
Content DB |
0 |
|
Site |
1 |
|
Web |
2 |
|
List |
3 |
When it comes to troubleshooting change token problems, it is also important to be able to analyze the change token itself – e.g. to see the date/time of the change it represents.
The following method allows to decrypt the change token into a better readable format:
{
string[] elements = cToken.ToString().Split(‘;’);
Console.WriteLine(cToken);
Console.WriteLine(“Version: “ + elements[0]);
Console.WriteLine(“Scope: “ + (SPChangeCollection.CollectionScope)Enum.ToObject(typeof(SPChangeCollection.CollectionScope), int.Parse(elements[1])));
Console.WriteLine(“Guid: “ + elements[2]);
Console.WriteLine(“ChangeTime: “ + new DateTime(long.Parse(elements[3])));
Console.WriteLine(“ChangeNumber: “ + elements[4]);
Console.WriteLine();
}
This code will create output like the following:
Version: 1
Scope: Site
Guid: a7e2f199-9176-42fc-9917-dabb913c66ee
ChangeTime: 25.05.2009 14:55:32
ChangeNumber: 1863
Version: 1
Scope: Web
Guid: dc4ffb77-6b1d-4437-88fb-6e3077128864
ChangeTime: 25.05.2009 08:01:09
ChangeNumber: 1832
Version: 1
Scope: List
Guid: ace2c0f6-ca8c-4765-93df-aa580d163b3e
ChangeTime: 25.05.2009 08:26:07
ChangeNumber: 1842
Change Log, Change Token and SharePoint Object Model
The Change log can be queried for changes at four different scopes: Content Database, site collection, site or list scope. Depending on the application and what scope the application is interested in defines which scope needs to be used. For example, if an application is only interested in changes in a specific list, it can query for changes at the list scope. On the other hand, if changes across the entire database need to be monitored, then the change log can be queried at the content database scope.
The GetChanges method can be used to query for changes:
SPSite mySite = new SPSite(“http://localhost”);
SPList myList = mySite.RootWeb.Lists[“MyList”];
// retrieve a change token that identifies the current point in time for
// this SPSite object. Result will be similar to this:
// 1;1;a7e2f199-9176-42fc-9917-dabb913c66ee;633788601321030000;1863
// the second “1” identifies the scope as Site Collection scope
SPChangeToken siteToken = mySite.CurrentChangeToken;
// retrieve a change token that identifies the current point in time for
// this SPList object. Result will be similar to this:
// 1;3;ace2c0f6-ca8c-4765-93df-aa580d163b3e;633788601321030000;1863
// the “3” identifies the scope as list scope
SPChangeToken listToken = myList.CurrentChangeToken;
// Get all changes available in the change log of the site collection
// be aware that only the first 1000 changes are returned.
SPChangeCollection changes = mySite.GetChanges();
// Get the first 1000 changes in the site collection since siteToken
SPChangeCollection changes = mySite.GetChanges(siteToken);
// Get the first 1000 changes in the list since listToken
SPChangeCollection changes = myList.GetChanges(listToken);
// this will not work as the scope of siteToken is for a site and cannot be
// used for a list
SPChangeCollection changes = myList.GetChanges(siteToken);
When you have a SPChangeCollection you can evaluate each change individually. Each SPChange object exposes the properties:
- ChangeToken – change token for this change
- ChangeType – see here
- SiteId – the unique ID identifying the site collection
- Time – the timestamp when this change happened
You can cast the individual SPChange objects to the exact change to get further information. For example, SPChangeWeb exposes the ID of the changed site. SPChangeFolder exposes the unique ID of the folder and the unique id of the site.
{
Console.WriteLine(change.ChangeToken);
DecryptChangeToken(change.ChangeToken);
Console.WriteLine(change.Time + ” – “ + change.GetType() + ” – “ +
change.ChangeType);
// get unique ID of the site collection the change was in
Guid changeSiteId = change.SiteId;
// handle changes to sites
if (change is SPChangeWeb)
{
SPChangeWeb changeWeb = change as SPChangeWeb;
// get unique ID of the changed site
Guid changeWebGuid = changeWeb.Id;
}
// handle changes to folders
if (change is SPChangeFolder)
{
…
}
…
}
Permalink
Hi Stefan,
I have been investigating the Change Logs in one of our Content Database due to performance problem. Regarding the "ChangeType", I did check all the ChangeType you point in this blog, but I need to map the EventType Database Value with the ChangeType Enumeration value. Most of the values I can find in the Internet for Add, Delete, Update, etc. But when I querry the Content DB in the EventCache table, (I don't work directly in the Content DB, just a copy of the Backup Database), I have a EventType "134217728", which I cannot find the enumeration mapping anywhere, could you please help me to find an exhaustive mapping between the Database EventType Value (from the EventCache) table, and the ChangeType enumeration?
Thank you in advance for your help.
Kind regards,
Thanh-Nu
Permalink
Here is the mapping:
Event Type Int Value (dec)
Add 4096
Modify 8192
Delete 16384
Rename 32768
Move Into 65536
Restore 131072
RoleAdd 262144
AssignmentAdd 524288
SystemModify 1048576
MemberAdd 2097152
MemberDelete 4194304
RoleDelete 8388608
RoleUpdate 16777216
AssignmentDelete 33554432
MoveAway 67108864
Navigation 134217728
Permalink
Hi Stefan,
Thanks for your quick reply; it helps us to interpret the peaks of our curve from the EventCache data. Strangely, during the last 3 months, now and then we have performance issues in our publishing site, each time the problem is related with SQL Server, and the work around consists of watching closely the CPU usage of the SQL Server, when it is too high, we run a SQL job that recalculate the statistic which helps SQL to optimize the use of caching, and this solve the performance issue. We apply this work-around, but still cannot understand what cause the performance issue, and why rerun the statistic optimization job in SQL helps to solve the problem. I am analyzing the EventCache table to find a correlation, and all peaks correspond with "Navigation" events, and those peaks matched with the periods where the performance issues happened. First I guess that when the content manager delete lots of sub sites in the content, the performances issues occur. (This is a direct editing publishing site, because the staging, and production environments strategies cannot survive due to content deployment problems together with variations 🙂
Now according to your mapping above, the peaks are really "Navigation" events and not "Delete" events. At this point, I get lost, and don't know what can cause the update of "navigation" to create such peaks recorded in the EventCache?
Have you ever run into such a case? With your experiences, I would hope you can enlighten us, because we are really sinking in the darkness with this recurrent performance issues.
Thank you so much for your time, and for sharing many precious information.
Permalink
Hi Thanh-Nu,
unfortuntelly I haven't looked into the details why Navigation events would be created as I never had a need to look into this. I would need to do some research on this topic. Best would be to open a support case with Microsoft to get this analyzed.
Cheers,
Stefan
Permalink
Hi Stefan,
I want to extract values from SP Change Log for audit purpose. I can easily get the changed values but didn't find any way where I can get the 'Before Change' value. Can you pls throw some light on this approach..Pls also let me know if considering SP Change log for auditing purpose is correct approach.
Permalink
Need your help in resolving below issue:
We have share point authoring and production Global portal setup. We have created below jobs to deploy content from authoring to production
1) Quick deploy Job- Runs every 15 min
2) Full deploy Job – We ran this job initially to move entire content from Authoring to Production.
3) Incremental deploy Job – Runs Every 1 hour
Incremental job is working on schedule basis (every 1hr) without any issues since 1 year (may be some minor issues which got fixed)…Now we are facing below issue for this job when it’s trying to import content into production
(Export is working fine, its able to export the content. But Import is throwing below error)
"Minor Version limit exceeded, publish to major version…"
Below steps we followed
1) We have checked both in authoring and production for minor version limit. We haven’t found any issues.
2) We have created new individual jobs, but its not successful
3) Finally, we have run Full Deploy Job we thought it may fix the issue. It exported all the content but import
getting below error
" FatalError: The URL 'Pages/xxxx.aspx' is invalid. It may refer to a nonexistent file or folder, or refer to a valid file or folder that is not in the current Web."
At this stage i have steped in to fix all the issues. Now we are facing new issue with export, when we run incremental job. Its throwing below
error:
[7/26/2013 9:00:08 AM]: Error: The changeToken refers to a time before the start of the current change log.
[7/26/2013 9:00:08 AM]: FatalError: The changeToken refers to a time before the start of the current change log.
at Microsoft.SharePoint.Deployment.SPExport.ThrowInvalidChangeTokenError(DateTime minChangeTime, Int32 minChangeNumber)
at Microsoft.SharePoint.Deployment.SPExport.GetIncrementalChanges()
at Microsoft.SharePoint.Deployment.SPExport.CalculateObjectsToExport()
at Microsoft.SharePoint.Deployment.SPExport.Run()
I have checked the setting in CA for Change log. It set to "Never" option for "Delete entries in change log". As per my analysis, until change token date refers to any date in change log this issue wont get fixed.
Running Full deploy job will reset change token date. Is there any risk running Full deploy job.
Need your help to solve the change token issue.
Sorry for such a lengthy description!!!
——————————————————————————–
Srini
Permalink
Hi Srini,
a couple of things:
1) never mix full and incremental deployment. You will end up with load of inconsistencies as full deploys less than incremental.
2) if you have (e.g.) a picture which exists as minor version 1.1 in the source and deploy this to the target it will be imported there with a minor version of 1.1 or similar.
Later when the item is deployed again – e.b. because referenced by a modified page – it will get imported as version 1.2 or similar although the version in the source has not been increased.
That can go on 254 times – then the maximum minor version limit has been reached and you run into this issue.
To avoid such problems you should ensure that referenced documents and items always exist as major version.
If you need additional help, please open a support case with Microsoft.
Cheers,
Stefan
Permalink
Hi Stefan,
This is just a great post that you have posted here as well as the one at this link (http://blogs.technet.com/b/stefan_gossner/archive/2008/03/13/moss-common-issue-incremental-deployment-fails-with-the-changetoken-refers-to-a-time-before-the-start-of-the-current-change-log.aspx).
However I am unable to figure out my issue. I have a content deployment job incremental and Full. Full deployment job is working fine, however incremental is failing for the same issue. I wanted to know the steps to solve the issue.
Please let me know is there a way to reset the ChangeToken time hold by the job. How do I make it work again
Permalink
Hi Saboor,
sounds as if you are running into this issue:
http://blogs.technet.com/b/stefan_gossner/archive/2010/12/29/interesting-changetoken-problem-when-mixing-complete-and-selective-deployment.aspx
Cheers,
Stefan
Permalink
Hi Stefan,
Is this a known issue? Our target content DB is bigger than our source content DB. We have daily scheduled content deployment job. What we found was that on the target site collection, random pages have higher version history when compared to same source page.
Why would the version on the target page automatically increase when source page version does not on deployment?
Appreciate your thoughts.
Thank you
Nishan
Permalink
Hi Nishan,
this is normal: content deployment deploys referenced items when deploying a page. E.g. if you create a new version of a page in the source by changing some text, then all the images referenced in the page will be deployed as well – and you will get a new version
of these images in the target.
So you might have many more versions of the referenced resources in the target than in the source.
Cheers,
Stefan
Permalink
Just to add: the version number on source and target are not related. You cannot expect to have the same versions on source and target.
If a page is referenced from another page, then it is deployed together with the page to ensure that no broken links exist in the target. That is required as content deployment cannot spy into the target to see if the page is there already or not. So to prevent
broken links content deployment will deploy pages where a hyperlink from another page points to as well.
If a page is linked from various different pages, then the number of versions of this page will be significantly higher than on the source. E.g. there might only be one version on the source if this is (e.g.) a legal disclaimer which got never changed. but
might have hundreds of versions on the target.
Its not that the page gets a new version for every reference – but for every deployment which includes at least one page referencing this specific page.