Challenge:
Before couple of weeks back, on our Publishing Instance (PI) we faced a strange issue. Whenever anyone published anything from our CM server via PI Server it was publishing item. But the tokens were as is for example $name. So, the live functionality stopped working.
After checking PI Logs we found following errors:
Template item {24C19083-4E4B-4F18-B4D8-4685B6B74CF3} not found for item /sitecore/content/Home/Test
We saw above error message for the first time. So, as you? After, that we tried to find any exceptions from log file. We found following. But they were after our schedule application pool recycle.
ManagedPoolThread #96 05:07:46 ERROR Failed to load item data
Exception: System.Exception
Message: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
Source: Sitecore.Kernel
at Sitecore.Data.DataProviders.Sql.DataProviderCommand.ExecuteReader()
at Sitecore.Data.DataProviders.Sql.SqlDataApi.CreateReader(String sql, Object[] parameters)
at Sitecore.Data.SqlServer.SqlServerDataProvider.LoadItemDefinitions(String condition, Object[] parameters, SafeDictionary`2 prefetchData)
Nested Exception
Exception: System.Data.DataException
Message: Error executing SQL command: SELECT [ItemId], [Order], [Version], [Language], [Name], [Value], [FieldId], [MasterID], [ParentID]
FROM (
SELECT [Id] as [ItemId], 0 as [Order], 0 as [Version], ” as [Language], [Name], ” as [Value], [TemplateID] as [FieldId], [MasterID], [ParentID]
FROM [Items]
UNION ALL
SELECT [ParentId] as [ItemId], 1 as [Order], 0 as [Version], ” as [Language], NULL as [Name], ”, NULL, NULL, [Id]
FROM [Items]
UNION ALL
SELECT [ItemId], 2 as [Order], 0 AS [Version], ” as [Language], NULL as [Name], [Value], [FieldId], NULL, NULL
FROM [SharedFields]
UNION ALL
SELECT [ItemId], 2 as [Order], 0 AS [Version], [Language], NULL as [Name], [Value], [FieldId], NULL, NULL
FROM [UnversionedFields]
UNION ALL
SELECT [ItemId], 2 as [Order], [Version], [Language], NULL as [Name], [Value], [FieldId], NULL, NULL
FROM [VersionedFields]
) as temp WHERE [ItemId] IN (SELECT [ID] FROM [Items] WITH (nolock) WHERE [ID] = @itemId)
ORDER BY [ItemId], [Order] ASC, [Language] DESC, [Version] DESC
Nested Exception
Exception: System.Data.SqlClient.SqlException
Message: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
Source: .Net SqlClient Data Provider
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning()
at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
at System.Data.SqlClient.SqlDataReader.ConsumeMetaData()
at System.Data.SqlClient.SqlDataReader.get_MetaData()
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
at System.Data.SqlClient.SqlCommand.ExecuteDbDataReader(CommandBehavior behavior)
at System.Data.Common.DbCommand.System.Data.IDbCommand.ExecuteReader()
at Sitecore.Data.DataProviders.Sql.DataProviderCommand.ExecuteReader()
ManagedPoolThread #96 05:07:47 ERROR Failed to sort items
Exception: System.Exception
Message: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
Source: Sitecore.Kernel
at Sitecore.Data.DataProviders.Sql.DataProviderCommand.ExecuteReader()
at Sitecore.Data.DataProviders.Sql.SqlDataApi.CreateReader(String sql, Object[] parameters)
at Sitecore.Data.SqlServer.SqlServerDataProvider.LoadItemDefinitions(String condition, Object[] parameters, SafeDictionary`2 prefetchData)
at Sitecore.Data.DataProviders.Sql.SqlDataProvider.PrefetchItems(String itemCondition, String fieldCondition, String childCondition, Object[] parameters)
at Sitecore.Data.DataProviders.Sql.SqlDataProvider.PrefetchItem(ID itemId)
at Sitecore.Data.DataProviders.Sql.SqlDataProvider.GetPrefetchData(ID itemId)
at Sitecore.Data.DataProviders.Sql.SqlDataProvider.GetItemDefinition(ID itemId, CallContext context)
at Sitecore.Data.DataProviders.DataProvider.GetItemDefinition(ID itemID, CallContext context, DataProviderCollection providers)
at Sitecore.Data.DataSource.GetItemInformation(ID itemID)
at Sitecore.Data.DataSource.GetItemDefinition(ID itemID)
at Sitecore.Data.Engines.TemplateEngine.AddSection(ItemDefinition item, Builder template)
at Sitecore.Data.Engines.TemplateEngine.BuildTemplate(ItemDefinition itemDefinition)
at Sitecore.Data.Engines.TemplateEngine.AddItemBasedTemplates(TemplateDictionary result)
at Sitecore.Data.Engines.TemplateEngine.InternalGetTemplates()
at Sitecore.Data.Engines.TemplateEngine.GetTemplate(ID templateId)
at Sitecore.Data.Managers.TemplateProvider.GetTemplateField(ID fieldId, Item item)
at Sitecore.Data.Fields.Field.GetTemplateField()
at Sitecore.Data.Fields.Field.GetSourceItem()
at Sitecore.Data.Fields.Field.GetInheritedValue()
at Sitecore.Data.Fields.Field.GetValue(Boolean allowStandardValue, Boolean allowDefaultValue)
at Sitecore.Data.Comparers.ComparerFactory.GetComparer(Item item)
at Sitecore.Data.Managers.ItemProvider.Sort(ItemList items, Item owner)
Nested Exception
Exception: System.Data.DataException
Message: Error executing SQL command: SELECT [ItemId], [Order], [Version], [Language], [Name], [Value], [FieldId], [MasterID], [ParentID]
FROM (
SELECT [Id] as [ItemId], 0 as [Order], 0 as [Version], ” as [Language], [Name], ” as [Value], [TemplateID] as [FieldId], [MasterID], [ParentID]
FROM [Items]
UNION ALL
SELECT [ParentId] as [ItemId], 1 as [Order], 0 as [Version], ” as [Language], NULL as [Name], ”, NULL, NULL, [Id]
FROM [Items]
UNION ALL
SELECT [ItemId], 2 as [Order], 0 AS [Version], ” as [Language], NULL as [Name], [Value], [FieldId], NULL, NULL
FROM [SharedFields]
UNION ALL
SELECT [ItemId], 2 as [Order], 0 AS [Version], [Language], NULL as [Name], [Value], [FieldId], NULL, NULL
FROM [UnversionedFields]
UNION ALL
SELECT [ItemId], 2 as [Order], [Version], [Language], NULL as [Name], [Value], [FieldId], NULL, NULL
FROM [VersionedFields]
) as temp WHERE [ItemId] IN (SELECT [ID] FROM [Items] WITH (nolock) WHERE [ID] = @itemId)
ORDER BY [ItemId], [Order] ASC, [Language] DESC, [Version] DESC
Nested Exception
Exception: System.Data.SqlClient.SqlException
Message: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
Source: .Net SqlClient Data Provider
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning()
at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
at System.Data.SqlClient.SqlDataReader.ConsumeMetaData()
at System.Data.SqlClient.SqlDataReader.get_MetaData()
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
at System.Data.SqlClient.SqlCommand.ExecuteDbDataReader(CommandBehavior behavior)
at System.Data.Common.DbCommand.System.Data.IDbCommand.ExecuteReader()
at Sitecore.Data.DataProviders.Sql.DataProviderCommand.ExecuteReader()
Totally new exception stacktrace. And we were not sure why they were in the logs? But by looking at StackTrace one thing we were sure, It is something to do with pref-etch cache building. And it had some co-relation with our issue. Which we fixed! Eager to know How? Would like to know the root cause? Would like to know the permanent solution? Easy, Easy we are going to share with you. Just keep reading!
Solution:
It was live issue, and to resolve it we tried following things:
- Cleared — items cache, data cache, SQL prefetch cache, and lot of cache for Master/web database etc – No luck yet
[To clear partial cache, We used Caching Manager -- http://marketplace.sitecore.net/en/Modules/Caching_Manager.aspx and our in-house tool. Also, this looked promising. But haven't tried it during our investigation -- http://marketplace.sitecore.net/en/Modules/Sitecore_Cache_Admin.aspx]
- Cleared full Cache — Using Cache.aspx — But no luck again
- We restarted our PI Application — App Pool Recycle and it worked!
So, after application pool recycle, Everything started working as it should be! All happy! But not we, and not you as well — Correct?. Because to fix any issue and make sure it doesn’t pops up again you should always try to understand root cause. And then try to find a permanent fix for it. Matches with your thinking? See, we got same thinking!
And guess what we did? Yes, you are right. We raised a Sitecore support case with our config files and log files. At the first instance they replied as follows:
Please try to increase the <setting name=”DefaultSQLTimeout” value=”00:05:00″ /> setting.
Then we checked our configuration settings, and we replied as below:
On our PI Server timeout is 50 Minutes.
<setting name=”DefaultSQLTimeout” value=”00:50:00″/>
<setting name=”DataProviderTimeout” value=”00:50:00″/>
And as we discussed earlier, It doesn’t seems to be a caching issue. Because we cleared full cache already and it didn’t work. And we think that Sitecore is storing Template information somewhere else and not in Caching layers.
And here comes the Sitecore reply, With as always through investigation and explanation:
I’ve investigated the scenario you provided me with and found that indeed SQL exception during templates initialization may break some part of Sitecore items until you recycle the AppPool. I’ve already registered this as a bug for current version of CMS and will inform you when it’s fixed.
Please note that Sitecore is affected by this issue only if exception is thrown during the templates initialization, which happens during the first request to the database items. The possibility of such event is low, but the issue still may happen.
Unfortunately, I don’t see easy ways how to fix the issue, so I cannot provide you with code based workaround. If the issue happened on your solution, you may call the database.Engines.TemplateEngine.Reset() method for each database object (it shouldn’t throw the NRE. If the exception is thrown, please check which property is null and let me know) to reload templates.
The templates subsystem uses its own internal caches in order to increase the performance and more effectively manage entities lifetime. That caches are managed by the TemplateEngine (which belongs to the database) and doesn’t interact with Sitecore Cache subsystem (I mean the Sitecore.Caching.CacheManager class and related ones).
Given that cache clearing doesn’t lead to the templates invalidation. We should explicitly call the database.Engines.TemplateEngine.Reset() method to reset the storage and initiate its entire reloading.
Basically, It has been reported as a BUG and it’s fix is not easy. But as an alternative way you can explicitly Invalidate templates by calling — database.Engines.TemplateEngine.Reset() method!
To be honest, We haven’t seen this error again. And haven’t tried this workaround. But Sitecore support guys reproduced the issue and tried this approach and it works! It worked for you and helped you to fix the live issue? If so, please share your experience with us!
In summary,
- Always understand root cause of any issue
- Try to understand the challenge and share your findings with Sitecore support guys as much as you can
- And Sitecore support guys rock, as always!
- Sitecore template has its own internal caches in order to increase the performance and more effectively mange entities lifetime. That cache are managed by the TemplateEngine!
Happy Sitecoring!
Filed under:
Publishing,
Sitecore,
Tips and Tricks Tagged:
Performance,
Publishing,
Sitecore Bugs