SCOM 2007 R2 : Configuration not loaded…again

Hi Guys,

If you follow my blog, we probably already read my previous post about : SCOM 2007 R2 : Configuration not loaded
Two days after this issue was resolved, my customer had again the exact same symptoms… First thing that I did, is to check the BaseManagedEntity table and  this time no inconsistency was found, everything seems to be ok.

After some search, I found that KB : Configuration may not update in System Center Operations Manager http://support.microsoft.com/kb/2635742/EN-US

The System Center Management Configuration service uses a timestamp to determine when new configuration data needs to be calculated for agents and management servers.  If the system clock on an agent is faster than the system clock on the Management Server, discovery data from this agent will set the timestamp for one or more managed instances hosted by that agent to the current agent system clock time.  The System Center Configuration Management service will delay calculating configuration updates for the instances on that agent until the system clock on the Management Server is current with the timestamp for that discovery data.  If the agent system clock was significantly faster than Management Server system time when discovery data was sent, or the agent continues to send data with a future timestamp, then it is possible that the management group would experience the symptoms listed above.

Setting the agent system clock time to match the Management Server system clock time will not reset the timestamp for the existing discovery data and the issue will remain until the Management Server system clock time exceeds the discovery data by the grooming interval, when the obsolete discovery data will be groomed normally.

Let check if we are in that case or not. The first thing to do is to run the following 3 queries :

Select GetUTCDate()as 'Current Time', MAX(TimeGeneratedOfLastSnapshot) as 'DiscoverySource Timestamp' from DiscoverySource Select GetUTCDate()as 'Current Time', MAX(timegenerated) as 'DiscoverySourceToTypedManagedEntity Timestamp' from DiscoverySourceToTypedManagedEntity Select GetUTCDate()as 'Current Time', MAX(timegenerated) as 'DiscoverySourceToRelationship Timestamp' from DiscoverySourceToRelationship

 

Capture3

As we could see, there is a difference of 3 hours 30 min between the current time and some data inserted in the DB. “We’ve got data from the future” Open-mouthed smile. We have now to identify which computer is causing that problem in our environment.

For that, we have to run 3 new queries : (I modified the queries to return the computer that has inserted data with more than 1 hour in advance.)

-- Find all computers with DiscoverySource Timestamp more than one day in future --
Select DisplayName, * 
from BaseManagedEntity
where BaseManagedEntityID in
 (select BaseManagedEntityId from BaseManagedEntity BME
  join DiscoverySource DS on DS.BoundManagedEntityId = BME.BaseManagedEntityId
  where DS.TimeGeneratedOfLastSnapshot > DATEADD (HOUR, 1, GETUTCDATE())
  and FullName like 'Microsoft.Windows.Computer%')

Capture4

-- Find all computers with DiscoverySourceToTypedManagedEntity Timestamp more than one day in future --
Select DisplayName, * 
from BaseManagedEntity
where BaseManagedEntityID in
 (select BaseManagedEntityId from BaseManagedEntity BME
  join DiscoverySourceToTypedManagedEntity DSTME on DSTME.TypedManagedEntityId = BME.BaseManagedEntityId
  where DSTME.TimeGenerated > DATEADD (HOUR, 1, GETUTCDATE())
  and FullName like 'Microsoft.Windows.Computer%')

This second query didn’t return any on environment. we will now run the last query :

-- Find all computers with DiscoverySourceToRelationship Timestamp more than one day in future --
Select DisplayName, * 
from BaseManagedEntity
where BaseManagedEntityID in
  (select BaseManagedEntityId from BaseManagedEntity BME
   join DiscoverySource DS on DS.BoundManagedEntityId = BME.BaseManagedEntityId
   join DiscoverySourceToRelationship DSR on DSR.DiscoverySourceId = DS.DiscoverySourceId
   where DSR.TimeGenerated > DATEADD (HOUR, 1, GETUTCDATE())
   and FullName like 'Microsoft.Windows.Computer%')

Capture5

It returned the exact same server that we had with the first query. Let’s have a quick look on that server. On the left the impacted server, on the right the Domain Controller located on the same site.

Capture

As you could see the time on the server is not correct. We have now to correct the time on the server, and also the existing data in the DB. For the time on the server, just modify it from the user interface, for the data in the data, we will have to run the following commands against the affected tables.

Update DiscoverySource
Set TimeGeneratedOfLastSnapshot = GETUTCDATE()
where TimeGeneratedOfLastSnapshot > GETUTCDATE()

Update DiscoverySourceToTypedManagedEntity 
Set TimeGenerated = GETUTCDATE()
where TimeGenerated > GETUTCDATE()

Update DiscoverySourceToRelationship 
Set TimeGenerated = GETUTCDATE()
where TimeGenerated > GETUTCDATE()

All data are now at the right time, a restart of the 3 SCOM services on the RMS should fix the problem.

Cheers

Christopher

Tweet about this on TwitterShare on FacebookShare on LinkedInShare on Google+Email this to someoneShare on TumblrPin on PinterestDigg thisShare on RedditFlattr the authorBuffer this pageShare on StumbleUpon

About Christopher Keyaert

Christopher Keyaert is a Consultant, focused on helping partners to leverage the System Center and Microsoft Azure cloud platform. He is also a Microsoft Most Valuable Professional (MVP) for Cloud and Data Center Management and a Microsoft Certified Trainer (MCT).
This entry was posted in Operations Manager. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *