Hi Guys,

First day back to work after the Xmas holidays and first error of the year
Below 3 events that I found on a customer server, that was not reporting to Operations Manager anymore.

Event Type:    Error
Event Source:    HealthService
Event Category:    Health Service
Event ID:    1220
Date:        1/3/2012
Time:        4:22:36 PM
User:        N/A
Computer:    xxxx
Description:
Received configuration cannot be processed. Management group "xxxx". The error is 0x80FF0066(0x80FF0066).

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Event Type:    Error
Event Source:    HealthService
Event Category:    Health Service
Event ID:    7022
Date:        1/3/2012
Time:        4:22:35 PM
User:        N/A
Computer:    xxxx
Description:
The Health Service has downloaded secure configuration for management group xxxx, and processing the configuration failed with error code 0x80FF0066(0x80FF0066).

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Event Type:    Warning
Event Source:    HealthService
Event Category:    Health Service
Event ID:    7004
Date:        1/3/2012
Time:        4:22:35 PM
User:        N/A
Computer:    xxxx
Description:
The Health Service received a secure message from management group xxxx which was encrypted using the wrong public key.  This message has been discarded and the public key will be re-published.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

The root cause is a missing registry key on the server. Take a look to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\HealthService\Parameters\Management Groups\UCBOM2007\SSDB\References

On the non working server :

On a working server :

When you compare the both servers, you notice that the following registry key is missing :

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\HealthService\Parameters\Management Groups\xxxx\SSDB\References\92f8f803-0763-f491-2480-274bfc4126f9]
@="01020202020202020202020202020202020202020200000000000000000000000000000000000000"

Just import the missing registry key from the working server to the non working server, and after a few seconds, the event below will appear, problem fixed

Event Type:    Information
Event Source:    OpsMgr Connector
Event Category:    None
Event ID:    21025
Date:        1/3/2012
Time:        4:28:37 PM
User:        N/A
Computer:    xxxx
Description:
OpsMgr has received new configuration for management group xxxx from the Configuration Service.  The new state cookie is "24 58 21 8C 7C D4 70 68 B2 87 1A 52 EA BE 7F 66 43 10 FC A7 "

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

If you want more information about this issue, I invite you to read the two followings posts :

http://thoughtsonopsmgr.blogspot.com/2009/02/eventid-7024.html
http://www.systemcentercentral.com/tabid/60/indexId/34058/tag/Forums+Operations_Manager/Default.aspx#vindex56241

Cheers
Christopher KEYAERT
http://twitter.com/keyaertc

operations manager · opsmgr · Scom

Hi All,

As you probably already noticed, Microsoft has released some days ago the new New Base OS MP 6.0.6957.0.

This management pack adds a lot of new cool functionalities that Kevin Holman explained on his blog : http://blogs.technet.com/b/kevinholman/archive/2011/09/30/opsmgr-new-base-os-mp-6-0-6956-0-adds-cluster-shared-volume-monitoring-bpa-and-many-changes.aspx

Now after several days, we’re starting to notice that this new management pack has also some “bugs” :

1: Report execution might fail because of lacking proper security settings on a Stored Procedure;
2: Importing the MP might fail when one has many overrides in place for diskspace. All because of a spelling error;
3: Knowledge is out of date for the new default values in the free space monitors;
4: The BPA monitors can be noisy for Server 2008R2 systems;
5: The “performance by utilization” report section dealing with Logical Disk % Idle time is upside down: the lowest idle time values are on top (100% Idle time is the lowest) and the highest idle time (anything close to 10% or even less) are on the bottom.
6: Impossible to deploy the new reports on SQL2005

Marnix Wolf and Bob Cornelissen started to report these bugs on their own blogs :

http://thoughtsonopsmgr.blogspot.com/2011/10/more-than-one-lets-ask-for-new-improved.html
http://www.bictt.com/blogs/bictt.php/2011/10/04/asking-for-a-quick-fix

Marnix also took the really good initiative to open a bug report on connect :

If you also noticed these bugs, please go to the link below and share your experience :

https://connect.microsoft.com/OpsMgr/feedback/details/692944/new-server-os-mp-version-6-0-6956-0-has-some-bad-bugs

This is not a flame or something like that, just an initiative to get a fixed and working MP very soon with all the great new stuff that was added and without at least most of the issues!

Cheers
Christopher KEYAERT
http://twitter.com/keyaertc

link · noisy · operations manager · opsmgr · Scom

Hi Everyone,

Last week, I’ve been called by one of my customer, who told me that they saw a huge performance degradation of their SCOM2007 R2 environment.

This customer has a quite big infrastructure, around 2200 servers, using Nworks MP, NetApp MP, Oracle Enterprise Manager connector, Opalis 6.3, Scheduling maintenance mode ….. This environment was working well since a long time and no major modification has been done so far.

The symptoms :

• Taking more 10 min for connecting to the console.
• Console navigation extremely slow or not responding.
• High CPU usage on the RMS.
• Very high CPU usage on the DB Cluster.
• Connection lost time to time to the DBs

After we checked everything we could, it was time to call the Microsoft Premier Support. After several days of investigation with them, we didn’t identity clearly the root cause. so, It was time to turn off one by one each connectors, each third party software that were connecting to Operations Manager.

It took time but we finally found which application was causing our nightmare since several days…. it was the Maintenance Mode Scheduling Tool.

This tool has been released part of the Administration Resource Kit for System Center Operations Manager 2007 R2 published by Operations Manager Product Team some months ago. More information available here : http://blogs.technet.com/b/jimmyharper/archive/2011/07/14/maintenance-mode-scheduling-tool.aspx

This is really an excellent tool that provide you an user interface for scheduling maintenance mode of your servers. My customer was using it a lot for disabling the monitoring during batch processing, planned reboot and so on.

Since we turned it off by disabling the rule Maintenance Mode Workflow, the Operations Manager infrastructure went back to business. So I continued to investigate on why this tools was causing all our troubles and I finally found this post : http://blogs.technet.com/b/momteam/archive/2011/06/21/schedule-maintenance-mode-reskit-tool-info.aspx

Best Practices:

• Do not go beyond 20 Jobs scheduled.  Anything over this will start to place too much load on your Root Management Server
• Do not schedule more then 20 items in one Job.  If you need to go over this please create a group and target this.  Note: The MP has a bug that only limits you to select up to six objects.  I have attached an updated MP that corrects this issue below.
• When scheduling a group make sure to select system.group then select the group.  If you select the group itself the tool lists the individual group membership.

My customer had more than 40 Jobs scheduled…. so we were clearly not respecting the best practice and that was explaining why our performance degradation came slowly day per day.

Currently, the only solution that we have for scheduling maintenance mode without using that’s tool is to come back to a PowerShell script that we will run from a windows schedule task.

Resources :
System Center Operations Manager 2007 R2 Evaluation : http://technet.microsoft.com/en-us/systemcenter/om/bb498233

Technet Forums : http://social.technet.microsoft.com/Forums/en-US/category/systemcenteroperationsmanager

Cheers
Christopher KEYAERT
http://twitter.com/keyaertc

high cpu usage · operations manager · performance degradation · scheduling tool · Scom

This session will provide you with an overview of the capabilities in OpsMgr 2012. With Operations Manager 2012 public beta, this session will get you started by explaining the new OpsMgr 2012 configuration – without an RMS!! We'll also look at configuring high availability and pooling of management servers, as well as other new concepts to be sure that you are ready to get the most out of the upcoming beta.

Watch the Video : Operations Manager 2012: Overview, Setup and Configuration

Resources :
System Center Operations Manager 2012 Beta : Installation step by step
System Center Operations Manager 2012 Beta : Available for download

Cheers
Christopher KEYAERT
http://twitter.com/keyaertc

capabilities · download · operations manager · opsmgr · Scom

As you are already aware, System Center Operations Manager 2012 Beta is now available for public download. The download link is available in my previous post : System Center Operations Manager 2012 Beta : Available for download

Once the file downloaded, you have to uncompressed it and let takes a look to the SCOM2012 installer.

Here the first screen of the new System Center Operations Manager 2012 installer.

So let’s go, click on install and we have now the license agreement.

In the following screen, you could choose which components you want to install. In my lab, I’m using a separate SQL Server which is also hosting the SQL Reporting service role. So I will uncheck the box for the Reporting Server component for the moment and we will install it later in this post, directly on the SQL server.

If you keep the box checked without having the Sql Reporting service installed on your current computer, you will be block later in the setup process.

So, we continue the setup process without the Reporting Server component. Here we just have to choose the Installation folder.

The new prerequisites checker is now in progress.

And the result. In my lab, two blocking prerequisites.

First one, I have to install the Report Viewer Controls. This is available as free download at the following url : http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=3841

The second one is the IIS 6 Metabase Compatibility role missing.

Go the Server Manager, add role and now check the IIS 6 Metabase Compatibility.

We recheck the prerequisites, and we just still have two warnings. As I’m using Virtual Machines in a lab environment, we could ignore it and continue the setup process.

Totally new environment, so we create a new management group.

We are now at the SQL part of the setup, I specified my SQL 2008 R2 server name and the setup directly detects that the SQL Server Full-Text-Search is not installed on it.

Go to the SQL Server and run the SQL installer.

In the setup, we just have to add a component to an existing instance and we check the Full-Text-Search component.

Go back to our SCOM2012 installation, the setup is now ok with the configuration of the Operational database and to let us continue.

Now, the parameters for the Warehouse database.

For the web console, we choose the Default Web Site :

Choose your authentication mode for the web console. As it’s for internal use only, the Mixed Authentication is the one that we have to choose.

The credentials part is one of the most interesting and important of the setup. I decided to use one dedicated account as Server Action Account. This account will be use for agent deployment, tasks, …. and a second account for reading and writing into the databases.

We have to create two news users domain.

Adding them as local admin of our future SCOM2012 management server and also as local Admin of the SQL Server.

Now, there is a part on which I’m not really sure. For installing the DB, is the setup will use my current credentials ? Or is the setup will use the service account that we provided. (I will update the post when I ‘ll receive the answer )

In doubt, we go to the SQL server credentials and we add the SCOM2012_DB account as SysAdmin. Like this, the setup process will be able to create the DBs and assign it the rights.

Go back to the SCOM Installation, We just have to provide the credentials of the two accounts that we created.

My lab VMs don’t have internet access, so I’m not able to participate to the Customer Experience Improvement Program, but if you could, please do it.

SCOM2012 seems to be part now of the Microsoft Update program. Option that I will test when I will have a lab with internet access.

The installation Summary where we could review all the information we provided.

Installation in progress……

Woua, setup completed

I opened the new SCOM2012 console… and first surprise…. I’ve got already some alerts

So, It’s seems that the Action Account that we provided during needs to have Read/Write to the WareHouse Db ? What for ? As we already provided a separate service account for the DBs access. Personally, I think that this is a little bug of this first beta release, I will retrieve it to the Microsoft Connect Site.

Ok, but how to solve this ? Simply go to the SQL server, in the credentials part. We open the properties of the Action Account (SCOM2012_AA) and in the User Mapping part, we set the same rights than the SCOM2012_DB account has on the OperationsManagerDW database.

As we already are in the SQL Studio, we could also remove the SysAdmin right that we granted to the SCOM2012_DB account during the setup process. This is not needed anymore.

Once done, all the alerts disappeared

Ok, so now, what’s about the reporting ? Go to the SQL server where we have the Sql Reporting Services installed and Run the SCOM2012 Setup, click on Install.

Select only the Reporting Server component.

Installation folder location.

Prerequisites checker.

We have to specify one management server of our SCOM2012 infrastructure.

We are now able to select the SQL Server instance for reporting services.

Red Cross, this is not good

Hopefully, Microsoft did a great work on the SCOM2012 installer, and simply by clicking on the Red Cross, we have the reason of the failure.

We simply have to start the SQL Server Agent service and to set it to Automatically.

We specify the DB reader account that we already used before.

If you server has internet access, please choose Yes.

As I said before, my lab servers don’t have internet access, so I will handle the updates manually.

The installation Summary report.

Go for the installation.

Once done, we’re now able to generate reports in your all new SCOM 2012 infrastructure.

Stay tune for next posts about SCOM2012

Cheers
Christopher KEYAERT
http://twitter.com/keyaertc

center operations · operations manager · opsmgr · reporting service · Scom

System Center Operations Manager 2012 Beta is now available for public download.

Here’s what’s new

• Rich application performance monitoring and diagnostics for .NET applications plus JEE application health monitoring
• Support for monitoring heterogeneous environments
• Integrated network device monitoring and alerts
• Simplified management infrastructure (no more RMS)
• Common console across datacenter and clouds with customizable dashboards

Microsoft Download link : DOWNLOAD
Microsoft Official announcement : Read it

Cheers
Christopher KEYAERT
http://twitter.com/keyaertc

beta · operations manager · opsmgr · rms · Scom

Hi all,

Today, I will present a little case that I need to deal with at one of my customer. My case Study is quite simple, I’ve got a Monitor Policy which creates an Incident/Ticket each time that a critical alert is raised in Operations Manager.

I will simplify this policy to the one below :

The scenario, a maintenance is planned on my Incident management platform, so at the convenience time I need to stop my Monitor Policy, wait the end of the maintenance window and after start my Monitor Policy again. As Opalis allows us to orchestrate our activities, I want to automate that task.

No object presents in the default integration packs allow us to do these actions easily.

· How could I stop an Opalis policy through another policy ?
· How could I schedule the start of an Opalis policy ?
· How could I pause my policy for some minutes ?

So we just have to create it with the Opalis Quick Integration Kit (QIK). J

Basically, I need to schedule an action, stop a policy, wait the end of the maintenance window and start my policy.

I invite you to read carefully the post of Adam Hall, Snr. Technical Product Manager for System Center Opalis/Orchestrator : Migrating a PowerShell script to an Opalis Activity

Thanks to this article and some PowerShell command I’ve been able to create three 3 new Opalis objects :

1. Sleep object: As input it simply takes a number of seconds and your policy will pause for the define time.
2. Start At: Specify a date and time for starting/continuing your policy.

3. Stop Policy: This object allows you to stop an Opalis policy from another one. It uses the sp_StopAllRequestsForPolicy SQL store procedure. As input it needs the Opalis Sql server name, database name, and the name of the policy that you want to stop.

Be carefull, the policy name that you specified must be unique, because this object will stop all the policies that are using the specified policy name. For example, if you’ve go more than one policy that is named “1.Monitor”, this object will stop all of them, and that certainly not what you want because that impact directly the others running policies. So be careful with it and always use an unique name for the policy that you want to top.
Now, we will use these news objects for stopping the Monitor Policy below during a maintenance window and restart it after, for that we created a Maintenance Window Policy.

Let’s configuring it

Monitor policy (Policy that we want to plan the stop)


Maintenance window policy

Start At: We decided to plan the stop of the Monitor policy the 05/17/2011 at 04:28 PM.

Stop Policy: We want to stop the Monitor policy (which is my OEA-IncidentCreation).

Sleep: Here we defined a maintenance window of 5 minutes.

Start Policy: Simply use the Trigger Policy object provided in the default integration pack for starting the Monitor Policy.

 

Let’s testing it

1. Now, we just have to start the Maintenance Window policy and go on weekend, Opalis will orchestrate the maintenance for us.

2. The Start At object will wait the specified date/time for releasing/continuing the rest of the policy.

3. At the defined date/time, the policy is released and our Monitor Policy well is stopped by the Maintenance Policy.

4. Our monitor policy is stopped

5. The sleep object will now pause the policy for the duration of your maintenance window.

6. When the maintenance window is ended, the monitor policy will start automatically.

7. Our Monitor Policy has been well started.

You can directly download the integration pack which is containing these news objects and the source code: Download.

Remember, this is provided “As is” without any support and/or guarantee.

Feel free to contact me through the blog comments for any questions and/or remarks.

Christopher KEYAERT

database name · integration kit · operations manager · opsmgr · Scom