Setting EM Blackouts from the GUI and Command Line

Oracle Enterprise Manager provides you with the ability to monitor your environments and alert you once specified thresholds have been reached.  Blackouts allow you to suspend monitoring so you do not get notified.   This is useful for scheduled maintenance windows, such as cold backups, where the application and/or database may not be available.

As well, blackouts also suspends data collection for the given targets.  This means that certain metrics such as availability will not be affected.

To create a blackout from the GUI, login to Enterprise Manager, navigate to the target you would like to blackout and at the bottom of the page under Related Links you will see a Blackouts link.  You will be brought to the following page:





On the screen above you can view any blackouts that may currently be in effect as well as create new ones.  To create a new blackout click on the create button.



On this page you can create a name for the blackout, with the default being “Blackout-<timestamp>”.  You can also select the targets you wish to set the blackout for.   As you can see from the screenshot I am going to set a blackout for an infrastructure application server (infra10g).

You can also provide a reason for the blackout by click on the Reason drop down list.  Quite a few are available, from Server Bounce to Security Patch.    Jobs can be disabled by deselecting the Run jobs during the blackout checkbox.  If your applying a security patch then you may not want a scheduled backup to run as it will either error or cause problems.



The next screen allows you to select which components within the target will be blacked out.  I selected a full blackout but you can select certain members if the outage will only affect specific components.



This screen allows you to schedule the blackout.  It can either start immediately or you can choose a date along with a duration.  Blackouts can be repeating as well, so you only have to create one for that monthly maintenance window for example.



The last screen provides a summary and once you have finished reviewing click on the Finish button.

You can also set blackouts from the command line, which is useful if you have some maintenance scripts which are not executed from Enterprise Managers job system.   I’ve only tested this on linux but it should be the same for windows.

To set a blackout for a list of targets:

emctl start blackout <Blackoutname> [<Target_name>[:<Target_Type>]]…. [-d Duration]

To set a blackout for all targets on a host:

emctl start blackout <Blackoutname> [-nodeLevel] [-d <Duration>]

-nodeLevel tells the agent to stop monitoring all targets on the server.

-d Duration allows you to set a duration in the format of [days] hh:mm.   ex. 1 02:05 means the blackout will last for 1 day, 2 hours and 5 minutes.

For example, to use this in a script in which all targets will be unavailable you would start a blackout at the beginning of the script and stop it at the end:

cd $AGENT_HOME/bin
./emctl start blackout alltargets-myserver –nodeLevel
<Perform Maintenance Tasks>
cd $AGENT_HOME/bin
./emctl stop blackout alltargets-myserver

Troubleshooting

In case you hit issues with blackouts take a look at the following notes:

Subject:     Agent Blackout Initiated By Emctl Command Not Ending      Doc ID:     559577.1

Subject:     EMDiagkit Download and Master Index   Doc ID:     421053.1

Subject:     How to Troubleshoot the EM 10gR1 Blackout Sub-system      Doc ID:     284024.1

Subject:     Troubleshooting Blackouts in EM 10g Grid Control using EMDiag Kit      Doc ID:     300671.1

They provide alot of information and solutions to different scenarios.   I hit an issue over the weekend in which the blackout didn’t end properly.  When I tried to stop it from the command line:

[oracle@myserver ~]$ /u01/app/oracle/product/agent10g/bin/emctl stop blackout alltargets-myserver
Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0.
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved.
Blackout stop Error : Blackout name alltargets-myserver is invalid

When trying to end the blackout via Enterprise Manager:

Error stopping the blackout on "infra10g": ORA-20710: Agent-side blackouts cannot be edited or stopped ORA-06512: at "SYSMAN.MGMT_BLACKOUT_ENGINE", line 501 ORA-06512: at "SYSMAN.MGMT_BLACKOUT_ENGINE", line 3262 ORA-06512: at "SYSMAN.MGMT_BLACKOUT", line 74 ORA-06512: at "SYSMAN.MGMT_BLACKOUT_UI", line 1167 ORA-06512: at line 1 .

To fix this problem I performed the following:
1. Shutdown the agent on the target server myserver
2. Removed the blackouts.xml file under $AGENT_HOME/sysman/emd
3. Used note 421053.1 to install the EMDiag kit
4. Logged in as sysman on the Enterprise Managers repository database and executed the following query:

select blackout_guid, blackout_name
from mgmt_blackouts;

BLACKOUT_GUID                    BLACKOUT_NAME
-------------------------------- ----------------------
30E2956CA329F0E59FBDF50951F2578E alltargets-myserver

5. Then executed:

exec mgmt_diag.KillBlackout(HEXTORAW(‘30E2956CA329F0E59FBDF50951F2578E’));

6. Restarted the agent on myserver and when I looked in Enterprise manager the blackout had cleared.

I have seen the command used above for some other scenarios but not this one specifically.   Before executing any commands in your environment please test first.