Grid Control 11g–Blocked Agents

Today I noticed that one of my Grid Control agents was down. From the Grid Control homepage for the target:



I thought it had just crashed so I checked the status from the server:

[oracle@myserver bin]$ ./emctl status agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 11.1.0.1.0
OMS Version : 11.1.0.1.0
Protocol Version : 11.1.0.0.0
Agent Home : /u01/app/oracle/product/grid11g/agent11g
Agent binaries : /u01/app/oracle/product/grid11g/agent11g
Agent Process ID : 29687
Parent Process ID : 29661
Agent URL : https://myserver:3872/emd/main/
Repository URL : https://myserver:4900/em/upload
Started at : 2011-06-23 02:06:56
Started by user : oracle
Last Reload : 2011-06-23 02:06:56
Last successful upload : (none)
Last attempted upload : (none)
Total Megabytes of XML files uploaded so far : 0.00
Number of XML files pending upload : 432
Size of XML files pending upload(MB) : 52.54
Available disk space on upload filesystem : 2.74%
Collection Status : Disabled by Upload Manager
Last attempted heartbeat to OMS : 2011-06-24 10:37:34
Last successful heartbeat to OMS : unknown
---------------------------------------------------------------
Agent is Running and Ready


Notice the “Last successful heartbeat to OMS” is unknown. If I try and force an upload it errors:

[oracle@myserver bin]$ ./emctl upload
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD upload error: uploadXMLFiles skipped :: OMS version not checked yet. If this issue persists check trace files for ping to OMS related errors.


Checking the agent logs ($GRID_HOME/agent11g/sysman/log/emagent.trc):

2011-06-24 10:40:03,074 Thread-1091160384 ERROR pingManager: Did not receive valid response to ping "ERROR-Agent is blocked. Blocked reason is: Agent is out-of-sync with repository. This most likely means that the agent was reinstalled or recovered. Please contact an EM administrator to unblock the agent by performing an agent resync from the console. Please contact EM adminstrator to unblock the agent"


So the agent is out of sync with with the repository and is now blocked. I remember back in the 9i days to clean up an agent was a bit of a pain but in 11g its pretty simple. Log back into Grid Control, click on the Setup link located in the top right hand corner. Next click on the Agents link in the blue bar, followed by the Misconfiguration Alerts and Blocked Agents tab.



If you click on the agent itself (instead of selecting it and clicking unblock) you will be brought to the agents page shown below:



To resynch it click on the Agent Resynchronization button.



Make sure the Unblock agent on successful completion… is selected and hit continue. Once completed you may notice that the agent is still showing as unreachable. If you check via the server again you’ll notice that its working fine.  Notice the Last successful heartbeat and last successful upload rows.

[oracle@myserver bin]$ ./emctl status agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 11.1.0.1.0
OMS Version : 11.1.0.1.0
Protocol Version : 11.1.0.0.0
.
.
Last successful upload : 2011-06-24 10:44:36
Total Megabytes of XML files uploaded so far : 19.37
Number of XML files pending upload : 0
Size of XML files pending upload(MB) : 0.00
Available disk space on upload filesystem : 2.76%
Last successful heartbeat to OMS : 2011-06-24 10:44:04
---------------------------------------------------------------
Agent is Running and Ready


After a short period of time it will show as up in the Grid Control Console as well.