Author Topic: False Alarms in Cluster  (Read 4236 times)

Offline Jordan

  • Contributer
  • *
  • Posts: 17
  • Country: gb
  • Karma: +0/-0
    • View Profile
False Alarms in Cluster
« on: September 27, 2017, 05:45:38 AM »
Hi all,

I have a bit of a strange issue that I'm sure I've seen before but can't remember how I fixed it (if I even did fix it!)...

We have a cluster of 2 controllers which when logged into show the group alarm status as Major. If I log into cluster element 1, the alarm status of 1 is clear, and 2 is major. If I then log into cluster element 2 the status of 1 is major and 2 is clear. Whichever element I log into always shows the other as having the alarm.

Any ideas what could be causing this and how to stop the false reporting of alarms? There is nothing wrong and no service impacting issues, but the alarm needs to be cleared.

Cheers,
Jordan


Offline sarond

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1404
  • Country: au
  • Karma: +73/-0
    • View Profile
Re: False Alarms in Cluster
« Reply #1 on: September 27, 2017, 05:57:50 AM »
What is the actual alarm showing? Not the group alarm, the Alarm Details.

Offline Jordan

  • Contributer
  • *
  • Posts: 17
  • Country: gb
  • Karma: +0/-0
    • View Profile
Re: False Alarms in Cluster
« Reply #2 on: September 27, 2017, 06:02:45 AM »
Hi Sarond, that's the issue - the alarm details page has no alarms on either controller

Offline sarond

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1404
  • Country: au
  • Karma: +73/-0
    • View Profile
Re: False Alarms in Cluster
« Reply #3 on: September 27, 2017, 06:10:29 AM »
From memory I think disabling SNMP and re-enabling can refresh the alarm state.
System Properties/System Administration/SNMP Configuration

Offline Jordan

  • Contributer
  • *
  • Posts: 17
  • Country: gb
  • Karma: +0/-0
    • View Profile
Re: False Alarms in Cluster
« Reply #4 on: September 27, 2017, 06:27:33 AM »
So, I've done that and it does clear the group alarm status on both controllers! But only for about 30 seconds  :( after that I get the original issue back again and again, neither controller has anything on the alarm details page.

Offline sarond

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1404
  • Country: au
  • Karma: +73/-0
    • View Profile
Re: False Alarms in Cluster
« Reply #5 on: September 27, 2017, 06:35:32 AM »
Does anything show if you ALARMS in the maintenance commands

Offline Jordan

  • Contributer
  • *
  • Posts: 17
  • Country: gb
  • Karma: +0/-0
    • View Profile
Re: False Alarms in Cluster
« Reply #6 on: September 27, 2017, 06:38:23 AM »
Both show no alarm

Offline sarond

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1404
  • Country: au
  • Karma: +73/-0
    • View Profile
Re: False Alarms in Cluster
« Reply #7 on: September 27, 2017, 06:45:52 AM »
OK, I'm at a bit of a loss now.

I'll let others chime in.

Offline Jordan

  • Contributer
  • *
  • Posts: 17
  • Country: gb
  • Karma: +0/-0
    • View Profile
Re: False Alarms in Cluster
« Reply #8 on: September 27, 2017, 06:53:12 AM »
Appreciate the help sarond, I thought you had the answer when I saw it go clear!

If anyone else has any ideas I'd appreciate it!

Offline ZuluAlpha

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 692
  • Country: us
  • Karma: +17/-0
    • View Profile
Re: False Alarms in Cluster
« Reply #9 on: September 27, 2017, 08:12:16 AM »
I typically resolve this how sarond states, but when you disable SNMP Trap Configuration, wait about 60 seconds before you re-enable it.

Offline Jordan

  • Contributer
  • *
  • Posts: 17
  • Country: gb
  • Karma: +0/-0
    • View Profile
Re: False Alarms in Cluster
« Reply #10 on: September 27, 2017, 09:18:35 AM »
Just tried again doing SNMP and SNMP Trap config waiting 60 seconds but still no luck.

Also for what it's worth we've tried a sync between the controllers and that didn't help either.

Offline lundah

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1215
  • Country: us
  • Karma: +66/-0
  • Senior Chief Grunt
    • View Profile
Re: False Alarms in Cluster
« Reply #11 on: September 27, 2017, 12:25:43 PM »
Just to be clear and cover all the bases, you're using the same SNMP read-only string on both controller and they're in the same Administrative Group, right? Is there a firewall between the controllers?

I see this all the time on larger (10+ nodes) clusters, but for only 2 nodes that seems odd.

Offline Jordan

  • Contributer
  • *
  • Posts: 17
  • Country: gb
  • Karma: +0/-0
    • View Profile
Re: False Alarms in Cluster
« Reply #12 on: September 28, 2017, 04:36:43 AM »
Hi Lundah....So, yesterday when the others were helping I had noticed that the read only string was different and correct that, then followed their advice which didn't work.

Just went to sanity check everything again before I replied to you and saw the read only string had changed back to what it was originally. Switched that back to what it needed to be, disabled the admin group, disabled SNMP, waited 60 seconds, enabled SNMP and enabled the admin group and now both nodes are showing as clear!

One thing I can't remember is if at the top of the web page it normally says "Group alarms clear" or something? Both nodes only say "Node '[node name]' Alarm Status: No Alarm". Is that right and it only changes to group if there's an alarm on one?

Thanks for all the help everyone!

Offline august

  • Sr. Member
  • ****
  • Posts: 258
  • Country: ca
  • Karma: +9/-0
    • View Profile
Re: False Alarms in Cluster
« Reply #13 on: September 28, 2017, 10:49:12 AM »
Major Alarm but System has no Alarms

Document ID HT1346  Version 5.0 Status Published  Published Date 09/06/2017
Products MiVoice Business,3300 ICP
Summary
Major Alarm but System has no Alarms
Question
Major Alarm but System has no Alarms
Answer
 

System is in a cluster and has a Major Alarm at top of ESM window and the Admin Group Alarm Summary shows a cluster element is in alarm, however Alarm Details show no alarms.

Solution

In the SNMP Configuration form

1. Verify "Enable SNMP Agent" is set to Yes

2. Verify all cluster elements have the same names in the Read only and Read/Write Community

NOTE: if "Accept Requests From All Managers" is set to YES - There is no need to configure IP addresses in the "Accept Requests from the following Managers" area of the form.

3. If all is configured properly - and there is still an active Admin Group alarm - Change the "Enable SNMP Agent" to "No", save, and then change back to "Yes" and save - the alarm should clear within a few minutes.


 

Sitemap 1 2 3 4 5 6 7 8 9 10