ICP Comms Card

[mvaughn]

we currently have 4 phone switches, almost daily we receive the following major alarm which clears itself after about 2 minutes. Because this is happening at different times and clears itself after a few minutes, it has been difficult to troubleshoot. Has anyone else experienced this alarm?

      Alarm details on Sheriff
      Date: Tue, 30 Jan 18 04:57:41 -0500

      System alarm status: Major
      Number of categories: 1

      1) Category Name: ICP Comms Card
         Total: 4
         Unavailable %: 25
         Alarm Level: Major
         Minor Threshold: 0
         Major Threshold: 25
         Critical Threshold: 100

[sunspark]

Do you have The E2T card ?

[sunspark]

we currently have 4 phone switches, almost daily we receive the following major alarm which clears itself after about 2 minutes. Because this is happening at different times and clears itself after a few minutes, it has been difficult to troubleshoot. Has anyone else experienced this alarm?

      Alarm details on Sheriff
      Date: Tue, 30 Jan 18 04:57:41 -0500

      System alarm status: Major
      Number of categories: 1

      1) Category Name: ICP Comms Card
         Total: 4
         Unavailable %: 25
         Alarm Level: Major
         Minor Threshold: 0
         Major Threshold: 25
         Critical Threshold: 100

----------------------------------------------------------------------------------------------------------
There is a network connectivity issue or the far end node is out of service. 1.Check your system logs to determine to which node(s) you have lost IP trunk connectivity.
2.Check your network connectivity between impacted nodes and confirm that the nodes are in service.
3.Check your system logs to determine when the trunk went down.
4.Check your audit logs for any IP trunk configuration changes. If a change has been made, return to the original configuration.
5.Check your audit logs for any service maintenance commands. For example:•Check if the IP trunk was forced busy by looking for the following maintenance command variance:◾BUSY
◾BUSY XNET
◾BUSY TRUNK

•Enter the following command to bring the IP trunk in service variance:◾RTS
◾RTS XNET
◾RTS TRUNK


6.If the alarm is not resolved, collect required System Diagnostics logs and contact Mitel product support.
 

[ZuluAlpha]

If you go to Admin Group Alarm Summary when you see this it will tell you to which node it thinks is having the issue. It looks like Sheriff itself isn't having the issue but is losing connectivity with a different node. If it's always the same node the issue may be in the path between those two. If it's cycling nodes there may be a network connectivity issue coming in to the Sheriff switch.

Also review the Maintenance Logs on Sheriff. It will tell you why the alarm was raised. Maybe it drops a few percent of packets during the busy parts of the day.

[mvaughn]

Thanks everyone for the feedback, I have identified the node that is losing connection which is caused by our network providers. Is it possible to to change the the timers on that particular node where it doesn't notify us until after an extended period of time since this is a node that is not heavily utilized?

[VinceWhirlwind]

I have changed heartbeat/resiliency settings for sets, not for trunks, but I imagine you can change them:
Check your Controller Registry form under the various "Link Maintenance" options.
But before you do that, do you have a network monitoring tool running in your environment?
If you don't, then run a continuous ping (from main network to the 4th controller's IP address), piped to a txt file, and collect a good long snapshot of your connectivity.
 
From either of the above you should get a good idea of:
1/ frequency of interruptions
2/ usual duration of interruptions.
 
Use that information to decide whether increasing the IP Trunk heartbeat timeout up from 30s is likely to reduce the frequency of your alerts.

[SARTECH87]

we had the same issue not long ago and it was discovered by one of my colleague that it relates to the inactivity timer on the IPX trunk. If the timer is not configured, the ICP COM Card will trigger an alarm. We could not figure it out until she took some notes. Our network has more than 300 elements but we would get those alarms randomly not knowing why.

Configure the inactivity timer to 1 hr or 30 min, it has to match at both ends ( meaning that if one of you MCD is for a satellite office, your main PBX at the main office must match the same inactivity timer as your satellite ). If that inactivity timer is not configure and no calls are made, the link will remain up indefinitely preventing calls to be made.

Just try it, it may solve your issue

[mvaughn]

we had the same issue not long ago and it was discovered by one of my colleague that it relates to the inactivity timer on the IPX trunk. If the timer is not configured, the ICP COM Card will trigger an alarm. We could not figure it out until she took some notes. Our network has more than 300 elements but we would get those alarms randomly not knowing why.

Configure the inactivity timer to 1 hr or 30 min, it has to match at both ends ( meaning that if one of you MCD is for a satellite office, your main PBX at the main office must match the same inactivity timer as your satellite ). If that inactivity timer is not configure and no calls are made, the link will remain up indefinitely preventing calls to be made.

Just try it, it may solve your issue

Where could I find the inactivity timer on the IPX Trunk?

[VinceWhirlwind]

It's actually set on the ICP/PBX network element, I think.

[mszodiac]

we had the same issue not long ago and it was discovered by one of my colleague that it relates to the inactivity timer on the IPX trunk. If the timer is not configured, the ICP COM Card will trigger an alarm. We could not figure it out until she took some notes. Our network has more than 300 elements but we would get those alarms randomly not knowing why.

Configure the inactivity timer to 1 hr or 30 min, it has to match at both ends ( meaning that if one of you MCD is for a satellite office, your main PBX at the main office must match the same inactivity timer as your satellite ). If that inactivity timer is not configure and no calls are made, the link will remain up indefinitely preventing calls to be made.

Just try it, it may solve your issue

We had the exact same problem.  We also have over 300 elements.  The MCD can only assign 255 simultaneous link handles.  If you run the command STATE XNET ALL, you'll see the Link Handle number.  When all 255 link handles are "established", no other XNET trunks work (out-of-service).  The solution was to set the inactivity timer to 1 hour at both ends.