IP phones rebooted for unknown reason

[mszodiac]

For some reason, all the IP phones on one of our sites rebooted today.  It was a short outage (but disruptive). The same thing has happened last week so I'm trying to find the root cause.

On 3300 ICP rel 6.0 system, the logs show:

CompleteFsmRequest: MID:987232a2 CID: 7efba70 PCC[ 0x7EFBA70 ]:Tx[ TDM, PortId:21, CompId:16, DN: ] Rx[ IP_PHONE, PortId=1116, CompId=1, DN:47004 ] completed in [ 10010 ] msec in state [ RX_CLOSED ] with a failure [ FSM_RX_CLOSE_TIMEDOUT ]
Module  Main
File Name and Line Number  IPSwSP_fsm.cpp;5266

Then:
ICP has lost contact with ( (08-00-0F-7B-2C-74)), network driver cluster pool free: 6127 and low water mark: 3896
Module  Main

And went on with the IP/XNET trunks dropping, then the IP phones re registering back to the primary controller.

Now if it was failure with the primary controller, I would have expected the phone to failover to the secondary controller.

Our phones are on a different VLAN than the ICPs.  The Cisco router and the switches do not have logs/error that can indicate a reason of the reboot.  No network changes were done that could have caused this either.

Has anyone seen this before? Any thoughts? Thanks for all your input.

[ralph]

I once had a similar issue with a school district.  Every Friday at around 1:00 a whole group of phones became unreachable.
Turned out there was something in the Cisco that reset the switch (or router - I don't remember which) at that time.

Other than that, all I can say is that every time I've seen something similar it has always been something in the network.
It could be something as stupid as someone pulling a cable but that would still be considered networking issues.

As for your log, although I'm not certain, I believe it's just telling you that it lost it's connection to the phone.  It's not telling you why and if it was a networking issue it couldn't tell you why.

Ralph

[mszodiac]

Thanks ralph.  That's the tough part -having to prove it's the network.  I know how to check logs/errors on the Cisco devices, pretty much layer 1, and layer 2, and nothing else.

What's odd is it randomly happens. 

[mszodiac]

Will the phones reboot if their DHCP IP lease expire?

Cisco forums says CDP protocol which is enabled globally by default can cause problems (ie. reboots) on the Mitel phones.  Has anyone experienced the same problem?

[acraig00]

we had a similar issue and the root cause was the switches.

We had some new lenovo machines that caused ipv6 broadcast storms when in sleep mode. https://communities.intel.com/message/225025

Fixing the network stopped the phone reboots.

[mszodiac]

This is awesome acraig00! The machines that are having problems are in our network are also Lenovo!

Do you know what kind of fix was implemented on the network?

[mszodiac]

Just an update:

The 2 sites that had LAN outages where the IP phones rebooted had new PCs installed in around the same time as the outages.... and they're HP 800 ed with the faulty chipset!

I've escalated the issue to our desktop team!

[acraig00]

the fix was disable sleep and ipv6 on those machines until a driver update comes out

[mszodiac]

The current driver of the NICs is 12.10.30.0.  We are updating ours to12.12.80.19 shortly. It doesn't look like there any more activities on the Intel forums pertaining to the driver issue after they recommended 19.0 update.