All VOIP Devices Lose Network Connection

[CharlesAB]

What variable -- active or passive -- on a DHCP based network will cause all Mitel VOIP handsets to:

• Lose network connectivity and drop active calls.
• Repeatedly cycle through the typical screens as if the phone was just connected to a live POE and network port.
• Not respond to actual power cycling, DHCP lease revocation and renewals or POE switch resets / restarts.
• Recover connectivity w/in a time band ranging from minutes to several hours.

Background: About two months ago and over five months of flawless production service, we had all Mitel 53xx phones on a client network go offline and start cycling through the power off, Waiting for LLDP, software version, Waiting for DHCP, DHCP: Discovery screens repeatedly. This issue occurs intermittently throughout the week and normally between 16:00 - 17:00 during the business day. There could be no DT for 4 / 5 days in a week. Or, the issue will manifest 4/5 days of the week.

DHCP is handled via the environment's router and we've successfully tested separate networks managed by the same router with no issues while the outage is occurring on the site's primary network.

While these Mitel 53xx (5320 for instance) handsets are provided with a local IP address, they connect up with the client's remote, cloud-based VOIP service provider.

We have worked with both the VOIP service provider as well as the router support group to verify settings, observe inbound DHCP packet traffic during outages, etc. Nothing specific sticks out. We have assigned static IP addresses to test phones as with no change in behavior during a given outage. Unfortunately, the phones simply will not connect up with the network.  W/o fail, the VOIP phones will regain connectivity at some point during the evening hours.

We have been working to identify the issue -- likely something on the network or w/in the supporting switches which is actively or passively interfering with the Mitel phones.

There is no VLAN configured to be used with the VOIP phones. There are a total of 157 devices on the network.

The supporting switches are old with firmware that's ~10 years in age (inherited this environment); the switches do not retain settings when reset or power cycled. These are known issues. This client's IT hardware budget is challenged and challenging.

Any / all assistance or ideas are welcome.

Thank you!

[johnp]

These are in sip mode?

[acejavelin]

So look at the symptoms and what causes them...

All phones lose connection... look at things common to all phones. Router, ISP/Uplink, core switch. You must have at least 3 switches at this location, are they daisy-chained, or is each one connected individually to a router or master switch port?

Repeatedly cycles through power-up sequence? This is caused by power issue or connectivity... If they stay at DHCP and just time-out, they are not getting a DHCP response from the router. Check your DHCP lease time and pool size... 157 phones with no vlans implies there could be at least 157 other network devices, almost 300? If so, this network should absolutely have VLANs and unique broadcast domains... Still, if it's a flat network verify your pool size is big enough because a /24 network won't cut it, and if it's a tight pool size, lower your lease time to 4 hours or something.

When you say there is no response to power cycling the phone, you mean it isn't resetting or it isn't working? if you power cycle the phone and it doesn't fix the problem but it just works later, this tells you the problem is NOT the phone... But a connectivity issue.

Recovery time of hours to minutes isn't very helpful in this case.

My guess is one of a few things... DHCP (as noted above), a bad patch cable that is common to all devices. or a defective switch... Since you implied that all devices do this at once, start at the most common devices. Although the problem can start at a device and spread throughout the network, it usually doesn't, but I would be lying if I said I had never seen that happen before. Connect a PC with Wireshark capturing data... You don't have to a packet master to use wireshark, just look for anything unusual when this occurs. I ran into a site where a smart TV in the conference room was flooding the network at random times, basically grinding everything to a halt then it would just start again. Nobody even knew the TV hanging on the wall was plugged into the network, but a firmware update later and haven't been back to that site in over year.

And now I am rambling... Need more coffee...