Phones causing broadcast storm?

[cholzhauer]

I have what seems to be a very strange problem.  We have about 65 phones on our 5000 and when I went to plug in four new phones yesterday, I managed to take other phones offline.  The phones exist on another switch in the same building (Procurve 5406s connected by 10GB fiber).  The phones rebooted themselves and would hang until I unplugged the new phones.  Normally the new phones that are plugged in don't even boot; they have a dimly lit LCD screen and all other buttons are on, as is typical when you first plug a 5340e in.

This seems like a networking issue to me, although I don't know why it would break all of a sudden.  The phones are in their own VLAN and the VLAN is identified as a voice VLAN on the switch.  I'm posting here to see if anyone can help me rule out the phone system as the culprit.  My vendor has mentioned we'll need to look at an add-on CPU card for the phone system soon and mentioned that we'd see odd issues if we didn't, but this issue doesn't seem like it could be related.

Any ideas/suggestions are appreciated.

Thanks

[Tech Electronics]

Cholzhauer,

Check the MAC addresses of the phones with the ones that go down and see if you have a conflict there. Have the new phones been pinned yet?

Thanks,

TE

[cholzhauer]

Hi,

The one phone I could not pin because it wouldn't boot.  I can check the other phones for a MAC conflict, but seeing as how I had multiple phones go down (and a computer that was on a different subnet) I sort of ignored that possibility

[Tech Electronics]

Cholzhauer,

Do you static assign the IP addresses or do you use the DHCP server off the 5000?

Thanks,

TE

[cholzhauer]

We use Windows Server 2012 R2 for DNS

[Tech Electronics]

Cholzhauer,

Um, I am going to take that as a no we don't statically assign IP addresses and that you meant to say the 2012 Server handles our DHCP requests for the phones.

So, have you tried just plugging in one of the phones at a time on the Procurve 5406s?

1. Take one 5340 and plug it in and see if the problem occurs.
  a. No Problem: Unplug the new phone and plug in another one.
  b. Phones Drop: Unplug the new phone and see if they come back, try another one of the new phones.

Repeat this process to see if the problem persists with all the new phones on the Procurve 5406s or just one of them. If it is just one of them get the phone replaced. If it happens on all of the phones then we go to step 2. If it happens on zero of the new phones on that switch then try two and then three at a time until it does happen.

2. Take on of the new 5340 phones and plug it into the switch of the phones that are dropping.
  a. No problem: Unplug the new phone and plug in another one.
  b. Phones Drop: Unplug the new phone and see if they come back, try another one of the new phones.

Repeat this process to see if the problem persists with all the new phones on the switch in the other building or just one of them. If it is just one of them get the phone replaced. If the problem persists on all of the phones then we need to look at other factors.

Another test you could try, if you have a PoE brick is plug in the Auxiliary NIC on the back of the HX-5000; if you have a CS-5000 then this is not an option as there is only one NIC. Then statically assign a known open IP address, subnet mask, and gateway, along with the VLAN ID information on each phone and see if the problem occurs. This will eliminate your physical switch that the 5000 resides on as the problem. 

As for adding a PEC or a PS-1 what other issues are you experiencing that your Vendor would suggest those as a solution to your problem?
How many phones do you have on the 5000?
How many voicemail ports do you have?
Are you having issues with the phones not responding to button presses?
Does your Auto Attendants seem to be having issues with callers or users are not able to access it?

Thanks,

TE

[cholzhauer]

Sorry, you're correct, everything is DHCP.

Note that everything works properly until we plug in these "new" phones.  Everything runs perfectly until we plug in these new phones, which would take our active count to about 68.  CPU usage on the system is under 10%

We probably have 60 voicemail boxes on the system. 

No issues with button presses.

The vendor has not suggested any fixes yet

[cholzhauer]

I can confirm the phones work...if I unplug a working phone and plug in this new phone, everything works

[Tech Electronics]

Cholzhauer,

I kind of figured you meant it was providing DHCP, but you never know sometimes. Anyway, the addition of a PEC or PS-1 would not solve your problem with the few amount of phones and resources needed for your configuration from what you have explained.

So, have been you able to determine at what point the "new" phones take out the existing phones? Did you try everything on Step 1?

Thanks,

TE

[cholzhauer]

This appears to be a wiring issue?  We can plug the phones directly into the switch and have no issues.  If we plug the phones into any one of four cables in the cubicles, we get broadcast issues and the CPU usage on the switches spikes to over 65%

[cholzhauer]

This ended up being a switch issue.  HP found that one of the power supplies was bad, even though the LED's on front of the switch and "sh power" didn't show anything out of the ordinary.