One-Way Audio on Internal 5240 Calls

[wkgeorge]

We've been fighting this issue for the better part of a year now.  We have 4 3300 controllers in our main office and pretty consistently have one-way audio issues between 5240 phones.  The controllers are all on the same subnet and are all running 4.2 SP2 PR1.  This issue seemed to start when we replaced all our old 3300 controllers with the newer MXe-III (we had to upgrade the hardware in order to have capacity for enough memory to go to 10.0)  We use a separate VLAN for all voice traffic.

We have gone round and round with our vendor, who is working with Mitel to figure out what is going on.  Mitel's final determination is that we have some servers on our network broadcasting ARP requests and it's confusing the phones. 

To quote Mitel: "From the captures we can see the following:
Once the phone has received the signalling to send audio to the other destination it sends an ARP requested as a broadcast message to the network:
10359 2012-06-28 15:14:41.468495000 Mitel_12:43:f3 Broadcast ARP Who has 20.1.1.143? Tell 20.1.1.109 60  That ARP request never receives an answer, therefore the phone can not send the audio."


The ball is in our court now and I still need to determine if their conclusion is valid.  My hunch is that it isn't, as they've been running us in circles with different wire captures etc for months now and our network really hasn't changed.

I'm just posting this in case anyone has any insight that might help us get to the bottom of this issue more quickly.

Thanks!

[wiseguy]

Sending packets might need MAC information, For example, the computers Matterhorn and Washington are in an office, connected to each other on the office local area network by Ethernet cables and network switches, with no intervening gateways or routers. Matterhorn wants to send a packet to Washington. Through other means, it determines that Washington's IP address is 192.168.0.55. In order to send the message, it also needs to know Washington's MAC address. First, Matterhorn uses a cached ARP table to look up 192.168.0.55 for any existing records of Washington's MAC address (00:eb:24:b2:05:ac). If the MAC address is found, it sends the IP packet on the link layer to address 00:eb:24:b2:05:ac via the local network cabling. If the cache did not produce a result for 192.168.0.55, Matterhorn has to send a broadcast ARP message (destination FF:FF:FF:FF:FF:FF) requesting an answer for 192.168.0.55. Washington responds with its MAC address (00:eb:24:b2:05:ac). Washington may insert an entry for Matterhorn into its own ARP table for future use. The response information is cached in Matterhorn's ARP table and the message can now be sent. Make sure that ARP can pass.

[JoeShmo]

(disreguard the question about internal/external calls.. you mentioned its internal)
Is this only on 5240 sets, or is that just the only phone you use?
And I'm guessing this only happens sometimes?  Otherwise you're running for most of the year with not being able to use phones?
Is this going on for internal calls only or both internal and external?  Internal only would imply problems between phones, and external would be indicate problems between the phone and the MCD.
Since you are running 4x Mxe IIIs because of needing more memory to get to ver 10 (aka 4.2), I'm guessing you have alot of phones (compared to us; we are running 4x MXe-II, but on ver 11 (5.0), and around 550 phones.
If you're not seeing a reply to some ARP packets, that could be an issue.  Not sure if its with the phone or what, but you mention that the phones are on a seperate VLAN.  How many phones, and how many voice vlans do you have?  We have about 400 phones on one VLAN, and I'm thinking of cutting that back drastically, as we have tons of problems with all the phones hitting the TFTP server at once, and seperate vlans & DHCP scopes will allow us to spread the load out better, and greatly reduce the broadcasts per vlan.

Also, I'm a network guy alot more than a phone guy.  Your voice network would have to be absolutely flooded and/or dropping packets for a phone to not respond to an ARP request.  How often can you reproduce this problem?  Do you ever have this issue with two phones on the same network switch?

[wkgeorge]

Hi and thanks for the replies.  We only use 5240 phones and have about 230 of them in this location.  It is an intermittent issue, happens a couple times per day and seems to happen more frequently to 10 or so extensions, though they may just be complaining about it the most.  I couldn't tell you for sure if any of the phones having this issue are on the same network switch or not.  So far the issue has been between phones on either the same or different controllers, though.  Rebooting the phone set always temporarily resolves the issue.

The voice traffic is all on one VLAN at this location.  What I don't understand is that Mitel had our support organization do a Wireshark capture on our network to see if they could blame our network, and they came back saying that 3 of our print servers were spitting out too many ARP requests and overwhelming our network.  The print servers are on the Data VLAN, so I think they plugged into the wrong port on our switch when they did the capture, otherwise they shouldn't have seen any ARP requests coming from our data VLAN.  Just To check, yesterday I ran a Wireshark capture on the voice VLAN and as I expected, no ARP requests appeared from the data VLAN.  And the ARP requests were pretty minimal on the voice VLAN.

As you mentioned, I think if we had enough ARP requests to overwhelm our network, we'd be seeing a lot more issues than some phone calls getting one-way audio.  Intuition tells me this is a software bug related to 5240 phones that began when we moved to the new Mxe Controllers, but that's just a hunch.

[LoopyLou]

You may be on to something as the 5240 is an older phone. They may not even test it on newer software.

Product support will use whatever capture they are sent so if the capture was taken in the wrong place it is possible the data they are working with is flawed. Its pretty easy to identify packets coming from other devices in a wireshark capture simply by looking at MAC addresses. I think Mitel phones are something like 08 00 OF ( or something like that ). Anything from other MAC addresses in the capture would be devices that should not be there if the voice VLAN is only for phone. There are some exceptions such as voicemail, Enterprise Manager Prairie Fyre etc

It would be helpful to see if all the phones were on a single switch as we did have a site at one point that had intermitant one way audio that turned out to be core switch hardware. I can't remember exactly what it was but when calls got above a certain number they sort of "overflowed" to use an additional piece of hardware which was defective. Problems only happened on the bad hardware. Anyway sorry can't be more helpful.