Insufficient bandwidth at the phone - not server

[Heritage02Rider]

We are a fairly new MITEL 5000 shop. We have about 50 users with about the same number of IP phones (5360's) phones with 1 GbE switches in them. Our network is running 1 GbE over HP ProCurve A5120's. We are converged, but the phones are on a separate VLAN from the data traffic. On a few phones, mostly the same ones, we are randomly getting a message on the phone itself (not on the 5000) indicating INSUFF BANDWITH and the phone gets a busy signal. After a few moments the call returns.

I have replaced the patch wires at both ends with no noticeable change. The issue cannot be recreated at will. We tried hammering the network connection at the same time we are on a call and the call is perfect. Not even a flutter.

Aside from replacing the wire from the server room to the end user, is there anything else I can/should be looking at? Could replacing the phones be worth a shot? If so, will MITEL actually replace them if I no longer get the issue? How would I prove it and is there a log somewhere of this actually occurring.

Any ideas are welcome.

[Hovus]

I'd check switch logs for Tx/Rx errors. Try moving cable to new switch port. Also make sure QoS is enabled and prioritized for voice. Most of the time I see insufficient bandwidth alarms on non-VLANed networks and most of the time it's negligible. If you're actually having service impacting issues as a result, I'd take a look at your network infrastructure as a whole. Check for patterns on time of day, people effected, etc.

[acejavelin]

Insufficient Bandwidth is a system alarm, you will get that on phones with the Administrator flag, it doesn't display on the phones causing the issue you would have to check the system logs to see which ones are the cause. If it is one phone, it could be a bad patch cable or something connected to the PC port of the phone as well.

If you are not having any call issues and cannot find a cause, another solution is to go into the system flags and turn off the Insufficient Bandwidth alarm... sometimes this can be a false condition.

[Heritage02Rider]

Insufficient Bandwidth is a system alarm, you will get that on phones with the Administrator flag, it doesn't display on the phones causing the issue you would have to check the system logs to see which ones are the cause. If it is one phone, it could be a bad patch cable or something connected to the PC port of the phone as well.

If you are not having any call issues and cannot find a cause, another solution is to go into the system flags and turn off the Insufficient Bandwidth alarm... sometimes this can be a false condition.

I have to correct you on this one. The phone that is having the issue at the time IS the phone indicating the message. The phone screen displays INSUFF BANDWIDTH (sp may not be exact) and the user gets a busy signal and the third party gets silence. After a moment or three, the phone returns to normal and the phone call continues.

I have checked the logs on the 5000 and cannot find any reference to this happening. Perhaps I have not checked in the correct places, but I did check the seemingly obvious places. Also, there is no alarms being registered on the 5000.

The issue is at the phone.

[Heritage02Rider]

I'd check switch logs for Tx/Rx errors. Try moving cable to new switch port. Also make sure QoS is enabled and prioritized for voice. Most of the time I see insufficient bandwidth alarms on non-VLANed networks and most of the time it's negligible. If you're actually having service impacting issues as a result, I'd take a look at your network infrastructure as a whole. Check for patterns on time of day, people effected, etc.

Last night I checked and you may be correct. The engineer who setup my switch stack may not have enabled the qos properly. I noticed that all my ports were untrusted. So I set the "qos trust dscp" on each port requiring a phone and will see how this affects the users having the issue. Obviously I will have to wait a few days to see if it comes back. Will write back when I have more information.

[Tech Electronics]

Heritage02Rider,

First off that only comes into play if you are giving the phones and controller DSCP values. If you are that is great! On the phone system though it will show up under the IP Connections settings instead of IP Settings for the System.

System > Devices and Feature Codes > IP Connections > P6xxx >

Audio RTP Type of Service   184
Data Type of Service       184

The phones will get this either through LLDP or you manually set them when you bring up the phone and those will be 46 in case you needed to know.

As for the Alarm 032 Insufficient Bandwidth you are correct that it will show at the phone that is having the problem. Have you looked at your Network Diagnostics Log yet to see what it is saying?

Here is the description of the Alarm just for future reference of readers to this post.

Alarm 032 indicates that the IP network does not currently have enough bandwidth to support the IP call that is connected. This alarm is generated if the IPRC/IPRA detects that the audio-receive timer has expired without receiving the first audio packet or that the Average In Time Frame Percentage has fallen below the threshold specified in Database Programming. When this alarm is generated, a transient message is displayed on the affected IP phone. In addition, the administrator station displays SYS ALARM #32 X INSUF BANDWIDTH where x is the extension number of the affected device.

NOTE: This alarm is displayed only if the Insufficient Bandwidth Alarm flag is enabled in Database Programming (System\Flags). If generated, this alarm indicates a network problem, which may result in packet loss which may affect audio quality. Make sure the affected device is not behind a firewall or NAT.

Message Print Manual Descriptive:

A032 ALARM Ext. <eeeee> Received Insufficient Bandwidth
FIELDS: eeeee The extension number of the device that received the insufficient bandwidth message.
CAUSE: The system detected that the IP endpoint's audio-receive timer has expired without receiving the first audio packet or the average-in-time-frame percentage has fallen below the threshold specified in the Voice Audio Session Configure Command.
ACTION: This indicates a network problem. Poor network condition could cause packet loss or a firewall prevents the callers from receiving audio packets. Dump the network diagnostics file for more information.
NOTE There is a system flag that can disable this alarm.
Thanks,

TE

[Heritage02Rider]

TE - Thanks. That is very detailed.

I did not have the Insufficient Bandwidth Alarm enabled, so I never saw the message on my phone. Also, I assume this prevents the log from show it as well. I have now enabled this feature.

As for the DSCP, I am not a qos expert, so let me ask a question here.

My settings for Type of Service are set as you have indicated: 184/184.

You mentioned that 'if you are giving the phone and controller DSCP values'; I guess I assumed that having these set along with the standard MITEL DHCP option 125 with dscp=46v46s26 has set the DSCP accordingly and the phones are tagging 46. Is there anyway to prove that the packets are tagged? Will Wireshark show this?

Basically, how do I know if I am tagging properly.

-Karl

[Heritage02Rider]

Follow-up - I did a WireShark and got the following:

Src (phone) = 192.168.200.x
Dst (5000) = 192.168.200.x
Differentiated Services Field: 0xb8 (DSCP 0x2e: Expedited Forwarding; ...)

Src (5000)
Dst (phone)
Differentiated Services Field: 0x00 (DSCP 0x00: Expedited Forwarding; ...)

Appears the phones are using qos and the switch is not.

modified...
Src (5000-PEC)
Dst (phone)
Differentiated Services Field: 0xb8 (DSCP 0x2e: Expedited Forwarding; ...)

Apparently the PEC card is doing qos since the UDP packets emanate from there.

[Tech Electronics]

Heritage02Rider,

Enabling the Insufficient Bandwidth Alarm will only post the alarm to the phones that are set as Administrators and has nothing to do with it showing up in the Error Logs/Message Print.

As for your setup just make sure that both the base and the PEC have 184 setup on their IP Connection, but that only comes into play at certain times, it is the phones that you want to make sure are using it and they are; which is good. Nice to see there is someone else who knows how to use Wireshark and when!

Did you look at your Network Diagnostics Log yet? If you don't know where to find it let me know and I will explain it to you. Well I will just go ahead and explain as it isn't that hard to find.

Download your Error Logs by going through Tools > Error Logs. It is easier to just download everything so do that. Once it is done if you open up the compressed folder you will find a log that should start with an E{MMDD}A.ndl or something like that; the main point is the .ndl which stands for Network Diagnostics Log.

Thanks,

TE

[Heritage02Rider]

Thanks for the information and all your help. I am much smarter now.

However either my system is different or I am an idiot. Not find the "Tools>Error Logs", I did find the "Operations>Error Information...". So I was only able to get the Exxxxxxx.ndl via a Freeze. The regular logs, of which I downloaded all of them, did not have this file. Ran into a few things I never saw before, but successfully recovered. (Once frozen I was unable to unfreeze until I figured out how to turn off Automatic Diagnostics Delivery - good thing this was before the NCAA tourney!)

So it appears the log file is cutoff. I starts mid-stream in a message and only goes back a couple days. (13K) But there was good information there. I believe the "qos trust" has solved the local issue. That was my main focus. My phones that are remote (i.e. Mexico) are still getting the Insufficient Bandwidth. I am now focusing on my VPN to do qos, however I am smart enough to be dangerous with Cisco. I may utilize my SmartNet for this one. That's why I pay for it after-all.

The message below was before my change to the switches:
Insufficient Bandwidth: 03-18-2014 at 09:10 AM
   IP Station Extension: '2191' at IP Address 192.168.200.22 on Board 7
   VoIP Configuration Information:
      N/A
   IP Audio Quality Snapshot Values:
      Reporting ICP IP address: 192.168.200.201
      Reporting IP Address: 192.168.200.22
      Reporting Signalling Port: 6921
      Reporting Voice Port: 50030
      Far End IP Address: 192.168.200.203
      Far End IP Port: 6728
      RX Codex: 1
      Delay: 0
      Jitter Instant: 0
      Jitter Average: 0
      Jitter Buffer Overflow: 0
      Jitter Buffer Underflow: 0
      Jitter Buffer Average Depth: 20
      Jitter Buffer Max Depth: 20
      Snapshot Packets Lost: 0
      Packet Loss Max Burst: 0
      Packets Out of Order: 0
   IP Audio Quality History:
      Packets Lost: 11
      Packets Received Total: 177739
      Jitter Buffer Histogram:
        10-19ms:   0
        20-39ms:   0
        40-59ms:   0
        60-99ms:   0
        100-200ms: 0
        200+ms:    0
      Packet Loss Histogram: consecutive packets
        1:    11
        2:    0
        3-5:  0
        6-10: 0
        10+:  0
   Audio Samplings:
   Rec:    Time   Jitter   Pck Lst   Pck Rx   InTime%
   1   5   0   0   250   100
   2   5   0   0   250   100
   3   5   5   0   250   100
   4   5   5   0   250   100
   5   5   0   0   250   100
   6   5   0   0   250   100
   7   5   5   0   250   100
   8   5   5   0   250   100
   9   5   5   0   250   100
   10   5   5   0   250   100
   11   5   5   0   250   100
   12   5   0   0   250   100

As for my remote phones, this is all they print out:

Insufficient Bandwidth: 03-19-2014 at 04:44 PM
   IP Station Extension: '2196' at IP Address 10.52.66.45 on Board 7
   VoIP Configuration Information (Audio Session 10):
      VoIP DSP Number: 10
      Audio Frames Per Packet: 2
      In-Time Frame Percentage Threshold: 60%
      In-Time Frame Timeout: 5 secs
      Audio TX IP Address/Port: 10.52.66.45:50160
      Speech Vocoder Type: G711 Mu-Law
      Speech RTP Payload: G711 MU-Law
      Other Device System Echo Canceller Profile: No Echo
      Echo Suppression: FALSE
      Echo Saturation: TRUE
      Echo Suppression Sensitivity Level: 0
      Conn Dir: Two-Way; Call Session: 2196 (GW) 2619

[Tech Electronics]

Heritage02Rider,

That will teach me to try and explain from memory, especially since I am not getting any younger, and you are correct it is Operations not Tools to get to the error logs; at least you found them and that is what matters.

Now, I don't know what kind of pipe you have at both ends between the US and Mexico, but let's just try a little experiment with your codecs before you start changing a lot of stuff around.

According to your Network Diagnostics File you are running G.711 for your remote phones so let's knock that down to G.729 and see if the frequency at which this happens either lessens or goes away all together; I will bet on it being reduced with some instances still occuring.

Go to System > IP-Related Information > IP Connections > Remote {or whatever your Mexico Phones are using}

Now, that I look at your Diagnostics again it looks as though you may have you Mexico Phones as local phones since it is showing 2 audio frames per packet. Hmm, you may just want to change your Mexico Phones over to the Remote IP Configuration or at least seperate them from the Local ones so you can change the Codec without affecting all the local phones as well. If you need help with this let me know and I can walk you through it.

Thanks,

TE

[rombrigs]

Tech Electronics,

I have been reading this thread with interest becaused I am having the same issue with one branch location (other 8x work fine).

I would appreciate your assistance in changing the phones in this hunt group to the Remote IP Configuration you suggested.  Thank you,

-R

[Tech Electronics]

Rombrigs,

If you believe that is what your problem is then sure I can tell you how to do it, but first you must gather some information for me so I can be a little more sure that it is the solution for you.

1. Do all the phones at this branch go across an external network to reach the phone system they are programmed on?
2. Look at the programming of one of those phones. System > Devices and Feature Codes > Phones > Ext nnnn > IP Settings, what number is next to Call Configuration
3. Also look at NAT Address Type and determine if it is Native or NAT

By default there are two Call Configurations:
1. Local
2. Remote

We want to ensure that Remote is what they are using if they are indeed Remote Phones. The differences between the two are an increase amount of Audio Frames per IP Packet which will increase latency but reduce Jitter and most of Network Congestion. Then you have an increase in minimum playback time which will also increase latency, but also reduce jitter. Now, there is the major difference of Codecs used to compress DTMF and Audio. With Local we use G.711 for both which does not compress the audio enough for the human ear to really notice. With the Remote however by default we change DTMF to RFC 2833 and Audio to G.729 and the audio can go down to G.729B (VAD) which uses silence suppression to reduce packet size and traffic. These however will provide a noticeable difference in what most humans perceive as quality of audio, especially G.729B (VAD) which messes with the comfort noise they like to hear.

Thanks,

TE

[rombrigs]

1. Yes, they go across a MPLS QOS enabled converged network via a T1
2. 1
3. Native

Thank you!  -R

[Tech Electronics]

Rombrigs,

I am surprised you are having issues with an MPLS circuit and since the phones are showing up as Native then it is considered one network. Do you have QoS set up on the local network at that branch and if so is it the same as all the other sites? Have you checked the metrics on that MPLS circuit to make sure there are no issues?

It could be that the data cable the phone is on is having some problems, the patch cord could be faulty, the port they are attached to could be faulty, the programming of the switch could be wrong. What all have you done to figure out the problem and what were your results?

If you want to continue on with the changes then the only thing I would do, without some actually testing on the network, would be to change the Call Configuration of the phones that have the problem the most to 2 and see if it continues.  Just keep in mind that your users may think that the audio difference is an new issue so you will have to let them know there will be a noticeable difference in the quality of the audio.

Thanks,

TE