Call quality/one way audio

[greenpaddyirishman]

Hi there...
So the past few weeks we have had users complaining about calls getting disconnected, static,call quality etc. We are trying to pin point the issue weather it be network based or local(3300). We have 25+ locations serviced by 3 MBG's with each location with a 3300.

In one particular location we have had the same 4 users complaining about calls getting disconnected. The call comes in fine to the secretary but once she transfers to the person it disconnects or cuts out after 30 secs or so. Also, direct calls to that branch experience the same cut off, static etc...now more offices are experiencing the same...its very sporadic to pin point or diagnose.

We operate in a MPLS environment with a 50MB pipe back to our 3 Main locations with MBG's...all other locations vary in range from 3MB to 12MB. QOS is set up on each branch router at 40%.

Anybody got any advise on where to start diagnosing further? We have engaged our phone service contractor but we want to make sure its not network based, we are not the phone guys!
PS..we do have access to all the MBG's and controllers if that helps.

Thanks in advance...

nosing these issues, we are not phone guys

[ralph]

It's going to be a network issue.
Odds are good that the QOS on the MPLS is not correct.

Ralph

[dilkie]

does kinda sound like network issues... but "static" is odd. Is this real "static", ie: scratchy audio, you can hear the person... or just white noise, which is actually what decryption errors in the audio stream sound like.

If these calls are going through MBG's, they have call stats in their logs. If you can provide a date/time of a particular event, you can find in the logs (or mitel support can) and see what's going on. This goes for disconnects too.

One thing, if you are prioritizing rtp over signaling and the pipe is full, sets will disconnect if their signaling "heartbeat" messages between the 3300 and the set is delayed for too long.

[greenpaddyirishman]

One thing, if you are prioritizing rtp over signaling and the pipe is full, sets will disconnect if their signaling "heartbeat" messages between the 3300 and the set is delayed for too long.

Thank you for the replies...since this has started a few weeks its hard to keep track of all the issues, exact dates, times, external/internal calls etc...apparently ATT was doing some construction on the line in one of the areas, the static issue has not been reported since, so at least for now we can rule that out.

The biggest complaint right now if one way audio, yesterday we had a person call our 1-888 number and dialed the extension in question. The person making the call could hear our employee but our employee could not hear her. The employee had the customer hang up and she called her back and the call went through fine.

I'm not too familiar about your comment regarding RTP and prioritizing, where could one look at that setup? As for looking at log files again I'm not too familiar with the MBG interface, there are so many log type to choose from!???

[greenpaddyirishman]

It's going to be a network issue.
Odds are good that the QOS on the MPLS is not correct.

Ralph, If all the branch routers have the same QoS as well as the main MPLS router(all set at 40%) wouldn't I see packet drops in the service-policy if this was indeed a QoS issue? I can attach the policy stats for the policy if that helps.

[ralph]

It's going to be a network issue.
Odds are good that the QOS on the MPLS is not correct.

Ralph, If all the branch routers have the same QoS as well as the main MPLS router(all set at 40%) wouldn't I see packet drops in the service-policy if this was indeed a QoS issue? I can attach the policy stats for the policy if that helps.

I would think the issue would be within the MPLS itself.  Not the router.  Been there, done that.
But that's speaking of static.  You may or may not see that in the VQ logs.

I'd expect the one way audio could be something else since it's inconsistent. 
I've very recently had an issue with one phone having inconsistent one way audio calls.  The phone was routed through a firewall VPN back to HQ.   Switched it to connect to a TW server and the problem went away.

Ralph

[VinceWhirlwind]

1. "static" is usually dropped packets - congestion and either QoS not working properly or simply not enough bandwidth provisioned, OR, a simple interface duplex mismatch somewhere on the critical path.
 
2. Calls dropping out after 30 seconds. Signalling issue, one end doesn't know the call has been established, so it is timing out.
 
3. One-way audio. Signalling issue or a firewall somewhere isn't letting through packets with cdertain RTP ports on them.

[mpprevost]

You are not alone my friend. A couple weeks ago we started having this very same issue minus the static. At first it was just in our water call center. They had about 15-20 calls that would either have no audio from the caller as soon as they picked up or they would lose the caller after about 15-30 seconds of being on the line. That same day it seemed to spread to the rest of the departments at our main building but didn't have any reports from remote buildings. All the issues were on inbound calls so we were assuming it was a PRI issue. I submitted a ticket with AT&T and of course they said it wasn't them. We later had one call reported that was ext to ext.  I got with the network admin who had some port mirrors setup to check for packet loss and found virtually zero. The port mirrors were setup to two of the controllers and one of the phones in the water department. We sent logs to Mitel and they saw nothing out of the ordinary.

At this point we have decided to replace the 3 controllers at our main building and replace the one at the public safety complex hoping it's an issue with the hardware on the controllers. We had planned on replacing them anyway as they're almost 10 years old, but since this issue popped up we put that on the fast track. I'll report back and let you know if it helped.

[VinceWhirlwind]

You can check how many packets were dropped on a call, after the event, by looking on the controller under "Voice Quality Statistics".
 
The problem calls I am looking at right now are showing "Jitter Buffer Underflow", so I am much better informed as to where the issue is occurring after looking at this form.

[mpprevost]

The problem calls I am looking at right now are showing "Jitter Buffer Underflow", so I am much better informed as to where the issue is occurring after looking at this form.

We looked at those as well. Some that were dropped were at 15k+, others were at 0.

[dilkie]

underrun is essentially equivalent to loss, a late packet got replaced with an audio gap, so if you are seeing a lot of underruns, network problems.

[mpprevost]

Just to give everybody an update as I was having this same issue. I'm not sure if any of this will help, but maybe so.

While replacing our controllers we found a ton of issues that had to be resolved. Our main controller had the NIC speed at 100/half instead of 1000/full. One of the controllers was on the wrong subnet mask (we recently had to rescope the building and missed it). AT&T checked their PRI circuit and since we're literally across the street they turned it down a little bit thinking it was coming in too hot. Since our issues were really only prevalent when our inbound call volume was high I had them check the cards within the CO considering one of those could be going bad. Also while replacing the controllers we found the dual framer that was being moved from old to new had a chip fall off of it, but after replacing it we didn't notice any improvement.

After all this our issues seemed to go way down with only a couple calls per day having issues. The phones that were consistently having issues we rebooted. Knock on wood we've been about 2 weeks now issue free. Hope this helps somebody.