IP phones buzz / feedback

[bhackbarth]

HX Controller cluster with v6, sets are 5330's. What is the correct troubleshooting procedure when you get a "buzz" echo that sounds almost like what you said into the microphone, only in "static-like buzz form"?  Speaking louder into the handset microphone makes it worse. Earpiece volume levels are set to modest values on the phones. Intercom calls don't seem to have any issues. On CO calls, it occurs randomly and sporadically on different sets, some calls are not affected and sound fine. It happens both with incoming loopstarts but also on ARS calls which are going out to a completely different node. I did try decreasing the CO transmit gain on the node where the complaint came from, but that didn't touch anything on the ARS calls as they head out to another node. If I go to the node which is taking the ARS calls and putting them out on the trunks, the buzz doesn't seem to happen there.  So I'm not sure this is a CO gain issue or what.  Everything used to work perfectly fine, no fiddling with gain settings was ever required at these sites. No service changes occured and no lines were moved, added or changed.  The only thing that has changed over time is the software version running on the nodes, which were just changed to 6.

I did run the hybrid balance / loop loss tests on all loopstarts /  nodes.  That seemed to fix an unrelated clear echo issue (clear audio echo) but not this buzzing feedback echo.  It almost seems as though running those tests made the buzz a bit more noticable, but that could be my imagination.

It seems to have cropped up with v6, but that could just be uncanny timing of the complaint coming in. Thanks for any wisdom.

[bhackbarth]

Update.  Desperate to find a resolution, I troubleshot and experiminted with a dozen combinations of settings. CO gain, Hybrid balance, and Echo Canceller mode. Then:

I changed System - VoIP Echo Canceller Setting to Basic.

Then under each trunk, I set the Hybrid Balance to 0.   My troubles suddenly went away. Couldn't reproduce the static sounding echo.

So then I went to the next site with the problem, rinsed and repeated. That trouble went away there too.

None of these nodes had an issue on 5.1 PR3 up to SP4.   Now, 1 in 11 calls will actually echo your voice back to you in full clarity and at full volume. Thankfully, those are rare. I can't reproduce this one by playing with the settings.

Suddenly, I feel like I am beta testing 6.0.   Tomorrow, off to share with the Mitel Tech Support and see what they say. I want to downgrade back to 5.1 SP4 but the release notes on 6 say careful doing that, the voicemail isn't region specific anymore but was back then.  Ah well, I found the bandaid. Thought I would share this story with everyone.

[DND ON]

I have the exact same problem with a site at v5.1 SP4 PR6, with 5320 phones and Loop Start trunks.

Reducing the trunk TX gain by a couple of dB helped some, but the problem still exists. VoIP Echo Canceller is set to Advanced. To clarify, did changing it to Basic help?

This issue has been discussed with Mitel, but they have no answers so far.

[acejavelin]

Every system with analog trunks that I have upgraded to v6 I have had issues talking on the analog trunks, just re-running the Hybrid Balance and Loop Loss tests and resetting the trunks to those settings seems to fix it. I wouldn't change the VoIP Echo Canceller setting until trying this...

[bhackbarth]

Thanks Ace,  yes, re-running those two tests is the first troubleshooting step I took.  I would then re-start DB programming to see the new Hybrid Balance and ERL values.   Unfortunately, the issue persists. It's like a very apparent side tone.  FWIW I have identified another site today running 6.0 that has side tone on its loop starts but not on direct IP calls. There is a ticket with Mitel for this. I will report back what they say.

[bhackbarth]

To clarify, did changing it to Basic help?

Yes.  Though far from perfect, in my case it reduced the side tone significantly enough (shortens the duration of it, if you will)  that I will call it a temporary bandaid fix until Mitel gives some advice for my ticket (tomorrow hopefully?). This bandaid only works if the Hybrid balance is 0, though. Changing that setting triggers a restart of call processing so plan the time accordingly.  In my case playing with the CO gain didn't seem to do a lot.  Each site with 6.0 has varying intensities of this problem. The one with the Arris voice modem (CO trunk is sourced 5 feet away) sounds the best, but the side tone is definitely still there!  While Mitel has had a couple releases disappear off Mitel On-Line in recent memory (3300 v6 SP1, 5000 5.1 SP3), and that 5000 v6 was immediately deferred until SP1,  I have faith in their support. They know their stuff. If there is something wrong, they will deliver a patch.

[DND ON]

Heard back from Mitel today. They suggest:

1. Change the LS trunk RX level to +4.
2. Run hybrid balance test.
3. Set IP phone handset volume to 4.

I'm on vacation for 2 weeks, so I won't get to this anytime soon.

[bhackbarth]

Thank you DND,  those suggestions helped a lot!  A critical step I was missing was running the Hybrid Balance test after adjusting the CO gain. Here are the settings that cleared up the issue completely (at one site so far):

Echo Canceller Mode: back to default (Advanced)
CO TX Gain: -4 dB
CO RX Gain: +4dB

Run the Hybrid Balance Test on all trunks
Run the Loop Loss Test on all trunks.

Problem = gone.  No more buzzing, static, or feedback on loop start calls. I will need to make some additional test calls to be sure the audio levels are good.    It appears that in 6.0 they must have changed the Advanced Echo Canceller gain values for loop starts on the back end.  This makes sense that the Basic echo canceller mode helped because that comes with a whole different set of gain values (as Mitel Tech support informed me).

[bhackbarth]

Another update.  Second location I did the same thing except I needed -5 dB for the transmit gain on all trunks.  Now audio is excellent on the second site too.  If appears that the higher the Effect Echo Return Loss (ERL), the lower the transmit dB must be. I'll see if that trend continues as I visit and test the other sites.

[DND ON]

Thanks for the update. Hey, if you could have this completely figured out by the time I'm back from vacation, I would be grateful. 

[bhackbarth]

I got all my sites sounding good.  Mitel has escalated my ticket to "level 4" or something like that.  I'll be sending them freezes and logs on request.

DND, basically what I did at each site:
-Run Hybrid balance all loop starts
-Run Loop Loss all loop starts
-Re-start DB programming
-Go to the trunk(s) and note the Echo Return Loss  (this indicates how clean your line is or how close you are to the CO card). If you are plugged directly into an Arris voice modem, you'll see this value between 0 and 10. This is a pretty clean line.
-If you have 0 - 10 for ERL, start On Line Monitor, pick a trunk to test and adjust the CO Receive Gain to +4 db.  Adjust the CO Transmit Gain to -4 db.
-Run Hybrid balance test on that trunk
-Run Loop Loss test on that trunk
Re-start DB Programming.  The ERL on that trunk will definitely change, but don't worry about that. Call into the system on that loop start trunk from the cellphone, and talk really loud into the Mitel handset microphone. Try to get it to produce static or feedback into the Mitel handset piece. If you still get feedback, hang up and lower the CO Transmit gain on that trunk one additional value (start with -5 db now). Re run the Hybrid balance test and loop loss test on that trunk.  Rinse and repeat until the issue is gone. I never had to go below -5 db transmit gain on any site.  Once it  sounds clear regardless of speech volume, go to the Trunks and apply the gains that worked, to all loop start trunks and run the hybrid balance / loop loss tests on all trunks.

To run hybrid / loop loss tests on only one trunk at a time, click on Trunks and right click the trunk from the right pane to get the context menu.

On my sites with Arris modems, +4 Receive and -4 Transmit gain did the trick. The ERL on these sites was around 8 on all trunks.  A couple of sites needed -5 db Transmit Gain to eliminate the static completely. These had higher ERL values greater than 10.  If you have any questions feel free to send me a message on the forum.

Since Mitel escalated my ticket twice since I filed it, I suspect a fix may come out for this.  Good luck!

[DND ON]

Just finished at my site; I've got settings all across the board, as much as -7 dB.

It sounds good to me, but these folks have had such a bad time with call quality that they’re super picky about any noise. We’ll see what happens, thanks for the updates.

[DND ON]

Miitel has released 5.1.0.56 SP4 PR7 to address audio issues. Just upgraded a site; we'll see what happens.

[bhackbarth]

And if I'm running 6.0?   Last I heard from Mitel then next planned release for that was end of year.

[marcolive]

Mitel released 6.0 PR2 monday with the same fix. We'll try it this week!