Jitter issues MItel3300 and 5320 Phones

[Montebond]

I have 20 building campus running 10g fiber between buildings all phones are on campus( no remote phones) I have HP ZL switches with a PRI for outbound/inbound calling. Recently the phone Vlan (20) has been experiencing unacceptable voice quality issues.  High jitter and digital noise, our Mitel Vendor almost pulled his arms out of this socket to point at our network (as with all vendors you have to prove to them its their issue as they will try to weasle out of any actual trouble shooting) SO here is the what I see
On the 3300 lots of these.....

Maintenance 3658 Info 2021/Oct/20 11:35:00 E2tSp IP Network Congestion Alert Jitter from phone 192.168.200.223 > 300ms Call... E2T Maintenance;0

on the switches where a 5320 is connected
Port   Bytes     Frames             Errors RX   Drops TX   FlowControl   Bcast Limit   Status   Mode   Mode
A5   78,149,627   1,315,784,895   0       22,037,624   off                         0               Up   100FDx   MDIX    192.168.200.202  08 00 0f 6d a8 21 LAN port regDN 1328,MITE...

Like Clockwork almost every single (we have 270 phones) port that has a phone connected shows 22 million drops   

We have a Barracuda f600 as Firewall with the gateway for the 192.168.200   

We have quality issues phone to phone and phone to outside caller, we have phone to phone quality issues on the same switch.

Vlan 20 is setup as such
 qos priority 6
   voice

on all HP switches.

I am open to any thoughts

            

[ZuluAlpha]

My first thought is your E2T card could be having issues, but that's not in play if you're experience the issue on network. What software release are you running?

If all phones experience this sudden drop at the same time that does seem to indicate a network issue. One thing I've randomly seen is that firewalls can cringe at large amounts of RTP packets. That's really just a shot in the dark though.

[Montebond]

After dumping every single switch's logs (all 40 of them) and sifting through them. I found a single phone had been plugged back into a switch (the connection wasn't even on the same vlan) even though Loop-protect was killing the second port it still made the voice quality close to unusable. These phones are extremely sensitive to any network anomaly  I can't imagine how these are usable on wan links or vpns. I also think the firmware in the phone should be able to detect a loop as well, I mean how hard is it to read LLDP and match chassisID

[ntnher@yahoo.com]

Hard to believe 1 phone would cause multiple issues for other phones. One bad phone couldn't have saturated most of the bandwidth in that vlan.

We have about 100 phones throughout 5 different cities across hundreds of miles of fiber on E-LAN network. All phones working well with no issue.  Your problem is most likely QoS. You have QoS setup?