ACD Phones Timing out...

[jbeiting]

We have ACD phones logging out on their own at seemingly random times.  Has anyone seen this or know what to do?  Mitel 3300 ICP with 5220/5224 phones.

John

[ralph]

This will happen if the agent doesn't answer the phone on an ACD call and the call is re-queued.
You can test this by calling the Path number and telling the agent not to answer.

Check your agent log out timer in the COS of the Agent.

Ralph

[Chakara]

  You can turn off the agent logout/no-answer option - but then I'm not clear what happens to a caller if an agent has wandered away.....

[jbeiting]

I apologize for the errant information.  They are TIMING out, not LOGGING out, and it happens sometimes while the agent doesn't have an active call, and it happens sometimes when they are in the middle of a call.  The only way that they can do anything if it times out during a call is to unplug the phone and plug it back in which, obviously, makes them drop the call that they're on.

[Chakara]

  I think we'll need a whole lot more info to figure this one out.  Where do the calls come from?  Is QoS all in place and working all the way from the handsets to the trunks (as well as to the handsets ICP)?  What, exactly, are the symptoms when a phone "Times" out?  Only ACD sets are affected - that seems very odd....

[swilson]

I work with jbeiting, we have been able to trace the occurrences back to following errors that always appear together.  We have our voice network on a separate vlan with QOS setup.  All calls come in on a DS3 that is routed trough an Adtran Atlas 890.  They then hit an auto attendant where callers can enter an extension or enter the day ACD queue.  Some of the symptoms of these problems are that they are unable to transfer or cancel a call that they are on.  Today someone wasn't able to logout of their phone.  It is basically locking up the phone and not allowing them to do anything.  Sometimes this corrects itself and other time the user reset their phone.  So far nothing that Mitel has recommended has worked and they are out of ideas so they recommended an upgrade from 7.1.4.9 to 8.  Our company doesn't want to pay for the upgrade unless they are sure it is going to work.

ACD ringing event reported twice - glare conditionTRACEBACK (see above for first level info):  bgrpmgr +2:$688E  bgrpmgr +2:$98EA  bgrpmgr +2:$A0C8  bgrpmgr +2:$D120

SWERR glare condition recovery,  OFFSET 2:$3E90,  PC: $25D908LOG_ID: bgrpmgr#24,  PROCEDURE: send_ring_warble,  SEVERITY: cp_majorPID: ($7, $56)OTHER_SWID: xnet_trunk 140PARTY: termFEATURE_ACTIVE:  21

Any advice would be great as this issue has been going on for a month now.

[ralph]

I've seen this on TDM devices but never on IP sets.
On the TDM sets, what it boiled down to is there was simply too many messages going back and forth to the set.
The set would have a PKM fully populated with DSS buttons.  The system/phone has to send messages back and forth to the phone telling it to light a button, ring, de-light a button etc.   Cutting down on the ammount of active buttons usually helped.  Also we had to be sure there were never more that 8 display sets on a MC330 card when this starting happening.

But... with IP sets...hummmmm.   Try setting a small worktimer on the ACD COS.  About 2 to 5 seconds should be enough.  The "Glare" logs suggest to me that the phone is trying to do something the same time a new message is coming in. i.e. "send_ring_warble".   Slow that message down just long enough for the phone to catch up and you may have it fixed.

Ralph

[Chakara]

  While I'm not happy with Mitel for implimenting the Software Assurance, as Mitel has done it, I wouldn't have a system out there without the Software Assurance.  At the very least - if you upgrade, then you can hold Mitel to the fire more in that they can't say you didn't do what they said.

  Good luck!

[swilson]

Unfortunately we are still have issues after we adjusted the work timer to 2.  Although we did go a whole day without any reports.  Anyone have any other ideas?

[ralph]

Is your PBX local to the phones or are the phones connected accross a WAN?
The next thing to take a look at is the network.   My guess would be that the phones are missing messaging IP packets.   In other words, the phone sends a message out to the 3300 and then is waiting for a response from the PBX.  If the packet is dropped or MAL-formed, I would think you could see similar issues.

Are users complaining of voice clipping at all?
Ralph

[swilson]

The PBX is local just on a different vlan than the phones.  We did have someone outside come in a assess the network and they didn't find any abnormalities(of course we didn't have any occurrences while they were here either).  We don't have any reports of clipping but we do have what we call dead air calls.  The agents answer the phone  and do not here anyone on the other side.  We have never been able to verify if the other caller could here us or not or if they even had a problem.  A couple of the calls we tracked down were cell phones and they told us they dropped the call.

We did have a different error message in the logs today.  We had 4 sets that weren't receiving phone calls with callers in queue and we had 4 sets of these errors messages at the time of occurrence.

HANDLER_ERR,  OFFSET 2:$6A,  PC: $2B6498LOG_ID: ss4lowlvl#30,  PROCEDURE: ss4lowlvl_quick_error_report,  SEVERITY: cp_majorPID: ($8, $5D7)SWID: brdcst_grp 269ss4 handler - gam_num_busy out of syncTRACEBACK (see above for first level info):

ss4lowlvl +2:$535E  cpaudits +2:$1E4A  cpaudits +2:$26AC  auditwork +2:$E2

[ralph]

This was in your log: "gam_num_busy"   - Seen this before.  Causes the phone to lock up but these were always on TDM phones.  4150 4025 etc.   Has to do with too many messages back and forth between the DNI card and the set.  But... you aren't usinng TDM phones.....

The dead air calls are not related. 

I still suspect a network problem.   You need to log into the rtc (not esm) and enter e2tshow.   This will show errors and packet loss on the network.  e2tclear clears the stats so you can get fresh info.

Next question: are you VLANed?   Are you in the same data switch as your controller?  Is the network interface to the 3300 hard coded at 100meg full on the data switch?  (if yes, this will cause problems.  set it to auto negotiate.)

Ralph