Issues on 6.3.9.108 Software

[acejavelin]

Anybody had any weird hunt group issues with this version of software?

We have a client that the every once in while will get a call into a hunt group, and it when they answer it the system just freezes, literally and figuratively... All activity on all phones just freezes for about 30 seconds, no one can make or receive a new call, and a system freeze occurs. This didn't start until after upgrading to 6.3.9.108 (they were previously on 6.3 build, but a year or two older version).

Oddly, this seems to only happen the day of a scheduled reset... It resets every Wednesday at 3am, and then the error occurs sometime during that day. We clear the freeze and it doesn't recur until the next week. Been going on for a couple months now.

Can't open a ticket direct with Mitel anymore, so before I reach out to our supplier's support team, any ideas here.

Freeze info:

-01:971- > -01:510- 07:09 04-21 M3259 INF EG CP Node 1 Enabled Day Mode
-01:971- > -01:511- 10:55 04-21 *** --- message_destination_t:
-01:971- > -01:512- 10:55 04-21 ***      ucd_hunt_device_t (ID(1:'5401'))
-01:971- > -01:513- 10:55 04-21 ***      depth:     0
-01:971- > -01:514- 10:55 04-21 *** --- entity_t:
-01:971- > -01:515- 10:55 04-21 ***      id:     ID(1:'5401')
-01:971- > -01:516- 10:55 04-21 ***      app_device_type:     121
-01:971- > -01:517- 10:55 04-21 *** --- device_entity_t:
-01:971- > -01:518- 10:55 04-21 ***      username:     Incoming
-01:971- > -01:519- 10:55 04-21 ***      pc_description:     ~Incoming HG
-01:971- > -01:520- 10:55 04-21 *** --- devices_queued_onto_me_list_t:
-01:971- > -01:521- 10:55 04-21 ***      is empty
-01:971- > -01:522- 10:55 04-21 *** --- devices_queued_onto_me_list_t end
-01:971- > -01:523- 10:55 04-21 ***      ACD hunt groups logged into []: None
-01:971- > -01:524- 10:55 04-21 ***      ACD hunt groups not logged into []: None
-01:971- > -01:525- 10:55 04-21 *** --- hunt_device_entity_t:
-01:971- > -01:526- 10:55 04-21 ***      hunt_list_extension:     'P4098'
-01:971- > -01:527- 10:55 04-21 ***      hunt_algorithm:     0
-01:971- > -01:528- 10:55 04-21 ***      validate_queue_operations:     FALSE
-01:971- > -01:529- 10:55 04-21 ***      no_answer_advance_timer:     180
-01:971- > -01:530- 10:55 04-21 ***      acd_hunt_group_flag:     FALSE
-01:971- > -01:531- 10:55 04-21 ***      acd_wrap_up_timer:     15
-01:971- > -01:532- 10:55 04-21 ***      priority_level:     0
-01:971- > -01:533- 10:55 04-21 ***      supervisor_list_extension:     'P4096'
-01:971- > -01:534- 10:55 04-21 ***      average_connect_time_per_call:     60
-01:971- > -01:535- 10:55 04-21 ***      hunt_campon_flag:     TRUE
-01:971- > -01:536- 10:55 04-21 ***      group_call_pickup_allowed:     FALSE
-01:971- > -01:537- 10:55 04-21 ***      use_acd_agent_id_flag:     FALSE
-01:971- > -01:538- 10:55 04-21 ***      reset_acd_last_call_time_flag:     FALSE
-01:971- > -01:539- 10:55 04-21 ***      send_campon_notification_to_dnd_members:     FALSE
-01:971- > -01:540- 10:55 04-21 ***      acd_agent_list_extension:     'P4097'
-01:971- > -01:541- 10:55 04-21 ***      audio_for_camped_on_announcement_calls:     0
-01:971- > -01:542- 10:55 04-21 ***      audio_for_calls_ringing_this_device:     2
-01:971- > -01:543- 10:55 04-21 ***      audio_for_calls_ringing_this_device_board_number:     8
-01:971- > -01:544- 10:55 04-21 ***      audio_for_calls_ringing_this_device_audio_source_ number:     0
-01:971- > -01:545- 10:55 04-21 ***      audio_for_calls_camped_onto_this_device:     3
-01:971- > -01:546- 10:55 04-21 ***      audio_for_calls_camped_onto_this_device_board_num ber:     8
-01:971- > -01:547- 10:55 04-21 ***      audio_for_calls_camped_onto_this_device_audio_sou rce_number:     0
-01:971- > -01:548- 10:55 04-21 ***      hunt_group_type:     0
-01:971- > -01:549- 10:55 04-21 *** --- ucd_hunt_device_entity_t:
-01:971- > -01:550- 10:55 04-21 ***      announcement_station:     ''
-01:971- > -01:551- 10:55 04-21 ***      announcement_timer:     18
-01:971- > -01:552- 10:55 04-21 ***      overflow_station:     ''
-01:971- > -01:553- 10:55 04-21 ***      overflow_timer:     72
-01:971- > -01:554- 10:55 04-21 ***      recall_destination:     '5400'
-01:971- > -01:555- 10:55 04-21 ***      recall_timer:     180
-01:971- > -01:556- 10:55 04-21 ***      is_voice_mail_hunt_group:     0
-01:971- > -01:557- 10:55 04-21 ***      mailbox_node_number:     0
-01:971- > -01:558- 10:55 04-21 *** --- device_t:
-01:971- > -01:559- 10:55 04-21 ***      this:     02209394
-01:971- > -01:560- 10:55 04-21 ***      device_class:     HUNT_GROUP_CLASS
-01:971- > -01:561- 10:55 04-21 ***      Hunt Groups Camped onto Me:          None
-01:971- > -01:562- 10:55 04-21 *** --- abstract_mc_device_t:
-01:971- > -01:563- 10:55 04-21 ***      destroy_call_control_on_return_from_process_messa ge:     FALSE
-01:971- > -01:564- 10:55 04-21 ***      destroy_datagram_control_on_return_from_process_m essage:     FALSE
-01:971- > -01:565- 10:55 04-21 *** --- ucd_hunt_call_control_t:
-01:971- > -01:566- 10:55 04-21 *** --- hunt_call_control_t:
-01:971- > -01:567- 10:55 04-21 *** --- abstract_device_call_control_t:
-01:971- > -01:568- 10:55 04-21 *** --- message_destination_t:
-01:971- > -01:569- 10:55 04-21 ***      abstract_device_call_control_t (ID(1:'@1492'))
-01:971- > -01:570- 10:55 04-21 ***      depth:     1
-01:971- > -01:571- 10:55 04-21 ***      parent_message_destination_ptr->get_id ():     ID(1:'5401')
-01:971- > -01:572- 10:55 04-21 ***      creation time: 04-21-21 10:55:39
-01:971- > -01:573- 10:55 04-21 ***       call_ids for this call control:
-01:971- > -01:574- 10:55 04-21 ***         call_id is ID(1:'@025X')
-01:971- > -01:575- 10:55 04-21 ***         call_id is ID(1:'@0260')
-01:971- > -01:576- 10:55 04-21 ***         call_id is ID(1:'@0261')
-01:971- > -01:577- 10:55 04-21 ***     state machine:
-01:971- > -01:578- 10:55 04-21 ***      [1] <merge_wait_hgs_t>
-01:971- > -01:579- 10:55 04-21 ***           u()
-01:971- > -01:580- 10:55 04-21 ***           b()
-01:971- > -01:581- 10:55 04-21 ***      [ 0 ]
-01:971- > -01:582- 10:55 04-21 ***           u()
-01:971- > -01:583- 10:55 04-21 ***           b()
-01:971- > -01:584- 10:55 04-21 *** --- abstract_device_call_control_t end
-01:971- > -01:585- 10:55 04-21 ***      processing_campon:     0
-01:971- > -01:586- 10:55 04-21 ***      most_recent_connect_request_indicated_that_a_merg e_would_follow:     FALSE
-01:971- > -01:587- 10:55 04-21 ***      originating_call_info is null
-01:971- > -01:588- 10:55 04-21 ***      connecting_call_info is null
-01:971- > -01:589- 10:55 04-21 ***      ACD Agents Already Been Rung: NONE
-01:971- > -01:590- 10:55 04-21 ***      Members Who Sent a Setup Reject: NONE
-01:971- > -01:591- 10:55 04-21 ***      Members Who Have Been Redirected: NONE
-01:971- > -01:592- 10:55 04-21 *** --- hunt_call_control_t end
-01:971- > -01:593- 10:55 04-21 ***      announcement_station_status:     0
-01:971- > -01:594- 10:55 04-21 ***      overflow_station_status:     0
-01:971- > -01:595- 10:55 04-21 *** --- ucd_hunt_call_control_t end
-01:971- > -01:596- 10:55 04-21 ***      CALL ID TO CALL CONTROL MAP
-01:971- > -01:597- 10:55 04-21 ***         ID(1:'@025X')
-01:971- > -01:598- 10:55 04-21 ***         ID(1:'@0260')
-01:971- > -01:599- 10:55 04-21 ***         ID(1:'@0261')
-01:971- > -01:600- 10:55 04-21 *** --- hunt_device_t:
-01:971- > -01:601- 10:55 04-21 ***      device_call_info_queue is empty
-01:971- > -01:602- 10:55 04-21 ***      ACD Wrap-Up Hunt Group Members: NONE
-01:971- > -01:603- 10:55 04-21 ***      Removed Hunt Group Members: NONE
-01:971- > -01:604- 10:55 04-21 ***      Hunt Group Members in DND/Offline: NONE
-01:971- > -01:605- 10:55 04-21 ***      Unavailable Hunt Group Members:
-01:971- > -01:606- 10:55 04-21 ***        ID: ID(1:'5308')  Agent ID: ''  ACD Calls: 0  Date/Time: 04-21-2021  03:00:31
-01:971- > -01:607- 10:55 04-21 ***        ID: ID(1:'5306')  Agent ID: ''  ACD Calls: 0  Date/Time: 04-21-2021  03:00:31
-01:971- > -01:608- 10:55 04-21 ***        ID: ID(1:'5303')  Agent ID: ''  ACD Calls: 0  Date/Time: 04-21-2021  03:00:31
-01:971- > -01:609- 10:55 04-21 ***        ID: ID(1:'5301')  Agent ID: ''  ACD Calls: 0  Date/Time: 04-21-2021  03:00:31
-01:971- > -01:610- 10:55 04-21 ***      Available Hunt Group Members: NONE
-01:971- > -01:611- 10:55 04-21 ***      offline_reported_to_status_module: 0
-01:971- > -01:612- 10:55 04-21 *** --- status_msg_t:
-01:971- > -01:613- 10:55 04-21 ***      update_messages_pending:     0
-01:971- > -01:614- 10:55 04-21 ***      current_status:     Busy
-01:971- > -01:615- 10:55 04-21 ***      previously_reported_status:     Busy
-01:971- > -01:616- 10:55 04-21 ***      current_num_secondary_calls:     0
-01:971- > -01:617- 10:55 04-21 ***      previous_num_secondary_calls:     0
-01:971- > -01:618- 10:55 04-21 ***      current_secondary_call_summary:     0
-01:971- > -01:619- 10:55 04-21 ***      previous_secondary_call_summary:     0
-01:971- > -01:620- 10:55 04-21 ***      current_status_for_park_key:     Change to Alerting Status
-01:971- > -01:621- 10:55 04-21 ***      previous_status_for_park_key:     Change to Alerting Status
-01:971- > -01:622- 10:55 04-21 *** --- request_list:
-01:971- > -01:623- 10:55 04-21 *** --- status_msg_t end
-01:971- > -01:624- 10:55 04-21 *** *************************************************************************************************
-01:971- > -01:625- 10:55 04-21 ***                                   F R E E Z E     S U M M A R Y                                 
-01:971- > -01:626- 10:55 04-21 *** **** REASON:   
-01:971- > -01:627- 10:55 04-21 ***                                  SAT Manager Freeze (
-01:971- > -01:628- 10:55 04-21 *** **** SITE INFO:
-01:971- > -01:629- 10:55 04-21 ***                 Company Name:   
-01:971- > -01:630- 10:55 04-21 ***                         Address:
-01:971- > -01:631- 10:55 04-21 ***                                  , 
-01:971- > -01:632- 10:55 04-21 ***                         Phone:   
-01:971- > -01:633- 10:55 04-21 ***                 Service Manager Name: 
-01:971- > -01:634- 10:55 04-21 ***                                 Phone:
-01:971- > -01:635- 10:55 04-21 ***                                 Email:
-01:971- > -01:636- 10:55 04-21 *** **** SYSTEM INFO:
-01:971- > -01:637- 10:55 04-21 ***                 License Number:  50998049
-01:971- > -01:638- 10:55 04-21 ***                 Version:         6.3.9.108
-01:971- > -01:639- 10:55 04-21 ***                 Product:         5000 CP
-01:971- > -01:640- 10:55 04-21 ***                 CPU Type:        MiVoice Office 250
-01:971- > -01:641- 10:55 04-21 *** *************************************************************************************************
-01:971- > -01:642- 10:55 04-21 *** Failed to start ADD post thread!
-01:971- > -01:643- 10:55 04-21 M3001 INF EG CP System History Of Node 1 Has Been Frozen From Node 1

[acejavelin]

Hmm... Another forum user reached out to me saying they have a similar issue with a client with the same software version and have a ticket open with Mitel with no information yet.

Debating on trying to downgrade now, haven't done a downgrade for years. :/

[Craig]

I have another update to share with you. Following another freeze this morning, an ADD was delivered to Mitel and it has since been escalated with design. I'll continue to keep you updated.

[acejavelin]

I have another update to share with you. Following another freeze this morning, an ADD was delivered to Mitel and it has since been escalated with design. I'll continue to keep you updated.

Sounds good... I got my ticket open and it's just been in "investigating" for 2 days... We will see what happens.

[Craig]

Still no updates from Mitel unfortunately, despite several freezes. Have you had anything at all?

[acejavelin]

Still no updates from Mitel unfortunately, despite several freezes. Have you had anything at all?

No, same here... Had a few automated diagnostic reports sent in, and then updated ticket with details from customer, but no interaction from support.

I have a feeling they are just bidding their time waiting for 6.3 SP6 that is supposed to be out this month, and then the answer will be "upgrade to this new version and then retest" and then we start over again. :/

[Craig]

Ace, I have not had any updates at all from Mitel. To be quite honest, they have been useless so far. We have, however, investigated this ourselves and believe we may have identified a 'fix'

I am not sure if this will apply to you but in our case, our customer could receive anywhere from 30-60 calls instantly via SIP Trunks. The Mitel was interpreting such a large volume of calls as a possible SIP attack and dropping packets as a result. We then adjusted 'Admission Control Low/High Threshold' to off (it is accessible via SIP Peers > Admission Control Low/High Threshold).

As soon as we made this change no freezes have taken place since. You will have to do it via Online-Monitor should you wish to try.

[acejavelin]

Ace, I have not had any updates at all from Mitel. To be quite honest, they have been useless so far. We have, however, investigated this ourselves and believe we may have identified a 'fix'

I am not sure if this will apply to you but in our case, our customer could receive anywhere from 30-60 calls instantly via SIP Trunks. The Mitel was interpreting such a large volume of calls as a possible SIP attack and dropping packets as a result. We then adjusted 'Admission Control Low/High Threshold' to off (it is accessible via SIP Peers > Admission Control Low/High Threshold).

As soon as we made this change no freezes have taken place since. You will have to do it via Online-Monitor should you wish to try.

Hmm... unlikely the same issue as this customer has a TDM PRI and seeing more than 7-10 calls up at one time would be unusual, much less ringing in at one time.

[Craig]

No worries. I thought I would pass it on just in case. Interestingly, I did have an update from Mitel yesterday.. They have suggested the following for us now (based on our logging): Then go under system/system information and set the value of "Spare Byte Data 4" to 16. They identified an underlying software issue.

Maybe this will help?

[acejavelin]

So after almost 3 months, I finally got Mitel to escalate this issue to engineering yesterday... And today I got an email that they found the problem. No descriptions, just a very cryptic answer...

I have reviewed the logs and found the issue. To fix this issue please login to db programming and select view/online monitor and click OK. Then go to system/system information and change the value of Spare Byte Data 4 from 0 to 16. Then do a backup save to the system and reboot the system for it to take affect.

Made the changes and will reboot over the weekend... No idea exactly I changed or why, but I guess we will go with it.

[Craig]

We also had that response as well.

[acejavelin]

We also had that response as well.

Yeah, I know... Probably could have saved my customer some pain if I had just done it when you posted it, but you never about messing around in the On-Line Monitor.