Mitel 3300 MXe random reboot

[fcutler]

Afternoon,

We just had our 3300 MXe reboot randomly. It is release 4.1 SP2 active software 10.1.2.13. The log entry I found that I think may be related is in bold below:

Boot File: /partition4/RTC8260
Boot Flag: 0x0 Main Maintenance;0
Software 763 Warning 2012/Nov/26 12:12:46 AppStartup Unexpected Result->SysServices::appStartupPart2 (172.16.8.2)
Abnormal reset, cause: (0x43) WATCH_DOG_CPLD_RESET Main appStartup.cpp;1596
Maintenance 762 Info 2012/Nov/26 12:12:46 AppStartup Progress Monitor->SysServices::appStartupPart2
MXe Platform (172.16.8.2) Booting Software version: 10.1.2.13

System Relink: No relink has been applied.
Call Control Relink: No relink has been applied.
Abnormal reset, cause: (0x43) WATCH_DOG_CPLD_RESET Main Maintenance;0
Software 761 Warning 2012/Nov/26 12:12:44 MLC JavaLayerLifecycleListener::handleNotification()->Received Startup notification for JavaLayer.
 Main ManagementLayerCoordinator.cpp;1491
Software 760 Warning 2012/Nov/26 12:12:37 MLC JavaLayer_PPC_StartUp->JavaLayer startup has been initiated. Main ManagementLayerCoordinator.cpp;1424
Software 759 Warning 2012/Nov/26 12:12:07 MLC InitSystemTimeZone()->Current Timezone is  America/Los_Angeles Main ManagementLayerCoordinator.cpp;2104

And in case this may be related I have recently deployed a new Oaisys Tracer server that is also hooked to the PRI via a splitter. When I was initially setting up the T1 parameters the same thing happened, though, this was after hours so not an issue. Could the T1 settings or PRI be the cause somehow?

Thanks,

Forrest

[ralph]

I doubt it's the T1 causing this.
I don't think there are enough logs here.  Can you post the 5 before it?

Ralph

[fcutler]

Below are events 751 - 771 from the logs. I find it strange that events 757 and 758 have a different time (20:11:38) than the previous and next entries.

Maintenance 771 Warning 2012/Nov/26 12:12:52 IP Networking - MIPS MIPS SP Startup, FAILURE - no Layer 2 Switch IP addr. Main Maintenance;0
Maintenance 770 Warning 2012/Nov/26 12:12:52 IP Networking - MIPS MIPS SP Startup, DB update with pre-existing Layer 2 Switch IP addr. Main Maintenance;0
Maintenance 769 Info 2012/Nov/26 12:12:50 IP Networking - NVS nvs_BerkConvert: Recovering existing IP Networking database. Main Maintenance;0
Software 768 Warning 2012/Nov/26 12:12:47 SYS_TimeCalls NotifyTimeChange
Time Change Could not be Notified Main SYS_TimeCalls.cpp;641
Software 767 Warning 2012/Nov/26 12:12:47 MemAudit memory audit started succesfully
 Main MemAudit.cpp;1049
Maintenance 766 Info 2012/Nov/26 12:12:47 ResourceMonitor Progress Monitor->ResourceAlarmStartup
ResourceAlarmStartup ready Main Maintenance;0
Software 765 Warning 2012/Nov/26 12:12:46 ProcessorCoordinator Progress Monitor->ProcessorCoordinator::ProcessorCoordinator
Processor Coordinator source Logs initialized. Main ProcessorCoordinator.cpp;1076
Maintenance 764 Info 2012/Nov/26 12:12:46 AppStartup Progress Monitor->SysServices::appStartupPart2 (172.16.8.2)
Boot Device: ata=0
Host Name:
Target Name:
Target IP address: 172.16.8.2:fffffC00
Host IP address: 172.16.4.160
Gateway IP address: 172.16.8.1
Boot File: /partition4/RTC8260
Boot Flag: 0x0 Main Maintenance;0
Software 763 Warning 2012/Nov/26 12:12:46 AppStartup Unexpected Result->SysServices::appStartupPart2 (172.16.8.2)
Abnormal reset, cause: (0x43) WATCH_DOG_CPLD_RESET Main appStartup.cpp;1596
Maintenance 762 Info 2012/Nov/26 12:12:46 AppStartup Progress Monitor->SysServices::appStartupPart2
MXe Platform (172.16.8.2) Booting Software version: 10.1.2.13

System Relink: No relink has been applied.
Call Control Relink: No relink has been applied.
Abnormal reset, cause: (0x43) WATCH_DOG_CPLD_RESET Main Maintenance;0
Software 761 Warning 2012/Nov/26 12:12:44 MLC JavaLayerLifecycleListener::handleNotification()->Received Startup notification for JavaLayer.
 Main ManagementLayerCoordinator.cpp;1491
Software 760 Warning 2012/Nov/26 12:12:37 MLC JavaLayer_PPC_StartUp->JavaLayer startup has been initiated. Main ManagementLayerCoordinator.cpp;1424
Software 759 Warning 2012/Nov/26 12:12:07 MLC InitSystemTimeZone()->Current Timezone is  America/Los_Angeles Main ManagementLayerCoordinator.cpp;2104
Software 758 Info 2012/Nov/26 20:11:38 OtpLogsClient Progress Monitor - SubscribeNewLog
Subscribed to * for Error VxOtpLogsClient Otp_Logs_Client.cpp;1403
Software 757 Info 2012/Nov/26 20:11:38 OtpLogsClient Progress Monitor - SubscribeNewLog
Subscribed to * for Warning VxOtpLogsClient Otp_Logs_Client.cpp;1403
Software 756 Warning 2012/Nov/26 12:02:16 Call Control - Software equip swid: $08000019, equip swid DN:                           , display digits: 5416036877                 display name:                     , call type: 12800                                                           Main log_outputs_util_en.c;1622
Software 755 Warning 2012/Nov/26 12:02:16 Call Control - Software call ann. buffer recovered
buffer 128 owned by non-existent pid ($5,$4ED)
 Main log_outputs_util_en.c;1622
Software 754 Warning 2012/Nov/26 11:52:37 SYS_TimeCalls setPoller
pingTimeServer returned error Main SYS_TimeCalls.cpp;534
Software 753 Warning 2012/Nov/26 11:52:37 SYS_TimeCalls pingTimeServer
S_sntpcLib_SERVER_UNSYNC(Server time is not in sync yet) Main SYS_TimeCalls.cpp;393
Software 752 Warning 2012/Nov/26 11:46:21 Call Control - Software equip swid: $0800001B, equip swid DN:                           , display digits:                            display name: DUNNING, DANIEL     , call type: 12800                                                           Main log_outputs_util_en.c;1622
Software 751 Warning 2012/Nov/26 11:46:21 Call Control - Software call ann. buffer recovered
buffer 599 owned by non-existent pid ($7,$E0)

[ralph]

I not sure that the time stamps is a problem.  The next few logs show the time being set so I'm thinking the time on boot is set to GMT and the to PST.  I think that explains the time stamp difference.

The reset cause doesn't jump out at me, though.
I don't have a code book to show what the cause code is for 'Abnormal reset, cause: (0x43)'

See what your resource recovery is set to.
Enter "program reboot dis" and post what it shows.

Ralph

[bobcheese]

have you checked the KPI list? I seem to remember an error like this being in there. Upgrade fixed the issue.

[fcutler]

Good to know about the timestamp, thanks. Below is the system response from PROGRAMMED REBOOT DISPLAY:

PROGRAMMED REBOOT SCHEDULE IS ON                                               
PROGRAMMED REBOOT SCHEDULED: WEDNESDAY 02:15:00                                 
PROGRAMMED REBOOT RESOURCE RECOVERY IS ON                                       
PROGRAMMED REBOOT RESOURCE: RECOVERY 02:15:00                                   
PROGRAMMED REBOOT COURTESY IS OFF 

bobcheese, I am not sure what the KPI list is or where to check it?

Thanks,
Forrest

[ralph]

KPI = Known Product Issues

PROGRAMMED REBOOT RESOURCE: RECOVERY 02:15:00                                   
PROGRAMMED REBOOT COURTESY IS OFF 

That's what I was looking for.   Turn the Programmed Reboot Courtesy ON.   That way if the system see's it has a problem it will try to wait until 2:15 am to reboot.

Ralph

[fcutler]

I turned on the programmed reboot courtesy as you suggested.

PROGRAMMED REBOOT SCHEDULE IS ON                                               
PROGRAMMED REBOOT SCHEDULED: WEDNESDAY 02:15:00                                 
PROGRAMMED REBOOT RESOURCE RECOVERY IS ON                                       
PROGRAMMED REBOOT RESOURCE: RECOVERY 02:15:00                                   
PROGRAMMED REBOOT COURTESY IS ON

Thank you for the help on this, I hope that it is what caused the reboot and this will prevent that from happening again. I am contacting our vendor to schedule an upgrade to MCD 5.0 asap now as well.

Cheers,
Forrest