Mitel Reboot issue

[comn8u]

All of my controllers rebooted at the same time this morning and I'm not sure why.  I was wondering if someone could help me with some of this log information.

Also, what is this "slowpoke was not set to run"?



Here are some of the log messages from one of the controllers:


EntryBlock - Enable Block State VM Not Ready with Mask 0x10
TerminateService: for client 2, disconnected. Mitai server not ready
Mitai Server is not ready to process OpenPBX : server state = MITAI_SERVER_WAITING
TerminateService: for client 2, disconnected. Mitai server not ready
Mitai Server is not ready to process OpenPBX : server state = MITAI_SERVER_WAITING
The link from MCECAAA MN3300 Controller at location 01 1 01 Active         to   MC266AA Communication Proc at location 01 1 10 Activeis now OPEN.
MC266AA Communication Proc at location 01 1 10 installed
TerminateService: for client 2, disconnected. Mitai server not ready
Mitai Server is not ready to process OpenPBX : server state = MITAI_SERVER_WAITING
EntryBlock - Disable Block State Resume Restore in progress with Mask 0x8
EntryBlock - Enable Block State Resume Restore in progress with Mask 0x8
EntryBlock - Disable Block State Resume Restore in progress with Mask 0x8
EntryBlock - Enable Block State Resume Restore in progress with Mask 0x8
Mitai Server is not ready to process OpenPBX : server state = MITAI_SERVER_WAITING
Mitai Server is not ready to process OpenPBX : server state = MITAI_SERVER_WAITING
TerminateService: for client 0, disconnected. Mitai server not ready
Mitai Server is not ready to process OpenPBX : server state = MITAI_SERVER_WAITING
MiTAI Server socket-listener started successfully
INFO: SSO Enabled.
INFO: SSO About to startup.
slowpoke was not set to run
Local date/time reset to 2012-JUL-13 11:44:39.00    FRI by ESM User.
The link from MCECAAx MN3300 Controller at location 01 1 01 Active         to   MC270Ax Universal T1 at location 03 1 02  is now in SCANNING mode.
The link from MCECAAx MN3300 Controller at location 01 1 01 Active         to   MC266Ax Communication Proc at location 01 1 13  is now in SCANNING mode.
The link from MCECAAx MN3300 Controller at location 01 1 01 Active         to   MCSTHAx ICP Communications at location 01 1 11  is now in SCANNING mode.
The link from MCECAAx MN3300 Controller at location 01 1 01 Active         to   MC266Ax Communication Proc at location 01 1 10  is now in SCANNING mode.
Restart information from hardware status register was  power up
LOGSYSRTR is ready to route entries Log name: maintenance
Local date/time reset to 2012-JUL-13 11:44:38.00    FRI by ESM User.
LOGSYSRTR is ready to route entries Log name: xxxxxx
LOGSYSRTR is ready to route entries Log name: xxxxxx
GPAGEMGR: Initializing data structures                                                                                                                                                                                                               
DBMS Status flag set.  DB Download Complete..
DBASRV created: PID = ($0005,$0034)
DBASRV created: PID = ($0004,$0033)
DBASRV created: PID = ($0003,$0032)
DBASRV created: PID = ($0002,$0031)
DBASRV created: PID = ($0001,$0030)
DBASRV created: PID = ($0008,$002F)
DBASRV created: PID = ($0007,$002E)
DBASRV created: PID = ($0006,$002D)
nvs_BerkConvert: Recovering existing IP Networking database.
NotifyTimeChangeTime Change Could not be Notified
memory audit started succesfully
Progress Monitor->ResourceAlarmStartupResourceAlarmStartup ready
Progress Monitor->ProcessorCoordinator::ProcessorCoordinatorProcessor Coordinator source Logs initialized.
Progress Monitor->SysServices::appStartupPart2 (10.10.60.120)Boot Device: ata=0Host Name: Target Name: Target IP address: 10.10.60.120:ffffff00Host IP address: 10.10.60.1Gateway IP address: 10.10.60.1Boot File: /partition4/RTC8260Boot Flag: 0x0
Unexpected Result->SysServices::appStartupPart2 (10.10.60.120)Abnormal reset, cause: (0x0) UNKNOWN
Progress Monitor->SysServices::appStartupPart2CXII Platform (10.10.60.120) Booting Software version: 10.1.1.11_2System Relink: No relink has been applied.Call Control Relink: No relink has been applied.Abnormal reset, cause: (0x0) UNKNOWN
JavaLayerLifecycleListener::handleNotification()->Received Startup notification for JavaLayer.
JavaLayer_PPC_StartUp->JavaLayer startup has been initiated.
InitSystemTimeZone()->Current Timezone is  US/Central
Progress Monitor - SubscribeNewLogSubscribed to * for Error
Progress Monitor - SubscribeNewLogSubscribed to * for Warning

---------------+--------+-------+-------+-------+---------+---------+---------- ICP Comms Card |     5  |     1 |  20 % | Minor |   > 0 % |    25 % |   100 % | ---------------+--------+-------+-------+-------+---------+---------+----------
---------------+--------+-------+-------+-------+---------+---------+----------                |Total In|      Unavailable      | Alarm Threshold Percentages    Category    | System | Total |    %  | Alarm |  MINOR  |  MAJOR  | CRITICAL
                                ------------ Current System Alarm :          |  MINOR   |       viewed from Active                                 ------------
IP Trunk link to icp 004 Building1 failed and link dropped.
IP trunk to 10.10.64.120:1067 has failed.
ICP has lost contact with (10.10.64.120:1067), network driver cluster pool free: 2965 and low water mark: 2853
TRAFORM: Traffic report for period 46 of session 1 completed.
TRAFORM: Traffic report for period 46 of session 1 begins.
TRAFORM: Traffic report for period 45 of session 1 completed.

[bobcheese]

"Abnormal reset, cause: (0x0) UNKNOWN" well......if it doesn't know why it rebooted how the hell are we? Sorry, couldn't resist. Safe to say I have not seen this error before. And you say more than 1 controller? Presume clustered then?

[ralph]

Restart information from hardware status register was  power up

This record might be a clue.
Are all your controllers off the same UPS?

Ralph

[comn8u]

The 5 controllers are clustered and are at separate facilities.  Originally I thought all went down, but only three of the five went down.  Still odd.  They are not on the same UPS.  We have been experiencing echo at all of our facilities lately, and our Mitel vendor was looking into it today.  It may have been a coincidence that they were looking into the echo issue and the controller restarts.  We experienced a power glitch on Thursday at three of our facilities, but the controller locations don't match up with the restart today.  All facilities lost their phones, but one of the facilities was where one of our PRI's exists.

bobcheese, yes, I agree with you.  I should have looked at that line to see "UNKNOWN".

[Mattmayn]

What release are you running? I had this problem a year or so ago with Mxe IIIs and I think there is a KPI on it. I will see if I can find it.

[comn8u]

The version of the Mitel Communications Director is: 4.1 SP1.

I have 4 other devices plugged into the backup battery that were not affected at the same time the Mitel controller rebooted. 

Maybe upgrade to another version?

[Mattmayn]

It wouldn't hurt to try a more recent release. I don't remember the exact release I was on when I had those problems, but it was somewhere around there. Sorry I can't be more exact.