Author Topic: MiCollab CPU Starvation alarm  (Read 7029 times)

Offline acejavelin

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 4064
  • Country: us
  • Karma: +129/-0
  • High-tech, heavy metal redneck!
    • View Profile
    • Like what I do and wanna help out? Send me a donation!
MiCollab CPU Starvation alarm
« on: March 01, 2016, 08:56:50 AM »
I have a fair sized vMCD and virtual MiCollab installation that periodically MiCollab keeps going into Major Alarm with a CPU Starvation alarm. The vSphere environment is pretty stout, hosting dozens of machines in a medical facility and I believe the IT staff when they say there is no resource issue in the VMware cluster.

We also get Sync Connectivity alarms from time to time, it always clears itself though, which is a little puzzling because this VM has WAN interface directly on the public internet that is used quite a bit for remote phones.

Any ideas?


Offline dilkie

  • Global Moderator
  • Sr. Member
  • *****
  • Posts: 324
  • Karma: +11/-0
    • View Profile
Re: MiCollab CPU Starvation alarm
« Reply #1 on: March 01, 2016, 09:00:58 AM »
Your IT staff is wrong... You don't have enough resources, hence the alarm.

The reason Mitel added the cpu and disk starvation detection alarms was because too many "trusted IT" organizations were  saying the same thing, yet the product and the user experience was saying otherwise.

Offline acejavelin

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 4064
  • Country: us
  • Karma: +129/-0
  • High-tech, heavy metal redneck!
    • View Profile
    • Like what I do and wanna help out? Send me a donation!
Re: MiCollab CPU Starvation alarm
« Reply #2 on: March 01, 2016, 11:15:08 AM »
Your IT staff is wrong... You don't have enough resources, hence the alarm.

The reason Mitel added the cpu and disk starvation detection alarms was because too many "trusted IT" organizations were  saying the same thing, yet the product and the user experience was saying otherwise.
Thanks... I will go back to them then.

Sent from my MotoG3 using Tapatalk


Offline martyn

  • Hero Member
  • *****
  • Posts: 688
  • Country: au
  • Karma: +10/-0
    • View Profile
Re: MiCollab CPU Starvation alarm
« Reply #3 on: March 01, 2016, 04:26:09 PM »
The vMCD should have been deployed via an OVA, if so then resource reservations would have been automatically set. If you are getting these alarms then it indicates that they have probably been removed (also making it an unsupported configuration).
There are clear requirements from Mitel on the host server requirements for a virtualised system, so I would point them in the direction of them, and have the resource reservations reinstated asap.

Offline acejavelin

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 4064
  • Country: us
  • Karma: +129/-0
  • High-tech, heavy metal redneck!
    • View Profile
    • Like what I do and wanna help out? Send me a donation!
Re: MiCollab CPU Starvation alarm
« Reply #4 on: March 01, 2016, 08:09:21 PM »
Hmm... I doubt they would change the resource reservations, but anything is possible. I will have to try and find the correct reservations and send it to the manager of the IT department and see what they he says.

Offline VinceWhirlwind

  • Hero Member
  • *****
  • Posts: 899
  • Country: au
  • Karma: +31/-0
    • View Profile
Re: MiCollab CPU Starvation alarm
« Reply #5 on: March 01, 2016, 10:40:39 PM »
I was seeing the same thing. I asked the IT people what process they ran every 12 hours at 11:45.
It turned out it was the VMWare environment doing the snapshotting.
Mitel told me they had to turn off the snapshotting.
Customer wasn't very happy as we had told them there was no such thing as "clustered" MiCollab servers as the VMWare snapshotting would take care of things.
I'm still not entirely sure why snapshotting has to be turned off.

Offline dilkie

  • Global Moderator
  • Sr. Member
  • *****
  • Posts: 324
  • Karma: +11/-0
    • View Profile
Re: MiCollab CPU Starvation alarm
« Reply #6 on: March 02, 2016, 07:08:17 AM »
snapshotting interferes a lot with the underlying guest OS. It's impossible to pass audio, for example, with snapshotting turned on as the guest OS dispatching is delayed a "lot"

You understand that taking a 'snapshot' of a running guest involves stopping it dead and recording all of the machine's state, all memory, all virtual h/w, etc... It's then diffed against the previous to get something to store... It's hard to maintain the responsiveness that even a single call requires (100 packets per second for each active call in MBG, for example) if you snapshot a running guest.

My observations are that even with "reservations", the physical h/w you are running on makes a difference and that reservation is not always honoured, that's what the starvation alarms are doing, performing *actual* measurements from within the guest OS, they aren't lying, that's what the guest is experiencing. Moreover, most of the problems I've seen are i/o issues, so much delay between the guest linux kernel and the virtual disk that the kernel gets blocked (lookup io_wait in the 'iostat' command), this is time lost to the entire guest.

I know on my el-cheapo vmware server, I've monitored disk performance (datastore delay) and seen 100's of milliseconds being charted. Even occasionally, 1000's of milliseconds. You know the guest VM's have to be suffering.

Offline VinceWhirlwind

  • Hero Member
  • *****
  • Posts: 899
  • Country: au
  • Karma: +31/-0
    • View Profile
Re: MiCollab CPU Starvation alarm
« Reply #7 on: March 02, 2016, 09:31:24 PM »
With snapshotting turned off, what is your plan for resiliency?

Offline dilkie

  • Global Moderator
  • Sr. Member
  • *****
  • Posts: 324
  • Karma: +11/-0
    • View Profile
Re: MiCollab CPU Starvation alarm
« Reply #8 on: March 03, 2016, 07:51:19 AM »
My plan? I run clustered product (vMBGs, vMCDs) on different vmware h/w to ensure availability.

Other than that, msl backup. Deploying a vMBG/vMCD to a replacement VM is pretty quick if you don't mind a short outage.

Also,  it's automatic snapshots that kill you (and if you read Mitel's Engineering guidelines, I'm pretty sure somewhere in there they recommend turning snapshots off), I *think* it's okay to manually take a snapshot during some quiet period, less folks affected.

Offline mszodiac

  • Sr. Member
  • ****
  • Posts: 232
  • Country: ca
  • Karma: +7/-0
    • View Profile
Re: MiCollab CPU Starvation alarm
« Reply #9 on: March 08, 2016, 09:41:31 PM »
Hmm... I doubt they would change the resource reservations, but anything is possible. I will have to try and find the correct reservations and send it to the manager of the IT department and see what they he says.

I was getting the same alarms.  I had the IT department run their own bandwidth/resource utilization on the virtual server, and sure enough, they saw the spikes the same time the alarms were happening.  The increased the reservations after.

Offline VinceWhirlwind

  • Hero Member
  • *****
  • Posts: 899
  • Country: au
  • Karma: +31/-0
    • View Profile
Re: MiCollab CPU Starvation alarm
« Reply #10 on: March 14, 2016, 11:52:26 PM »
snapshotting interferes a lot with the underlying guest OS. It's impossible to pass audio, for example, with snapshotting turned on as the guest OS dispatching is delayed a "lot"

My plan? I run clustered product (vMBGs, vMCDs) on different vmware h/w to ensure availability.

Just to clarify 2 points -
The MiCollab is relevant to passing audio because MiCollab's built-in MBG does something in relation to the voice stream for external calls, right?
I understand MCD clustering, of course, but we were told the MiCollab doesn't do it.

Offline acejavelin

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 4064
  • Country: us
  • Karma: +129/-0
  • High-tech, heavy metal redneck!
    • View Profile
    • Like what I do and wanna help out? Send me a donation!
Re: MiCollab CPU Starvation alarm
« Reply #11 on: March 15, 2016, 08:38:57 AM »
Just to clarify 2 points -
The MiCollab is relevant to passing audio because MiCollab's built-in MBG does something in relation to the voice stream for external calls, right?
I understand MCD clustering, of course, but we were told the MiCollab doesn't do it.
Depends on your configuration... for Teleworker phones it obviously is a part of the voice stream, but it could be as well if you have SRC (Secure Recording Connector) for real-time call recording or if the MBG is acting as the session border controller for SIP trunking. It could also come into play with conferencing via the AWV blade.

Offline VinceWhirlwind

  • Hero Member
  • *****
  • Posts: 899
  • Country: au
  • Karma: +31/-0
    • View Profile
Re: MiCollab CPU Starvation alarm
« Reply #12 on: March 15, 2016, 07:32:49 PM »
So an out-of-hours snapshot of the MiCollab should be reasonably safe?

Offline dilkie

  • Global Moderator
  • Sr. Member
  • *****
  • Posts: 324
  • Karma: +11/-0
    • View Profile
Re: MiCollab CPU Starvation alarm
« Reply #13 on: March 15, 2016, 11:26:55 PM »
So an out-of-hours snapshot of the MiCollab should be reasonably safe?

I would think so, but turn off that "automatic snapshotty" feature that keeps dynamic track of the state of the vm... vmotion or something like that (forgot the name, and it's late)


 

Sitemap 1 2 3 4 5 6 7 8 9 10