AWC under attack

[ricvil]

Hi,

We have been getting hammered by attacks over the internet the last few days to our Audio, Web and Video conferencing server (v9.1.3.205).  The initial symptoms were email alerts indicating "TPS-7005 NO ports available".  I confirmed no calls were happening during the times of the alarm so I focused on doing packet captures on traffic coming from the internet.  It became clear that during the times the alerts were generated, some inbound traffic to UDP port 10074 causes the AWC server to generate enormous amounts of packets outbound (tens of thousands).  This is clearly a sign of a DDoS amplification style attack (a few packets sent our way, and we send tens of thousands back to the source).

The process listening on that port is "/usr/awc/tns/tp240dvr".

Has anybody else experienced this?  Does anybody know what I lose in AWC if I close off that UDP port to the internet?  I closed it and was able to have regular conferences so I don't know what it really is supposed to be doing.

Thanks,
Ricardo

[Monkeytail]

Hi Ricardo,

Had the same thing last 2 days.
Bin chasing it for 48 hours.
Mitel came with a patch.
3 Files need to be replaced.
Contact your Mitel support person to help you get the patch.

After tracing the issue to AWV we first stopped the service to have some network stability back.
We have about 50 systems running for multiple clients and it was madness
If your Mitel support can't help you please contact the company I work for. Maybe we can work something out.
You can reach us at +31880405858

Good luck.
Thijs

[ricvil]

Thank you Monkeytail.  I will contact our Mitel vendor for the patch.

[Monkeytail]

Hi Ricardo,

Please let us know if your Mitel vendor was able to help you.

Thanks,
Thijs

[Monkeytail]

Hi Ricardo,

Mitel just released 2 articles on KMS.   
The issue is fully addressed in MiCollab 9.5.

Document ID AL420
High Volume Traffic Detected on MiCollab Servers and MiVoice Business Express Servers

Document ID SO6795
High Traffic Volume Detected and Resource Alarms reported on MiCollab and MiVoice Business Express Servers


Resolution
Alter system configuration to bring MiCollab onto the LAN, fronted with an MBG proxy server.
Or
Alter firewall rules to block ports to the DMZ or LAN (10070, 10073, 10074)
Or
Apply the patch appropriate to your MiCollab version. Instructions and patch files are attached.


Thijs

[ricvil]

Hi Monkeytail,

I really appreciate this.  I downloaded and applied the patches from the KMS articles.  It was clear what Mitel did.  The new binaries only listen on the loopback interface (127.0.0.1) and not on the LAN or WAN ones.

For example, before:
# netstat -aupn | grep 10070
udp        0      0 0.0.0.0:10070           0.0.0.0:*                           22044/muxer

and after the patch:
# netstat -aupn | grep 10070
udp        0      0 127.0.0.1:10070         0.0.0.0:*                           6108/muxer

Thanks!

[Monkeytail]

New update received from Mitel

An Fix Pack for MiCollab 9.4 is to be released.

Prior to its release, please follow the instructions in SO6795

[Mecii57]

I am new to this site and we are having the same issue. Where do I get these patches and where do I find the Document ID AL420 and Document ID SO6795.

Thank you,
Mecii57

[Monkeytail]

Hi Mecii57,

The patch and the documents are available on Mitel KMS.
Your Mitel vendor should be able to help you.

Thijs

[Monkeytail]

Mitel released a new patch.

A software upgrade fix is targeted for 9.4. SP1 FP1 aiming for mid-March release.

For now:

Run the patch script on your server. This patch is applicable to all systems running 8.0 to 9.4, MiCollab or MiVB-x.

Patch available in KMS SO6795

[Monkeytail]

New Micollab version released.

A Fix Pack for MiCollab 9.4 (9.4.1.102) has been released.