Networked Nodes, dropped calls

[Camoron]

Hi all,

Have 4 branches networked together and the connections seem stable now, I believe every port is forwarded that ought to be (though would appreciate any insight into that) but we occasionally have some dropped calls between branches. I checked out the logs to try to see what is happening but while I can see in the log that there's an error, the error is not verbose at all and I'm trying to figure out what is actually happening. I saw a number of errors occurring overnight while the system was not in use, but just witnessed one on site with the client while they were on the phone with another branch. The call dropped (no "insufficient bandwidth" alarm or anything), and they were able to call right back and complete the call. I immediately refreshed the log and caught it happening, but again, I don't know what the error is or why it's happening... seems to happen a lot on port 5570, which the client says is forwarded.

Here's the error, 192.168.4.254 is the remote node they were on the line with when the call dropped, this is from the local node's logs:


[18Nov15 09:32:12] ipra_app[ERR]: cc_socket_mgr_t: recv() returned 0!

[18Nov15 09:32:12] ipra_app[INFO]: cc_socket_mgr_t: socket 49 closed connection to 192.168.4.254:5570, error:  Unknown Error! (304)

[18Nov15 09:32:13] ipra_app[INFO]: cc_socket_mgr_t: socket 49 established call-control connection to 192.168.4.254:5570!

[Tech Electronics]

Camoron,

Using the Help [F1] function in database programming gives you this.

5570 is the TCP Call Control Port that off-node IP resources use to connect call control with the local IP resource. The TCP Call Control Port cannot conflict with other port numbers on the IP connection. The IP connection’s TCP Call Control Port must be kept in sync throughout the network and corresponds to the off-node IP connection’s Remote Listening Port.

Did you look at the other system that it lost connection to and see what the error  logs showed there?

Thanks,

TE

[Camoron]

Hi, thanks for the reply...
I thought this was solved as the client has been stable for a few weeks but had another hiccup the other day so I'm back on site looking into things.

Looking through logs for other branches reveals a lot of different errors using any number of different ports that I didn't know Mitel even used... the branch I am at now, however, only seems to have errors on port 5570

here's some of the log for one of the branches:

[01Dec15 02:13:53] ipra_app[INFO]: cc_socket_mgr_t: socket 37 timed out!

[01Dec15 02:13:53] ipra_app[INFO]: cc_socket_mgr_t: socket 37 closed connection to 192.168.2.254:49556, error:  EKATIMEOUT! (701)

[01Dec15 02:13:53] ipra_app[INFO]: cc_socket_mgr_t: socket 33 accepted call-control connection from 192.168.2.254:49557!

[01Dec15 03:26:26] ipra_app[ERR]: cc_socket_mgr_t: recv() returned 0!

[01Dec15 03:26:26] ipra_app[INFO]: cc_socket_mgr_t: socket 38 closed connection to 192.168.4.254:53832, error:  Unknown Error! (304)

[01Dec15 03:26:26] ipra_app[INFO]: cc_socket_mgr_t: socket 33 accepted call-control connection from 192.168.4.254:43374!

[01Dec15 03:34:25] ipra_app[ERR]: cc_socket_mgr_t: recv() returned 0!

[01Dec15 03:34:25] ipra_app[INFO]: cc_socket_mgr_t: socket 37 closed connection to 192.168.2.254:49557, error:  Unknown Error! (304)

[01Dec15 03:34:26] ipra_app[ERR]: cc_socket_mgr_t: recv() returned 0!

[01Dec15 03:34:26] ipra_app[INFO]: cc_socket_mgr_t: socket 38 closed connection to 192.168.4.254:43374, error:  Unknown Error! (304)

[01Dec15 03:34:27] ipra_app[ERR]: cc_socket_mgr_t: recv() returned 0!

[01Dec15 03:34:27] ipra_app[INFO]: cc_socket_mgr_t: socket 34 closed connection to 192.168.0.254:50881, error:  Unknown Error! (304)

[01Dec15 03:34:34] ipra_app[INFO]: cc_socket_mgr_t: socket 33 accepted call-control connection from 192.168.2.254:40223!

[01Dec15 03:34:34] ipra_app[INFO]: cc_socket_mgr_t: socket 33 accepted call-control connection from 192.168.4.254:59670!

[01Dec15 03:34:36] ipra_app[INFO]: cc_socket_mgr_t: socket 33 accepted call-control connection from 192.168.0.254:46715!

[01Dec15 04:26:41] ipra_app[INFO]: cc_socket_mgr_t: socket 34 timed out!

[01Dec15 04:26:41] ipra_app[INFO]: cc_socket_mgr_t: socket 34 closed connection to 192.168.2.254:40223, error:  EKATIMEOUT! (701)

[01Dec15 04:26:41] ipra_app[INFO]: cc_socket_mgr_t: socket 33 accepted call-control connection from 192.168.2.254:59321!

[01Dec15 04:27:08] ipra_app[INFO]: cc_socket_mgr_t: socket 37 timed out!

[01Dec15 04:27:08] ipra_app[INFO]: cc_socket_mgr_t: socket 37 closed connection to 192.168.4.254:59670, error:  EKATIMEOUT! (701)

[01Dec15 04:27:12] ipra_app[INFO]: cc_socket_mgr_t: socket 33 accepted call-control connection from 192.168.4.254:53990!

[01Dec15 04:27:42] ipra_app[ERR]: cc_socket_mgr_t: recv() returned 0!

[01Dec15 04:27:42] ipra_app[INFO]: cc_socket_mgr_t: socket 37 closed connection to 192.168.4.254:53990, error:  Unknown Error! (304)

[01Dec15 04:27:42] ipra_app[INFO]: cc_socket_mgr_t: socket 33 accepted call-control connection from 192.168.4.254:53991!

[01Dec15 04:54:26] ipra_app[INFO]: cc_socket_mgr_t: socket 34 timed out!

[01Dec15 04:54:26] ipra_app[INFO]: cc_socket_mgr_t: socket 34 closed connection to 192.168.2.254:59321, error:  EKATIMEOUT! (701)

[01Dec15 04:54:26] ipra_app[INFO]: cc_socket_mgr_t: socket 33 accepted call-control connection from 192.168.2.254:37782!

[01Dec15 05:10:03] ipra_app[INFO]: cc_socket_mgr_t: socket 37 timed out!

[01Dec15 05:10:03] ipra_app[INFO]: cc_socket_mgr_t: socket 37 closed connection to 192.168.4.254:53991, error:  EKATIMEOUT! (701)

[01Dec15 05:10:03] ipra_app[INFO]: cc_socket_mgr_t: socket 33 accepted call-control connection from 192.168.4.254:46861!

[01Dec15 05:13:36] ipra_app[INFO]: cc_socket_mgr_t: socket 37 timed out!

[01Dec15 05:13:36] ipra_app[INFO]: cc_socket_mgr_t: socket 37 closed connection to 192.168.4.254:46861, error:  EKATIMEOUT! (701)

[01Dec15 05:13:36] ipra_app[INFO]: cc_socket_mgr_t: socket 33 accepted call-control connection from 192.168.4.254:36333!

[01Dec15 05:41:49] ipra_app[INFO]: cc_socket_mgr_t: socket 34 timed out!

[01Dec15 05:41:49] ipra_app[INFO]: cc_socket_mgr_t: socket 34 closed connection to 192.168.2.254:37782, error:  EKATIMEOUT! (701)

[01Dec15 05:41:50] ipra_app[INFO]: cc_socket_mgr_t: socket 33 accepted call-control connection from 192.168.2.254:44891!

[01Dec15 05:51:42] ipra_app[INFO]: cc_socket_mgr_t: socket 37 timed out!

[01Dec15 05:51:42] ipra_app[INFO]: cc_socket_mgr_t: socket 37 closed connection to 192.168.4.254:36333, error:  EKATIMEOUT! (701)

[01Dec15 05:51:43] ipra_app[INFO]: cc_socket_mgr_t: socket 33 accepted call-control connection from 192.168.4.254:55657!

[01Dec15 06:29:18] ipra_app[ERR]: cc_socket_mgr_t: recv() returned 0!

[01Dec15 06:29:18] ipra_app[INFO]: cc_socket_mgr_t: socket 37 closed connection to 192.168.4.254:55657, error:  Unknown Error! (304)

[01Dec15 06:29:18] ipra_app[INFO]: cc_socket_mgr_t: socket 33 accepted call-control connection from 192.168.4.254:47036!

[01Dec15 06:32:39] ipra_app[INFO]: cc_socket_mgr_t: socket 34 timed out!

[01Dec15 06:32:39] ipra_app[INFO]: cc_socket_mgr_t: socket 34 closed connection to 192.168.2.254:44891, error:  EKATIMEOUT! (701)

[01Dec15 06:32:39] ipra_app[INFO]: cc_socket_mgr_t: socket 33 accepted call-control connection from 192.168.2.254:55829!

[01Dec15 07:00:29] ipra_app[ERR]: cc_socket_mgr_t: recv() returned 0!

[01Dec15 07:00:29] ipra_app[INFO]: cc_socket_mgr_t: socket 37 closed connection to 192.168.4.254:47036, error:  Unknown Error! (304)

[01Dec15 07:00:29] ipra_app[INFO]: cc_socket_mgr_t: socket 33 accepted call-control connection from 192.168.4.254:33145!

[01Dec15 07:45:56] ipra_app[INFO]: cc_socket_mgr_t: socket 34 timed out!

[01Dec15 07:45:56] ipra_app[INFO]: cc_socket_mgr_t: socket 34 closed connection to 192.168.2.254:55829, error:  EKATIMEOUT! (701)

[01Dec15 07:45:57] ipra_app[INFO]: cc_socket_mgr_t: socket 33 accepted call-control connection from 192.168.2.254:45186!

[01Dec15 08:03:59] ipra_app[ERR]: cc_socket_mgr_t: recv() returned 0!

[01Dec15 08:03:59] ipra_app[INFO]: cc_socket_mgr_t: socket 37 closed connection to 192.168.4.254:33145, error:  Unknown Error! (304)

[01Dec15 08:03:59] ipra_app[INFO]: cc_socket_mgr_t: socket 33 accepted call-control connection from 192.168.4.254:39748!

[01Dec15 08:05:07] ipra_app[INFO]: cc_socket_mgr_t: socket 34 timed out!

[01Dec15 08:05:07] ipra_app[INFO]: cc_socket_mgr_t: socket 34 closed connection to 192.168.2.254:45186, error:  EKATIMEOUT! (701)

[01Dec15 08:05:46] ipra_app[INFO]: cc_socket_mgr_t: socket 33 accepted call-control connection from 192.168.2.254:41670!

[01Dec15 08:05:52] ipra_app[INFO]: cc_socket_mgr_t: socket 33 accepted call-control connection from 192.168.2.254:41668!

[Tech Electronics]

Camoron,

If you are not at the latest PR for the version you are on then I suggest you upgrade them to that and see if that fixes the problem otherwise I think you will need to get Mitel involved on fixing your issue here.

Thanks,

TE

[Camoron]

Every other branch is on 6.0.11 except the main branch which is on 6.0.10. The main branch is also the one with the least amount of errors appearing in their ipra logs. Maybe there is a correlation. I'll look into it, thanks... unfortunately we don't have a certified Mitel technician anymore so we are unable to get tech support from Mitel.

[Camoron]

Over the last week or two this has degenerated into complete inability to call between certain branches (Busy).

Branches 2 and 3 have some problems connecting to other branches. Branch 4 is the most recently added one, and branch 2 and 3 have  hard time connecting to it, though it can connect to them...

here's some stuff from the message print for node 3

-03:655- 13:03 12-08 M2039 INF HW CP Dead Socket Response From node: 4 Reason(304): Broken pipe
-03:656- 13:03 12-08 M2036 INF HW CP Established IP LINK To node 4
-03:657- 13:08 12-08 M2039 INF HW CP Dead Socket Response From node: 4 Reason(304): Broken pipe
-03:658- 13:08 12-08 M2036 INF HW CP Established IP LINK To node 4
-03:659- 13:24 12-08 M2037 INF HW CP Closed IP LINK To node 4 - Reason(701): Socket timed out
-03:660- 13:24 12-08 M2036 INF HW CP Established IP LINK To node 4
-03:661- 13:25 12-08 M2039 INF HW CP Dead Socket Response From node: 4 Reason(304): Broken pipe
-03:662- 13:25 12-08 M2036 INF HW CP Established IP LINK To node 4
-03:663- 13:26 12-08 M5122 WRN HW CP Nodes Did Not Respond To Broadcast Message From 'PP022' : Nodes 4


As you can see it's only having problems with node 4, but the Network Information monitor shows Node 3 to Node 4 is busy as well as Node 3 to Node 2. It also shows Node 2 to Node 1 (the hub of the VPN through which all connections traverse) is busy.