Phones auto logging out - any ideas?

[pstones]

Getting a lot of errors as shown below in the logs.  This is when phone's are auto logging out for some reason.  The POE switches we have have enough power for all the ports.  There is no pattern with regards to it being during day or night.  Anyone got any ideas what is causing this?

2014/Apr/28   09:26:08   MOBILITY   Hot Desk : 2453 logged out from 8158 by system during registration
2014/Apr/28   08:57:35   MOBILITY   Hot Desk : 2228 logged out from 1974 by system during registration
2014/Apr/27   13:21:21   MOBILITY   Hot Desk : 2238 logged out from 8163 by system during registration
2014/Apr/26   19:29:24   MOBILITY   Hot Desk : 2284 logged out from 8099 by system during registration
2014/Apr/26   19:13:08   MOBILITY   Hot Desk : 2271 logged out from 8069 by system during registration
2014/Apr/26   19:01:57   MOBILITY   Hot Desk : 2294 logged out from 8073 by system during registration
2014/Apr/26   18:43:25   MOBILITY   Hot Desk : 2453 logged out from 8158 by system during registration
2014/Apr/26   18:26:39   MOBILITY   Hot Desk : 2281 logged out from 8155 by system during registration
2014/Apr/26   18:25:37   MOBILITY   Hot Desk : 2317 logged out from 8132 by system during registration
2014/Apr/26   18:17:32   MOBILITY   Hot Desk : 2261 logged out from 1981 by system during registration
2014/Apr/26   18:17:25   MOBILITY   Hot Desk : 2402 logged out from 8046 by system during registration
2014/Apr/26   18:17:21   MOBILITY   Hot Desk : 2448 logged out from 8171 by system during registration
2014/Apr/26   18:17:00   MOBILITY   Hot Desk : 2334 logged out from 8075 by system during registration
2014/Apr/26   18:16:44   MOBILITY   Hot Desk : 2285 logged out from 8131 by system during registration
2014/Apr/26   18:16:28   MOBILITY   Hot Desk : 2236 logged out from 1954 by system during registration
2014/Apr/26   18:00:06   MOBILITY   Hot Desk : 2469 logged out from 8020 by system during registration
2014/Apr/26   17:59:54   MOBILITY   Hot Desk : 2323 logged out from 8028 by system during registration
2014/Apr/26   17:51:56   MOBILITY   Hot Desk : 2411 logged out from 8039 by system during registration
2014/Apr/26   17:46:08   MOBILITY   Hot Desk : 2220 logged out from 8172 by system during registration
2014/Apr/26   17:43:58   MOBILITY   Hot Desk : 1038 logged out from 8022 by system during registration
2014/Apr/26   17:43:32   MOBILITY   Hot Desk : 2354 logged out from 8026 by system during registration
2014/Apr/26   17:43:19   MOBILITY   Hot Desk : 2270 logged out from 8120 by system during registration
2014/Apr/26   17:35:28   MOBILITY   Hot Desk : 2428 logged out from 1964 by system during registration
2014/Apr/26   17:34:46   MOBILITY   Hot Desk : 2337 logged out from 8166 by system during registration
2014/Apr/26   17:34:18   MOBILITY   Hot Desk : 2389 logged out from 8150 by system during registration
2014/Apr/26   17:27:40   MOBILITY   Hot Desk : 2322 logged out from 8078 by system during registration
2014/Apr/26   17:27:19   MOBILITY   Hot Desk : 2434 logged out from 8200 by system during registration
2014/Apr/26   17:26:45   MOBILITY   Hot Desk : 2419 logged out from 8036 by system during registration
2014/Apr/26   17:18:52   MOBILITY   Hot Desk : 2468 logged out from 8002 by system during registration
2014/Apr/26   17:00:10   MOBILITY   Hot Desk : 3269 logged out from 8173 by system during registration
2014/Apr/26   16:59:38   MOBILITY   Hot Desk : 2320 logged out from 8082 by system during registration
2014/Apr/26   16:59:31   MOBILITY   Hot Desk : 2297 logged out from 8088 by system during registration
2014/Apr/26   16:51:50   MOBILITY   Hot Desk : 2440 logged out from 1959 by system during registration
2014/Apr/26   16:51:26   MOBILITY   Hot Desk : 2324 logged out from 1960 by system during registration
2014/Apr/26   16:51:05   MOBILITY   Hot Desk : 1021 logged out from 8160 by system during registration
2014/Apr/26   16:42:42   MOBILITY   Hot Desk : 2256 logged out from 8113 by system during registration
2014/Apr/26   16:26:47   MOBILITY   Hot Desk : 2207 logged out from 8048 by system during registration
2014/Apr/26   16:24:52   MOBILITY   Hot Desk : 2346 logged out from 8090 by system during registration
2014/Apr/26   16:24:45   MOBILITY   Hot Desk : 2245 logged out from 8033 by system during registration
2014/Apr/26   16:24:36   MOBILITY   Hot Desk : 2340 logged out from 8105 by system during registration
2014/Apr/26   15:59:19   MOBILITY   Hot Desk : 2250 logged out from 8058 by system during registration
2014/Apr/26   15:57:13   MOBILITY   Hot Desk : 2264 logged out from 8038 by system during registration
2014/Apr/26   15:56:34   MOBILITY   Hot Desk : 2470 logged out from 8064 by system during registration
2014/Apr/26   15:56:16   MOBILITY   Hot Desk : 2381 logged out from 8003 by system during registration
2014/Apr/26   15:40:02   MOBILITY   Hot Desk : 2416 logged out from 1947 by system during registration
2014/Apr/26   15:31:09   MOBILITY   Hot Desk : 2455 logged out from 1978 by system during registration
2014/Apr/26   15:22:03   MOBILITY   Hot Desk : 2230 logged out from 8162 by system during registration
2014/Apr/26   15:22:02   MOBILITY   Hot Desk : 2471 logged out from 1943 by system during registration
2014/Apr/26   15:21:57   MOBILITY   Hot Desk : 2240 logged out from 8152 by system during registration
2014/Apr/26   15:14:41   MOBILITY   Hot Desk : 2379 logged out from 1977 by system during registration
2014/Apr/26   15:14:23   MOBILITY   Hot Desk : 2404 logged out from 8076 by system during registration
2014/Apr/26   14:59:28   MOBILITY   Hot Desk : 2392 logged out from 8097 by system during registration
2014/Apr/26   14:50:53   MOBILITY   Hot Desk : 2394 logged out from 8196 by system during registration
2014/Apr/26   14:48:30   MOBILITY   Hot Desk : 2363 logged out from 1961 by system during registration
2014/Apr/26   14:40:13   MOBILITY   Hot Desk : 2441 logged out from 1973 by system during registration
2014/Apr/26   14:39:32   MOBILITY   Hot Desk : 2459 logged out from 8004 by system during registration
2014/Apr/26   14:39:31   MOBILITY   Hot Desk : 2203 logged out from 8011 by system during registration
2014/Apr/26   14:23:23   MOBILITY   Hot Desk : 2300 logged out from 8040 by system during registration
2014/Apr/26   14:14:33   MOBILITY   Hot Desk : 2255 logged out from 8107 by system during registration
2014/Apr/26   13:24:00   MOBILITY   Hot Desk : 2357 logged out from 8123 by system during registration
2014/Apr/26   13:22:44   MOBILITY   Hot Desk : 2331 logged out from 8117 by system during registration
2014/Apr/26   13:22:18   MOBILITY   Hot Desk : 3263 logged out from 8207 by system during registration
2014/Apr/26   13:13:25   MOBILITY   Hot Desk : 2407 logged out from 1970 by system during registration
2014/Apr/26   12:54:12   MOBILITY   Hot Desk : 2445 logged out from 8094 by system during registration
2014/Apr/26   12:54:04   MOBILITY   Hot Desk : 2280 logged out from 8153 by system during registration
2014/Apr/26   12:27:08   MOBILITY   Hot Desk : 3248 logged out from 1939 by system during registration
2014/Apr/26   12:26:44   MOBILITY   Hot Desk : 2362 logged out from 1949 by system during registration
2014/Apr/26   12:26:10   MOBILITY   Hot Desk : 2350 logged out from 8164 by system during registration
2014/Apr/26   12:26:00   MOBILITY   Hot Desk : 2417 logged out from 1971 by system during registration
2014/Apr/26   12:07:28   MOBILITY   Hot Desk : 2321 logged out from 8093 by system during registration
2014/Apr/26   11:41:44   MOBILITY   Hot Desk : 2380 logged out from 8091 by system during registration
2014/Apr/26   11:25:44   MOBILITY   Hot Desk : 2329 logged out from 8032 by system during registration
2014/Apr/26   11:25:28   MOBILITY   Hot Desk : 2289 logged out from 8021 by system during registration
2014/Apr/26   11:01:08   MOBILITY   Hot Desk : 2452 logged out from 8080 by system during registration
2014/Apr/26   11:00:36   MOBILITY   Hot Desk : 2228 logged out from 8149 by system during registration
2014/Apr/26   10:23:20   MOBILITY   Hot Desk : 2314 logged out from 8129 by system during registration
2014/Apr/26   10:05:49   MOBILITY   Hot Desk : 2290 logged out from 8031 by system during registration
2014/Apr/26   09:57:54   MOBILITY   Hot Desk : 2254 logged out from 8109 by system during registration
2014/Apr/26   09:57:11   MOBILITY   Hot Desk : 2275 logged out from 1948 by system during registration
2014/Apr/26   08:48:03   MOBILITY   Hot Desk : 2458 logged out from 8061 by system during registration
2014/Apr/26   08:04:40   MOBILITY   Hot Desk : 1005 logged out from 8190 by system during registration
2014/Apr/26   06:01:11   MOBILITY   Hot Desk : 1019 logged out from 8184 by system during registration
2014/Apr/25   22:41:48   MOBILITY   Hot Desk : 3260 logged out from 8177 by system during registration
2014/Apr/25   18:53:32   MOBILITY   Hot Desk : 2359 logged out from 8125 by system during registration
2014/Apr/25   18:45:12   MOBILITY   Hot Desk : 2218 logged out from 8098 by system during registration
2014/Apr/25   17:53:24   MOBILITY   Hot Desk : 2383 logged out from 8138 by system during registration
2014/Apr/25   17:07:53   MOBILITY   Hot Desk : 2456 logged out from 8174 by system during registration
2014/Apr/25   16:59:51   MOBILITY   Hot Desk : 2244 logged out from 8217 by system during registration
2014/Apr/25   16:34:01   MOBILITY   Hot Desk : 2344 logged out from 8024 by system during registration
2014/Apr/25   16:18:42   MOBILITY   Hot Desk : 2414 logged out from 8101 by system during registration
2014/Apr/25   16:01:33   MOBILITY   Hot Desk : 2391 logged out from 8161 by system during registration
2014/Apr/25   15:52:49   MOBILITY   Hot Desk : 2460 logged out from 8067 by system during registration
2014/Apr/25   15:52:28   MOBILITY   Hot Desk : 2475 logged out from 1995 by system during registration
2014/Apr/25   15:25:20   MOBILITY   Hot Desk : 2390 logged out from 8042 by system during registration
2014/Apr/25   15:25:06   MOBILITY   Hot Desk : 2252 logged out from 1932 by system during registration
2014/Apr/25   15:15:47   MOBILITY   Hot Desk : 2457 logged out from 8115 by system during registration
2014/Apr/25   15:15:07   MOBILITY   Hot Desk : 2409 logged out from 1953 by system during registration
2014/Apr/25   14:56:06   MOBILITY   Hot Desk : 2228 logged out from 8149 by system during registration
2014/Apr/25   14:48:09   MOBILITY   Hot Desk : 2386 logged out from 1965 by system during registration
2014/Apr/25   12:43:00   MOBILITY   Hot Desk : 2472 logged out from 8079 by system during registration
2014/Apr/25   11:15:04   MOBILITY   Hot Desk : 2412 logged out from 8017 by system during registration
2014/Apr/25   10:58:45   MOBILITY   Hot Desk : 2439 logged out from 8057 by system during registration
2014/Apr/25   10:42:14   MOBILITY   Hot Desk : 2416 logged out from 1947 by system during registration
2014/Apr/25   10:24:20   MOBILITY   Hot Desk : 2289 logged out from 8021 by system during registration
2014/Apr/25   09:59:06   MOBILITY   Hot Desk : 3262 logged out from 8062 by system during registration
2014/Apr/25   09:01:48   MOBILITY   Hot Desk : 2272 logged out from 8052 by system during registration
2014/Apr/25   08:55:12   MOBILITY   Hot Desk : 2249 logged out from 8066 by system during registration
2014/Apr/25   05:33:19   MOBILITY   Hot Desk : 3247 logged out from 8182 by system during registration
2014/Apr/25   05:07:51   MOBILITY   Hot Desk : 1035 logged out from 8206 by system during registration
2014/Apr/25   04:42:55   MOBILITY   Hot Desk : 3244 logged out from 8205 by system during registration
2014/Apr/25   03:10:56   MOBILITY   Hot Desk : 3242 logged out from 8179 by system during registration
2014/Apr/25   03:10:35   MOBILITY   Hot Desk : 3260 logged out from 8177 by system during registration
2014/Apr/25   03:03:47   MOBILITY   Hot Desk : 1019 logged out from 8184 by system during registration
2014/Apr/25   01:50:24   MOBILITY   Hot Desk : 2474 logged out from 8211 by system during registration
2014/Apr/25   01:34:55   MOBILITY   Hot Desk : 1029 logged out from 8188 by system during registration
2014/Apr/24   19:08:11   MOBILITY   Hot Desk : 2287 logged out from 8169 by system during registration
2014/Apr/24   17:32:41   MOBILITY   Hot Desk : 2365 logged out from 8009 by system during registration
2014/Apr/24   17:25:04   MOBILITY   Hot Desk : 1020 logged out from 8270 by system during registration
2014/Apr/24   17:23:41   MOBILITY   Hot Desk : 2473 logged out from 8045 by system during registration
2014/Apr/24   17:23:19   MOBILITY   Hot Desk : 3269 logged out from 8173 by system during registration
2014/Apr/24   16:38:06   MOBILITY   Hot Desk : 2352 logged out from 8051 by system during registration
2014/Apr/24   16:14:23   MOBILITY   Hot Desk : 2362 logged out from 1952 by system during registration
2014/Apr/24   16:14:13   MOBILITY   Hot Desk : 2263 logged out from 1941 by system during registration
2014/Apr/24   16:13:40   MOBILITY   Hot Desk : 2229 logged out from 8157 by system during registration
2014/Apr/24   16:05:22   MOBILITY   Hot Desk : 2407 logged out from 1970 by system during registration
2014/Apr/24   15:58:44   MOBILITY   Hot Desk : 2279 logged out from 8095 by system during registration
2014/Apr/24   15:57:06   MOBILITY   Hot Desk : 2466 logged out from 8081 by system during registration
2014/Apr/24   15:46:41   MOBILITY   Hot Desk : 2250 logged out from 8058 by system during registration
2014/Apr/24   14:30:29   MOBILITY   Hot Desk : 2337 logged out from 8166 by system during registration
2014/Apr/24   14:21:48   MOBILITY   Hot Desk : 2394 logged out from 8196 by system during registration
2014/Apr/24   13:05:16   MOBILITY   Hot Desk : 2296 logged out from 1983 by system during registration
2014/Apr/24   12:39:54   MOBILITY   Hot Desk : 2392 logged out from 8097 by system during registration
2014/Apr/24   12:39:54   MOBILITY   Hot Desk : 2432 logged out from 8055 by system during registration
2014/Apr/24   12:39:12   MOBILITY   Hot Desk : 2391 logged out from 8161 by system during registration
2014/Apr/24   12:30:39   MOBILITY   Hot Desk : 2261 logged out from 1981 by system during registration
2014/Apr/24   12:21:46   MOBILITY   Hot Desk : 2220 logged out from 8172 by system during registration
2014/Apr/24   10:36:46   MOBILITY   Hot Desk : 2470 logged out from 8064 by system during registration
2014/Apr/24   10:29:07   MOBILITY   Hot Desk : 2210 logged out from 8259 by system during registration
2014/Apr/24   10:10:12   MOBILITY   Hot Desk : 2216 logged out from 8192 by system during registration
2014/Apr/24   10:09:53   MOBILITY   Hot Desk : 2203 logged out from 8011 by system during registration
2014/Apr/24   09:33:50   MOBILITY   Hot Desk : 2424 logged out from 8268 by system during registration
2014/Apr/24   07:54:55   MOBILITY   Hot Desk : 2453 logged out from 8158 by system during registration
2014/Apr/24   03:18:27   MOBILITY   Hot Desk : 1019 logged out from 8184 by system during registration
2014/Apr/24   00:49:09   MOBILITY   Hot Desk : 2222 logged out from 8213 by system during registration
2014/Apr/23   18:43:06   MOBILITY   Hot Desk : 2357 logged out from 8123 by system during registration
2014/Apr/23   18:22:28   MOBILITY   Hot Desk : 2359 logged out from 8125 by system during registration
2014/Apr/23   18:15:18   MOBILITY   Hot Desk : 2379 logged out from 1977 by system during registration
2014/Apr/23   18:14:37   MOBILITY   Hot Desk : 2346 logged out from 8090 by system during registration
2014/Apr/23   18:05:54   MOBILITY   Hot Desk : 2314 logged out from 8129 by system during registration
2014/Apr/23   17:06:07   MOBILITY   Hot Desk : 2446 logged out from 8203 by system during registration
2014/Apr/23   17:05:10   MOBILITY   Hot Desk : 2244 logged out from 8217 by system during registration
2014/Apr/23   16:48:46   MOBILITY   Hot Desk : 2459 logged out from 8004 by system during registration
2014/Apr/23   16:31:17   MOBILITY   Hot Desk : 2453 logged out from 8158 by system during registration
2014/Apr/23   15:46:26   MOBILITY   Hot Desk : 2245 logged out from 8033 by system during registration
2014/Apr/23   15:46:22   MOBILITY   Hot Desk : 2284 logged out from 8099 by system during registration
2014/Apr/23   15:00:25   MOBILITY   Hot Desk : 2262 logged out from 8092 by system during registration
2014/Apr/23   14:26:33   MOBILITY   Hot Desk : 2361 logged out from 1975 by system during registration
2014/Apr/23   13:41:18   MOBILITY   Hot Desk : 3265 logged out from 8005 by system during registration
2014/Apr/23   13:17:40   MOBILITY   Hot Desk : 2413 logged out from 1985 by system during registration
2014/Apr/23   12:12:07   MOBILITY   Hot Desk : 2313 logged out from 8170 by system during registration
2014/Apr/23   11:45:42   MOBILITY   Hot Desk : 2236 logged out from 1954 by system during registration
2014/Apr/23   11:22:15   MOBILITY   Hot Desk : 1028 logged out from 1935 by system during registration
2014/Apr/23   10:03:23   MOBILITY   Hot Desk : 2289 logged out from 8021 by system during registration
2014/Apr/23   09:47:54   MOBILITY   Hot Desk : 2409 logged out from 1953 by system during registration
2014/Apr/23   09:46:54   MOBILITY   Hot Desk : 2457 logged out from 8115 by system during registration
2014/Apr/23   09:20:01   MOBILITY   Hot Desk : 2223 logged out from 8029 by system during registration
2014/Apr/23   08:24:19   MOBILITY   Hot Desk : 2242 logged out from 8167 by system during registration
2014/Apr/23   06:55:26   MOBILITY   Hot Desk : 1035 logged out from 8206 by system during registration
2014/Apr/23   06:08:54   MOBILITY   Hot Desk : 2222 logged out from 8213 by system during registration
2014/Apr/23   05:06:16   MOBILITY   Hot Desk : 3268 logged out from 8216 by system during registration
2014/Apr/23   04:23:35   MOBILITY   Hot Desk : 1019 logged out from 8184 by system during registration
2014/Apr/23   03:40:02   MOBILITY   Hot Desk : 3244 logged out from 8205 by system during registration
2014/Apr/23   03:31:08   MOBILITY   Hot Desk : 2474 logged out from 8188 by system during registration
2014/Apr/22   12:29:51   MOBILITY   Hot Desk : 2256 logged out from 8113 by system during registration

[ralph]

Looks like something is causing the phones to de-register.
When  ever I see this it's always something in the network.
Take a look at your maintenance / software logs.
You may see where the phones lost connection or heartbeat.


Ralph

[pstones]

You are right I'm seeing heartbeat problems just before un-registration.

#2014/Apr/26   18:17:32   MOBILITY   Hot Desk : 2261 logged out from 1981 by system during registration
2014/Apr/26   18:17:25   MOBILITY   Hot Desk : 2402 logged out from 8046 by system during registration
2014/Apr/26   18:17:21   MOBILITY   Hot Desk : 2448 logged out from 8171 by system during registration
2014/Apr/26   18:17:00   MOBILITY   Hot Desk : 2334 logged out from 8075 by system during registration
2014/Apr/26   18:16:44   MOBILITY   Hot Desk : 2285 logged out from 8131 by system during registration
2014/Apr/26   18:16:28   MOBILITY   Hot Desk : 2236 logged out from 1954 by system during registration
2014/Apr/26   18:13:17   Heartbeat Server   ICP has lost contact with (10.80.10.253 (08-00-0F-7A-7B-90)), network driver cluster pool free: 3014 and low water mark: 2575
2014/Apr/26   18:13:16   Heartbeat Server   ICP has lost contact with (10.80.10.235 (08-00-0F-7B-77-32)), network driver cluster pool free: 3014 and low water mark: 2575
2014/Apr/26   18:13:15   Heartbeat Server   ICP has lost contact with (10.80.10.228 (08-00-0F-7B-77-09)), network driver cluster pool free: 3014 and low water mark: 2575
2014/Apr/26   18:11:22   MOBILITY   Hot Desk : Logged out from 1913 - Associated hot desk user was 1015
2014/Apr/26   18:11:22   MOBILITY   Hot Desk : Logout of 1015 from 1913 initiated
2014/Apr/26   18:04:12   Heartbeat Server   ICP has lost contact with (10.80.10.230 (08-00-0F-7B-6F-5A)), network driver cluster pool free: 3014 and low water mark: 2575
2014/Apr/26   18:04:01   Heartbeat Server   ICP has lost contact with (10.80.10.144 (08-00-0F-7B-73-A3)), network driver cluster pool free: 3014 and low water mark: 2575
2014/Apr/26   18:04:00   Heartbeat Server   ICP has lost contact with (10.80.10.171 (08-00-0F-7B-7E-0C)), network driver cluster pool free: 3014 and low water mark: 2575
2014/Apr/26   18:04:00   Heartbeat Server   ICP has lost contact with (10.80.10.109 (08-00-0F-7B-7D-BA)), network driver cluster pool free: 3014 and low water mark: 2575
2014/Apr/26   18:03:59   Heartbeat Server   ICP has lost contact with (10.80.10.52 (08-00-0F-7B-77-41)), network driver cluster pool free: 3014 and low water mark: 2575
2014/Apr/26   18:03:58   Heartbeat Server   ICP has lost contact with (10.80.10.35 (08-00-0F-7B-7D-BF)), network driver cluster pool free: 3014 and low water mark: 2575

[martyn]

Are all phones in the same subnet? I notice that all of the ones that are listed there are, but is every phone in the same subnet? Also is the controller in that subnet?

[pstones]

Same subnet, same vlan

[v2win]

Its looks similar to a problem I had recently with network congestion.

Can you check the switch uplink ports for errors?

[jrg0852]

I would take a problem phone and connect direct to 2nd LAN of 3300 with power brick. If it still happens, obviously system related. If not, POE switches/network.

[pstones]

It does not happen when patched directly into the mitel on a power brick.  I have asked Mitel if we can change the time-out settings on the system as it will be a nightmare to investigate structured cabling / network side of things..

[ralph]

I stumbled across a related document on MOL. (Article ID # 08-5191-00104)
Here's the relevant section:

2.  recommend (as a test) relaxing the heartbeat timers in the Controller Registry form
       they are:
          Link Maintenance - Heartbeat Rx Link Failure Time-out (1/10 seconds) default = 300; try 400
          Link Maintenance - Phone Inactivity Tx Timer (1/10 seconds) default = 300; try 400

As it suggest, try it as a test but I wouldn't set it too high.  It may cause other issues.

Ralph