Network Problems

[JeffonJohnsIsland]

Hello 

Systems are MiVoice Office 250, HX controllers, v6.3.7.58 to 6.3.7.99.

I am hoping one or more of you experts can either verify the networking is correct, or point out errors in the configuration below.

Company I work for went from two nodes to four.  All nodes/sites are connected via fiber.  I have two worries about the networking at this time:

1. In the SA&D GUI, on the System Information tab, in Networking Information N1-3 show N4 as only having VM instead of VM and IP.  Links show IP connections between all nodes as good (no red.)  N4 shows all as having VM and IP, with all links good (no red.)  This mystifies and frustrates me, perhaps it knaws at the little bit of OCD I have, especially since I don't want to start adding phones and then have to redo a bunch of work.

2. I see various errors in SA&D GUI, System Output.  Some of the errors are:
    N1
    -01:520- > -01:514- 11:43 06-19 M8027 SVR HW SS [7.0.0]: Illegal Data Byte Sent To Board: [C8,07,19,03,47,03,30,00,64,03,32,00,00,03,33,00,00,09,34,00,00,00,0A,00,FF,00,FF,02,47,00,02,35,03,03,36,00,50,05,37,C0,A8,C9,CA,03,38,C4,82,05,39,00,00,80,EA,05,3A]
   
   -01:520- > -01:334- 08:41 06-18 M2039 INF HW CP Dead Socket Response From node: 4 Reason(304): Broken pipe
   
    -01:520- > -01:335- 08:42 06-18 M2038 INF HW CP Failed Session Startup With Node 4
   
    -01:520- > -01:336- 08:42 06-18 M2038 INF HW CP Failed Session Startup With Node 4
   
    -01:520- > -01:337- 08:43 06-18 M2036 INF HW CP Established IP LINK To node 4

   N4
   -04:231- > -04:248- 17:18 06-18 M1002 INF DB CP Import Of Device Info From Node 1 Started
   -04:231- > -04:249- 17:18 06-18 *** off_node_device_t::store:  Invalid Field ID = 4
   -04:231- > -04:250- 17:18 06-18 *** off_node_device_t::store:  Invalid Field ID = 4
   -04:231- > -04:251- 17:18 06-18 *** off_node_device_t::store:  Invalid Field ID = 4
   -04:231- > -04:252- 17:18 06-18 M1003 INF DB CP Import Of Device Info From Node 1 Completed

N1, Syslog

un 19 06:25:05 Mitel5000BOFF2 kernel: imklog 3.18.6, log source = /proc/kmsg started.
Jun 19 06:25:05 Mitel5000BOFF2 rsyslogd: [origin software="rsyslogd" swVersion="3.18.6" x-pid="641" x-info="http://www.rsyslog.com"] restart
Jun 19 07:25:01 Mitel5000BOFF2 disk_info.ccare: /dev/hda1 (/) has 607M free space - 39% used (374M/990M)
Jun 19 07:25:01 Mitel5000BOFF2 disk_info.ccare: tmpfs (/lib/init/rw) has 50M free space - 0% used (0/50M)
Jun 19 07:25:01 Mitel5000BOFF2 disk_info.ccare: tmpfs (/dev/shm) has 50M free space - 0% used (0/50M)
Jun 19 07:25:01 Mitel5000BOFF2 disk_info.ccare: /dev/ram0 (/var/log/intl) has 2.2M free space - 42% used (1.6M/3.9M)
Jun 19 07:25:01 Mitel5000BOFF2 disk_info.ccare: /dev/ram1 (/var/log/diag) has 3.7M free space - 1% used (29K/3.9M)
Jun 19 07:28:39 Mitel5000BOFF2 file_check.ccare: [/etc/ppp/options] differs from backup
Jun 19 07:28:39 Mitel5000BOFF2 mem_info.ccare: 99MB RAM - 93MB used - 5MB free - 6MB buffers - 35MB cached
Jun 19 09:07:49 Mitel5000BOFF2 lighttpd[21920]: (connections.c.1702) SSL: 1 -1 error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
Jun 19 09:17:25 Mitel5000BOFF2 lighttpd[21920]: (connections.c.1702) SSL: 1 -1 error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
Jun 19 09:41:22 Mitel5000BOFF2 lighttpd[21920]: (connections.c.1702) SSL: 1 -1 error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
Jun 19 09:41:23 Mitel5000BOFF2 lighttpd[21920]: (server.c.1444) [note] sockets disabled, connection limit reached
Jun 19 09:41:26 Mitel5000BOFF2 lighttpd[21920]: (server.c.1398) [note] sockets enabled again
Jun 19 09:42:04 Mitel5000BOFF2 lighttpd[21920]: (connections.c.1702) SSL: 1 -1 error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
Jun 19 09:42:04 Mitel5000BOFF2 lighttpd[21920]: (connections.c.1702) SSL: 1 -1 error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
Jun 19 09:42:04 Mitel5000BOFF2 lighttpd[21920]: (connections.c.1702) SSL: 1 -1 error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
Jun 19 09:42:04 Mitel5000BOFF2 lighttpd[21920]: (connections.c.1702) SSL: 1 -1 error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
Jun 19 12:45:26 Mitel5000BOFF2 sshd[18317]: error: Could not get shadow information for admin
Jun 19 12:45:26 Mitel5000BOFF2 sshd[18317]: Accepted password for admin from 172.26.106.66 port 29074 ssh2
Jun 19 12:45:26 Mitel5000BOFF2 sshd[18321]: error: setlogin failed: Function not implemented
Jun 19 12:45:27 Mitel5000BOFF2 sshd[18317]: lastlog_openseek: Couldn't stat /var/log/lastlog: No such file or directory
Jun 19 12:45:27 Mitel5000BOFF2 sshd[18317]: lastlog_openseek: Couldn't stat /var/log/lastlog: No such file or directory

CP_SYSTEM_LOG

2019-06-17 11:21:27 SSL] SSL_accept() failed - returned -1, error code 5
[2019-06-17 11:21:27 CP Master SSL Socket] SSL negotiation failed

I have pasted in a bunch of the networking information below.  There are two primary sections below, network information from following networking starting at System, Devices and Feature Codes, Node IP Connection Groups and the second section starting at System, Devices and Feature Codes, Nodes.  I traced the programming by following the sections, clicking each result to reach the next section.  I hope it's something you can understand.


Why the one node showing only VM?

Are the errors a problem I should rectify before I proceed with setting up the phones?

From the bottom of my heart, thanks in advance for any help you can offer!

Jeff

I almost felt like I was making ASCII art (never have before), sorry about the lousy formatting. 




   Node                                                                                          1                     2                                   3                      4   
                                                                                                BO                        BW                              Ah                       Zi   
                                                                                               192.168.201.51    192.168.201.52           10.250.7.50      10.7.6.50   
                                                  Nodes (Node Route Group)         97001                   97002                           97003              97004   
                                                  Processor Module                         P6000                   P6100                           P6300              P6400   
                                                  Expansion Card                         P6001                   P6101                           P6301              P6401   
                     
               
                     
D&FC   Node IP Connection Groups*      Off-Node IP Connections                              P8000                           P8000              P8000   
                                                                              Extension                              P6000                           P6000              P6000   
                                                                           IP Connections                              97001                           97001              97001   
                                                                                                                    P6000                           P6000              P6000   
                                                       Node IP Connection Group                              P8000                           P8000              P8000   
                                                                 Remote IP Address                              192.168.201.51                   192.168.201.51     192.168.201.51   
                     
D&FC   Node IP Connection Groups*   Off-Node IP Connections          P8000                                               P8001             P8001   
                                                                           Extension          P6100                                               P6100             P6100   
                                                                             IP Connections          97002                                               97002             97002   
                                                                                                P6100                                               P6100             P6100   
                                                   Node IP Connection Group          P8000                                               P8001             P8001   
                                                             Remote IP Address          192.168.201.52                                       192.168.201.52  192.168.201.52   
                     
D&FC   Node IP Connection Groups*   Off-Node IP Connections          P8002           P8002                                                P8002   
                                                                           Extension          P6300           P6300                                                P6300   
                                                                                                97003           97003                                                97003   
                                                                         IP Connections          P6300           P6300                                                P6300   
                                                    Node IP Connection Group          P8002           P8002                                                P8002   
                                                              Remote IP Address          10.250.7.50   10.250.7.50                                        10.250.7.50   
                     
D&FC   Node IP Connection Groups*   Off-Node IP Connections          P8001           P8001                              P8002      
                                                                            Extension          P6400           P6400                              P6400      
                                                                         IP Connections          97004           97004                              97004      
                                                                                                P6400           P6400                              P6400      
                                                    Node IP Connection Group          P8001           P8001                              P8002      
                                                               Remote IP Address          10.6.7.50           10.6.7.50                              10.6.7.50      
                     
D&FC   Nodes                                                                                                 97001                              97001               97001   
      Node Trunk/IP Conn Groups                                                                      P8000                              P8000               P8000   
      Node IP Connection Groups                                                                              P8000                              P8000               P8000
      Off-Node IP Connections                                                                              P6000                              P6000               P6000   
      IP Connections                                                                                              97001                              97001               97001   
                                                                                                                    P6000                              P6000               P6000   
      Node IP Connection Group                                                                              P8000                              P8000               P8000   
      Remote IP Address                                                                                      192.168.201.51                      192.168.201.51  192.168.201.51   
                     
D&FC   Nodes                                                                            97002                                                          97002               97002   
      Node Trunk/IP Conn Groups                                                 P8000                                                          P8001               P8001   
      Node IP Connection Groups                                                         P8000                                                          P8001               P8001   
      Off-Node IP Connections                                                         P6100                                                          P6100               P6100   
      IP Connections                                                                         97002                                                          97002               97002   
                                                                                               P6100                                                          P6100               P6100   
      Node IP Connection Group                                                         P8000                                                          P8001               P8001   
      Remote IP Address                                                                 192.168.201.52                                          192.168.201.52 192.168.201.52   
                     
D&FC   Nodes                                                                            97003                   97003                                                   97003   
      Node Trunk/IP Conn Groups                                                 P8002                   P8002                                                   P8002   
      Node IP Connection Groups                                                         P8002                   P8002                                                   P8002   
      Off-Node IP Connections                                                         P6300                   P6300                                                   P6300   
      IP Connections                                                                         97003                   97003                                                   97003   
                                                                                               P6300                   P6300                                                   P6300   
      Node IP Connection Group                                                         P8002                   P8002                                                   P8002   
      Remote IP Address                                                                10.250.7.50           10.250.7.50                                           10.250.7.50   
                     
D&FC   Nodes                                                                           97004                   97004                               97004      
      Node Trunk/IP Conn Groups                                                P8001                   P8001                               P8002      
      Node IP Connection Groups                                                        P8001                   P8001                               P8002      
      Off-Node IP Connections                                                        P6400                   P6400                               P6400      
      IP Connections                                                                        97004                   97004                               97004      
                                                                                              P6400                   P6400                               P6400      
      Node IP Connection Group                                                        P8001                   P8001                               P8002      
      Remote IP Address                                                                10.6.7.50           10.6.7.50                               10.6.7.50      
                  

[Tech Electronics]

JeffonJohnsIsland,

Wow, that was hard to read.

Can you provide screenshots of what you think is "wrong". From what I could read and understand everything looks fine.

Thanks,

TE

[JeffonJohnsIsland]

TE,

Sorry about the ton of information. 

I will attach the pic and post a few of the error messages I see.

The difference between the two screen captures is N4 System Information, Networking Information shows N4 VM and IP, whereas N1 only shows VM for N4.  Does that make sense?

Jeff

[JeffonJohnsIsland]

TE,

These are the errors I have seen and cause me a bit of concern.

N1
    -01:520- > -01:514- 11:43 06-19 M8027 SVR HW SS [7.0.0]: Illegal Data Byte Sent To Board: [C8,07,19,03,47,03,30,00,64,03,32,00,00,03,33,00,00,09,34,00,00,00,0A,00,FF,00,FF,02,47,00,02,35,03,03,36,00,50,05,37,C0,A8,C9,CA,03,38,C4,82,05,39,00,00,80,EA,05,3A]
   
   -01:520- > -01:334- 08:41 06-18 M2039 INF HW CP Dead Socket Response From node: 4 Reason(304): Broken pipe
   
    -01:520- > -01:335- 08:42 06-18 M2038 INF HW CP Failed Session Startup With Node 4
   
    -01:520- > -01:336- 08:42 06-18 M2038 INF HW CP Failed Session Startup With Node 4
   
    -01:520- > -01:337- 08:43 06-18 M2036 INF HW CP Established IP LINK To node 4

   N4

   -04:197- > -04:177- 15:00 06-19 *** ip_port_device_t received unresolved datagram message
   -04:197- > -04:178- 15:00 06-19 *** datagram message type = 3587
   
   -04:231- > -04:248- 17:18 06-18 M1002 INF DB CP Import Of Device Info From Node 1 Started
   -04:231- > -04:249- 17:18 06-18 *** off_node_device_t::store:  Invalid Field ID = 4
   -04:231- > -04:250- 17:18 06-18 *** off_node_device_t::store:  Invalid Field ID = 4
   -04:231- > -04:251- 17:18 06-18 *** off_node_device_t::store:  Invalid Field ID = 4
   -04:231- > -04:252- 17:18 06-18 M1003 INF DB CP Import Of Device Info From Node 1 Completed

N1, Syslog

un 19 06:25:05 Mitel5000BOFF2 kernel: imklog 3.18.6, log source = /proc/kmsg started.
Jun 19 06:25:05 Mitel5000BOFF2 rsyslogd: [origin software="rsyslogd" swVersion="3.18.6" x-pid="641" x-info="http://www.rsyslog.com"] restart
Jun 19 07:25:01 Mitel5000BOFF2 disk_info.ccare: /dev/hda1 (/) has 607M free space - 39% used (374M/990M)
Jun 19 07:25:01 Mitel5000BOFF2 disk_info.ccare: tmpfs (/lib/init/rw) has 50M free space - 0% used (0/50M)
Jun 19 07:25:01 Mitel5000BOFF2 disk_info.ccare: tmpfs (/dev/shm) has 50M free space - 0% used (0/50M)
Jun 19 07:25:01 Mitel5000BOFF2 disk_info.ccare: /dev/ram0 (/var/log/intl) has 2.2M free space - 42% used (1.6M/3.9M)
Jun 19 07:25:01 Mitel5000BOFF2 disk_info.ccare: /dev/ram1 (/var/log/diag) has 3.7M free space - 1% used (29K/3.9M)
Jun 19 07:28:39 Mitel5000BOFF2 file_check.ccare: [/etc/ppp/options] differs from backup
Jun 19 07:28:39 Mitel5000BOFF2 mem_info.ccare: 99MB RAM - 93MB used - 5MB free - 6MB buffers - 35MB cached
Jun 19 09:07:49 Mitel5000BOFF2 lighttpd[21920]: (connections.c.1702) SSL: 1 -1 error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
Jun 19 09:17:25 Mitel5000BOFF2 lighttpd[21920]: (connections.c.1702) SSL: 1 -1 error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
Jun 19 09:41:22 Mitel5000BOFF2 lighttpd[21920]: (connections.c.1702) SSL: 1 -1 error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
Jun 19 09:41:23 Mitel5000BOFF2 lighttpd[21920]: (server.c.1444) [note] sockets disabled, connection limit reached
Jun 19 09:41:26 Mitel5000BOFF2 lighttpd[21920]: (server.c.1398) [note] sockets enabled again
Jun 19 09:42:04 Mitel5000BOFF2 lighttpd[21920]: (connections.c.1702) SSL: 1 -1 error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
Jun 19 09:42:04 Mitel5000BOFF2 lighttpd[21920]: (connections.c.1702) SSL: 1 -1 error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
Jun 19 09:42:04 Mitel5000BOFF2 lighttpd[21920]: (connections.c.1702) SSL: 1 -1 error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
Jun 19 09:42:04 Mitel5000BOFF2 lighttpd[21920]: (connections.c.1702) SSL: 1 -1 error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
Jun 19 12:45:26 Mitel5000BOFF2 sshd[18317]: error: Could not get shadow information for admin
Jun 19 12:45:26 Mitel5000BOFF2 sshd[18317]: Accepted password for admin from 172.26.106.66 port 29074 ssh2
Jun 19 12:45:26 Mitel5000BOFF2 sshd[18321]: error: setlogin failed: Function not implemented
Jun 19 12:45:27 Mitel5000BOFF2 sshd[18317]: lastlog_openseek: Couldn't stat /var/log/lastlog: No such file or directory
Jun 19 12:45:27 Mitel5000BOFF2 sshd[18317]: lastlog_openseek: Couldn't stat /var/log/lastlog: No such file or directory

CP_SYSTEM_LOG

2019-06-17 11:21:27 SSL] SSL_accept() failed - returned -1, error code 5
[2019-06-17 11:21:27 CP Master SSL Socket] SSL negotiation failed

Jeff

[acejavelin]

The first part is likely a communication issue occurred to between this node and node 4... The VPN dropped or had other issues for a short time, don't let the "illegal data bytes" thing bother you, it just means invalid messages were attempted to be processed. The other possibility is during the sync some things couldn't sync due to a difference in software versions. Although +/- 2 minor versions is allowed, it can cause sync errors. If the two are not the same version you should upgrade both the latest version.

The SSL issues are different, this could be an older browser was attempting to access the system or a hacking attempt, or less likely a corruption of the system software. I would suggest opening a ticket with Mitel support.

[Tech Electronics]

JeffonJohnsIsland,

Typically it is best to use the other view with the dot and lines instead of the 3 circles view; it provides more information. From what I am seeing it thinks that Node 4 is just a Voice Processor and not a controller with Voice Processing.

If the connectivity issue is a one-time thing then most likely it was a temporary connectivity issue and not a problem.

As for the web server [lighttpd] have you made sure that all of your SSL Certificates are current and refreshed; you can do that within the web browser portion of each controller. If that problem continues to persist you will need to call Mitel Tech Support to help resolve it.

Thanks,

TE