IP Trunk Problem: RMESS VERIFY PBX fails with Ethernet Out of Service

[GMan]

I have three 3300 MCD's, all running Release 6.0 PR1 (12.0.0.52).  They are clustered, and SDS sharing is implemented and working.  I am attempting to replace T1 links between the PBX's with IP trunking.  To that end I have programmed the IP trunks as per Mitel's documentation (something I have done previously at other sites without problem).

When I issue the RMESS VERIFY PBX x LAN maintenance command it fails when issued from PBX 1 to PBX's 1 (itself), 2, and 3, and also when issued from PBX 2 to PBX's 1, 2 (itself) and 3.  The command returns RMESSAGE: Ethernet Out of Service, Failed to connect to PBX x.  When the command is issued from PBX 3 to PBX 3 (itself), the verify works - it returns RMESSAGE: Verification successful to PBX 3.  When issued from PBX 3 to PBX's 1 or 2, it fails, returning Unsolicited STH Close, Failed to connect to PBX x.  This leads me to believe that the IP trunks are functional on PBX 3, although they can't connect to PBX's 1 and 2 because the trunks are not functioning on those PBX's.

Issuing the STATE XNET ALL IP maintenance command on PBX 1 or 2 fails with ICP/PBX not programmed for XNET IP - State Aborted.  When issued from PBX 3, the command returns a table listing all PBX's, all showing a state of Idle.

Other factors:
- I can log into the System Administration Tool of any of the PBX's from a laptop connected to any of the subnets to which the PBX's are connected.
- I can successfully access forms on all remote PBX's from any of the PBX's using Application Reach Through.
- All PBX's are licensed as Enterprise systems.
- SDS Sharing has been running for 2+ years without any problems.
- PBX 1 and PBX 2 are located at the same site, and their Ethernet ports are connected to the same layer 2 switch.  Therefore there are no issues with firewalls/gateways blocking connectivity between these switches.
- Rebooting all three PBX's did not resolve the problem.

I had this problem a number of years ago at another site, but don't recall the solution; I had Mitel tech support investigate.  I have a vaguely recollection that it had something to do with a service not running on the PBX.  Does anyone have any experience with this who could assist?  Much appreciated.

[lundah]

Check the programming in ICP/PBX Networking on all 3 systems, make sure the correct system is flagged as local in each. Double check your License and Options forms and System Capacity as well, just to be sure.

[GMan]

Check the programming in ICP/PBX Networking on all 3 systems, make sure the correct system is flagged as local in each. Double check your License and Options forms and System Capacity as well, just to be sure.

Thanks for the suggestions.

All is good in the ICP/PBX Networking forms - have checked them a number of times, as well as the IP/XNET Trunk Profiles, IP/XNET Trunk Groups and Network Elements forms.

The License And Options forms and System Capacity forms are fine as well - although I don't believe there is any special licensing required for IP trunks.  All System Capacity values are fine as well; I'm nowhere near using up the assignments in any category.

[lundah]

Yeah the Enterprise licensing should be all you need to do IP networking between nodes. The only other time I've seen what you're seeing is when the ICP/XNET forms aren't correct. Basically what I think the system is telling you is that it's not optioned or licensed properly for IP trunks, but it sounds like in reality you are set up correctly. Is a DBMS check passing? Next thing I would suspect is database corruption. Run a "DBMS Check Full", and maybe try a backup/restore on one of the problem nodes and see if that changes anything.

[GMan]

Is a DBMS check passing? Next thing I would suspect is database corruption. Run a "DBMS Check Full", and maybe try a backup/restore on one of the problem nodes and see if that changes anything.

The daily DBMS checks are passing consistently.  I've done a full check, and they pass as well.  A backup/restore - possibly with a re-load of software as well may be the solution, as all services would be starting fresh at that point, but it would be a bit of work, as one of the systems has flexed dimensions.  I'm thinking of de-clustering and rebuilding the ICP/PBX Networking forms on the problem switches, thinking if there is corruption that may solve it.