Calls dropping...no alarms but TSP trace shows temp failure

[evan631]

We have a office that is complaining of dropped calls and some other issues. We are using 3303s with PRIs
Provider is saying no issues on PRIs.

I ran a edt trace tsp l2l3 PLID and see a few entries with "Temporary Failures"  I don't have called numbers and times to match up with logs.

Could someone help out and let me know what these entries mean?  Where can I go from here to troubleshoot? 

I also see this in the logs:

SWERR range_check_error, OFFSET 2:$3252, PC: $66E898 LOG_ID: calllogs#140, PROCEDURE: calllogs_generate_one_log, SEVERITY: cp_major PID: ($1, $1A3) id: $03AC TRACEBACK (see above for first level info): cpcrrm +2:$DDA cpcrrm +2:$1000

OS (Message) warning: Receiver process not found. Redun code: $02 $02 $02 TX/RX msid: $00 $01 $00 $00 $00 $00 $01 $01 $00 $00 TX/RX pid: ($0003,$0223) cpprog ($0004,$05A2) ---------- appl/func: $01 $CC active pid: ($0003,$0223) cpprog

Greatly appreciated! 

2018-06-21  10:26:49  Link 1  - To Mitel   - HLen 4, ILen 13, IType 5
SAPI 0 TEI 0 C/R 1 P/F 0 NR 114 NS 82 INFO Dest PD=0x8 CR=0x168  PROGRESS(0x3)
1   00001000 Information Element: Cause: 
2   IE Length                   : 2 octets
3   1------- Extension bit      : Not Continued
    -00----- Coding Standard    : CCITT
    ---0---- spare 
    ----0010 Location           : Public network serving the local user
4   1------- Extension bit      : Not Continued
    -0101001 Cause code         : temporary failure                           
1   00011110 Information Element: Progress Indicator:
2   IE Length                   : 2 octets
3   1------- Extension bit      : Not Continued
    -00----- Coding Standard    : CCITT
    ---0---- Spare              : Spare
    ----0100 Location           : Public network serving the remote user
4   1------- Extension bit      : Not Continued
    -0001000 Description        : In-band information now available

++

018-06-21  10:26:53  Link 2  - To Mitel   - HLen 4, ILen 9, IType 5
SAPI 0 TEI 0 C/R 1 P/F 0 NR 55 NS 90 INFO Dest PD=0x8 CR=0x14e  DISCONNECT(0x45)
1   00001000 Information Element: Cause: 
2   IE Length                   : 2 octets
3   1------- Extension bit      : Not Continued
    -00----- Coding Standard    : CCITT
    ---0---- spare 
    ----0010 Location           : Public network serving the local user
4   1------- Extension bit      : Not Continued
    -0101001 Cause code         : temporary failure             

++

Thanks for the help!

-Evan

[acejavelin]

This means that there is a temporary failure at the physical layer on the ISDN network.

This is typically a cabling problem or equipment failure and is usually accompanied by other indications such as a high slip rate or errored seconds, do a "DTSTAT READ plid" to see if there is anything there or more technically "EDT SHOW FRAMER STATS PRI" and look for non-zero values in BPV1, BPV2, CRC1, and CRC2. Note that these may not actually raise an alarm in the system if it isn't severe enough.

Usually this is a patch cable or Dmarc extension with a bad connection, although I have seen a bad port cause it as well. From experience I find this type of issue is rarely the Mitel's PRI card, meaning I have never seen it be the cause unless there is an outside influence like lightning damage, water damage, or someone tried to shove a 6-pin cable into the 8-pin connector and physically damaged it, but is sometimes the interface card the Telco uses.

[evan631]

Thanks for the reply!

DTSTAT comes up clean on both PRIs
EDT show framer stats shows.  What's interesting here is there is only Link 2 but we have two. Not seeing non-zero in those fields.


Link FMSL TACW SGCW CLCW LIUC CFCW CTxP
2:    0xA0 0x10   0x90   0xA0 0x 0  0x 0   0x88


Link ALW1 SYNC OVFL MFSC PSW1 PSW2 FRBC RBOM RSSW OCFC ALRM BPV1 BPV2 CRC1 CRC2
2:    0x00  0x01  0x02 0x00 0x00   0xCC  0x00  0x00   0x08  0x00  0x1C  0x00 0x00 0x00 0x00

Do you still think this is a cabling issue?

Some additional info...we reboot the PBX every night, and on a few occasions some of the PRI channels on link 1 did not come up clean and were labled "unseizable".  However I have not seen that issue in a few weeks.

Any other thoughts?  We have the provider looking at something but I doubt they will find anything.  Might need to get cabling people in and run some tests.

Thanks again!

-Evan

This means that there is a temporary failure at the physical layer on the ISDN network.

This is typically a cabling problem or equipment failure and is usually accompanied by other indications such as a high slip rate or errored seconds, do a "DTSTAT READ plid" to see if there is anything there or more technically "EDT SHOW FRAMER STATS PRI" and look for non-zero values in BPV1, BPV2, CRC1, and CRC2. Note that these may not actually raise an alarm in the system if it isn't severe enough.

Usually this is a patch cable or Dmarc extension with a bad connection, although I have seen a bad port cause it as well. From experience I find this type of issue is rarely the Mitel's PRI card, meaning I have never seen it be the cause unless there is an outside influence like lightning damage, water damage, or someone tried to shove a 6-pin cable into the 8-pin connector and physically damaged it, but is sometimes the interface card the Telco uses.

[ZuluAlpha]

Are only external calls dropping?

[evan631]

I do believe it is External only. 

Are only external calls dropping?

[acejavelin]

It's cabling or an equipment failure... Most likely a patch cord or demarc extension cabling issue, meaning the physical connection from the telco's demarcation point (typically an Adtran or other gateway device, or an NIU/ONT) to the Mitel's PRI card.

1   00001000 Information Element: Cause: 
2   IE Length                   : 2 octets
3   1------- Extension bit      : Not Continued
    -00----- Coding Standard    : CCITT
    ---0---- spare 
    ----0010 Location           : Public network serving the local user
4   1------- Extension bit      : Not Continued
    -0101001 Cause code         : temporary failure             

0101001 = 41... so this is a Q931 Cause Code 41 of Temporary Failure of the physical link.

Cause No. 41 - temporary failure.

This cause indicates that the network is not functioning correctly and that the condition is not likely to last a long period of time; e.g., the user may wish to try another call attempt almost immediately.

What it means:
This means that there is a temporary failure at the physical layer on the ISDN network.