After reboot, phones show "press hold to send pin"... gurus help!

[cpegeno]

Hey all,
The issue has returned... and I ensure that all Secondary elements were set, as well as syncing to ALL sites, etc.

Details:

I work on a 55 site cluster, with VMCD brain, and have an issue that I cannot wrap my head around.

When the client reboots their VMCD on Wednesday night, many folks come in on Thursday morning to their IP PHONES displaying "press hold to send pin". I have verified that these users (all on different locals) have secondary elements setup in the vmcd programming.
I am unsure where to look first to start isolation on this. Our company doesn't maintain their network, and they have setup their own IP helper on their network. Their system is on 7.0...
About a year and a half ago, we showed them how to build simple users and MaC work, since they have an IT dept of 8 people and we service over 4000 users at their sites.

Has anyone else had this issue?
Please give me a little guidance... I know that the main telephony/it person says that these phones that he has built, some do this every time, some are intermittent, and above all, he has built other users that are not affected at all by this.

Please help!
Geno

[ralph]

Did you log into the secondary controller to be sure you see the phones programmed in there?

Ralph

[VinceWhirlwind]

Are the software versions the same across the cluster and vMCD?

[cpegeno]

The extensions are BOTH on the Vmcd AND the locals. So resiliency is good. I guess that a few of the extensions/phones were fine. One of the clients' phones screens broke. They replaced the set, and now that particular phone is doing this issue...
They just  blew out the MAC and put it to a new phone. Plot thickens...

Also,
active software on Vmcd is 13.0.0.83... active on one of the local boxes (which has some users doing this) is also 13.0.0.83

Has me stumped.
Gene

[cpegeno]

Long story short...
There are 50+ sites all clustered together, and it has become apparent that this trouble is NOT with any particular site, nor software. It is completely random.
I am curious if they have changed something in their network configuration that is timing out the cutover/passing of IP info from the Local or the Virtual...
I was under the impression that the phones IP via the local network, but pass call traffic via SIP/etc from the phone to the vmcd... I guess I need to read more into this.
If you have any suggestions on this topic please let me know!
Could this be an enterprise manager issue??
Any help is good help,
Geno

[cpegeno]

Next question...
Captain Obvious, maybe I am overlooking something?

Can COS options make the phones NOT re-register to VMCD during fail/reboot??

Please help.

Geno

[eugenej]

This is very odd.

So first thing is when the phone shows use superkey to send pin, is the phone's MAC address still configured in it's primary MiVB?
Also are you sure your cluster is working correctly? Verify the IP parameters of the phone after reboot. In other words specifically look at the MiVB it is pointing to. How do the phones get their MiVB IP info etc.

Use superkey to send pin means the phone is "talking" to a MiVB but the MiVB has no record of it so it won't assign the phone it's ext.

A 55 system cluster is fairly big. Is this a fully meshed system.

A wil stab but if the MAC exists I think your phone is likely talking to another system after reboot and the cluster is not directing the phone to home base since the remote directory number (read cluster) is not working properly.

Hope this helps

[cpegeno]

We have a sched reboot set for tonight. I'm hoping logs will show more info. The phones, if unplugged/replugged in... will reconnect. It's almost as if the failover happens from the V to the local, but then when the Virtual is reacquired the phone doesn't transition 100%. Rebooting re-establishes the connection.
I wish I knew more about what in the customer's network could stutter/impair this.

[eugenej]

We have a sched reboot set for tonight. I'm hoping logs will show more info. The phones, if unplugged/replugged in... will reconnect. It's almost as if the failover happens from the V to the local, but then when the Virtual is reacquired the phone doesn't transition 100%. Rebooting re-establishes the connection.
I wish I knew more about what in the customer's network could stutter/impair this.

You haven't really answered any of the questions posed. Perhaps you don't know the answer to them.
So let's take this in chunks:
So if the vm reboots or goes down, some but not all phones go to user superkey to send pin? Once the phone reboots when in this state, it comes right again?

[ralph]

The only reason a phone will ask you to "Press Hold to Enter Pin" is when the phone is trying to register to a system that doesn't know about that phone.  You're going to need to find it.  Usually this happens when there are 2 IP addresses in the DHCP options and there is something wrong with the resiliency.  Check your DHCP.  Are there two controllers in the options.  If yes, remove one of them.
You might also be able to check the IP inventory of each system to look for that MAC address.  That would tell you what system it's registering to.  I'm not sure on this but you may be able to check the config while it's in that state. (Hold vol up and Vol down, release the Vol down, enter 234)  Then check the IP config to see if it tells you what system it's trying to register to.

Ralph

[cpegeno]

I think that this is what is happening...
I do not have access to the clients network/ring... but I arrived onsite for one of the trouble phones today.
I went into the VMCD and tried to blow out/erase the phone/user etc... It threw a weird message about local mcd programming.
I then logged into the local, and when trying to erase that, I got another message about remnants on a secondary controller (which this message is normal). After a bit of phenagling (*sp), I finally had no trace/remnants of the extension anywhere (LOC FEAT EXTN 7xxxx).
Also, there was NO trace in the directories.
I went ahead, defaulted the hardware/phone. I rebuilt the set in the virtual, with secondary elements (kill/build), and assigned keys.
I walked over to the set, pressed *** 7XXXX, hold... and the display did not change. I am starting to believe that this set, on the network, is having problems talking through the client's network back to the Virtual (main brain).
I repeated these steps, 3 times, each about a minute apart. The phone finally pinned up. I started to pack up my things to leave, and checked the phone, and the display was back to "press hold to send pin". Rinse/repeat. Phone stayed up for the remaining half hour that I hung around.
These DHCP settings, are in the controller setup? Or is this something that would be in the Client's network? I understand using the up/down and 234 to see if the ip in the phone is pointing to the local pbx or the virtual... But that leads me back to the first question in understanding what is actually happening... is the phone to have a local ip but be pointed to the Virual (offsite) controller?, ONLY to move to the local in a resiliency case? (Thinking out loud)...
I have my fingers crossed.

[eugenej]

Ok so you haven't really answered some of my and Ralph's question directly but kinda in a round about way.

You need to be methodical otherwise you gonna be chasing your tail.

From the start (forget for a moment you have resiliency):

- When a phone boots up it gets info of where AN ICP is - it may or may not be it's home ICP. This is especially true in a cluster unless phones are VLANed in such a way that the phone gets its relevent ICP info from DHCP for instance.
-FACT: if your phone shows "Use Superkey to Send PIN" then it is talking to a MCD/MIVB (virtual or physical) of some description.
-FACT: your phone can have up to 4 ICP addresses it will/can connect to in a listed order. ICP1, ICP2, ICP3, ICP4 (this may be slightly different if you have very old software)
-FACT: When your phone is registered (you do *** 7xxxx hold/superkey) the MiVB/MCD (virtual or physical) will assign your phone's MAC address to the extension number that matches what you entered (7xxxx). So if you now go to the system and look at ext 7xxxx, you will see your phone's MAC address in there.
-FACT: If at some point this registered phone goes back to displaying "Use Superkey to Send Pin" then it means that your MAC address either disappeared from the MIVB/MCD  (its home MCD where the extension is/was created) OR your phone is no longer talking to that particular MCD/MiVB but another.

So at this point you need to check (this is where resiliency comes in):

1. Is the phone's MAC address still populated in the corresponding MCD system that has the extension number created on it? If No then someone or something is deleting or removing it. If it is still there see point 2.
2. Your phone is now likely been instructed or decided to look for an alternative system. An alternative system will only be possible if any of the other ICP's have been sent to the phone. That is ICP2, ICP3 or ICP4. These are sent to the phone either by DHCP, Static coded OR by virtue of the fact that you have resiliency (second MCD configured for the user).
---At this point it is important to know/understand what your phone has in it's ICP list - check this please.
3. Your phone will try connect to ICP2 for instance. If ICP2 is in fact your resilient system as you programmed it then the phone should "failover" and still have it's extension number. If ICP2 is the resilient MCD (second element) but the phone shows "Use Superkey to send pin" then you're cluster is not working properly. Note, yOur phone will only try the next ICP in the list, example ICP2 if it cannot reach ICP1.

Lot's more info on this but like I stated earlier you need to confirm a few things first.
a. Does your MAC address remain in the extension field when your phone exhibits this behaviour
b. what is your phone's ICP list.
c. If possible, figure out what is handing out DHCP and with that how option 125 is configured (content) AND check that your phone does not have any static settings on it, especially for ICP1-4

Hope this helps

[VinceWhirlwind]

In addition to Eugene's advice, don't rely on any assumption about where the handset is getting its IP address from.
Identify all your DHCP servers, do a packet trace on the handset switchport during boot-up, and confirm exactly where the active accepted lease has come from.
And log into the phone settings directly and check what controller IP address it received from DHCP.

[ralph]

In addition to Eugene's advice, don't rely on any assumption about where the handset is getting its IP address from.
Identify all your DHCP servers,

Very good point.

Ralph

[cpegeno]

Ok, sorry for not being clear, in the chain of posts.

1. I have logged into both the Vmcd and the 2nd controller and have confirmed that the Telephone directory, and user/services programming is present in both, and that sharing/sds is talking. I performed a kill/build to ensure that this was working well.
2. The software versions are all the same, currently 13.0.0.83. Including the sw on the Vmcd.
3. The phone displays Press HOLD to send pin. This does NOT affect ALL phones. It only affects about 1% or less of the entire client base (roughly <30 phones total of over 3900+ users).

I'm not on an engineering status, and I guess I do not understand exactly how the phones "REDIRECT" upon restoral of the VMCD (Main Brain) after it reboots, and shifts over to the Local (onsite) controller. Where/how can I read about "READ CLUSTER"?
I see that you advise "during boot" check IP of ICP on phone. Are there supposed to be multiple ICPs defined? This build was not done by me, I inherited 90% of it. The cluster had been fine for about 3 years, and in the last two months something has this handful of phones not coming up (without a physical reboot/unplug per phone AFTER the reboot).

Ralph/Vince/EugeneJ... I will keep you informed, and thanks for the info in the above thread. I guess I will have to see what the client is doing with his phones during his build, and will have to have them leave the phones in bad status until I can look at them.

I appreciate your patience greatly,
Geno