Cisco Voice Servers Version 11.5 Could Not Load modules.dep

About 6 months ago we updated 3/4 of our Cisco Telephony environment from 8.5 to 11.5. The only reason we didn’t do it all is because UCCX 11.5 wasn’t out yet so it went to 11. While there were a few bumps in the road; resizing VMs, some COP files, etc. the update went well. Unfortunately once it was done we starting having a glorious issue where after a reboot the servers sometimes failed to boot, presenting “FATAL: Could not load /lib/modules/2.6.32-573.18.1.el6.x86_64/modules.dep: No such file or directory”. Any way you put it, this sucked.

The first time this happened I call TAC and while they had seen it, they had no good answer except for rebuild the VM, restore from backup. Finally after the 3rd time (approximately 3 months after install) the bug had been officially documented and (yay) it included a work around. The good news is that the underlying issue at this point has been fixed in 11.5(1.11900.5) and forward so if you are already there, no problems.

The issue lies with the fact that the locked down build of RHEL 6 that any of the Cisco Voice server platforms are built on don’t handle VMware Tools updates well. It’s all good when you perform a manual update from their CLI and use their “utils vmtools refresh” utility, but many organizations, mine included, choose to make life easier and enable vCenter Update Manager to automatically upgrade the VMware tools each time a new version is available and the VM is rebooted.

So how do you fix it? While the bug ID has the fix in it, if you aren’t a VMware regular they’ve left out a few steps and it may not be the easiest thing to follow. So here I’m going to run down the entire process and get you (and chances are, myself when this happens in the future) back up and running.

0. Go out to the cisco.com site and download the recovery CD for 11.5. You should be able to find that here, but if not or if you need a different version browse through the downloads to Downloads Home > Products > Unified Communications > Call Control > Unified Communications Manager (CallManager) > Unified Communications Manager Version 11.5 > Recovery Software. Once done upload this to any of the datastores available to host your failing VM resides on.
1. If you’ve still got the VM running, shut it down by right clicking the VM>Power>Power Off in the vCenter Web UI or the ESXi embedded host client.
2. Now we need to make a couple of modifications to the VM’s settings to tell it 1) attach the downloaded ISO file and check the “Connected at boot” box and 2) Under VM Options> Boot Options to “Force BIOS setup” at next boot. By default VMs do not look at attached ISOs as the first boot device. Once both of these are done it’s time to boot the VM.
3. I personally like to launch the VMware Remote Console first and then boot from there, that way I’ve already got the screen up. After you power on the BIOS in a VM is the same old Phoenix BIOS we all know and love. Simply tell the VM to boot to CD before hard drive, move to Exit and “Save and Exit” and your VM will reboot directly into the recovery ISO.
4.  Once you get up to the Recovery Disk menu screen as shown to the left we need to get out to a command prompt. To do this hit Alt-F2 and you’ll be presented with a standard bash prompt.
5. So the root cause of all this issue is that the initramfs file is improperly sized after an automatic upgrade of VMware tools has been processed. So now that we have our prompt we first need to verify that we are actually seeing the issue we expect. To do this run the command “ find / -name initramfs* .” This command should produce the full path and filename of the file. So to get the size of this file you now need to run an ls -lh against it. In my example your full command would be “ ls -lh /mnt/part1/boot/initramfs-2.6.32-573.18.1.el6.x86_64.img .” If you aren’t particularly used to the Linux CLI once you get past …initr you should be able to hit tab to autocomplete. This should respond by showing you that that file is incorrectly sized somewhere between 11-15 MB.
6. Now we need to perform a chroot on the directory that contains boot objects. In most cases this should simply be “ chroot /mnt/part1 “

7. Finally we need to manually re-run the VMware Tools installer to to get the file properly sized. These are included locally on the Recovery Disk so just run the command “ /usr/bin/vmware-config-tools.pl -d ” There are various steps throughout the process where it is going to ask for input. Unless you know you have a reason to differ just hit enter at each one until it completes.

Once the VMware Tools installation is done up arrow to where you checked the size of initramfs…img file above and rerun the command. You should now see file size changed to 24 MB or so.

8. Now we just need to do a little clean up before we reboot. You need to make sure you go into Settings for your VM and tell it not to connect the ISO at boot. Once you make that change you should be able to flip back over to your console and simply type reboot  or shutdown -r 0  to reboot back to full functionality.

 

Vegas Baby! Heading to CiscoLive! 2016

As 2016 moves into April we find ourselves ready to go into the conference season once again. For the past couple of years I’ve been to VMworld because that is where my work has had me focused, but for the same reason I will be heading the Cisco Live in Las Vegas, NV this year. The event will be held at the Mandalay Bay Resort July 10-14. Yes it will be hot, but let’s be honest you are going to be inside most of the time. This is the 2nd time I’ve attended Cisco Live US (you may see it referred to as #CLUS quite a bit) and if this is anything like the last time it’s going to be great. I have been particularly impressed with the content they make available and the community that has grown around it.

What to do

The first and foremost thing you should check out at Cisco Live is the always excellent sessions throughout the conference. If you are new to conferences this is actually something to consider sooner than later; the session catalog is currently up and the scheduler will open on May 3. I recommend that if you have any particular sessions or focus you are looking at with this trip go ahead and have a list done early and then be ready on the 5/3, many popular sessions will fill up quickly and nobody wants to wait in the overflow line. 😉

To be honest if you just look at the scope of topics covered in the session list it is a bit overwhelming. While I’m no grizzled veteran of conferences by any means what I’ve found best is to pick a focus or two and then start there. For example this year we have a big focus on upgrading our edge security and our production datacenter to include Cisco UCS solutions. What sessions I pick will almost entirely be from either the Security and Datacenter & Virtualization tracks to support those goals. Keep in mind all of these sessions will be available to you online after the fact so keep in mind the people giving them as well.

cae

If you have never been to one of the major tech conferences (20k attendees and up) there is never really a shortage of things to do, ranging from the educational to the social to just straight fun. Cisco Live is in my opinion a great event with a better than most mix of content and social, the highlight of which is the Customer Appreciation Event. The CAE this year will be held at the T-Mobile Arena and features concerts with Maroon 5 and Elle King. I saw Maroon 5 in a very cold field  a couple of years ago and they are a pretty good show and I’ve really liked what I’ve heard from Elle King on the radio.

Besides the concert event there will be no shortage of things to do if you are socially inclined. Besides the mixers each evening there are a wide array of events from different vendors in the Cisco ecosystem each evening. Many of these are by invite only so now would be an appropriate time to be reaching out to Account Execs you have at the various vendors and see if they are doing anything there.

 

20130627_173819000_iOSGo forth and be social

This will be my 6 tech conference in 4 years and while the content of the sessions is great and extremely helpful like I mentioned above all of that content is available online, 24/7/365 after the conference. What is not is the ability to meet and have conversations with some of the best minds of our chosen field. My very first major conference was CLUS 2013 in Orlando, FL and as I got myself out of my shell and started to meet people I was frankly floored by the combined brain power in such a small area. The way I’ve often put this to people is that the entire state of West Virginia, where I am from, has a total of 3 CCIEs in it. While this is not a normal demographic, there are only 50,000 some worldwide. At one point that first year I found myself  sitting in a discussion where out of 20 people I was the only person NOT a CCIE and really it is amazing what you can absorb in the social settings at Cisco Live. If you are willing to put yourself out there and not be the cave-dwelling geek many of us are naturally drawn to be you will find a community of people who will readily accept you in.

So how do I find such social people and befriend them? Well fear not there are plenty of ways. To start with if you are just starting out in your tech career the very first advice is to get yourself on twitter if you haven’t already. I literally setup my twitter account walking down the main concourse of CLUS 4 years ago and it has presented no end of enjoyment, help and opportunity since. Once you have said account head on over to Tom Hollingsworth’s site and sign yourself up for the annual twitter list.

Now that you are in the social mood right off the bat one of the first places I will be locating is the Social Media Hub. This is pretty much the main congregation area for the socials types. At some point in the early evening Sunday there will be an opening Tweetup there, if you attend be sure to say hi!

If you are interested in going yourself but haven’t registered yet you can do so on the Cisco Live 2016 website.

3 steps to really reset a Cisco 7900 Phone

Recently had some issues with one of our phones at the office and you know how it goes, reboot it. What you may not know is that there are different levels of “reboot” for the 7900 series phones, each of which are a little more pervasive. In this post I’ll outline how to go about performing these 3 ways to reset your desk phone to cure what may or may not be ailing you.

I. The Simple Reset

Sure you could go into ccmadmin and hit the reset button but that doesn’t work as well if you are standing right in front of it.  A quick reset can be performed by doing the following directly from the device

  1. Hit the settings button on the device
  2. Hit **#** on the keypad
  3. You should then see the screen display the “Resetting…” message followed by a reboot

II. Configuration Erase

When you boot your 7900 series IP phone as part of the boot sequence it reaches out to your Publisher’s TFTP server to grab a copy of either its specific configuration file or if none exist the default configuration file. Once this occurs it is stored locally to allow for quicker subsequent reboots. From time to time this locally cached copy will get gummed up and it is necessary to erase it and have it download a fresh copy. To do this the steps are

  1. Hit the settings button on the device
  2. Hit the **# buttons in order, afterwards you will see “Settings Unlocked!” display on the screen and a “More” soft button appear on the screen
  3. Hit the “More” soft button followed by the “Erase” soft button.
  4. You should then see the screen display the “Resetting…” message followed by a reboot

III. Factory Reset

This is the big daddy, if neither of the previous fixes worked then this process will erase not only the configuration but any firmware updates you have pushed to it as well, resulting in a phone as fresh as when it left the factory from a software perspective. To perform this process do the following steps:

  1. Unplug the power cable and/or the switch cable if using PoE
  2. Plug the device back in, pressing and holding the “#” key before the Speaker button flashes on and off
  3. Continue to hold the # button until each line button flashes on and off in sequence (amber).
  4. Next release the # and in order hit 123456789*0#
  5. After the sequence is done correctly the line buttons will flash red and then the phone will reboot.
  6. The phone will go through multiple reboot processes as various firmware loads and configuration files are downloaded.
  7. Do not remove power in any way until the reset process is completed in its entirety. You will know that this is done when the phone either correctly registers to CUCM or display the “Registering…” message on the screen.

That’s it, if you’ve made it this far without fixing your issue then you either need to get back in CUCM and check you configurations of the device or contact TAC for a replacement device.

Configuring Networking for Nimble-vSphere iSCSI

One of my last tasks for 2014 was integrating a new Nimble Storage array into our environment. As this is the first of these I’ve encountered and I haven’t been able to take the free one day Nimble Installation and Operation Professional (NIOP) course they provide I was left to feeling my way through it with great help from their documentation and only ended up calling support to resolve a bug related to upgrading from 2.14 of the Nimble OS. On the network side our datacenter is powered by Cisco Nexus 3000 series switches, also a new addition for us recently. These allowed us to use our existing Cat6 copper infrastructure while increasing our bandwidth to 10 GbE. In this post I’m going to document some of the setup required to meet the best practices outlined in Nimble’s Networking Best Practices Guide when setting up your system with redundant NX-OS switches. Go Read More

Updating the Code of a ipbase Licensed Cisco Catalyst 3750X Switch Stack

Here in the office the Access Layer of our switching infrastructure is handled completely with a 7 unit stack of Cisco 3750X switches.  There is no need for these to do any routing other than intervlan so when purchased 3 years ago we just ordered the IP Base licensing level.  Well from what I can tell there is a universal code base and a licensed feature level of each code revision.  The universal naming convention looks like c3750e-universalk9-mz.122-55.SE1 while the ipbase looks like  c3750e-ipbasek9-mz.150-2.SE6.  What I found is that I do not have the ability to download the universal code of later releases due to my licensing level and possibly the lack of SmartNet I keep on these, but I do have access to the ipbase code.  When attempting to update the code on this stack I was presented with the error


After some searching I found reference to others trying to go from IP Base to Advanced IP Services code having to put the /allow-feature-upgrade switch on the archive download-sw code in order to allow the upgrade as well as it seems a downgrade.  Evidently this feature came about with IOS version 12.2(35).  Now the upgrade progressed and I have happy little upgrades switches.


Another note about this upgrade I found in the official release notes is any upgrade from 15.0(2)SE to later will result in a microcode upgrade which when unmitigated will lead to an exceptionally long restart of the switch.  You can mitigate this either by using the /force-ucode-reload parameter when downloading the code to the devices or by using the archive download-sw /upgrade-ucode privileged EXEC mode command afterwards.

Cisco Unified Contact Center Express Scripting Stuff

I recently participated in one of Cisco’s Collaboration User Group (CUG) briefings on scripting their Contact Center Express product.  While I took the official class, UCCXD, I haven’t really had to do much with it because we had a vendor taking care of stuff for us.  Well, that has now gone by the wayside so I’m trying to get myself back up to date on how to manage the application powering our call center.  I had forgotten how much of a PITA it was to find the scripting documentation on Cisco’s site, so after finding it I thought it prudent to Provide a listing and some links here so I can find it later (and for you if you need it.)

The scripting documentation for UCCX is organized into 3 separate documents.  The general link for all supported version is to here.  The links below are for the current 9.0 release.

One thing I learned in the CUG briefing is that included in each version’s documentation bundle, but evidently not linked anywhere that I can find on Cisco’s site, is a Sample Script Repository.  This is updated for each release.

Finally if you work with Cisco collaboration/ voice products much I highly recommend that you think about joining CUG as it has proven to be a very nice resource not only with troubleshooting problems with production release software but also to gain access to betas of what’s coming next.

Allowing Supervisors to Modify Skill Levels in UCCX 9

Since we installed Cisco’s Call Manager Express call center system a couple of years ago I could set my watch by the requests from our group of supervisors to modify the skill level of our various agents for the various Customer Service Queues (CSQs).  Generally at the same time they will request access to do this themselves.  Imagine my excitement when UCCX 9 was released and one of the features was a mobile browser application called, creatively, Mobile Skill Manager to do just that.  Further you can imagine my chagrin when after upgrading we quickly realized that the app doesn’t work particularly well at this point, either in a mobile browser or through any of the major standard browsers.

So to twitter I went trying to find a way to make this happen and lo and behold I found the answer within the System Parameters of UCCX.  Start by logging into the web interface and look in System> System Parameters.  Then under the Application Parameters section you will find an option called “Supervisor Access.”  By default this will be set to No Access to Teams, and if you want to provide access you will need to choose one of the other two options depending on  your need, Access to All Teams or Access to Supervisor’s Teams only.  For us we chose the former because we are a relatively small call center where all the Supervisors cross train.

uccx9-after-screenshotSo what does this do?  Changing this setting allows Supervisors access to a subset of the menu when they log in with their own credentials at the /appadmin web link, specifically it allwos them access to the RmCm Subsystem which controls the various settings related to CSQs, Resources (Agents), and Skills.  You may want to provide this access with a little guidance because with this they will be able to create and delete CSQs, Skills and Resources as well and most likely you won’t want them to do this.

While I am happy to have this option, I believe we can do it better.  In a perfect world this the base functionality would be built into the Supervisor Desktop application or the new Finesse web interface, with a capability to turn access on and off.  Further I’ve heard tale of an IP service application being developed by CTI Logic to allow desktop phone access to perform this task.  Both of those would be extremely nice to have as less interfaces for the user to know is always a good thing.