Quieting the LogPartitionLowWaterMarkExceeded Beast in Cisco IPT 9.0.x Products

As a SysAdmin I’m used to waking up, grabbing my phone and seeing the 20 or so e-mails that  the various systems and such have sent me over night, gives me an idea of how the day will go and what I need start with. Every so often though you get that morning where the 20 becomes 200 and you just want to roll over and go back to bed. This morning I had about 200, the vast majority of which was from my Cisco Unified Contact Center Express server with the subject “LogPartitionLowWaterMarkExceeded.” Luckily I’ve had this before and know what to do with it but on the chance you are getting it too here’s what it means and how to deal with it in an efficient manner.

WTF Is This?!?

Or at least that was my response the first time I ran into this. If you are a good little voice administrator one of the first things you do when installing your phone system or taking one over due to job change is setup the automatic alerting capability in the Cisco Unified Real Time Monitoring Tool (or RTMT, you did install that, right?) so that when things go awry you know in theory before the users do. One of the downsides to this system is it is an either on or off alerting system meaning what ever log events are saved within the system are automatically e-mailed at the same frequency.

This particular error message is the by-product of a bug (CSCul18667) in the 9.0.x releases of all the Cisco IP Telephony products in which the JMX logs produced by the at the time new Unified Intelligence Center didn’t get automatically deleted to maintain space on the log partition. While this has long since been fixed phone systems are one of those things that don’t get updated as regularly as they should and such it is still and issue. The resulting effect is that when you reach the “warning” level of partition usage (Low Water Mark) it starts logging ever 5 minutes that the level has been reached.

Just Make the Screaming Stop

Now that we know what the issue is how do we fix it?

Go back to the RTMT application, and connect to the affected component server. Once there you will need to navigate to the Trace & Log Central tool then double-click on the Remote Browse option. remote-browse
Once in the Remote Browse dialog box choose “Trace Files” and then we really only need one of the services selected, Cisco Unified Intelligence Center Serviceability Service and then Next, Next, Finish. select-cuic
Once it is done gathering all of the log files it will tell you your browse is ready. You then need to drill all the way down through the menu on each node until you reach “jmx.” Once you double-click on jmx you will see the bonanza of logs. It is best to just click one, Ctrl+A to select all and then just hit the Delete button. browse-to-node
After you hit delete it will probably take it quite a while to process through. You will then want to click on the node name and hit refresh to check but when done you should be left with just the currently active log file. Afterwards if you have multiple nodes of the application you will need to repeat this process for the other. all-clean

And that’s it really. Once done the e-mail bleeding will stop and you can go about the other 20 things you need to get done this day. If you are experiencing this and if possible I would recommend being smarter than me and just update your CIPT components to a version newer than 9.0 (11.5 is the current release), something I am hoping to begin the process of in the next month or so.

3 steps to really reset a Cisco 7900 Phone

Recently had some issues with one of our phones at the office and you know how it goes, reboot it. What you may not know is that there are different levels of “reboot” for the 7900 series phones, each of which are a little more pervasive. In this post I’ll outline how to go about performing these 3 ways to reset your desk phone to cure what may or may not be ailing you.

I. The Simple Reset

Sure you could go into ccmadmin and hit the reset button but that doesn’t work as well if you are standing right in front of it.  A quick reset can be performed by doing the following directly from the device

  1. Hit the settings button on the device
  2. Hit **#** on the keypad
  3. You should then see the screen display the “Resetting…” message followed by a reboot

II. Configuration Erase

When you boot your 7900 series IP phone as part of the boot sequence it reaches out to your Publisher’s TFTP server to grab a copy of either its specific configuration file or if none exist the default configuration file. Once this occurs it is stored locally to allow for quicker subsequent reboots. From time to time this locally cached copy will get gummed up and it is necessary to erase it and have it download a fresh copy. To do this the steps are

  1. Hit the settings button on the device
  2. Hit the **# buttons in order, afterwards you will see “Settings Unlocked!” display on the screen and a “More” soft button appear on the screen
  3. Hit the “More” soft button followed by the “Erase” soft button.
  4. You should then see the screen display the “Resetting…” message followed by a reboot

III. Factory Reset

This is the big daddy, if neither of the previous fixes worked then this process will erase not only the configuration but any firmware updates you have pushed to it as well, resulting in a phone as fresh as when it left the factory from a software perspective. To perform this process do the following steps:

  1. Unplug the power cable and/or the switch cable if using PoE
  2. Plug the device back in, pressing and holding the “#” key before the Speaker button flashes on and off
  3. Continue to hold the # button until each line button flashes on and off in sequence (amber).
  4. Next release the # and in order hit 123456789*0#
  5. After the sequence is done correctly the line buttons will flash red and then the phone will reboot.
  6. The phone will go through multiple reboot processes as various firmware loads and configuration files are downloaded.
  7. Do not remove power in any way until the reset process is completed in its entirety. You will know that this is done when the phone either correctly registers to CUCM or display the “Registering…” message on the screen.

That’s it, if you’ve made it this far without fixing your issue then you either need to get back in CUCM and check you configurations of the device or contact TAC for a replacement device.

Allowing Supervisors to Modify Skill Levels in UCCX 9

Since we installed Cisco’s Call Manager Express call center system a couple of years ago I could set my watch by the requests from our group of supervisors to modify the skill level of our various agents for the various Customer Service Queues (CSQs).  Generally at the same time they will request access to do this themselves.  Imagine my excitement when UCCX 9 was released and one of the features was a mobile browser application called, creatively, Mobile Skill Manager to do just that.  Further you can imagine my chagrin when after upgrading we quickly realized that the app doesn’t work particularly well at this point, either in a mobile browser or through any of the major standard browsers.

So to twitter I went trying to find a way to make this happen and lo and behold I found the answer within the System Parameters of UCCX.  Start by logging into the web interface and look in System> System Parameters.  Then under the Application Parameters section you will find an option called “Supervisor Access.”  By default this will be set to No Access to Teams, and if you want to provide access you will need to choose one of the other two options depending on  your need, Access to All Teams or Access to Supervisor’s Teams only.  For us we chose the former because we are a relatively small call center where all the Supervisors cross train.

uccx9-after-screenshotSo what does this do?  Changing this setting allows Supervisors access to a subset of the menu when they log in with their own credentials at the /appadmin web link, specifically it allwos them access to the RmCm Subsystem which controls the various settings related to CSQs, Resources (Agents), and Skills.  You may want to provide this access with a little guidance because with this they will be able to create and delete CSQs, Skills and Resources as well and most likely you won’t want them to do this.

While I am happy to have this option, I believe we can do it better.  In a perfect world this the base functionality would be built into the Supervisor Desktop application or the new Finesse web interface, with a capability to turn access on and off.  Further I’ve heard tale of an IP service application being developed by CTI Logic to allow desktop phone access to perform this task.  Both of those would be extremely nice to have as less interfaces for the user to know is always a good thing.