Quieting the LogPartitionLowWaterMarkExceeded Beast in Cisco IPT 9.0.x Products

As a SysAdmin I’m used to waking up, grabbing my phone and seeing the 20 or so e-mails that  the various systems and such have sent me over night, gives me an idea of how the day will go and what I need start with. Every so often though you get that morning where the 20 becomes 200 and you just want to roll over and go back to bed. This morning I had about 200, the vast majority of which was from my Cisco Unified Contact Center Express server with the subject “LogPartitionLowWaterMarkExceeded.” Luckily I’ve had this before and know what to do with it but on the chance you are getting it too here’s what it means and how to deal with it in an efficient manner.

WTF Is This?!?

Or at least that was my response the first time I ran into this. If you are a good little voice administrator one of the first things you do when installing your phone system or taking one over due to job change is setup the automatic alerting capability in the Cisco Unified Real Time Monitoring Tool (or RTMT, you did install that, right?) so that when things go awry you know in theory before the users do. One of the downsides to this system is it is an either on or off alerting system meaning what ever log events are saved within the system are automatically e-mailed at the same frequency.

This particular error message is the by-product of a bug (CSCul18667) in the 9.0.x releases of all the Cisco IP Telephony products in which the JMX logs produced by the at the time new Unified Intelligence Center didn’t get automatically deleted to maintain space on the log partition. While this has long since been fixed phone systems are one of those things that don’t get updated as regularly as they should and such it is still and issue. The resulting effect is that when you reach the “warning” level of partition usage (Low Water Mark) it starts logging ever 5 minutes that the level has been reached.

Just Make the Screaming Stop

Now that we know what the issue is how do we fix it?

Go back to the RTMT application, and connect to the affected component server. Once there you will need to navigate to the Trace & Log Central tool then double-click on the Remote Browse option. remote-browse
Once in the Remote Browse dialog box choose “Trace Files” and then we really only need one of the services selected, Cisco Unified Intelligence Center Serviceability Service and then Next, Next, Finish. select-cuic
Once it is done gathering all of the log files it will tell you your browse is ready. You then need to drill all the way down through the menu on each node until you reach “jmx.” Once you double-click on jmx you will see the bonanza of logs. It is best to just click one, Ctrl+A to select all and then just hit the Delete button. browse-to-node
After you hit delete it will probably take it quite a while to process through. You will then want to click on the node name and hit refresh to check but when done you should be left with just the currently active log file. Afterwards if you have multiple nodes of the application you will need to repeat this process for the other. all-clean

And that’s it really. Once done the e-mail bleeding will stop and you can go about the other 20 things you need to get done this day. If you are experiencing this and if possible I would recommend being smarter than me and just update your CIPT components to a version newer than 9.0 (11.5 is the current release), something I am hoping to begin the process of in the next month or so.

Unsupported Configuration when using VUM for a Major Upgrade

I’ve recently been working on getting my environment upgraded from vSphere 5.1 to 5.5. Last week I replaced one vCenter server with a clean install and upgraded another, in process implementing home brewed certificates thanks in no small part to Derek Seaman’s excellent SSL toolkit and tutorials. With that done and nice and clean this week I turned towards getting the ESX hosts updated. Like all right thinking folks, I typically like to use vSphere Update Manager for this task in a vCenter supported environment.

The first host went very well and was up and patched without issue. After that the wheels fell off for the other two. I was continuously getting “Unsupported configuration” when I would try to scan the host, if I tried to push through and Remediate it would fail with “Software or system configuration of host <hostnamehere> is incompatible. Check scan results for details.” Nice error messages right? I tried a few things, reinstalling the management agents via VMware KB 1031919, rebooting the host, etc. After no luck there I logged a case with VMware where we began trying to find more information in the vua.log and verifying the correct fdm agent is installed via the esxcli software vib list | grep fdm command. In the end we were able to find my issue but I’ll be honest the documentation and logging in this scenario is pretty bad.

When Veeam Backup & Replication creates a vPowerNFS share, mounting your backup datastore as an addressable datastore to your host that is added in at least one way as a series of lines in the /etc/vmware/esx.conf file as shown below:

 

tasks-and-events-outputThis is great except as I’ve moved from Veeam server to Veeam server with different names and I dismounted and removed the different datastores from the hosts the old lines of config weren’t removed from esx.conf.  Further after finally seeing the “Error in ESX configuration file (esx.conf)” we got lead down the rabbit hole of the preprocessing of a VUM upgrade. Evidently one of the first steps (at the 12% mark of the remediate task in my case) is to run a variant of the esxcfg-info CLI command which in my case was producing this:

scan-afterwhere backupserver.domain.local was the name of an old Veeam server we had used. When the unfiltered esxcfg-info command it would begin listing but would eventually bomb with the same error.

After seeing the command output I opened up the esx.conf file with vi, found the offending lines of configuration and removed them. After saving the file I was able to scan the host again and the scan reported the host as being non-compliant instead of incompatible, just what we were looking for.  A remediation then was successful and I was back in business. One item of note if you find yourself wanting to try this yourself is make sure you take a backup of the esx.conf file as a miss step here could result in production datastore being unavailable. For those not too familar with Linux style commands you can do this easily with

original-errorConclusion

In the end what I do know is that the act of adding a NFS datastore to an ESX host and then later removing it both from ESXi configuration as well as the underlying DNS zone is what caused the blocking of my upgrade. Now what I don’t know if this is due to it being programmatically added by Veeam and then manually removed at a later date or if this is a situation that is common to the use of NFS datastores in general.  More importantly, it would be great if VMware would work on how it is reporting such configuration issues. Even taking me out of the equation, if it takes your own Support Engineer 1.5 hours to track it down it isn’t documented enough.

Fully Install VMware Tools Via Yum in CentOS

I’ll be the first to admit that I know far less about Linux than is necessary to be good at it and more than necessary to be dangerous at it.  That said, if nothing else, I do try to learn more about it.  I find that in general I’ve basically committed to CentOS as my flavor of choice with it being the underpinnings of every non-appliance installation I’ve got.  Alot of this has to do with the fact that my first experiences were with RedHat and the subsequent RHEL, so with CentOS being the server side, open source derivative of RHEL it makes sense that that’s where I’d go.  In the vSphere world as you get further down the rabbit hold of monitor systems for your infrastructure you’ll find that for most things to even begin to operate effectively you’ve got to have VMware tools installed.  While there are various instruction sets out there floating around for how to get these on both, through the “Install VMware Tools” GUI and via yum (the RHEL package installation system) I’ve found that your mileage may vary greatly.

Below is a list of commands that I’ve finally got happy with to get these installed and allow for complete control over the VM much like you do with your Windows VMs via the VI client.  With the exception of a couple of modifications regarding your revisions of vSphere and CentOS you can pretty much copy and paste this into your elevated prompt (root) on your linux box and get all the information and monitoring you need.

1. Add the VMware GPG keys
rpm --import http://packages.vmware.com/tools/keys/VMWARE-PACKAGING-GPG-DSA-KEY.pub
rpm --import http://packages.vmware.com/tools/keys/VMWARE-PACKAGING-GPG-RSA-KEY.pub

2. Copy the following to create a yum repository with all of the relevant information. You will need to change the ESXi version (red) and CentOS base (blue) to match what you run:
echo "[vmware-tools]" >> /etc/yum.repos.d/vmware-tools.repo
echo "name=VMware Tools" >> /etc/yum.repos.d/vmware-tools.repo
echo "baseurl=http://packages.vmware.com/tools/esx/5.0/rhel6/$basearch" >> /etc/yum.repos.d/vmware-tools.repo
echo "enabled=1" >> /etc/yum.repos.d/vmware-tools.repo
echo "gpgcheck=1" >> /etc/yum.repos.d/vmware-tools.repo

3. Install all portions of the VMware Tools:
yum -y install vmware-tools*

And that’s pretty much it.  Once done you’ll probably immediately notice that it shows as you are running a 3rd Party version of the tools, but now you’ll see the IP address of the box in the VM summary screen.  Further you’ll now be able to monitor heartbeat and view performance data for your VMs, which is very nice to have.  In my environment I immediately began getting issue notifications via Veeam ONE letting me know about issues I didn’t even know I had.

A lot of the other guides on how to do this have you use the command yum install vmware-tools-core, but I find that to be pretty incomplete as there are various plugins that allow for greater management and utilities such as auto update abilities.  You can see a whole list of what’s possible and cherry pick if you like by running the command yum search vmware-tools* once you’ve added your repository (step 2).