Fun with the vNIC Shuffle with Cisco UCS

Here at This Old Datacenter we’ve recently made the migration to using Cisco UCS for our production compute resources. UCS offers a great number of opportunity for system administrators, both in deployment as well as on going maintenance, making updating the physical as manageable as we virtualization admins are getting used to with the virtualized layer of the DC. Of course like any other deployment there is always going to be that one “oh yeah, that” moment. In my case after I had my servers up I realized I needed another virtual NIC, or vNIC in UCS world. This shouldn’t be a big deal because a big part of what UCS does for you is it abstracts the hardware configuration away from the actual hardware.

For those more familiar with standard server infrastructure, instead of having any number of physical NIC in the back of the host for specific uses (iSCSI, VM traffic, specialized networking, etc) you have a smaller number of connections as part of the Fabric Interconnect to the blade chassis that are logically split to provide networking to the individual blades. These Fabric Interconnects (FI) not only have multiple very high-speed connections (10 or 40 GbE) but each chassis typically will have multiple FI to provide redundancy throughout the design. All this being said, here’s a very basic design utilizing a UCS Mini setup with Nexus 3000 switches and a copper connected storage array:

ucs-design

So are you starting to thing this is a UCS geeksplainer? No, no my good person, this is actually the story of a fairly annoying hiccup when it comes to the relationship between UCS and VMware’s ESXi. You see while adding a vNIC should be as simple as create your vNICs in the Server Profile, reboot the effected blades and new NIC(s) are shown as available within ESXi, it of course is not that simple. What happens in reality when you add new NICs to an existing Physical NIC to vSwitch layout is that the relationships are shuffled. So for example if you started with a vNIC (shown as vmnicX in ESXi), vSwitch layout that looks like this to start with

1-before

After you add NICs and reboot it looks like this

2-after

Notice the vmnic to MAC address relationship in the 2. So while all the moving pieces are still there different physical devices map to different vSwitches than as designed. This really matters when you think about all the differences that usually exist in the VLAN design that underlay networking in an ESXi  setup. In this example vSwitch0 handles management traffic, HQProd-vDS handles all the VM traffic (so just trunked VLANS) and vSwitch1 handles iSCSI traffic. Especially when things like iSCSI that require specialized networking setup are involved does this become a nightmare; frankly I couldn’t imagine having to do this will a more complex design.

The Fix

So I’m sure you are sitting here like I was thinking “I’ll call support and they will have some magic that with either a)fix this, b) prevent it from happening in the future, or preferably c) both. Well, not so much. The answer from both VMware and Cisco support is to figure out which NICs should be assigned to which vSwitch by reviewing the MAC to vNIC assignment in UCS Manager as shown and then manually manage the vSwitch Uplink assignment for each host.

3-corrected

4-correctedesx

As you may be thinking, yes this is a pain in the you know what. I only had to do this with 4 hosts, I don’t want to think about what this looks like in a bigger environment. Further, as best I can get answers from either TAC or VMware support there is no way to make this go better in the future; this was not an issue with my UCS setup, this is just the way it is. I would love it if some of my “Automate All The Things!!!” crew could share a counterpoint to this on how to automate your way out of this but I haven’t found it yet. Do you have a better idea? Feel free to share it in the comments or tweet me @k00laidIT.

Getting the Ball Rolling with #vDM30in30

Ahh, that time of year when geeks pull that long forgotten blog site out of the closet, dust it of and make promises of love and content: #vDM30in30. If you aren’t familiar with the idea, vDM30in30 is short for Virtual Design Master 30 blog posts in 30 days, an idea championed by Eric Wright of discoposse fame to get bloggers out there to work their way through regular generation of content. As you can see from this site new content is pretty rare so something like this is a welcome excuse to focus and get some stuff out there. vDM30in30 runs through the month of November and the best way to follow along with the content is to track the hashtag on twitter.

So What’s the Plan?

I’m a planner by nature so if I don’t at least have a general idea this isn’t going to work at all. The good news is I’ve got quite a few posts that I’ve been meaning to work on for some time so I’m going to be cleaning out my closet this week and get those out there. So the full schedule is going to look like this:

  • Week of Nov 1: random posts I’ve never quite finished but need to be released
  • Week of Nov 7: focus on all the new hotness coming from Veeam Software
  • Week of Nov 14: VMware’s upcoming vSphere 6.5 release
  • Week of Nov 21: randomness about community, career and navel gazing in general

I’m really looking forward to participating this year as I do believe that a lot of growth comes from successfully forming out thoughts and putting them down. Hope you find some of this hopeful, if there is anything you’d like to see in the space feel free to comment.

Lots of new stuff coming from Veeam

Veeam had what they called “THEIR BIGGEST EVENT EVER” and while it at times did seem to be really heavy on the sales for the sake of sales pitch, there was a lot of stuff to legitimately be excited about for those of us who use their products. From the features coming in Veeam Backup & Replication in version 9.5 in a couple of months through the first new feature of next year’s version 10 all in total there were 5 major announcements here today that those of us using the product can make use of. In this post I’m going to run briefly through these and in the coming months will provide some deeper insights when possible.

Veeam Backup & Replication / Veeam ONE 9.5 (October 2016)

  • Nimble Storage Integration- Nimble with be the next vendor after EMC, NetApp and HP storage systems that will allow Veeam to interact at the array level, allowing for backups from snapshot. If you are a Nimble customer (like me) this is going to be some good stuff
  • Advanced usage of Windows Server 2016 ReFS- This is the real gravy here for anybody who is having to work with any kind of synthetic operations with their backup files. Through an integration Veeam has with Microsoft when ReFS is used to back your Veeam repositories your weekly rollups are going to take a heck of a lot less time and as well as see less storage consumption for long terms “weekly fulls”.  This is due to ReFS’ basic mechanism where file copies and moves never actually move data, it just moves the pointers. An example I’ve seen is on a 10 GB change rate backup the weekly full went from 35 minutes on NTFS to 5 minutes on ReFS. Now move that out to a real production dataset and you are really talking about something. There will be a lot more of this in follow-up posts.
  • Direct Restore to Microsoft Azure – If you are resource constrained (which you usually are in a situation where you need a restore) Veeam now has the ability to restore a VM (even if it is vSphere based) directly to Azure. Pretty cool and I think probably the first of what we’ll see on this thread
  • vCloud Director Integration
  • VeeamONE 9.5 – If your organization needs to work with charge back this is something that is directly supported in VeeamONE. If you haven’t played with VeeamONE yet, please do so, I’ve yet to meet anyone who hasn’t found one problem with VeeamONE when first installed in their virtualization environment

Veeam Agents (November-December 2016)
agent versions

Expanding on the Veeam Endpoint for Windows (and now Linux) Veeam has come out with a Veeam Agents for Windows and Linux product. While Endpoint is and will still be available for standalone installations, we finally have an enterprise managed version we’ve been looking for and we truly can have one centrally managed Veeam installation for our virtual, physical and workstation backups. As you can see there’s still a lot to like about the Free version including the new ability to restore directly to Azure or Hyper-V, the paid versions give us server grade capabilities such as Application-aware processing and transaction log processing. Further one I’m excited about as part of my use case for this is for my mobile workforce is the ability for workstations and remote office servers to cache their backups locally when they aren’t connected to the Internet and then ship them back to the corporate office or Cloud Connect repository when once again connected. This is good stuff that has been a long time coming.

Veeam Availability Console (Q1 2017)

I truly want to believe this is the first edge of “one UI to rule them all”, but the Veeam Availability Console is a web-based console to let you monitor and manage all of your Veeam resources; VBR, Agent, Cloud Connect, etc. This is an evolution of the managed backup portal available to Service Providers for a bit now and allows it to be moved downstream to the Enterprise. Let me  reinforce the emphasis on the Enterprise, while included in licensing you are going to have to be so big of an organization/installation to be allowed access to it. Hopefully as subsequent versions are released that will trickle down more.

Veeam Availability Orchestrator (Q1 2017, beta soon)

Veeam for a DevOpsy world. VAO will allow you to automate many of the processes you need to do with Veeam based upon your disaster recovery plan. Let’s say your plan requires you do so many backups, so many replicas, regular testing and comply with documentation practices. Orchestrator is going to allow you to take all that on paper and define it in workflows so in theory you are always in compliance, and if you aren’t have the documentation to show you where you aren’t. I’ve seen quite a few things about this, things that are going to be available to everybody to test soon, and they are all very powerful things.

Veeam Office 365 E-mail Backup (Q4 2016)

Of the new products announce this is the biggie. For those of us who have already began or have done Exchange migrations to Office 365, Veeam now has the ability to backup those mailboxes to your local repositories so that you always know that data is there. I don’t know how those conversations have gone for you but this is a major pain point for us in going to the cloud. Pricing or even how it is going to be sold still isn’t set but what is known is that when released the end of this year it will be free for a year for all Veeam customers with an active support contract and for 3 years for those with Enterprise Plus licensing.

Again, while I have no knowledge that it will happen I have to believe this is the first baby step into a whole host of things to make our cloudy life better in the future with Sharepoint, OneDrive and anything else coming down the road.

Veeam Backup & Replication integration with IBM storage (????, preview May 2017)

Finally the last announcement was the first related to Veeam Backup version 10, in this case the next storage vendor integration. This integration is going to work with any IBM product based on their Spectrum Virtualize software and should work like any other of their integrations. With this we also go to learn that the first technical preview of v10 will coincide with VeeamON 2017 in New Orleans, so mid May 2017.

VMware Tools Security Bug and Finding which VMware Tools components are installed on all VMs

Just a quick post related to today’s VMware security advisories. VMware released a pair of advisories today, CVE-2016-5330 and CVE-2016-5331 and while both are nasty their scopes are somewhat limited. The 5331 issue is only applicable if you are running vCenter or ESXi 6.0 or 6.0U1, Update 2 patches the bug. The 5330 is limited to Windows VMs, running VMware Tools, and have the option HGFS component installed. To find out if you are vulnerable here’s a Power-CLI script to get all your VMs and list the installed components. Props to Jason Shiplett for giving me some assistance on the code.

While the output is still a little rough it will get you there. Alternatively if you are just using this script for the advisory listed you can change  where-object { $_.Name -match $componentPattern }  to  where-object { $_.Name -match "vmhgfs" } . This script is also available on GitHub.

The Unofficial Official CiscoLive! US Gatherings Page

Here’s the list of all the outside of business hours events that I and others know of at CiscoLive 2016. If you know of others please DM or tweet me @k00laidIT and I’ll get them added.

 

Saturday 7/9/2016
Adventure to  National Atomic Testing Museum

  • 2 PM
  • 755 E Flamingo Rd, Las Vegas, NV 89119 (Map)
  • #clatomic 

Sunday 7/10/2016
#CLUS Sunday Tweetup

  • 5:30 PM
  • Social Media Central, Bayside Foyer, Mandalay Bay

Monday 7/11/2016
Veeam & Nimble Integration party at Cisco Live!

Tuesday 7/12/2016
SD-WAN Mixer with Packet Pushers’ Ethan Banks

Meraki After Party

Wednesday 7/13/2016
Customer Appreciation Event

The Basics of Network Troubleshooting

The following post is something I wrote as an in-house primer for our help desk staff. While it a bit down level from a lot of the content here I find more and more the picking and reliably going with a troubleshooting methodology is somewhat of a lost art. If you are just getting started in networking or are troubleshooting connectivity issues at your home or SMB this would be a great place to start.

We often get issues which are reported as application issues but end up being network related. There are a number steps and logical thought processes that can make dealing with even the most difficult network issues easy to troubleshoot. The purpose of this post is to outline many of the basic steps of troubleshooting network issues, past that it’s time to reach out and ask for assistance.

  1. Understand the basics of OSI model based troubleshooting

    The conceptual idea of how a network operates within a single node (computer, smartphone, printer, etc.) is defined by something called the OSI reference model. The OSI model breaks down the operations of a network into 7 layers, each of which is reliant on success at the layers below it (inbound traffic) and above it (outbound traffic). The layers (with some corresponding protocols you’ll recognize) are:

    7. Application: app needs to send/receive something (HTTP, HTTPS, FTP, anything that the user touches and begins/ends network transmission)
    6. Presentation: formatting & encryption (VPN and DNS host names)
    5. Session: interhost communication (nothing to see here:))
    4. Transport: end to end negotiations, reliability (the age old TCP vs. UDP debate)
    3. Network: path and logical addressing (IP addresses & routing)
    2. Data Link: physical addressing (MAC addresses & switches)
    1. Physical: physical connectivity (Is it plugged in?)

    The image below is a great cheat card for keeping these somewhat clear:

    OSI_2014
    Image source: http://www.gargasz.info/osi-model-how-internet-works/

    How OSI is used today is as a template for how to understand and thus troubleshoot networking issues. The best way to troubleshoot any IT problem that has the potential to have a network issue is from the bottom of the stack upwards. Here are a few basic steps to get you going with troubleshooting.

  2. Is it plugged in?

    This may seem like a smart ass answer, but many times this is just the case. Somebody’s unplugged the cable or the clip has broken off the Cat6 cable and every time somebody touches the desk it wiggles out. Most of the time you will have some form of a light to tell you that you have both connectivity to the network (usually green) and are transmitting on the network (usually orange).

    This troubleshooting represents layer 1 troubleshooting.

  3. Is the network interface enabled?

    So the cable is in and maybe you’ve tried to plug the same cable from the wall into multiple devices; you get link lights on other devices but no love on the device you need. This may represent a Data Link issue where the Network Interface Card (NIC) has been disabled in the OS. From the client standpoint this would be within Windows or Mac OSX or whatever, on the other side it’s possible the physical interface on the switch that represents the other end of the wire may be disabled. Check out the OS first and then reach out to your network guy to check the switch if need be.

  4. Can the user ping it?

    Moving up to the Network layer, the next step is to test if the user can ping the device which they are having an issue with. Have the user bring up a command prompt and ping the IP address of the far end device.

  5. Can you ping it?

    By the very nature of you being an awesomesauce IT person you are going to have more ability to test than the user. To start with, see if you can ping it from your workstation. This will rule out user error and potentially any number of other issues as well. Next if you can’t, are you on the same subnet/VLAN as the device you are trying to access? If not try to access a device in the same subnet as the endpoint device you are testing and ping it from there. That may give you some insight into having issues with default gateway configuration or underlying routing (aka Layer 3) issues.

  6. Can you ping it by name?

    Let’s say you can ping it by IP address from all of the above. If the user is trying to access something by name, say server1.foo.com have them ping that as well. It’s possible that while the lower three layers of the stack are operating well, something has gone awry with DNS or other forms of naming that happen at the Presentation layer.

  7. Application firewalls and the like

    Finally we’ve reached the top of the stack and we need to take a look at the individual applications. So far you’ve verified that the cable’s plugged in, the NICs on both sides are enabled and you can ping between the user and the far device by both IP and hostname but still the application won’t work so now’s when we look at the actual application and immediately start rebooting things.

    Just kidding 🙂 No now we need to look at services that are being present to the network. If we are troubleshooting an e-mail issue is the services running on the server and can we connect to it. When talking about TCP/IP-based traffic (meaning all traffic) all application layer traffic occurs over either a TCP or UDP protocol port. This isn’t something you physically plug-in, but rather it is a logical slot that an application is known to talk on, kind of like a CB radio channel. For example SMTP typically runs on TCP port 25, FTP 21, printing usually on 9100. If you are troubleshooting an e-mail issue bring up a command prompt and try to connect to the device via telnet like “telnet server1.foo.com 25.” If the SMTP server is running on that port at the far end then it will answer, if not the connection will time out.

  8. Call in reinforcements

    If you’ve got this far it’s going to take a combination of multiple brains and probably some application owners/vendors to unwrangle the mess those crazy users have made. Reach out to your network and application teams or call in vendor support at this point.

Network troubleshooting isn’t hard, you just have to know where to start.

Vegas Baby! Heading to CiscoLive! 2016

As 2016 moves into April we find ourselves ready to go into the conference season once again. For the past couple of years I’ve been to VMworld because that is where my work has had me focused, but for the same reason I will be heading the Cisco Live in Las Vegas, NV this year. The event will be held at the Mandalay Bay Resort July 10-14. Yes it will be hot, but let’s be honest you are going to be inside most of the time. This is the 2nd time I’ve attended Cisco Live US (you may see it referred to as #CLUS quite a bit) and if this is anything like the last time it’s going to be great. I have been particularly impressed with the content they make available and the community that has grown around it.

What to do

The first and foremost thing you should check out at Cisco Live is the always excellent sessions throughout the conference. If you are new to conferences this is actually something to consider sooner than later; the session catalog is currently up and the scheduler will open on May 3. I recommend that if you have any particular sessions or focus you are looking at with this trip go ahead and have a list done early and then be ready on the 5/3, many popular sessions will fill up quickly and nobody wants to wait in the overflow line. 😉

To be honest if you just look at the scope of topics covered in the session list it is a bit overwhelming. While I’m no grizzled veteran of conferences by any means what I’ve found best is to pick a focus or two and then start there. For example this year we have a big focus on upgrading our edge security and our production datacenter to include Cisco UCS solutions. What sessions I pick will almost entirely be from either the Security and Datacenter & Virtualization tracks to support those goals. Keep in mind all of these sessions will be available to you online after the fact so keep in mind the people giving them as well.

cae

If you have never been to one of the major tech conferences (20k attendees and up) there is never really a shortage of things to do, ranging from the educational to the social to just straight fun. Cisco Live is in my opinion a great event with a better than most mix of content and social, the highlight of which is the Customer Appreciation Event. The CAE this year will be held at the T-Mobile Arena and features concerts with Maroon 5 and Elle King. I saw Maroon 5 in a very cold field  a couple of years ago and they are a pretty good show and I’ve really liked what I’ve heard from Elle King on the radio.

Besides the concert event there will be no shortage of things to do if you are socially inclined. Besides the mixers each evening there are a wide array of events from different vendors in the Cisco ecosystem each evening. Many of these are by invite only so now would be an appropriate time to be reaching out to Account Execs you have at the various vendors and see if they are doing anything there.

 

20130627_173819000_iOSGo forth and be social

This will be my 6 tech conference in 4 years and while the content of the sessions is great and extremely helpful like I mentioned above all of that content is available online, 24/7/365 after the conference. What is not is the ability to meet and have conversations with some of the best minds of our chosen field. My very first major conference was CLUS 2013 in Orlando, FL and as I got myself out of my shell and started to meet people I was frankly floored by the combined brain power in such a small area. The way I’ve often put this to people is that the entire state of West Virginia, where I am from, has a total of 3 CCIEs in it. While this is not a normal demographic, there are only 50,000 some worldwide. At one point that first year I found myself  sitting in a discussion where out of 20 people I was the only person NOT a CCIE and really it is amazing what you can absorb in the social settings at Cisco Live. If you are willing to put yourself out there and not be the cave-dwelling geek many of us are naturally drawn to be you will find a community of people who will readily accept you in.

So how do I find such social people and befriend them? Well fear not there are plenty of ways. To start with if you are just starting out in your tech career the very first advice is to get yourself on twitter if you haven’t already. I literally setup my twitter account walking down the main concourse of CLUS 4 years ago and it has presented no end of enjoyment, help and opportunity since. Once you have said account head on over to Tom Hollingsworth’s site and sign yourself up for the annual twitter list.

Now that you are in the social mood right off the bat one of the first places I will be locating is the Social Media Hub. This is pretty much the main congregation area for the socials types. At some point in the early evening Sunday there will be an opening Tweetup there, if you attend be sure to say hi!

If you are interested in going yourself but haven’t registered yet you can do so on the Cisco Live 2016 website.

Quieting the LogPartitionLowWaterMarkExceeded Beast in Cisco IPT 9.0.x Products

As a SysAdmin I’m used to waking up, grabbing my phone and seeing the 20 or so e-mails that  the various systems and such have sent me over night, gives me an idea of how the day will go and what I need start with. Every so often though you get that morning where the 20 becomes 200 and you just want to roll over and go back to bed. This morning I had about 200, the vast majority of which was from my Cisco Unified Contact Center Express server with the subject “LogPartitionLowWaterMarkExceeded.” Luckily I’ve had this before and know what to do with it but on the chance you are getting it too here’s what it means and how to deal with it in an efficient manner.

WTF Is This?!?

Or at least that was my response the first time I ran into this. If you are a good little voice administrator one of the first things you do when installing your phone system or taking one over due to job change is setup the automatic alerting capability in the Cisco Unified Real Time Monitoring Tool (or RTMT, you did install that, right?) so that when things go awry you know in theory before the users do. One of the downsides to this system is it is an either on or off alerting system meaning what ever log events are saved within the system are automatically e-mailed at the same frequency.

This particular error message is the by-product of a bug (CSCul18667) in the 9.0.x releases of all the Cisco IP Telephony products in which the JMX logs produced by the at the time new Unified Intelligence Center didn’t get automatically deleted to maintain space on the log partition. While this has long since been fixed phone systems are one of those things that don’t get updated as regularly as they should and such it is still and issue. The resulting effect is that when you reach the “warning” level of partition usage (Low Water Mark) it starts logging ever 5 minutes that the level has been reached.

Just Make the Screaming Stop

Now that we know what the issue is how do we fix it?

Go back to the RTMT application, and connect to the affected component server. Once there you will need to navigate to the Trace & Log Central tool then double-click on the Remote Browse option. remote-browse
Once in the Remote Browse dialog box choose “Trace Files” and then we really only need one of the services selected, Cisco Unified Intelligence Center Serviceability Service and then Next, Next, Finish. select-cuic
Once it is done gathering all of the log files it will tell you your browse is ready. You then need to drill all the way down through the menu on each node until you reach “jmx.” Once you double-click on jmx you will see the bonanza of logs. It is best to just click one, Ctrl+A to select all and then just hit the Delete button. browse-to-node
After you hit delete it will probably take it quite a while to process through. You will then want to click on the node name and hit refresh to check but when done you should be left with just the currently active log file. Afterwards if you have multiple nodes of the application you will need to repeat this process for the other. all-clean

And that’s it really. Once done the e-mail bleeding will stop and you can go about the other 20 things you need to get done this day. If you are experiencing this and if possible I would recommend being smarter than me and just update your CIPT components to a version newer than 9.0 (11.5 is the current release), something I am hoping to begin the process of in the next month or so.

Updating the Photo Attributes in Active Directory with Powershell

Today I got to have the joys of needed to once again get caught up on importing employee photos into the Active Directory photo attributes, thumbnailPhoto and jpegPhoto. While this isn’t exactly the most necessary thing on Earth it does make working in a Windows environment “pretty” as these images are used by things such as Outlook, Lync and Cisco Jabber among other. In the past the only way I’ve only ever known how to do this is by using the AD Photo Edit Free utility, which while nice tends to be a bit buggy and it requires lots of repetitive action as you manually update each user for each attribute. This year I’ve given myself the goal of 1) finally learning Powershell/PowerCLI to at least the level of mild proficiency and 2) automating as many tasks like this as possible. While I’ve been dutifully working my way through a playlist of great PluralSight courses on the subject, I’ve had to live dangerously a few times to accomplish tasks like this along the way.

So long story short with some help along the way from Googling things I’ve managed to put together a script to do the following.

  1. Look in a directory passed to the script via the jpgdir parameter for any images with the file name format <username>.jpg
  2. Do an Active Directory search in an OU specified in the ou parameter for the username included in the image name. This parameter needs to be the full DN path (ex. LDAP://ou=staff,dc=foo,dc=com)
  3. If the user is found then it will make a resized copy of the image file into the “resized” subdirectory to keep the file sizes small
  4. Finally the resized image is then set as the both the thumbnailPhoto and jpegPhoto attribute for the user’s AD account

So your basic usage would be .\Set-ADPhotos.ps1 -jpgdir "C:\MyPhotos" -OU "LDAP://ou=staff,dc=foo,dc=com" . This should be easily setup as a scheduled task to fully automate the process. In our case I’ve got the person in charge of creating security badges feeding the folder with pictures when taken for the badges, then this runs at 5 in the morning each day automatically.

All that said, here’s the actual script code:

 

Did I mention that I had some help from the Googles? I was able to grab some great help (read Ctrl+C, Ctrl+V) in learning how to piece this together from a couple of sites:

The basic idea came from https://coffeefueled.org/powershell/importing-photos-into-ad-with-powershell/

The Powershell Image Resize function: http://www.lewisroberts.com/2015/01/18/powershell-image-resize-function/

Finally I’ve been trying to be all DevOpsy and start using GitHub so a link to the living code can be found here: https://github.com/k00laidIT/Learning-PS/blob/master/Set-ADPhotos.ps1

A how-to on cold calling from the customer perspective

Now that I’m back from my second tech conference in less than two months I am fully into the cold call season and I am once again reminded why I keep meaning to buy a burner phone and setup a Gmail account before I register next year. It seems every time I get back I am destined to months of “I am so glad you expressed deep interest in our product and I’d love to tell you more about it” when the reality is “I am calling you because you weren’t nimble enough to lunge away from our team of booth people who are paid or retained based on as many scans they can get. Most often when I get these calls or e-mails I’ll give each company a courteous thanks but no thanks and after that the iDivert button gets worn out.

The genesis of this post is two-fold. First a cold call this morning that was actually destined for my boss but when informed he wasn’t here went into telling how glad the person was that I had personally expressed interest in their product, WTF? This first event reminded me of a second, where a few months ago I was at a mixer preceding a vendor supplied training when I was approached by a bevy of 20 something Inside Sales Engineers and asked “what can I do to actually get you to listen?” From this I thought that just in case a young Padawan Sales Rep/Engineer happens to come across this, here are those ways to make your job more efficient and to stop alienating your potential customers.

Google Voice is the Devil

I guess the first step for anybody on the calling end of a cold call scenario is to get me to answer the phone. My biggest gripe in this regard and the quickest way to earn the hang up achievement is the currently practice of many of startups out there to use Google Voice as their business phone system. In case you don’t know with Google Voice they do local exchange drop offs when you call outside of your local calling area, meaning that when you call my desk I get a call with no name and a local area code, leaving me with the quandary of “is this a customer needing support or is this a cold call?” I get very few of the former but on the off-chance it is I will almost always answer leaving me hearing your happy voices.

I HAVE AN END CALL BUTTON AND I AM NOT AFRAID TO USE IT, GOOD DAY TO YOU SIR/MADAM!

You want to know how to do this better? First don’t just call me. You’ve got all my contact info so let’s start with being a little more passive and send me an e-mail introducing yourself and asking if I have time to talk to you. Many companies do this already because it brings with it a good deal of benefits; I’ve now captured your contact info, we’re not really wasting a lot of time on each other if there is zero interest, I don’t have to drop what I am dealing to get your pitch. If this idea just absolutely flies in the face of all that your company holds dear and you really must cold call me then don’t hide behind an anonymous number, call me from your corporate (or even better your personal DID) with your company’s name plastered on the Caller ID screen so at least I have the option to decide if it’s a call I need to deal with.

A Trade Show Badge Scan List Does Not Mean I am (or anybody else is) Buying

I once again had an awesome time at VMworld this year but got to have an experience that I’m sure many other attendees have had variants of. There I was, happily walking my way through the show floor through a throng of people, when out of my peripheral vision a booth person for a vendor not to be named literally stepped through one person and was simultaneously reaching to scan my badge while asking “Hi, do you mind if I scan you?” Yes, Mr./Ms. Inside Sales person, this is the type of quality customer interaction that happened that resulted on me being put on your list. It really doesn’t signify that I have a true interest in your product so please see item one above regarding how to approach the cold call better.

I understand there is an entire industry built around having people capture attendee information as sales leads but this just doesn’t seem like a very effective way to do it. My likelihood of talking to you more about your product is much higher if someone with working knowledge of your product, say an SE, talks to me about your product either in the booth or at a social event and then the communication starts there. Once everybody is back home and doing their thing that’s the call I’m going to take.

Know Your Product Better Than I Do

That leads me to the next item,  if by chance you’ve managed to cold call me, get me to pick up and finally manage to keep me on the line long enough to actually talk about your product, ACTUALLY KNOW YOUR PRODUCT. I can’t tell you how many times I’ve received calls after a show and the person on the other end of the line is so blatantly doing the fake it until you make it thing it isn’t funny. Keep in mind you are in the tech industry, cold calling people who most likely are fairly tech savvy and capable of logical thought, so that isn’t going to work so well for you. Frankly, my time is a very, very finite resource and even if I am interested in your product, which is why I took your call, if I’m correcting the caller that is an instant turn off.

I get that the people manning the phones aren’t going to be Senior Solutions Architects for your organization but try this on for size; if you’ve got me talking to you and you get asked something you don’t know, don’t be afraid to say you don’t know. This is your opportunity to bump me up the chain or to loop in a more technical person to the call to get the discussion back on the right track. I will respect that far more than if you try to throw out a BS answer. Meanwhile get as much education as you can on what you’re selling. I don’t care if you are a natural sales person, you aren’t going to be able to sell me my own pen in this market.

Employees != Resources

So you’ve got yourself all the way through the gauntlet and you’ve got me talking and you know your product, please don’t tell me how you can get some resources arranged to help me with designing my quote so the deal can more forward. I was actually in a face to face meeting once where the sales person did this, referring to the technical people within the organization as resources and I think my internal response to this can best be summed up in GIF form:

obama_kicks_door

This absolutely drives me bonkers. A resource is an inanimate object which can be used repeatedly without consequence except in the inevitable end result where the resource breaks. What you are calling a resource is a living, breathing, most likely highly intelligent human being who has all kinds of responsibilities, not just to you but to his family, community and any other number things. By referring to them as this, and therefore showing that you think of them as something that can be used repeatedly without consequence, you are demeaning that person and the skill set he or she has, and trust me that person is most likely who we as technical professionals are going to connect with far more than we are with you.

So that’s it, Jim’s guide to getting me on the phone. I’m sure as soon as I post this many other techniques will come to my mind and I’ll have to update this. If you take this to heart, great, I think that is going to work out for you. If not, well, I still hope I’ll remember to buy that burner phone next May and the Gmail account is already setup. 😉