Hi all, I’m writing this to document a fix to an interesting challenge that has pretty much been my life for the last 24 hours or so. Through a comedy of errors and other things happening, we had a situation where the upstream CA from our VMware Certificate Authority (and other things) became very unavailable and the certificate authorizing it to manage certificates expired. Over the course of the last couple of days I’ve had to reissue certificates for just about everything including my Nimble Storage array and as far as vSphere goes we’ve had to revert all the certificate infrastructure to essentially the same as the out of the box self-signed guys and then reconfigure the VMCA as a subordinate again under the Root CA.

Even after all that I continued to have an issue where my Production VVOLs storage was inaccessible to the hosts. That’s not to say they weren’t working, amazingly and as a testament to the design of how VVOLs works my VMs on it ran throughout the process, but I was very limited in terms of the management of those VMs. Snapshots didn’t work, backups didn’t work, for a time even host migrations didn’t work until we reverted to the self-signed certs.

Thanks for a great deal of support and help from both VMware support and Nimble Storage Support we were finally able to come up with a runbook in dealing with a VVOL situation where major certificate changes occurred on the vSphere side. There is an assumption to this process that by the time you’ve got here all of your certificates both throughout vSphere as well as with the Nimble arrays are good and valid.

1. Unregister the VASA provider and Web Client integration from the Nimble array. This can be done either through the GUI in Administration>VMware Integration by editing your vCenter, unchecking the boxes for the Web Client and VASA Provider and hitting save. This can also be done via the CLI using the command

   vcenter --unregister <vcenter_name> --extension vasa
   vcenter --unregister <vcenter_name> --extension web

2. Register the integrations back in. Again, from the GUI simply just check the boxes back and hit save. If successful you should see a couple of little green bars briefly appear at the top of the screen saying the process was successful. From the CLI the commands are pretty similar

   vcenter --register <vcenter_name> --extension vasa
   vcenter --register <vcenter_name> --extension web

3. Verify that your VASA provider is available in vCenter and online. This is just to make sure that the integration was successful. In either the Web Client or the HTML5 client go to vCenter> Configure> Storage Provider and look for the entry that matches the name of your array group and in the URL has the IP address of your array’s management interface. This should show as online. As you have been messing with certificates its probably worth looking at the Certificate Info tab as well while you are here to verify that the certificate is what you expect.
4. Refresh the CA Certificates on each your hosts. Next, we need to ensure that all of the CA certificates are available on the hosts to ensure they can verify the certificates presented to them by the storage array. To do this you can either right-click each host > Certificates > Refresh CA Certificates or if you navigate to the configuration tab of each host, go to Certificate there is a button there as well. While in the window it is worth looking at the Status of each host’s certificate and ensure that it is Good.
   
5. Restart the vvold service on each host. This final step was evidently the hardest one to nail down and find in the documentation. The simplest method may be to simply reboot each of your hosts as long as you can put them into maintenance mode and evacuate them first. The quicker way and the way that will let you keep things running is to enter a shell session on each of your hosts and simply run the following command:

   /etc/init.d/vvold restart

   Once done you should see a response like the feature image on this post and a short while later your VVOLs array will again become available for each host as you work on them.

That’s about it. I really cannot thank the engineers at VMware (Sujish) and Nimble (Peter) enough for their assistance in getting me back to good. Also I’d like to thank Pete Flecha for jumping in at the end, helping me and reminding me to blog this.

If nothing else I hope this serves as a reminder to you (as well as myself) that certificates should be well tended to, please watch them carefully. 😉

I don’t know about the rest of you but printing has long been the bane of my existence as an IT professional. Frankly, I hate it and believe the world should be 100% paperless by this point. That said, throughout my career, my users have done a wonderful job of showing me that I am truly in the minority on this matter so I have to do my part in making sure they are available.

As any Windows SysAdmin knows installing the actual print driver and setting up a TCP/IP port aren’t even half the battle. From there you got to get them shared and have the users actually connect to them so that they can use them. It’d be awesome if they would all just sit down say “I have no printers, let me go to Active Directory and find some” but I’ve yet to have more than a handful of users who see this as a solution; they just want the damned things there and ready to rock and roll.

In the past, I’ve always managed this with a series of old VBS scripts, which still works but requires tweaks from time to time. It’s possible to do this kind of stuff with Powershell these days as well as long as your user has the Active Directory module imported (Hint: they probably don’t). There are also any number of other 3rd party and really expensive Microsoft systems (Hi SCCM!) that will do this as well. But luckily we’ve had a little thing called Group Policy Preferences around for a while now too and it will do everything we need to make this really manageable, with a nice pretty GUI that you can even teach the Help Desk Intern how to manage.

1. Setup the Print Server(s)- This is the same old, same old. Pick a server or set of servers and setup all your printers and share them. This gives you centralized queue management and all the goodies we know and love.
2. Create Security Groups- Unless you work in a 10 person office most people won’t necessarily need every printer. I like to create Security groups, 1 per printer, and then assign everybody who needs that printer to the security group. I typically also like to set up these groups with a prefix, usually “prnt” so that they are all grouped together but that’s just me. Set these up now and we’ll use them in a minute.
3. Create a new GPO- Truthfully this a personal preference, but I typically like to create a separate GPO for each major task I want to achieve aside from baseline things I through in a domain default policy.
4. Navigate to Users>Preferences>Control Panel Settings>Printers- Cool, it’s a blank screen! Let’s fill this sucker up with some printing goodness. Start by right-clicking the screen and choosing New>Shared Printer.
5. Once here you will the default action is Update. While there is an option for Create we want to leave the setting at the default because this will allow you more flexibility in the future while still letting you accomplish your goal now.
6. Go ahead and fill in the share path with the full UNC path to the shared printer leaving everything else blank then click on the “Common” tab.
7. This is where the magic happens so everybody only gets what they need. Check the box for “Item-level targeting” at the bottom and then click the now available button
8. In the now open Targeting Editor window click the “New Item” button and choose “Security Group.” Note: I like to do this task with Security Groups but as you can see there are lots of options to choose from. You may want to do the assignment based on Active Directory Sites if you have a rotating band of workers for example. Do what fits your organization.
9. Hit the browse “…” button and go find your group you want to have this printer added for then hit OK all the way back out to the GPO screen.

That’s it! you can essentially rinse and repeat these instructions for as many printers and print servers as you need to support. There really isn’t even any server magic to the printing, for all GP Preferences cares these can all be printers shared off individual workstations. I wouldn’t do that, but you know… My one real gripe with this is there doesn’t seem to be a way to script your way out of the process yet. I was able to bulk install the printers and create the ports on the print server but doing this work out of the GUI essentially means exporting the preferences list to an XML file, editing it and then importing it back in. Eww.

P.S. ProTip: Use Delete All For Print Server Migrations

So the idea spark for this post was a need to recreate all the logical printers in response to an office reorganization. The old names made no sense so we just blew them away and created new. One thing I did find out is that since Windows Server 2012 you can create a Printer Preference with type Delete and choose “Delete all shared connections.” Coupled with the Common options of “Apply once and do not reapply” this can be a very effective way to manage a print server migration, reorganization, or any other number of goals I can think of. If you do choose to do this be sure to 1) make sure any version of this you were using to do the “old printers” is gone before you set this to run and 2) you mess with the order of the Printer Preferences so it is number 1 in the order. In addition, when I was looking to use it I created it and then immediately right-click > Disabled the preference until I was really ready for it to go.

Just a quick post with a script I’ve just written. Living in WV we from time to time have to let staff know that the offices will be closed for various reasons, from heavy snow to chemical companies dumping large quantities of chemicals into the area’s water supply. For this reason, we maintain a basic emergency staff notification process that requires an authorized person to send an e-mail to a certain address and that will carpet bomb staff who chose to opt-in to receive text messages and e-mails to their personal (as opposed to business) e-mail addresses. This is all powered by using creating hidden mail contacts on our Exchange server for the personal address as well as the e-mail address that corresponds to the users’ mobile provider. These addresses are all then dynamically added to a distribution list that is restricted by who can send to it.

To be honest the system is mostly automatic with the exception of needing to makes sure new contacts get put in and old contacts get taken out. Taking them out by GUI is pretty simple, just right click delete but it seems to be lots of steps to add them in. So in the script below I’ve automated the process of interrogating the Admin entering them and then using that information to automatically create the contacts and then hide them from the Global Address List.

#New-DRContact.ps1
#Allows for the creation of Exchange 2010-2016 Mail Contacts for both a mobile phone and a personal e-mail address
#If you wish these addresses to be listed in the GAL comment the Set-MailContact lines

$OU = Read-Host -Prompt "OU to put the contact in (format domain/staff/contacts)"
$FirstName = Read-Host -Prompt "What is the user's first name"
$LastName = Read-Host -Prompt "What is the user's last name"
$Mobile = Read-Host -Prompt "What is the mobile number e-mail address (Common Domains- txt.att.net, vtext.com, pcs.ntelos.net, sprintpcs.com)"
$Personal = Read-Host -Prompt "What is the user's personal e-mail address"

if ($Mobile) {
    $MobileCName = $FirstName + " " + $LastName + " Mobile"
    $MCAlias =$FirstName.ToLower().Substring(0,1) + $LastName.ToLower() + "mobile"
    New-MailContact -Name $MobileCName -Alias $MCAlias -ExternalEmailAddress $Mobile -OrganizationalUnit $OU
    Set-MailContact $MCAlias -HiddenFromAddressListsEnabled $true    
}

if ($Personal) {
    $PersonalCName = $FirstName + " " + $LastName + " Personal"
    $PCAlias =$FirstName.ToLower().Substring(0,1) + $LastName.ToLower() + "personal"
    New-MailContact -Name $PersonalCName -Alias $PCAlias -ExternalEmailAddress $Personal -OrganizationalUnit $OU
    Set-MailContact $PCAlias -HiddenFromAddressListsEnabled $true

Now in order to make this work you need to either have an Exchange Shell window open or be remotely connected. As I am getting to where I have a nice, neat PowerShell profile on my laptop I like to stay in it so I remote in. I’ve automated that process in this Open-Exchange.ps1 script.

#Open-Exchange
#Prompts for servername and credentials and then gives you a remote Exchange Shell connection

$Server = Read-Host -Prompt "What's the server name"
$UserCredential = Get-Credential
$Session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri http://$Server/PowerShell/ -Authentication Kerberos -Credential $UserCredential
Import-PSSession $Session

Now if you’d like to save yourself the need to cut and paste these locally you can find these scripts and few others I’ve been writing on my GitHub repo.

I am lucky enough to be a father to a wonderful 5-year-old daughter, fresh into her Kindergarten year of school. Recently she came home with the dramatic cry of a 5-year-old, upset that her class has a Leader of the Month award and she didn’t win it. Once the sobbing subsided she got around to asking me how to be a leader, one of those basics of life type questions that all parents know and yet always get thrown by. How do I boil down the essence of leadership to something she not only can understand but can apply herself?

Thanks to the reoccurring themes of Special Agent Oso I got the idea to try to condense leadership to 3 simple steps. Simplistic I know, but the more I thought about it the more I realized that not only would it get her on the right track but that, to be honest, there are a great number of adults in leadership positions that experience differing levels of success with them. So thanks to my daughter and our good friend Oso I present Jim’s 3 simple steps to being a good leader.

Step 1! Have A Good Attitude

Seriously, there are so many studies/articles on the effect that a leader’s public attitude has on the productivity and efficiency of their team. If those linked articles aren’t enough for you Google it, there are a lot more. I know we all have our days when it all falls apart and have experienced these myself, but when those days stretch into weeks or months that team you are leading or even being a part of is going to start to fall apart as well. Some days it is going to require you to put a good face on what you are internally feeling, but coming in with a bounce in your step and not being negative is a great first step.

One of the realizations that I’ve had recently is that with any position there comes a time when you have to decide to either have some internal reconciliation with the job you have and make peace with the issues you have with it or decide to make the jump to something new. I’m not saying you should try to make the things you dislike better, you absolutely should, but staying in something you actively dislike and have lost the willpower to try to fix not only hurts you it hurts those around you.

Step 2! Listen

So for a 5-year-old trying to get them to listen to anything that isn’t on Netflix or YouTube is a challenge. Guess what? The same is true for me and you and those around you in the workplace from time to time. As framed to her, listening means making sure you pay attention not only to your teachers when they talk (and you absolutely must do this) but also to those around you that may be having issues. The teacher is talking about how to make the perfect “R” but your neighbor is having a problem? If she’s got it down this is a great opportunity to help!

As this applies to us how many of us go into meetings or conversations with our co-workers with a preset agenda or outcome of it already in our mind? Worse still is when we try to find the implied meaning of what is unsaid in a conversation. We as workers and leaders do better when we open our minds and actually listen to the words that are being spoken, then ask for clarification as needed. I know I have to work hard to quit relying on my own biases or trying to guess what the speaker means simply because we don’t want to ask “Ok, I didn’t get that, can you expand on this?”

Step 3! Apply What You’ve Learned

As we complete our journey through leadership we need have to apply what we’ve learned in the first two steps. We’ve spent the last 20 minutes learning how to make that perfect “R” and now we have to practice in class. My 2.0 may have this down, but she see’s her neighbor is trying to get help from the teacher. With a great attitude and an understanding of what’s expected this is her time to shine and help out her classmate!

At work this is important too. We have a great attitude now and we really want to be a helper, getting things done, and we’ve listened to those around us and learned not only about the needs and processes of our organizations but also the needs of our co-workers and staff. With all this ammo we are now in a position to truly make a difference because regardless of who you are helping when the knowledge gets distributed the potential for greater outcomes goes up. For those of us in traditional IT or other “support departments” this is where the true value lies. If we can’t learn, interpret and apply technology to the organization and staff’s needs then we aren’t supporting anyone.

Conclusion

So there you have it. Yes I know it’s simple, yes I know there are other factors (ego, time management, etc) that go into it as well but as in all things, you have to master the basics first. How many of you that have made it to this point were shaking your head and considering interactions with others as you read them? How many read these and had a “doh” thought about something you yourself have done? I know I did in thinking about it and there’s nothing wrong with that as long as you learn from it.

I had/have the honor today of presenting a couple vBrownBag sessions while here at VMworld. The first of these was about my journey from living in an area with little to know Tech Community resources available to becoming a part of the bigger global community and why that’s a good thing. As I feel this has really changed my career and enabled me to grow my skills as an IT professional in ways I never thought possible this is a subject I’m pretty passionate about.

So does that paragraph sound familiar to you? If so please consider watching the presentation below (it’s only 10 minutes) and start your own journey. If you need help along the way reach out to me on twitter @k00laidIT.

I write this while traveling to sunny and amazingly hot Las Vegas for the 2017 edition of VMworld US. I hope to provide feedback and news throughout the conference, highlighting not only the excellent content and programs but also the best the virtualization community has to offer.

Today will be a travel day as well as a day to meet up with friends, new and old. Tomorrow, the Sunday before the conference, is when the real fun begins with things like Opening Acts for me, TAM and partner content for others as well as a number of social events.

What We Know So Far

Yesterday was the day that Vmware went on a killing spree, announcing the depreciation of Windows based vCenter, the flash based vSphere web client and the vmkLinux APIs and its associated driver ecosystem. All of these enter the depreciated state with the next major version of vSphere and then will be gone for ever and ever in the revision after that. Each of these are significant steps towards the evolution of vSphere as we know it, and when coupled with the advances in PowerCLI in version 6.5 the management of our in house infrastructure has been changed for the better.

These announcements came rapid fire on the Friday before Vmworld with the death of the Windows based vCenter coming first. As we have had versions of varying success of the vCenter Server Appliances (VCSA) for over 5 years now it’s been a long time coming. I myself migrated two years ago and while it was good then with the latest 6.5 version, with its PhotonOS base, excellent migration wizard and in appliance vCenter Update Manager support it has show it is definitely the way forward.

The flash client was the next announcement to come and again, we are looking at an depreciation that needs to happen and is most definitely going to be a good thing but does come with some apprehension. With most things that have been depreciated by Vmware we’ve had at least 1 feature rich version of the replacement out and stable before they announced the predecessor’s demise. This isn’t the case with the flash based web client. While the latest builds are getting very, very good there are still major things that either are quirky or simply aren’t there yet. The good news to this is we have been given almost immediately assurances by everyone involved with the product management that we the vSphere admins will never be left without a GUI management ability for any given task we have today and I for one believe them. The last components of what is known as the HTML5 client in my opinion simply can’t come enough, I’m tired of having to hop through multiple GUIs and browsers to be able to perform basic tasks in my daily work life.

Finally the day was finished with the announced depreciation of the non-native Linux drivers. To be honest I didn’t know that these were even still a thing as every Linux VM I’ve rolled for the past many years have been able to work with the native drivers. I’m sure there are those that at this point may still need additional time but the as the removal is still a couple of versions off this should be something can be mitigated now that the end is known.

Conclusion

With all of these preconference announcements related to Vmware’s flagship product is this going to be the year where Vmworld is chocked full of improvements to vSphere. This will be my 3rd one in 4 years and each year I’ve felt their focus was elsewhere. While vSAN, NSX, and the like are definitely where the company’s seeing growth all of these things rely on vSphere as an underlay. I for one would be happy to see a little love shown here.

With that happy thought I’m going to shut it down and land. For those coming to Vmworld this weekend safe travels and for those at home look for more info as its known here on koolaid.info.

One of the biggest headaches I not only have and have heard about from other Veeam Backup & Replication administrators have is backup server migrations. In the past I have always gone the “All-in-One” approach, have one beefy physical server with Veeam directly installed and housing all the roles. This is great! It runs fast and it’s a fairly simple system to manage, but the problem is every time you need more space or your upgrading an old server you have to migrate all the parts and all the data. With my latest backup repository upgrade I’ve decided to go to a bit more of a distributed architecture, moving the command and control part out to a VM with an integrated SQL server and then letting the physical box handle the repository and proxy functions producing a best of both worlds setup, the speed and simplicity of all the data mover and VM access happening from the single physical server while the setup and brains of the operation reside in a movable, upgradable VM.

This post is mostly composed of my notes from the migration of all parts of VBR. The best way to think of this is to split the migration into 3 major parts; repository migration, VBR migration, proxy migration, and VBR migration. These notes are fairly high level, not going too deep into the individual steps. As migrations are complex if any of these parts don’t make sense to you or do not provide enough detail I would recommend that you give the fine folks at Veeam support a call to ride along as you perform your migration.

I. Migrating the Repository

1. Setup 1 or more new repository servers
2. Add new repository pointing to a separate folder (i.e. D:\ConfigBackups) on the new repository server to your existing VBR server exclusively for Configuration Backups. These cannot be included in a SOBR. Change the Config Backup Settings (File > Config Backup) to point to the new repository. This is also probably a good time to go ahead and run a manual Config Backup while you are there to snapshot your existing setup.
3. Create one or more new backup repositories on your new repository server(s) to your existing VBR server configuration.
4. Create Scale Out Backup Repository (SOBR), adding your existing repository and new repository or repositories as extents.
5. All of your backup jobs should automatically be changed to point to the SOBR during the setup but check each of your jobs to ensure they are pointing at the SOBR.
6. If possible go ahead and do a regular run of all jobs or wait until your regularly scheduled run.
7. After successful run of jobs put the existing extent repository into Maintenance Mode and evacuate backups.
8. Remove existing repository from the SOBR configuration and then from the Backup Repositories section. At this point no storage of any jobs should actually be flowing through your old server. It is perfectly fine for a SOBR to only contain a single extent from a data locality standpoint.

II. Migrate the Backup and Guest Interaction Proxies

1. Go to each of your remaining repositories and set proxy affinity to the new repository server you have created. If you have previously scaled out your backup proxies then you can ignore this step.
2. Under Backup Proxy in Backup Infrastructure remove the Backup Proxy installation on your existing VBR server.  Again, if possible you may want to run a job at this point to ensure you haven’t broken anything in the process.
3. Go to each of your backup jobs that are utilizing the Guest Processing features. Ensure the guest interaction proxy at the bottom of the screen is set to either your new repository server, auto or if scaled out another server in your infrastructure.

III. Migrate the Veeam Backup & Replication Server

1. Disable all backup, Backup Copy and Agent jobs on your old server that have a schedule.
2. Run a Config Backup on the old server. If you have chosen to Encrypt your configuration backup the process below is going to be a great test to see if you remember or documented it. If you don’t know what this is go ahead and change it under File>Manage Passwords before running this final configuration backup.
3. Shutdown all the Veeam services on your existing backup server or go ahead and power it down. This ensures you won’t have 2 servers accessing the same components.
4. If not already done, create your new Veeam Backup and Replication server/VM. Be sure to follow the guidelines on sizing available in the Best Practices Guide.
5. Install Veeam Backup, ensuring that you use the same version and update as production server. Safest bet is to just have both patched to the latest level of the latest version.
6. Add a backup repository on your new server pointing to the Config Backup repository folder you created in step 2 of the Migrating the Repository step.
7. Go to Config Backup and hit the “Restore” button.
8. As the wizard begins choose the Migrate option.
9. Change the backup repository to the repository created in step 5, choose your latest backup file which should be the same as the one created in step 2 above.
10. If encrypted, specify your backup password and then choose to overwrite the existing VeeamBackup database you created when you installed Veeam in step 4. The defaults should do this.
11. Choose any Restore Options you may want. I personally chose to check all 4 of the boxes but each job will have its own requirements.
12. Click the Finish button to begin the migration. From this point if any screens or messages pop up about errors or issues in processing it is a good idea go to ahead and contact support. All this process does is move the database from the old server to the new, changing any references to the old server to the new along the way. If something goes wrong it is most likely going to have a cascade effect and you are going to want them involved sooner than later.

IV. Verification and Cleanup

1. Now that your server has been migrated it’s a good idea to go through all the tabs in your Backup Infrastructure section, ensuring that all your information looks correct.
2. Go ahead and run a Config Backup at this point. That’s a nice low-key way to ensure that all of the basic Veeam components are working correctly.
3. Re-enable your disabled backup, backup copy and Agent jobs. If possible go ahead and run one and ensure that everything is hunky dory there.

Gotchas

This process when working correctly is extremely smooth. I’ll be honest and admit that I ran into a what I believe is a new bug in the VBR Migration wizard. We had a few SureBackup jobs that had been setup and while they had been run they have never been modified again since install. When this happens VBR notes the job_modified field of the job config database record as NUL. During the migration the wizard left those fields blank in the restored database, which is evidently something that is checked when you start the Veeam Backup Service. While the Service in the basic services.msc screen appears to be running under the hood you are only getting partial functionality. In my case support was able to go in and modify the database and re-include the NUL data to the field, but if you think you might have this issue it might be worth changing something minor on all of your jobs before the final configuration backup.

Conclusion

If you’ve made it this far, congrats! You should be good to go. While the process seems daunting it really wasn’t all that bad. If I hadn’t run into an issue it wouldn’t have been bad at all. The good news is that at this point you should be able to scale your backup system much easier without the grip and rip that used to be required.

If you work in the virtualization or datacenter field (are they really different anymore?) you probably know that VMworld US 2017 is next week, August 27-31. While VMware may not be the only option out there when it comes to virtualization anymore VMworld is still the defacto event for people in the field. This conference’s definition of community is unrivaled in scope with just as much if not more going on outside of the conference agenda as  in it.

As with all things worth doing conference attendance probably needs a checklist. Have you done yours? If not here are the high points of mine. I’m not going to bore you with “Jim will be attending session so and so”; well except for VMTN6699U and VMTN6700U you should totally join me at those sessions, but these are pretty general things I try to do each time.

• Take Your Vitamins– I hate to say it but the Vegas Flu is a real thing. Between being in the recirculated air of a jumbo jet for any number of hours to bookend event and being in the recirculated air of a Vegas hotel/casino/conference center I always seem to get at least a mild head cold at some point during the week. Start about now taking whatever version of Vitamin C supplement you like and do so throughout the event to help head this issue off.
• Bring Sharable Power- The average conference attendee has 3 devices on them at all times, phone, tablet and laptop. These things will start to get low on battery about midday and that just won’t do. In theory lots of places will have power outlets but with 25,000+ attendees they are still in short supply. I typically bring a big battery pack, a travel surge protector and USB power cables for everything under the sun so that I can plug in and share at sessions and keynotes.
• Get There Early and Be Ready To Learn– While the conference doesn’t start in earnest until Monday the 27th I always try to arrive midday Saturday because there is so much going on before the conference starts. One of the highlights of the entire conference to me each year is Opening Acts, a series of panel sessions put on by VMunderground and vBrownBag on Sunday afternoon. These sessions always prove to be insightful and are traditionally more career-centric or more wide-ranging than your typical VMworld session. The fact that this is followed by the always awesome VMunderground party that night is not lost on me either. Also, if you are a VMware TAM customer there is exclusive content for you on Sunday afternoon.
• Be Comfortable Being Yourself– So what do you wear? My friend Matt Crape covered this well in his recent post but I would like to add that go with what makes you most comfortable networking with your peers. If you are good with shorts and a t-shirt, go for it. Me personally I’m a golf shirt and jeans kind of guy so that’s most of what you’ll see from me. Your days at VMworld are most likely going to be between 15-20 hours so go with what feels good unless that’s naked. Nobody needs to see that. 😉
• Get Out and Be Social– This is not a “Woo Hoo, It’s Vegas So Let’s Party” topic. Yes, you can do that if that’s your prerogative, but keep in mind some of the smartest minds in your chosen career are going to be here and out at both events in the evening as well as in the hang space during the day. Go meet people as they are typically pretty nice and cool. While the VMworld sessions are what’s being sold as the content of the conference I will book very few of those, choosing instead to spend my time learning from others how they are dealing with many of the same issues that I have and make connections that can prove helpful down the road.
  Where to go be social? During the day the HangSpace/ VM Village is the place to go. In the evenings there is a never-ending list of gatherings to find your way to. I personally will be making sure I attend the Veeam party and VMunderground as they are my 2 evening must do’s each year and are typically among the biggest. Past that I’ll just go with the flow.
• Be Social Online Too– If you are a tweeter be sure to use not only the #VMworld hashtag but also that of whatever session or event you are currently in. If you look around it will typically be on a wall somewhere. This will help you extend the conversation during the session. If you aren’t on twitter yet you may want to consider that, often this is a great way to see what your colleagues are saying about announcements and such in real time. It also serves a great way to meet up with others at the conference.
• Get Some Sleep When Possible– I know this sounds counter-intuitive to the previous topic but if you are a 40-year-old like me this week will catch up to you. It is definitely possible to do events and conference from 7:30 AM to after midnight each day and while that’s a lot of fun, by Wednesday there are so many zombies walking around Mandalay Bay it looks like an episode of the Walking Dead. If you’ve been working on the session builder already take a look at your schedule and make room for you to sleep in a morning sometime midweek. You can catch up on the sessions once you get back.

While there’s more than that for me those are the basics. If you are going please hit me up @k00laidIT on twitter, I’d love to have a coffee, a beer or just a conversation with you. Have a great time!

P.S. Wear comfortable shoes!

As June is here we are deep into tech conference season already so I find myself behind the curve somewhat with this post, but here we are. I am extremely fortunate to have an employer who understands the value of attending Tech Conferences for IT Professionals and I’ve been able to attend at least one each year since 2014; going back and forth between CiscoLive and VMworld with a sprinkling of VeeamON and more local events such as vBrisket and VMUGs for good measure. As a “Hyper-Converged Admin” my choice of which “biggie” conference is done each year by looking at where my projects land; last year was CiscoLive due to a lot of Voice and Security Projects, this year VMworld due to lots of updates coming down the pike there and a potential VDI project.

The problem when you have a conference with north of 25,000 attendees is that you are limited in where you put these on. While Cisco does tend to move around some, VMworld has typically either been in San Francisco or Las Vegas. With the Moscone Center closed again this year for renovation we find pretty much all of the big guys are back in Las Vegas, with both CiscoLive and VMworld at Mandalay Bay once again as well as AWS re:Invent and Dell/EMC World in town this year as well. If you haven’t been to one of these Tech Conferences before or to Las Vegas both can be both exciting and overwhelming, but with a little help from others and some decent tips neither are that big of a deal.

Las Vegas Basics

So for a small town guy like me Las Vegas is very cool town, but tiring. The common thread I feel and have heard others voice as well is that Las Vegas is deceptively large because all of the hotels on the strip are so massive. While you can see from your Mandalay Bay window that New York New York is just the next block, it is probably about a mile away if walking there. Why this is important is that if you look at the list of hotels on each conference’s list you’ll see lots of options, but getting to that 8AM session may require a 30+ minute walk or even longer shuttle ride if you chose to stay at the Cosmopolitan (my personal favorite of all Las Vegas hotels but prohibitively far away). Couple that with temperatures in the triple digits during summer and proximity becomes more important.

Hotel Choices

So the first tip for any of these conferences is get a hotel as close as possible. For CiscoLive and VMworld keep in mind that you can move freely between the Mandalay Bay, Delano the Luxor and the Conference Center without ever stepping foot outside.  I would highly recommend trying to be in one of these. If you are booking late and the conference is out of rooms it’s worth trying to book directly through the hotel as they don’t let the events have the whole place. That said you are still going to be in for a hike. For example I stayed in the Mandalay Bay last year and it was approximately 1800 steps from my room to the entrance to conference.

Many of the vendor types that seemingly live their lives at these types of events like to opt for either the nearby Marriott Courtyard Las Vegas South or the Holiday Inn at Desert Club Resort for those that like a kitchen. From either of these you’re a quick Uber or Lyft away from the Conference Center entrance but don’t have to deal with the hustle and bustle of staying on the Strip if you don’t want to.

Getting Around

Speaking of Uber and Lyft, getting around with out walking is a bit of a consideration as well, both for the daily commute as well as for the various events. Traffic in the afternoons into the early morning is pretty impressive on the actual strip so to be honest I’ve not heard good things about trying to rely on the conference shuttles when available. Further I’ve heard many complaints from those who are locals that drive in and try to find parking.

Where that leaves you is 1) ride sharing service, 2) using the monorails, or 3) walking. Uber is nice because they are pretty knowledgeable about routing you around traffic regardless of time of day. Keep in mind when it comes to this and Mandalay Bay there are actually two defined Uber pickup/drop off spots, one outside of the conference center and another around the valet area underneath the hotel drop off area. These are impressively far apart so be sure you know where you want picked up before you request a ride.

The monorails are also nice but short. For those of you going to CLUS this is a good way to get to the Customer Appreciation Event as it will drop you off close to the T-mobile Arena.

Finally walking is a decent option, especially after dark for the various vendor events, but I do recommend if you are going to do it find a buddy or 3 or 4. I’ve never personally seen violence on the strip but you hear about it and there are lots of “character buskers” dressed like everything from Michael Jackson to Spongebob that will harass you.

One final note, while first impressions are important there really isn’t any point to being that person in the fancy shoes unless you’ve got booth duty. I typically while go buy a new pair of good running shoes a week or two before the conference so I can break them in and then that’s what I wear. If you are a step tracker kind of person like me expect 20,000 and up each day so take care of your feet.

Things To Do

Seriously, there’s plenty to do even if you weren’t at a conference already providing lots to do. Regardless of your interest if the conference doesn’t have you jam-packed enough you can find something you like here.

If you are new to IT or are just starting to get your name out there the most important things to do outside of the sessions is to get out there and be social. Both of the conferences we are talking about here have a great community that surround it with some wonderful people in it. The first step if you aren’t already would be to get yourself on twitter and follow the hashtag stream for your event (#CLUS for CiscoLive US, #VMworld for VMworld) , not only while you are there but before especially as many outside events will be planned then. Be sure to find the social area for your given conference and go make friends. Outside of the standard conference hours you’ll find that many of the Vendors will have events planned for attendees. If you have partners or vendors you work heavily with its worth asking your SE if they are doing anything.

CiscoLive Basics

CiscoLive will be held this year June 25-29th and promises to be a great show once again. While I have really enjoyed all of the conferences I’ve attended CLUS  was my first and near to my heart. First off of all those I’ve been to this one feels more academic than others. There aren’t really as many softball sessions and the sessions are a bigger part of the focus for the event than other. That said, they do a very good job of supporting the social community by having a Social Media Hub right in the middle of it all with special events for the twitterazzi most days. I highly recommend showing up and if nothing else walking up and just introducing yourself, trust me, you’ll fit right in there somewhere especially if you bring a Kilt. 😉 If you can come in early on Sunday the annual Tweetup Sunday afternoon is always a good time to make friends.

If you are going to CiscoLive you should have at this point booked most of your sessions. A couple of points here. First do not overbook yourself on sessions. While the pressure is always there to make sure you are getting all the education out of it as possible every session these days is recorded and can be watched later. My decision on if I’m going to do a particular session is based on if the subject is directly related to something that’s got me stumped and I want the opportunity to touch base with the speaker. Past that I’ll watch most after the fact. A better use of your time is getting out and networking, soaking up some of the distributed information there and will in many cases serve as a resource after the fact. I’ve yet to leave an event and not come home to do some kind of redesign based on things I’ve learned from the community.

A highlight for anybody who’s been to CLUS is always the Customer Appreciation Event. This year Bruno Mars will take over the T-mobile Arena and I am legitimately bummed that I will be missing it. The celebrity keynotes are always very good as well and usually provide a different view on how technology interacts with the world. I truly enjoyed listening to Kevin Spacey last year and this year they’ve booked Bryan Cranston.

Regarding keynotes, I typically like watch these in the social areas rather than packing myself into the keynote halls. The seating is better, there’s fewer people and usually refreshments are close at hand, plus you can find a surface to put your computer/iPad on to take notes and/or live tweet the talk.

VMworld Basics

As much as the focus on CiscoLive is on the direct educational benefit the focus from VMworld is more on learning from the community. With the conference officially running from August 27-31 there just as many official conference sessions as there are at CiscoLive, but I find there to be more lower level, marketing style sessions at VMworld. What makes up for it though is any number of community learning opportunities surrounding it. If you can swing coming in either Saturday or very early Sunday the vBrownbag/VMunderground Opening Acts is always a great place to learn about what is coming next in virtualization and technology. Speaking of vBrownBag, these guys have a stage running concurrent to the conference with session about anything you can conceive of all week long. Historically the vBrownBag stage has been found in the Hang Space (VMworld for social media area) but this year is still to be determined.

Another thing you’ll find is the potential to have your evenings books is exceptionally high with multiple vendor events every single night, traditionally starting with vBeers on Saturday evening. At some point as we get closer to the conference VMworld will fill a website with information and registration links for many of the gatherings to make scheduling easy. The Veeam, VMunderground and vExpert/VCDX/VMUG parties are always the most talked about. There is also the annual VMworld Party with typically big name acts but at the time of this writing there really isn’t any information about this yet. Be sure to follow along online and on social media to find out soon enough.

Conclusion

With all that being said, just go enjoy yourself as you are meant to do. There’s a reason that Denise Fishburne refers to CiscoLive as “Geek Summer Camp” because it does feel that way, regardless of the conference you’re attending. Everybody does things their own way. As I’ll be attending VMworld this year if you are there and want to say hi feel free to reach out and find me on twitter @k00laidIT.

About 6 months ago we updated 3/4 of our Cisco Telephony environment from 8.5 to 11.5. The only reason we didn’t do it all is because UCCX 11.5 wasn’t out yet so it went to 11. While there were a few bumps in the road; resizing VMs, some COP files, etc. the update went well. Unfortunately once it was done we starting having a glorious issue where after a reboot the servers sometimes failed to boot, presenting “FATAL: Could not load /lib/modules/2.6.32-573.18.1.el6.x86_64/modules.dep: No such file or directory”. Any way you put it, this sucked.

The first time this happened I call TAC and while they had seen it, they had no good answer except for rebuild the VM, restore from backup. Finally after the 3rd time (approximately 3 months after install) the bug had been officially documented and (yay) it included a work around. The good news is that the underlying issue at this point has been fixed in 11.5(1.11900.5) and forward so if you are already there, no problems.

The issue lies with the fact that the locked down build of RHEL 6 that any of the Cisco Voice server platforms are built on don’t handle VMware Tools updates well. It’s all good when you perform a manual update from their CLI and use their “utils vmtools refresh” utility, but many organizations, mine included, choose to make life easier and enable vCenter Update Manager to automatically upgrade the VMware tools each time a new version is available and the VM is rebooted.

So how do you fix it? While the bug ID has the fix in it, if you aren’t a VMware regular they’ve left out a few steps and it may not be the easiest thing to follow. So here I’m going to run down the entire process and get you (and chances are, myself when this happens in the future) back up and running.

	0. Go out to the cisco.com site and download the recovery CD for 11.5. You should be able to find that here, but if not or if you need a different version browse through the downloads to Downloads Home > Products > Unified Communications > Call Control > Unified Communications Manager (CallManager) > Unified Communications Manager Version 11.5 > Recovery Software. Once done upload this to any of the datastores available to host your failing VM resides on.
	1. If you’ve still got the VM running, shut it down by right clicking the VM>Power>Power Off in the vCenter Web UI or the ESXi embedded host client.
	2. Now we need to make a couple of modifications to the VM’s settings to tell it 1) attach the downloaded ISO file and check the “Connected at boot” box and 2) Under VM Options> Boot Options to “Force BIOS setup” at next boot. By default VMs do not look at attached ISOs as the first boot device. Once both of these are done it’s time to boot the VM.
	3. I personally like to launch the VMware Remote Console first and then boot from there, that way I’ve already got the screen up. After you power on the BIOS in a VM is the same old Phoenix BIOS we all know and love. Simply tell the VM to boot to CD before hard drive, move to Exit and “Save and Exit” and your VM will reboot directly into the recovery ISO.
	4.  Once you get up to the Recovery Disk menu screen as shown to the left we need to get out to a command prompt. To do this hit Alt-F2 and you’ll be presented with a standard bash prompt.
	5. So the root cause of all this issue is that the initramfs file is improperly sized after an automatic upgrade of VMware tools has been processed. So now that we have our prompt we first need to verify that we are actually seeing the issue we expect. To do this run the command “find / -name initramfs* .” This command should produce the full path and filename of the file. So to get the size of this file you now need to run an ls -lh against it. In my example your full command would be “ls -lh /mnt/part1/boot/initramfs-2.6.32-573.18.1.el6.x86_64.img .” If you aren’t particularly used to the Linux CLI once you get past …initr you should be able to hit tab to autocomplete. This should respond by showing you that that file is incorrectly sized somewhere between 11-15 MB.
	6. Now we need to perform a chroot on the directory that contains boot objects. In most cases this should simply be “chroot /mnt/part1 “
	7. Finally we need to manually re-run the VMware Tools installer to to get the file properly sized. These are included locally on the Recovery Disk so just run the command “/usr/bin/vmware-config-tools.pl -d ” There are various steps throughout the process where it is going to ask for input. Unless you know you have a reason to differ just hit enter at each one until it completes. Once the VMware Tools installation is done up arrow to where you checked the size of initramfs…img file above and rerun the command. You should now see file size changed to 24 MB or so.
	8. Now we just need to do a little clean up before we reboot. You need to make sure you go into Settings for your VM and tell it not to connect the ISO at boot. Once you make that change you should be able to flip back over to your console and simply type reboot  or shutdown -r 0  to reboot back to full functionality.