Upgrade ESXi firmware and Drivers while you sleep – Parallel execution

Hello

Again writing after a long time, but nothing worth writing came up.

Finally after a long wait, there was a requirement to update Firmware and Driver of over 100 ESXi hosts, with a strict deadline.

Being a brand ambassador of Laziness and procrastination (along with being CTO Ambassador… Just in case you missed it), I figured out it is lot of work.

Thus started writing a script to do the task with minimum manual intervention and also keep it fast.

In the process created 2 scripts

  1. Script to create threads so that the activity is performed in parallel
  2. Script to upgrade drivers and firmware.

So the plan here was we have 18 clusters, we will take 1 host from each cluster for upgrade at same time so the time required will be reduced by 18 times.

Please get in touch with me, if you have similar activity or need any help

 

  1. Generic : in any given task where we want to perform parallel execution, we generally have a array of objects against which we run a set of command.

$names = @(“a”, “b”, “c”, “d”, “e”)

get-job | Remove-Job

foreach($n in $names)

{

    Write-host “Starting thread $n”

    Start-Job -filepath “C:\DriveD\Threads\ExecThread.ps1” -ArgumentList $n -Name $n

}

 

Here I am creating a basic array of 5 elements for which I will create threads,

Next I Iterate in the array and start a new job for each element. I am also passing name of the object as argument.

Start-Job is a command in Powershell which is used to create a parallel thread, you need to provide path of the script which will run for each thread. Also I am giving each batch a Name

This will now run an instance of ExecThread.p1 for each object in parallel.

 

I am not doing much in ExecThread.PS1

$passedValue = $args[0]

for($i = 0; $i -le 10; $i++)

{

 Write-host “$passedValue $i”

         sleep 2

}

 

For each object a new instance of this script is run which will accept  object name as input and display the counter value and go to sleep for 2 secs.

Now the running jobs and the output of this script will not be visible on the screen, as this runs in memory. To see the output you need to have a job manager, this job manager is created using Get-Job and Receive-Job commands as follows.

do{

$a = get-job

$runningJobs = 0;

       foreach($c in $a)

       {

        $jobState = $c.State;

        if($jobState -eq “Running”)

        {

            $runningJobs = $runningJobs + 1;

        }

        else

        {

            $name = $c.Name;

            write-host “$name = Job Name”

            $outp = Receive-Job -Name $name;

                

             write-host $outp;

            Remove-Job -Name $name;

        }

 

       }

write-host “Running Jobs $runningJobs”;

sleep 10;

 

}while($runningJobs -ne 0);

 

Here I am just checking the state of the jobs and displaying the number of running jobs, once the job is complete I retrieve the job status and display on screen. This can also be done using logging if you wish.

 

  1. Putting this to use in production – Here I am creating a log of log files as there are 18 cluster and all 18 cluster will start upgrade at same time. I want to keep track of all the 18 threads at same time also check for errors.

One of the Pre-req is you create a script using RacAdm and upload the firmware on all the hosts using iDRAC

$datetime = Get-Date -format “-MMdd-HHmmss”

$starttime = Get-Date -format “HH:mm”

 

$folderPathObj = pwd;

$rootFolderPath = $FolderPathObj.Path;

$logFolder = $rootFolderPath+”\LOGs”+$datetime;

New-Item -ItemType Directory -Path $logFolder -Force

$transFile = $logFolder+”\Console.log”

New-Item -ItemType File -Path $transFile -Force

 

Start-Transcript -Path $transFile -Verbose;

connect-viserver -server <VCNAME> -username “Username” -Password “Password”

 

$clusterList = Get-Cluster

get-job | Remove-Job

foreach($cluster in $clusterList)

{

       Write-host “Starting thread for Cluster”

       Start-Job -filepath “H:\vmware dump\ValidationScript\ExecuteUpgrade.ps1” -ArgumentList $arglist -Name $cluster

}

 

As you can see there is not much of difference from what we did in generic script apart from creating log folders and log files. Executing the thread on other hand here is a lot of job.

 

We put the server in maintenance –> wait till all the VMs are moved to other host –> upgrade the drivers à restart the host –> Upgrade the firmware –> wait till the server is back up –> bring the server out of maintenance.

here is the snippet of what I am doing

$ClusObj = get-cluster $clsname;    

$HostList = $ClusObj | get-vmhost;

foreach($hst in $HostList)

{

       $hstName = $hst.Name;     

        Write-host “Entering Host $hstName in Maintenance” -ForegroundColor Yellow;

        $hst | set-VMhost -state Maintenance -Evacuate | out-Null

        do{ sleep 10;

            $hstState = (get-vmhost $hst).ConnectionState;

            $hstStateStr = [string]$hstState;

            Write-host “.” -NoNewline;

        }while($hstStateStr -ne “Maintenance”);

         Write-host “Host $hstName entered Maintenance” -ForegroundColor Yellow;

             # Do similar loop for each step mentioned above like

       # Upgrade Host

       # Restart Host etc

}

Stop-Transcript;

Again a long post, the script helped us to save a lot of time while minimizing errors.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s