Re-installing / Deploying 4000 EPOPs Agents in under 60 Minutes

Note: This is a very Long and a boring post with no screenshots

Its been long that I got some new topic to blog. This one came when we decided to move a VCenter from old setup of vROPs to a new Setup.

The customer is heavily relying on EPOPs data to monitor the environment. (The dashboard for unified view coming soon)

One of the major challenge we had was how do we upgrade re-configure 4000+ EPOPs agents to point to new vROPs setup.

To add to the challenge there were a few SR raised with GSS which needed us to completely re-install the EPOPs agents,

modify the config file and re-start the service.

These being production servers and my dashboards need the service level data coming from the EPOPs, Also the server monitoring could not stay down for more than two hours.

Now we needed a way where we could do this with script in parallel and at the same time be fast.

Powershell was first option but it proved to be slow when I tried to copy the binnary from the source to destination.

So I decided to try the good old batch script combined with PSEXEC.exe

To make the script even faster the approach was to take stage all the binnaries before the actual execution.

It also was needed one more script which could do actual un-installation and configuration of the EPOPs agents from inside the server, so that all the VMs can execute in parallel.

 

Following are the Steps I followed

  1. Create a script to copy the Binnary and Scripts on each Server
  2. Create a script to Execute un-install and re-install of the EPOPs agents, and do the configuration of EPOPs agents
  3. Create a Script to Execute the re-install script on all the servers from a common server.

Step 1: Creating a Script to Copy the Binaries and Re-install script to all 4000 VMs.

After a lot of trial and error this is what I found to be fastest

  1. Copy all the binaries in one folder on central machine,
  2. Mount that folder as a temp directory “t:\” on the target server, using admin username and password.
  3. Copy the folder locally in the VM.

Below is the Snippet of the script,

@echo off

set counter=0

set SCRIPTFOLDER=D:\Binnarycopy\File_copy

For /F “tokens=1,2 delims=,” %%G IN (Input.txt) DO (

echo VMCode :%%H

set username= ADMIN%%H

set password=<YourPassword>

net use t: \\%%G\c$ /User:username password

xcopy “D:\patchcopy\File_copy\ReinstallEPOps.bat” “t:\Windows\VMware\” /S /Y /F /D

xcopy “D:\patchcopy\File_copy\ vRealize-Endpoint-Operations-Management-Agent-x86-win-6.6.1-6091118.exe” “t:\Windows\VMware\” /S /Y /F /D

net use t: /delete

set /a counter=counter+1

)

Understanding the Script:

I have created an input file “Input.txt” with 2 inputs VMcode – a 5 digit number and IP Address.

You don’t need to give VMcode, In my case I need that to create the admin Username as every VM has a different admin password.

set counter=0

set SCRIPTFOLDER=D:\Binnarycopy\File_copy

For /F “tokens=1,2 delims=,” %%G IN (Input.txt) DO (

echo VMCode :%%H

In the Above lines I am just setting a counter to 0 reading input and displaying the VMcode for journaling purpose.

 

set username= ADMIN%%H

set password=<YourPassword>

Now I set the username and password to use to connect. As I said in your case it could be one common username password.

%% G is the IP address of the VM, you can also work with FQDN.

Next step is to mount a shared drive and copy the Binaries and the re-install script.

net use t: \\%%G\c$ /User:username password

xcopy “D:\patchcopy\File_copy\ReinstallEPOps.bat” “t:\Windows\VMware\” /S /Y /F /D

xcopy “D:\patchcopy\File_copy\ vRealize-Endpoint-Operations-Management-Agent-x86-win-6.6.1-6091118.exe” “t:\Windows\VMware\” /S /Y /F /D

net use t: /delete

set /a counter=counter+1

)

Increase the Counter and run for next VM.

Step 2: creating script for re-installation of EPOPs agents

This script assumes that all the required binaries are already available in the VM.

Following are the steps that we need to do in the script.

  1. Stop the running EPOPs service.
  2. Uninstall the EPOPs agent using force uninstall
  3. Delete the old token file.
  4. Install the EPOPs agents.
  5. Add the services to the configuration file.
  6. Re-start the service.

Below is the Snippet of the script

//

echo “starting uninstall” > log.txt

SC stop “End Point Operations Management Agent”

timeout  30

“C:\ep-agent\unins000.exe” /VERYSILENT

timeout  30

echo “Removing stale folders” >> log.txt

rmdir “c:\ep-agent”

rmdir “c:\ProgramData\VMware\EP Ops Agent\epops-token”

echo “Installing New Agent” >> log.txt

“C:\Windows\VMware\vRealize-Endpoint-Operations-Management-Agent-x86-win-6.6.1-6091118.exe” /VERYSILENT -serverAddress <vROPs FQDN> -username admin -securePort 443 -password <vROPs Password> -serverCertificateThumbprint 3E:0A:57:65:BC:EB:03:72:8B:33:49:DB:4B:F4:6A:EA:C4:DC:3A:91

timeout  60

echo. >> c:\ep-agent\conf\agent.properties

echo windows.services.discover=IISADMIN,OracleServicecbprod,OracleOraDb10g_home1TNSListener,COMSysApp,MSDTC,EventSystem,RpcSs,SENS,wuauserv,VMTools,Spooler,IP_FastPrint,IP_PrintServer >> c:\ep-agent\conf\agent.properties

SC stop “End Point Operations Management Agent”

timeout  30

SC start “End Point Operations Management Agent”

//

Understanding the Script:

echo “starting uninstall” > log.txt

SC stop “End Point Operations Management Agent”

timeout  30

In the Above line I am Stopping the service and waiting for 30 seconds for the service to stop completely.

“C:\ep-agent\unins000.exe” /VERYSILENT

timeout  30

echo “Removing stale folders” >> log.txt

rmdir “c:\ep-agent”

rmdir “c:\ProgramData\VMware\EP Ops Agent\epops-token”

Here we uninstall the EPOPs agents and remove the token files.

Next step is to install the epops agents with the parameters like vROPs FQDN, username password, and certificate thumbprint.

“C:\Windows\VMware\vRealize-Endpoint-Operations-Management-Agent-x86-win-6.6.1-6091118.exe” /VERYSILENT -serverAddress <vROPs FQDN> -username admin -securePort 443 -password <vROPs Password> -serverCertificateThumbprint 3E:0A:57:65:BC:EB:03:72:8B:33:49:DB:4B:F4:6A:EA:C4:DC:3A:91

Last part of the script is to add the services to conf file and restart the service.

echo windows.services.discover=IISADMIN,OracleServicecbprod,OracleOraDb10g_home1TNSListener,COMSysApp,MSDTC,EventSystem,RpcSs,SENS,wuauserv,VMTools,Spooler,IP_FastPrint,IP_PrintServer >> c:\ep-agent\conf\agent.properties

SC stop “End Point Operations Management Agent”

timeout  30

SC start “End Point Operations Management Agent”

Step 3: – Creating a script for executing the Binaries from the central server.

This script is very similar to the copy script, instead of copying the file we will execute the command.

Below is the snippet of the script

//

@echo off

set SCRIPTFOLDER=D:\patchcopy\File_copy

For /F “tokens=1,2 delims=,” %%G IN (input.txt) DO (

echo VMcode ID :%%H

set username= ADMIN%%H

set password=<VM Password>

%SCRIPTFOLDER%\psexec.EXE -s -accepteula \\%%G -u username -p password -d cmd.exe /c “C:\Windows\VMware\ReinstallEPOps.bat”

)

//

 

order to Run the Script : –

  1. First Run the Script to Copy the binaries to the remote servers.
  2. Run the script to execute the re-install script on all the servers.

 

Note:- try to find a central machine such that there is no firewall between the VMs for faster execution.

You will also need psexec.exe in the same folder as the script on the central server.

Future Enhancements in progress :

Creating Menu Driven approach, Scheduler for upgrade and extending script for Linux.

 

 

 

 

 

 

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s