How to trigger a Aria Orchestrator Workflow by Aria Operations for Logs

Recently I had to troubleshooting a storage issue in a VMware vSphere environment. It turned out that the situation could be defused when the ESXi host enters Maintenance Mode automatically when a certain error log entry is generated. So I set to work and automated the task. To do so, VMware Aira Suite is quite useful. The process actually requires most of the products of it: Aria Operations for Logs (AOL), Aria Operations (AOps) and Aria Automation Orchestrator (AO). In the following post I will explain how all these products can work together to automate individual tasks based on log entries.

Unfortunately, you have to take a few steps to achieve this goal. These are:

• Integrate AOps into AOL
• Create an AOL alert based on log entry
• Deploy AO and connect to vCenter
• Install Management Pack for VMware Aria Automation Orchestrator
• Create workflow in AO
• Add workflow to package and add it to AOps actions
• Create a AOps alert that triggers the workflow

The sequence of the steps is variable. It should be obvious which steps build on each other.

I describe here my specific use case to enter maintenance mode automatically on a ESXi when the host throws a certain error.

Integrate AOps into AOL

It is required that AOL is able to forward Alerts to AOps. To obtain this function, AOps must be integrated into AOL. It an easily be done beneath Integrations –> VMware Aria Operations in AOL. Make sure, Enable alert integration is enabled.

Create an AOL alert based on log entry

As we want to trigger a workflow based on a log entry, we need to define a AOL alert for this specific log entry. This can be done by querying AOL to find the entry of choice. Based on this query an alert can be created.

Make sure, Send to VMware Aria Operations is selected.

Some important notes here:

• The name of this alert will used in AOps to define the symptom. Therefore, name the alert descriptively. If you use for example more AOL instances, add the instance name to the alert name.
• You can select a Fallback object BUT it essential that the forwarded alert is connected to the source object. Simply because AO workflow will be triggered on this object.
• AOps will not trigger AO workflow if the alert is still active on the object. Therefore, I enabled Auto cancel.

Deploy AO and connect to vCenter

This task is necessary if you do not have any AO instance in your environment. I would recommend setting up a separate instance if you are already using Aira Automation to clearly distinguish between the different use cases.

Installation is straight forward. Just download Automation Orchestrator Appliance and deploy the .ova file. I will not go into detail since this is well documented.

After deployment, two steps are necessary for our use case. We need to add Authentication with vSphere and we need to add vSphere vCenter using a pre-defined workflow.

In summary, the following steps need to be completed:

1. Go to https://fqdn_of_AO_appliance/vco-controlcenter and press Configure Authentication Provider.
2. Select vSphere as Authentication mode and fill in Host address and Default tenant. In my case this is SSO domain of my vCenter. In Admin group you can select a user group that contains your admins of choice.
3. Log into https://fqdn_of_AO_appliance/orchestration-ui using a member of the Admin group you selected in previous step.
4. Go to Workflows and search for workflow Add a vCenter Server instance. Run this workflow.
   
5. Fill in IP or host name of the vCenter Server instance to add and hit the checkbox if you want AO to ignore certificates if necessary.
6. On tab Set the connection properties uncheck the first checkbox and provide username and password.

Note: You can also add a vCenter instance to AO using the Management Pack we install in the next step. Nevertheless, I prefer it here because it is done just after deployment and no additional software is involved.

Install Management Pack for VMware Aria Automation Orchestrator

To be able to start Aria Orchestrator workflows in Aria Operations, you need to install and configure the Management Pack for VMware Aria Automation Orchestrator. At the time of writing, current version was 3.3.

Installation is straight forward. Download the Management Pack from VMware Marketplace and install the .pak-file in Operations.

For configuration, add a new Integrations account beneath Data Sources and select VMware Aria Automation Orchestrator Adapter.

Configure the Management Pack for VMware Aria Automation Orchestrator is well documented. Just one note: In my use case I used Orchestrator Credential and entered a vSphere-account for simplicity. This works because we added Authentication with vSphere after AO deployment.

Create workflow in AO

Now let’s create a workflow in Aria Orchestrator that we want to trigger in Aria Operations when a specific log entry is intercepted in Aria Operations for Logs. In my example this is a workflow for setting maintenance mode on the host. To promote an understanding of how to create a own workflow, I will show how I set up my workflow.

First, I duplicated a build-in maintenance mode workflow. Since I know exactly how maintenance mode should be set, I removed all input variables except the hostname.

Because of the reduced input data, I adjusted the code to fulfil my requirements.

var maintenanceSpec = new VcHostMaintenanceSpec();
maintenanceSpec.vsanMode = new VcVsanHostDecommissionMode();
maintenanceSpec.vsanMode.objectAction = "ensureObjectAccessibility";

task = host.enterMaintenanceMode_Task(0, true, maintenanceSpec);

Add workflow to package and add it to AOps actions

After you successful tested your workflow in AO, you need to create a package. A package is the layer that can be handed over to Aria Operations. Go to Assets -> Packages. Create a new package using a meaningful name, as this must be entered in Aria Operations.

Add your workflow in tab Content.

Do not skip to add dependencies!

The next step is about as important as it is unintuitive. The package just created must be announced to AOps. This task is done at the adapter instance. In AOps go to Inventory -> open Adapter Instances -> open Orchestrator Adapter Instance.

Take care to mark just Orchestrator Adapter Instance and Orchestrator on the right side as seen in the screenshot. Then press the small gear wheel and select Configure Package Discovery.

In the appearing window, add the name of your AO package in a new line.

Next step is to add the content of this package to available actions for the Resource Type – in my example Host System. To do so, click the gear wheel again and press Create/Modify Workflow Action on vCenter Resources. In the appearing windows mark your workflow and add the resource type of choice.

After this and a few moments, a new alert action should be available in Aria Operations.

Create a AOps alert that triggers the workflow

So, action is available yet. Before we can create an alert, we need to build remaining necessary parts of an alert. These are: recommendation and symptom. Let’s create a recommendation that allows to trigger the action in form of a AO workflow. For this, start the wizard for Recommendations and select VMware Aria Automation Orchestrator as Adapter Type and the just imported action as Action.

As you probably know, a AOps alert is defined by symptoms. Therefore, we need a symptom for our AOps alert. Go to Symptom Definitions and select Message Events. Then press ADD.

Select the Base Object Type of your choice. In my example, Host System. Drag and drop Notification.

Give it a name and enter the name of the alert you defined in Aria Operations for Logs as content of the Event Message.

Now we have everything to create an AOps alert, so let’s put it together. Start the alert creation wizard and give it a name. Select the appropriate Base Object Type. Beneath Symptoms / Conditions drag and drop your custom symptom.

Next on Recommendations, drop your one. Then select the policy, this alert should be active in. Optionally set notifications.

Since recommendations of new alerts are not automated by default, do some testing here! The last step is to enable automation for this alert. For this, edit the policy you added the alert to and activate Automate.

With this final setting, the AO workflow will be triggered when AOL catches the specified log entry.

Troubleshooting tips

As this solution is quite complex, you will certainly have to search for errors during implementation.

• Be aware of the History feature in AOL alerts!
  
• Configure Package Discovery in AOps has some issues. At least in my environment. Sometimes added package names disappeared respectively were not processed. Just try it again.
• Same with Create/Modify Workflow Action on vCenter Resources. Sometimes I had to do is multiple times. When Resource Type and Target Resource Type appears correct in the window, grab a coffee before you check the existence of the action. It takes some time.

Notes

• Before you active automation, be sure you tested your solution carefully!
• For some more advance AOps stuff, check my post go beyond standard Super Metrics.