Skip to content

Lab 4 - Agentic response with ThousandEyes and dynamic network path data

Prerequisites

Before starting this lab, ensure you have completed: - Lab 1 - Webex bot and notification workflow (for receiving agent messages) - Lab 3 - AI Agent workflow, RADKit, and MCP server must be configured and working

Recap

So far you have:

  • Integrated a network topology's events into Cisco Workflows
  • Configured deterministic automated response in Workflows
  • Configured agentic operational response into Workflows
  • Integrated inventory and remote access with RADKit

Overview

In this lab, you will build on the agentic response, but instead trigger troubleshooting off an assurance event from ThousandEyes for digital user experience degrading.

By the end of this lab, you will:

  • Have ThousandEyes integrated into Cisco Workflows
  • Enrich the event data with network path data from ThousandEyes path info
  • Observe how effective troubleshooting is with just an event from ThousandEyes and device access

The following diagram illustrates the event flow we are building:

flowchart LR
    A[ThousandEyes Agent] -->|webhook| B[Cisco Workflows]
    B -->|MCP| C[RADKit]
    C --> D[Device]

Step 1: Setting up ThousandEyes Agent

1.1 Register the agent appliance

  1. Go to https://198.18.1.202/login, accept the certificate warning, and login with admin / welcome.
  2. Set the new password and click change password. Cisco123! will work for the password policy.
  3. Go to Agent > Account Group Token, add token pl4okteoylmox9t60vi1ghz456ixeoa7, and click Continue.
  4. Click Complete. Don't worry about the Gateway not pingable and NTP server errors. They will resolve.
  5. Go to Network and update the hostname to <your_name>-thousandeyes and click Save Changes.

1.2 Validate registration

  1. In ThousandEyes.com, go to Network & App Synthetics > Agent Settings.
  2. You should see your agent in the list, under the hostname field.
  3. Notice how the agent name has a random ID suffix created during registration due to everyone participating in the lab having the same OS hostname. Let's rename this agent name as well for better identification. Click the agent and change Agent Name to be <your_name>-thousandeyes and Save Changes.

Step 2: Configuring ThousandEyes Testing

2.1 Configure the HTTP Server test

  1. Login to https://www.thousandeyes.com and go to Network & App Synthetics > Test Settings > Add New Test > HTTP Server.
  2. Configure URL of https://cisco.webex.com
  3. Set the test name to something unique, like _webex
  4. Run the test every 1 minute
  5. Select Agents. Select your Agent name <your_name>-thousandeyes. Make sure you change the Default interface selection to eth1 198.18.13.202 so the test runs out the access network across R3-R2-R1 instead of across the mgmt network. Click Close.
  6. Click Deploy.

2.2 Configure an alert for the test

  1. Manage > Alert Rules > Add New Alert Rule
  2. Set these parameters:
    • Alert Type: Network > Agent to Server
    • Alert Rule Name: <your_name> - Congestion Alert
    • Severity: Major
    • Tests: <Select your test name from 2.1>
    • Agents: <Select your agent>
  3. Change Alert Detection to Manual
  4. Set the rules to:
    • 'Any' (not all) conditions met by the same 1 agent
    • Latency >= 200ms (You will need to click the icon next to - to switch from low/medium/high to static numbers)
    • Jitter >= 200ms (You will need to click the icon next to - to switch from low/medium/high to static numbers)
    • Packet Loss >= 5%
    • Error is present

ThousandEyes alert config

  1. Stay on this screen through the next step.

Tip

Adaptive alerting is a neat feature, but it requires a day to run to build normality for the anomaly detection. We don't have that much time here, so even in a world of predictive AI, we're going with old-school manual thresholds.

2.3 Configuring the webhook integration in Workflows

  1. Now we need to create a new webhook in Workflows for our ThousandEyes event. In Meraki Dashboard, go to Automation > Rules > Webhooks > + New webhook
  2. Name it <your_name>-te
  3. Save and then go back and view it to get the URL and save this to your local notepad for later reference.

2.4 Configuring the webhook integration in ThousandEyes

  1. Click the Notifications tab below the alert name you specified.
  2. Enable Send emails to your personal email, if desired. This is sometimes helpful to see quickly when alerts change state.
  3. Under Integrations click Manage integrations and go to Integrations 2.0.
  4. Choose Custom Webhook
  5. Name the webhook <your_name>-wh.
  6. Define the target as the Cisco Workflows webhook URL you created earlier in this lab.
  7. No Auth Type is needed since the API key is in the URL, so you can leave the Auth Type as Custom.
  8. Click Save and assign operation
  9. Set Operation Name to <your_name>-congestion-json and choose the Preset Configuration of Splunk. We aren't sending to Splunk but the preset for Splunk is a nice simple JSON format that Cisco Workflows and our AI agent will nicely process. It should prepopulate the Content-Type header for application/json which we want.
  10. Click Save.
  11. Go back to the Alert Rules tab (you may need to refresh the page). Set the Integration to your newly created webhook. Click Save Changes.

ThousandEyes Alert Notifications

Alert Rule Notifications tab with webhook integration selected

ThousandEyes Integration Setup

Integrations 2.0 - Custom Webhook operation setup

2.5 Create ThousandEyes API Token

The ThousandEyes workflows require an API Bearer Token to query path visualization data from the ThousandEyes API.

  1. In ThousandEyes, click your name in the top right of the screen, or go to Manage > Account Settings > Users and Roles.
  2. Scroll down to the OAuth Bearer Token section.
  3. Click Create to generate a new token. You will need to check your email and pass the token from email back to the web browser.
  4. Copy the token and save it securely - you will need this when importing the workflows.

ThousandEyes OAuth Bearer Token creation screen

Warning

The token is only displayed once. Make sure to copy it before closing the dialog.

2.6 Importing a workflow for ThousandEyes event handling

  1. In Cisco Workflows, go to the Automation Workspace where the workflows are listed.
  2. Add the git repo for ThousandEyes workflows like you did in Lab 3:
    • Display Name: LTRAI-1487 - ThousandEyes
    • Use the same Account Keys from Lab 3
    • REST API Repository: api.github.com/repos/ciscomanagedservices/ciscolive26_cw_ai_agents
    • Branch: main
    • Code Path: workflows/ThousandEyes
  3. Click Actions > Import Workflow > From Git and import the GetThousandEyesPathInfo workflow.
  4. When prompted for the te_bearer variable, enter the ThousandEyes OAuth Bearer Token you created in the previous step.
  5. Next, import the ThousandEyesAlertWebhook workflow.

2.7 Configuring the trigger for the workflow

  1. In Cisco Workflows, go to Automation > Rules
  2. Click the Automation rules tab, then click + Add automation rule
  3. Configure the rule:
    • Type: Webhook rule
    • Title: <your_name>-te-alert
    • Webhook: Select your webhook (<your_name>-te)
    • Workflow: Select ThousandEyes alert webhook
  4. Click Save

Step 3: Create congestion on the network to generate an incident

3.1 Create impairment

  1. Let's create impairment on R2 in the middle of the router chain. ssh cisco@198.18.1.102
  2. Run event manager run CONGESTION_ON which triggers an applet to apply interface policing and shaping on Gig1 and Gig2 to impair the access network.

3.2 Validating the troubleshooting run

  1. It will take a few minutes for ThousandEyes to see the result, since we have a 2 minute alert. Any one have any good stories or jokes? If not, Steve is going to play music he likes which may not be to your favor.
  2. If you signed up for an email alert, you should receive it soon. You can view the triggered workflow by going to Automation > Run Monitoring, searching for ThousandEyes alert webhook, then click view run details.
  3. It should now be troubleshooting the network. This takes some time to analyze output and determine the action to take for remediation--roughly 5 minutes with current early 2026 models. To watch for progress as it troubleshoots you can click ... on the AI Agent subworkflow in the webhook workflow run, and then click Update I_messages variable to watch for the latest updates in realtime. You will want to change the iteration in the main Agent Iteration Loop to the last or second to last iteration (sometimes the last iteration is still processing and won't have content).
  4. Alternatively, you can just wait for Webex Teams summary messages at milestones.
  5. Eventually you should see it request a task to be approved. Go to Automation > User Tasks to see if it is requesting a task approval. If it needs more info from you, it may also request something in Prompts but for this use case we don't expect it to need a prompt as it should have all the info it needs from ThousandEyes + RADKit to identify what it needs to solve the issue.
  6. Once you see the change request, take a look at it. You should see a well curated explanation of the symptom, isolation to the root cause, and suggested fix if the change is approved. It even qualifies the risk of the change.
  7. Click Approve. It will take a few more minutes, but then the policy should be removed and ThousandEyes alert should clear.
  8. The agent may keep assessing. We often see that after the fix it analyzes like a Problem Manager would, to determine how to prevent this issue from ever occurring again. See if you see that and a second change request.

3.3 Troubleshooting

If you are not seeing an alert trigger or the workflow is not running, try these troubleshooting steps:

Troubleshooting Checklist

  • ThousandEyes Alert Rule Configuration: In ThousandEyes, go to Manage > Alert Rules > <your Rule>. Verify you have tests selected, verify your alert conditions (latency > 200ms, etc.), make sure it says 'Any' not 'All', and click the Notifications tab to verify you have your email and integration set.
  • Check ThousandEyes Alerts: In ThousandEyes, click Alerts. Do you see your alert?
  • Check ThousandEyes Test Status: Go to Network & App Synthetics > Views, select your test, then Agent to Server. Do you see a spike in latency?
  • Verify Automation Rule in Cisco Workflows: Verify your automation rule is enabled, set to the webhook, and applied to ThousandEyes alert webhook.
  • Check Webhook History: Go to Automation > Rules > History. Do you see your webhook event?
  • Check ThousandEyes Connector: If you don't see your webhook but have an alert, go back to Integrations 2.0 > Connectors (https://app.thousandeyes.com/manage/integrations/v2/connectors). Does your connector go to the right URL? Does it have assigned operations?
  • Retrigger the Alert: If you need to retrigger the alert, SSH to R2 and run event manager run CONGESTION_OFF then event manager run CONGESTION_ON.

Summary

That's it--great job!

You have successfully configured:

Component Status Purpose
ThousandEyes Connected Detects network performance degradation and triggers alerts
Cisco Workflows Automated response Now performing automated actions on devices
RADKit Integrated Provides secure device access for troubleshooting

And with that, you have seen event stimulus to drive agentic network troubleshooting, all the way through root cause resolution.

Now--imagine the power of it having the ThousandEyes alert, network path, and access to device level events/logs. What other types of your network issues do you think you can use Cisco Workflows ambient agent to solve for you?

Cleanup

In order to avoid triggering unnecessary email alerts and workflow runs, please delete your ThousandEyes alert rule after you have successfully completed the lab.

  1. In ThousandEyes, go to Manage -> Alert Rules
  2. Find your alert rule (e.g., <your_name> - Congestion Alert)
  3. Click the ... menu on the right side of your alert rule
  4. Click Delete to remove the alert rule

Next Steps

In the next lab, you will learn how to create custom tools to extend the AI Agent's capabilities. This allows you to integrate additional data sources and APIs that the agent can use during troubleshooting.

Take Home Ideas

We encourage you to continue to test and enhance the power of agentic troubleshooting at home. Some other things to consider trying are:

  • Integrate your knowledge base to augment and refine specific policies or processes
  • Integrate with your enterprise ITSM change management, such as ServiceNow
  • Add additional observability sources like AppDynamics, Datadog, or your own monitoring tools
  • Combine multiple event sources to give the agent richer context for troubleshooting
Updated at 2026-03-18 16:54:41