Skip to content

Lab 1 - Setting up Network Topology

Overview

In this lab, you will configure the foundational infrastructure required for agentic network operations.

By the end of this lab, you will have:

  • Access to your dCloud lab environment
  • Syslog forwarding from network device alerts to Splunk
  • Splunk configured to trigger webhooks to Cisco Workflows on specific events
  • Cisco Workflows configured with a workflow triggered by device events
  • External tool integration with Cisco Workflows for Webex Teams notifications

This will serve as the detect & notify aspects of these labs.

The following diagram illustrates the event flow we are building:

flowchart LR
    A[Device] -->|syslog| B[Splunk]
    B -->|webhook| C[Cisco Workflows]
    C -->|notification| D[Webex Teams]

Step 0: Sign up for a Webex account

This account will let you join our session's discussion space where you can ask questions and obtain info that we send to you throughout the session. You will also need this account for generating notification API keys in some of the labs.

  1. If you already have a Webex account, login within it. If you need to register, go to https://signup.webex.com/sign-up and then validate your email.
  2. Join our room for discussion but opening the Cisco Events app for Cisco Live, and in the sidebar go to My sessions> Hands on building cognitive network operations agents with Cisco workflows for AI driven troubleshooting. and click the webex icon in the lower right part of that box.
  3. Put your email associated with your webex account in for joining session LTRAI-1487.

Step 1: Access Your Lab Environment

Your lab environment is hosted on Cisco dCloud and has been pre-provisioned for this session. You will access it through https://lab-assistant.com/, which provides instant access to running lab pods.

1.1 Checking out your personal Lab Pod

This step has already been completed for you by your lab instructors at your station. If you prefer to use your personal laptop instead of the workstation in front of you, and have Cisco Secure Client and SSH installed, you can use your own laptop. If you choose to use your own device, let the instructor know and we will mark your station's pod number available so you can check the pod out from your personal laptop by following steps 1.1 and 1.2.

  1. Go to https://lab-assistant.com/ and enter the access code provided by your instructors.
  2. The documentation button is where your lab guide lies, which is this document that you may already be reading.
  3. Check out a pod by clicking the :: POD :: button, and selecting an available pod number from the list of Available Pods shows in green.
  4. Note: If performing this lab outside of a proctored Cisco Live! intructor led lab, your instructor should have shared a dCloud topology with your cisco.com account, and VPN details will instead be available in the dCloud session info.

1.2 Connecting to your Pod

This step has already been completed for you already, unless you are choosing to use your personal laptop instead for managing the lab, or if your session becomes disconnected.

  1. In the Pod Details section, you will see the VPN details. Clicking Connect VPN should auto connect VPN, if you need to connect. You can also VPN to the defined VPN URL, and enter the VPN credentials available from this section.

1.3 Validating VPN connectivity

  1. Validate your VPN connectivity to the lab pod. You should be on the 198.18.1.x/24 network, which is our management network. Try to ping 198.18.1.200 which should be successful.

1.4 Famaliarize yourself with the lab topology

lab topology

  1. We have some workstations in the lower right corner of the topology diagram. These are your user access workstations, where users are using cloud/SaaS application services such as Webex.
  2. These users route over an access network routed by R3 -> R2 -> R1 to get out to the public Internet.
  3. The user access network has a ThousandEyes agent that will run synthetic tests for user experience after you provision it later in the lab.
  4. We have a Splunk server that can take syslog info (events) from devices and forward actionable faults on to Cisco Workflows, which will orchestrate our automated fault analysis and response.
  5. We have some utility servers to help integrate Workflows. Specifically, we have a remote access server for Cisco Workflows (used in lab 2), Cisco RADKit for connectivity via MCP (used in labs 3+).

Step 2: Configure Router Syslog

In this step, you will configure router R3 to send syslog messages to Splunk. These syslog messages will be used to trigger automated workflows when interface state changes occur.

2.1 Connect to Router R3

  1. Open PuTTY or your favorite terminal from your workstation
  2. SSH to router R3:
ssh cisco@198.18.1.103
Password: cisco

2.2 Configure Syslog

Enter configuration mode and apply the following configuration:

enable
configure terminal

! Enable syslog at debugging level to capture interface events
logging trap debugging

! Include hostname in syslog messages for identification
logging origin-id hostname

! Set source interface to mgmt interface
logging source-interface GigabitEthernet3

! Configure Splunk as the syslog destination
logging host 198.18.1.210

! Optional: you may want to set the timestamps for logging and debugging:
service timestamps log datetime msec
service timestamps debug datetime msec 

end
write memory

2.3 Verify Syslog Configuration

show logging

Confirm that: - Logging trap level is set to debugging - Logging host 198.18.1.210 is listed

You may optionally want to clear the log's boot messages to see new events easier. 

clear logging
<enter to confirm>

Step 3: Setting up Splunk syslog ingestion

Now we need to get the syslog data from the router into splunk.

We must create a place for the syslog messages to be stored within Splunk. We will use an app to extend splunk to ingest Cisco syslog with Add-on for Cisco Network Data which gives us the cisco:ios sourcetype for data parsing of syslog messages. This has been pre-installed in Splunk already for you. We have also pre-installed the Cisco Networks app if you want to take advantage of syslog visualizations/trending from this data, but that is beyond the intent of this course.

We will store the syslog messages from our monitored routers into a special syslog index, so let's first create that index in splunk, to separate it from the other non-syslog data that may also be already coming into splunk.

3.1 Create a syslog index

  1. In a web browser, login to http://198.18.1.210:8000 with admin / cisco.

  2. If Splunk isn't responding, ssh to 198.18.1.210 with admin / cisco and make sure that the splunk service is running with:

    bash cd /opt/splunk/bin sudo systemctl start splunk

  3. Setup an index by going to: Settings > Data > Indexes. Click the New index button in the upper right corner.

  4. Set the index name to syslog and the App to Cisco Networks
  5. Click Submit.

3.2 Setup splunk to listen to syslog

We need to ensure Splunk is listening for syslog traffic.

  1. Go to Settings > Data > Data inputs > UDP.
  2. In upper right, click New Local UDP. Configure the listener for UDP, port 514. Hit Next.
  3. Set the source type to cisco:ios. You will have to type this in manually, it is not available as a choice through the navigation tree in the pulldown.

  4. Set the App context to Cisco Networks, Host method as IP, index to syslog.

    Splunk data input

  5. Click Review and validate your configuration then click Submit.

3.3 Optional syslog ingestion validation

  1. If you want to validate that you are getting syslog messages into splunk, ssh cisco@198.18.1.203, and generate an interface down event.

    configure terminal
interface Loopback0
shut
! Wait threee seconds
no shut

  2. Browse to the search and reporting for the Cisco Networks app that the syslog messages are going to. Apps > Cisco Networks > Search

  3. Type index=syslog into the search bar. You should see a few syslog events generated from that interface state change. If you don't, first ensure that you are in the correct App (Cisco Networks) for the search and that you setup the index and data inputs for the right app, as well.

    Tip

    The configuration we have done so far gets the events into Splunk, but does not yet trigger any outbound webhooks to automation. Before we can set that up, we need to work backwards from Cisco Workflows and set some API keys up, so let's park our work in Splunk for a minute.

    Splunk syslog index search

Step 4: Setting up a Webex API account

We need to setup a Webex API before we can build our workflow notification.

  1. Sign up for an account at the Webex Developer Portal if you don't already have one.
  2. In the upper right, click your photo/avatar, and choose My Webex apps and then click Create a new app.
  3. Select Create a Bot since we just need to send/receive messages.
  4. Name your bot workflows-lab and set the username to something globally unique across all Webex users like <your_name>-<company_name>-workflows-bot. You must select an icon to submit. Choose one of the default robot icons, or feel free to upload a fun robot icon if you desire. Add a brief 10+ character description. Click Add bot.
  5. Copy your Bot access token and Bot username to a safe place.

4.1 Create a Webex Space for Notifications

Create a Webex space where the bot will send notifications. You will use this space throughout the remaining labs.

  1. Open Webex (app or web at https://web.webex.com)
  2. Click + to create a new space
  3. Name the space <your_name>-workflows-lab
  4. Add your bot to the space by pasting its username (e.g., <your_name>-<company_name>-workflows-bot@webex.bot). It will not autocomplete with typeahead, so hit enter and then you should be able to click the Add button. Ensure that the bot user has been added to your room--you should see 2 members in the room now. If not, workflows won't be able to have programatic access to find/post to the room.
  5. Note the space name - you will use this in later labs for AI Agent notifications

Step 5: Building a basic response workflow

Here we will first build a basic workflow to acknowledge that we can get an event to trigger a workflow. It won't do any troubleshooting/repair just yet.

5.1 Signing up for Meraki Dashboard

  1. Sign up for an account at meraki.cisco.com with your email address if you don't already have one. You will have to confirm the email invite, and then inut the MFA code in another email link to finish registration.
  2. Login to meraki.cisco.com.
  3. If you get a dialog, click the Setup systems manager screen and click Next.
  4. You will see an Automation section in the left sidebar, which is where we will mostly spend our time. This is the Cisco Workflows app.

5.2 Setting the webex API key

  1. In Workflows, go to Automation > Variables. Click +New Variable.
  2. Name the key <yourName>-webex, set String Type to Secure String, leave scope as Global and put the Webex access token in the value. Click Save.

5.3 Configuring a workflow

  1. Go to Automation > Workspace. Click the +Create button in the upper right, and choose Workflow with Automation since we will be attaching a webhook to this workflow.
  2. Name it <your_name>-notify and click Continue.
  3. You will be given an empty canvas.
  4. In the left panel are prebuilt modules you can use to call functions. Let's send a Webex message when we trigger an alert. In the left panel, navigate to Activities > Cisco Webex > Webex - Search for Room and drag this activity box into the middle panel workspace canvas.
  5. Click on the Webex - Search for Room activity block. Set the Search Room Name input to <your_name>-workflows-lab.
  6. For the Access Token, click the variable reference icon on the right of the text box and select your Webex API key variable.
  7. Next, drag a Webex - Post Message to Room activity block onto the canvas and connect it after the Search for Room block.
  8. Click on the Webex - Post Message to Room activity block and configure:
    • Access Token: Click the variable reference icon and select your Webex API key variable
    • Room ID: Click the variable reference icon and navigate to Activities > Webex - Search for Room > Room ID
    • Markdown Message: Enter ## It Worked! then press enter a few times, then click the variable reference icon on the right and add the variable Webhook > Output > Request Headers
    • This will take the JSON from the webhook and send it in a Webex Teams message

  9. Configure the workflow target. Click off the activity back to the workflow canvas and scroll down to Target. Since all activities in this workflow use the same target, we can set it once at the workflow level:

    • Select Execute on this target
    • Click the + to add a new target
    • Create an HTTP endpoint with protocol of HTTPS
    • Set host address to webexapis.com
    • Set No account keys to True
    • Check Disable server certification validation
    • Click Save

    Workflow target configuration for webexapis.com

  10. You should now be able to click Validate, and Run in the upper right and receive a message in your Webex space.

    Webex Send Message activity configuration with access token and message settings

Yay, your first workflow has been created and tested!

Step 6: Hooking the workflow up to a webhook

Now let's hook the webhook up, so we can trigger our test message from a device's syslog event.

6.1 Setting up a webhook in Cisco Workflows

  1. Go to Automation > Rules and then click the Webhooks section in the header bar and click + New webhook.

    Automation UI for Workflows
    Adding a webhook

  2. Name the webhook <your_name>-splunk-webhook. Keep the content type as application-json and click Save.

  3. Click back into the webhook you just created. You should now see the Webhook API Key and Webhook URL populated. Grab the Webhook URL (which has the API key embedded in the URL query) and stash it in a local notepad--we will need to put them back in Splunk in a minute.

6.2 Setting up the Meraki automation

  1. Go to Automation > Rules > Add automation
  2. Set type to webhook and title it Syslog: %LINEPROTO-5-UPDOWN
  3. Set the selected Webhook to the webhook you just created.

  4. Apply to the workflow you just created: <your_name>-notify and click Save.

    Automation configuration with webhook type and workflow selection

6.3 Finishing the Splunk webhook alert

Now we need to setup alerts for interesting syslog messages to trigger webhooks into Cisco Workflows for response.

We will create a saved search that triggers a webhook when matching syslog events are detected.

  1. Go back to Splunk and navigate to Settings > KNOWLEDGE > Searches, reports and alerts.
  2. IMPORTANT! Make sure you change the App context in the upper header to Cisco Networks before creating the alert, so it associates to the right app's index.
  3. Click New Alert in the upper right.
  4. Set the inputs to the following and save it:
    • Title: %LINEPROTO-5-UPDOWN webhook
    • Search: index=syslog "%LINEPROTO-5-UPDOWN" "changed state to down"
    • App: Cisco Networks
    • Alert type: Real-time
    • Trigger alert when: Per-Result
    • Expires: 1 second
    • Trigger Actions: Webhook

  5. In the Webhook configuration, paste your Webhook URL from step 6.1.

    Troubleshooting Tip

    You may want to also add a Trigger action for Add to Triggered Alerts so that you can verify the alert is triggering correctly by checking Activity > Triggered Alerts. Alerts will not appear there unless this action is configured.

    Webhook Troubleshooting

    To see if the syslog generated an alert, run the Run from the Searches, Reports, and Alerts page and change the search duration from Real-time to Presets> All-time. To troubleshoot webhook issues from within Cisco Workflows, go to Automation > Rules, then click the History tab to see incoming webhook requests and their status. This will tell you if the webhook is making it into Workflows from Splunk.

Step 7: Testing the event triggering workflow

7.1 Shutdown an interface

We will test on R3 since that is the router where we configured syslog forwarding in Step 2.

  1. You can generate a test syslog message by shutting down the loopback0

    clear logging
enable
configure terminal
interface Loopback0
shutdown

This should produce a new log as seen in show logging with contents of %LINEPROTO-5-UPDOWN.

7.2 Validation

  1. Check Splunk for the syslog event:

    • View the results of the index=syslog "%LINEPROTO-5-UPDOWN" saved search from the Searches, Reports, and Alerts page
    • Click Run to execute the search
    • Change the search scope (right of search bar) from Real-time to All-time to see historical results
    • You may need to wait a couple minutes for the event to get picked up and indexed (typically ~60 seconds end-to-end)

  2. Check Cisco Workflows for the triggered workflow:

    • If you see interface results in the saved search, go to Meraki Dashboard
    • Navigate to Automation > Run Monitoring to see if your workflow ran
    • Click on the workflow run and select View run details in the lower right
    • Click the Webex activity box to see the payload of the syslog message that was sent to Webex

  3. Check Webex for the notification:

    • You should have received a Webex message with JSON payload of the syslog event
    • Notice the extra info that the Cisco Networks app parses from the syslog format

    Troubleshooting

    If you aren't seeing end-to-end notification, isolate where the messaging is not making it. Start at Splunk and troubleshoot within these logical areas:

    • IOS syslog logging or Splunk data ingestion - Is the syslog reaching Splunk?
    • Search/reporting/webhook in Splunk - Is the alert triggering? To help isolate, check Splunk's saved search to see if the syslog is picked up, and the Activity > Triggered Alerts page to see if the webhook went out to Cisco Workflows.
    • Workflows rule history - Is the workflow being triggered? If you see the webhook with rule match count of None, Workflow got the webhook but didn't match a valid workflow to run. Check that your workflow doesn't have errors.

Summary

You have successfully configured the foundational infrastructure for agentic network operations:

Component Status Purpose
dCloud Lab Connected Provides isolated lab environment
Router Syslog Configured Sends interface events to Splunk
Splunk Webhook Active Triggers Cisco Workflows on interface down events
Cisco Workflows Active Workflow sends notification of event to Webex Teams

In the next lab, you will configure the Cisco Workflows agent to analyze incoming events and take automated remediation actions, instead of just being notified.

Updated at 2026-03-18 16:54:41