Dashboards

Build operational Splunk dashboards with panels, tokens, drilldowns, and saved searches for real-time visibility.

Simple Explanation (ELI5)

A Splunk dashboard is a page that runs multiple searches at once and shows you all the results together. Instead of running 10 separate searches every morning, you open one page and see everything in one place — and it updates automatically.

Technical Explanation

Splunk dashboards are composed of panels, each backed by a saved search or inline SPL. Dashboards can be created via the visual editor (no XML needed) or directly in XML for full control. Dashboard Studio is the modern drag-and-drop dashboard builder with CSS-level customization. Classic dashboards use SimpleXML. Tokens enable interactivity — a dropdown controls a time range or host filter across all panels simultaneously via token substitution.

Dashboard Anatomy

Creating a Dashboard (Simple XML)

<dashboard version="1.1" theme="dark">
  <label>Application Error Monitor</label>
  <description>Real-time error rates by service</description>

  <!-- Token input: filter by environment -->
  <fieldset submitButton="false">
    <input type="dropdown" token="env" searchWhenChanged="true">
      <label>Environment</label>
      <choice value="prod_app">Production</choice>
      <choice value="staging_app">Staging</choice>
      <default>prod_app</default>
    </input>
  </fieldset>

  <row>
    <!-- Single value panel: total errors -->
    <panel>
      <single>
        <title>Total Errors (Last 1h)</title>
        <search>
          <query>index=$env$ level=ERROR earliest=-1h | stats count</query>
          <earliest>-1h</earliest>
        </search>
        <option name="colorBy">value</option>
        <option name="rangeColors">["0x65ba45","0xf59e0b","0xef4444"]</option>
        <option name="rangeValues">[0,100,500]</option>
      </single>
    </panel>

    <!-- Line chart panel: errors over time -->
    <panel>
      <chart>
        <title>Error Rate Over Time</title>
        <search>
          <query>index=$env$ level=ERROR | timechart span=5m count by service</query>
          <earliest>-1h</earliest>
        </search>
        <option name="charting.chart">line</option>
      </chart>
    </panel>
  </row>

  <row>
    <!-- Table panel: top error hosts -->
    <panel>
      <table>
        <title>Top Error Hosts</title>
        <search>
          <query>index=$env$ level=ERROR
| stats count as errors by host, service
| sort -errors | head 10</query>
          <earliest>-1h</earliest>
        </search>
        <option name="drilldown">row</option>
        <drilldown>
          <link target="_blank">
            search?q=index=$env$ level=ERROR host=$row.host$&earliest=-1h
          </link>
        </drilldown>
      </table>
    </panel>
  </row>
</dashboard>

Saved Searches for Dashboards

# In Splunk UI: Activity → Searches, Reports, and Alerts → New Report
# Or reference an existing named search in a panel:

<search ref="Error Rate by Service" />

# Schedule the underlying saved search to run every 5 minutes
# and cache results — dashboard panels load instantly from cache

Debugging Scenarios

• Panel shows "No results": Check the panel's inline SPL by clicking the magnifying glass icon. Verify token is substituting correctly (inspect query with $tokenName$ resolved).
• Token not passing between panels: Ensure the drilldown action sets the correct token name and the dependent panel references the same token in its search.
• Dashboard loads slowly: Pin panels to cached saved searches scheduled at intervals. Avoid long-running inline queries with wide time ranges.
• Chart type wrong: Some SPL outputs (multi-value) don't render as expected in line charts — use timechart for time-series and chart for field-based charts.

Real-world Use Case

An SRE team built a "Service Health" dashboard with a single environment token. It showed error counts (single value panel), error rate over time (line chart), regional breakdown (choropleth map), and top failing endpoints (table with drilldown to full search). The dashboard was pinned as the on-call engineer's first screen, reducing time-to-context during incidents from 10 minutes to under 60 seconds.

Interview Questions

Summary

Effective Splunk dashboards turn raw log data into operational awareness. Tokens make them interactive, drilldowns make them investigative, and scheduled saved searches make them fast. Design dashboards for the person who will use them, not for aesthetic completeness.