Skip to content

Build Grafana Dashboards

Overview

This repository is the central source of truth for all Grafana resources (dashboards, alerting rules, etc.) at KNMI.

The primary goal of this centralized approach is to:

  • Provide a consistent and simple experience for all KNMI teams.
  • Massively reduce the complexity and maintenance burden of managing Grafana resources.
  • Create a clear chain of ownership and responsibility.

This project solves the challenges of the previous decentralized model, where dashboards were scattered across different repositories and required complex, custom-built deployment tools.

Key Features of this Workflow

  • Single Source of Truth: All (production) dashboards and alerts are managed here, within source control.
  • Simplified Access: Viewing dashboards is public for anyone on the KNMI VPN. No account or login is required to view or edit in the playground.
  • "Playground" Development: You can build and test your dashboards live in the production Grafana instance within a dedicated Playground folder before committing the code here.
  • Standard Tooling: We use gcx and jsonnet for managing dashboards and alerts, and Terraform for other resources like contact points (not yet supported by gcx).

Workflows: Developing Dashboards and Alerts

You can create or modify dashboards and alerts using two primary workflows.

Required Tags

All dashboards require the owner: prefix tag to be present. Service dashboards (dashboards that are placed in the services folder) additionally require the application: and service: prefix tag.

For example, a Grafana service dashboard that is part of the Observability Platform would require the following tags:

1
2
3
owner:Cloud Platform Team
Application:Observability Platform
service:grafana

ClickOps Workflow

Important

The Playground folder rotates on a bi-weekly schedule! The next rotation date is displayed in the folder name. All dashboards and alerts will be deleted from the folder. If you are still working on a dashboard, either:

  • Create a Draft Merge Request with the dashboard JSON.
  • Save the JSON file on your local filesystem.

This workflow mostly resembles the workflow in the legacy Grafana setup. Key difference here is that the complete development happens in one central Grafana instance, instead of having to switch between prd and dta contexts. You use the Grafana UI in the Playground folder to build your resource, then export its JSON and add it to this repository.

Create a New Dashboard

  1. Navigate to the Playground folder in Grafana
  2. Create a new dashboard in the top right corner.
  3. Add panels, rows and queries to build your dashboard. Other team members can preview it live using the same URL.
  4. Once you're happy, go to Export (top right corner) > Export as code
  5. Copy the entire JSON model - pick the v1 resource model, other models are not supported.

Modify an Existing Dashboard

  1. Find the JSON file for the dashboard in this repository (or copy the JSON model from the existing dashboard in the UI).
  2. Navigate to the Playground folder in Grafana and paste the JSON into a new dashboard.
  3. Make your modifications in the UI.
  4. Once complete, export the new JSON model from the UI.
  5. Paste the new JSON back into the original file in this repository, overwriting the old content.
  6. Create a Merge Request with your changes.

As-Code Workflow

For advanced use cases (like creating reusable dashboard templates), you can write the jsonnet or raw JSON files directly.

  1. Create your .dashboard.jsonnet or .alerts.jsonnet file in the appropriate directory.
  2. Build your dashboard as code, using templates from the runbooks/lib/ directory, or write your own.
  3. To test locally, you can run gcx dev serve to start a local dev server with live-reload capabilities.
  4. For new dashboards, see the 'How to deploy a new dashboard' section, for modifying, save your changes and create an MR.

Best Practices and Guidelines

Use Environment Selectors

Do not create separate dashboards for DEV and PRD. This creates duplication and maintenance overhead.

Instead, your dashboard should be environment-agnostic. Use a Grafana variable (e.g., $service or $environment) as a selector at the top of your dashboard. Your queries should then use this variable to filter metrics (e.g., ...{env="$env", ...}).

You can link an alert to a specific dashboard panel, but the process requires care. The uid of a dashboard changes when it moves from the Playground to its final provisioned location.

To link an alert to a dashboard panel safely:

  1. Step 1: Create your dashboard in the Playground, export the JSON, and create a Merge Request for it.
  2. Step 2: Wait for the dashboard MR to be merged and deployed. Your dashboard now has a stable, permanent uid.
  3. Step 3: Create your alert in the Playground. In the alert configuration, you can now link to the permanent dashboard and panel.
  4. Step 4: Export the alert group JSON and create a new Merge Request for the alert.

Grafana Grid System

  • The width of the dashboard is divided into 24 columns
  • The height of a panel is represented in integers, where 1 represents 30 pixels

source