Skip to content

Local Development

Prerequisites

  • mise installed and configured
  • AWS CLI configured with appropriate permissions
  • Session Manager plugin installed (automatically handled by the script)

Preparing AWS Credentials for local development

Using granted, run:

1
assume -x # Select a SRE SSO Admin Role

This will export the necessary AWS environment variables to your current environment which then get set in the Docker compose services.

If using another method, ensure the following environment variables are set:

1
2
3
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
export AWS_SESSION_TOKEN=...

Docker Compose

The docker-compose files in the project root can be deployed locally to simulate various AWS-based observability platform components. The various components can be configured in their respective subfolders before running the docker commands.

You need to rebuild and redeploy the docker-compose stack after updating the configuration!

Non-clustered deployment

File: docker-compose.yml

A lightweight version where no clustering is enabled. Easy to deploy and interact with. After deployment, you should find each component at the following URLs:

  • Grafana: localhost:3001
  • (Alloy) CloudWatch collector: localhost:12346
  • (Alloy) Metrics: localhost:12345
  • (Prometheus) Metrics storage: localhost:9090
  • (Prometheus) Metrics quarantine: localhost: 9091

Deployment command:

1
docker compose up --build -d

CloudWatch collector gossip cluster deployment

File: alloy-cluster.docker-compose.yml

A version of the local development stack for testing clustering behaviour of the configuration found in ../alloy/config. This stack contains a CloudWatch collector cluster of three nodes. After deployment, you should be able to find each component at the following URL:

  • Grafana: localhost:3001
  • (Alloy) CloudWatch collector: localhost:12346
  • (Alloy) Metrics:
  • alloy-1: localhost:12346
  • alloy-2: localhost:12347
  • alloy-3: localhost:12348
  • (Prometheus) Metrics storage: localhost:9090
  • (Prometheus) Metrics quarantine: localhost: 9091

Deployment command:

1
docker compose -f alloy-cluster.docker-compose.yml up --build -d

Bastion Connection

To connect to Grafana locally through the bastion:

1
2
3
4
5
# Basic usage with defaults
mise run connect-bastion

# Custom configuration
mise run connect-bastion --instance "my-bastion-*" --service "grafana.example.com" --local-port 8080

HK: Git Hook Management (Experimental)

This project uses hk for managing the Git hooks and ensuring code quality through automated checks. This tool is added as an experiment to this project.

For a quick setup, run:

1
2
3
4
5
# Picks up the mise.pre-commit.toml file
export MISE_ENV=pre-commit
mise install
# Will execute hooks with mise exec. With this, it won't be necessary to activate mise in order to run hooks with mise tools.
hk install --mise

For more installation and usage instructions, refer to the jdx hk documentation.

Local Alerting Development

This section describes how to develop and test Grafana alerts locally using Terraform. This assumes that you already started up a local stack using the docker-compose.yml file and pushed the Grafana dashboards at least once using mise run dashboards:push.

Note

Be aware that this has limited capabilities compared to the cloud environment as most datasources (e.g: OpenSearch) are not available locally.

Local Development Setup

  1. The local Grafana instance runs on http://localhost:3001 with default credentials admin:admin.
  2. Run the task mise run tf:context local to switch to the local context and initialize the backend.

  3. Run Terraform commands:

    1
2
3
    cd grafana/provisioning
terraform plan
terraform apply

Testing Alerts Locally

  1. After running terraform apply, verify alerts appear in the local Grafana UI
  2. Check the alert rule group in Grafana to verify the configuration matches expectations.
  3. Check if the correct Contact Point is set for the alert rule.

How It Works

When local_dev_config.enabled is set to true:

  • The Grafana provider connects to http://localhost:3001 instead of the cloud environment
  • Prometheus datasources are not provisioned (they require cloud credentials)
  • Alerts are created against the local Grafana instance

Jsonnet Development

This project uses Jsonnet for rendering the Grafana dashboards and configurations. This sections will help you set up your local development environment for Jsonnet.

This guide assumes you are using VS Code as your IDE. For other IDEs, please refer to the respective documentation.

Set up VS Code Language Server

  1. Install the vscode-jsonnet extension.
  2. In the VS Code settings, search for @ext:grafana.vscode-jsonnet set the following:
  3. in Jsonnet -> Language Server -> Jpath, add the following paths:
    1. lib
    2. libsonnet
    3. vendor
  4. In Jsonnet -> Language Server -> Tank Mode: set disabled.

Troubleshooting the Extension

Jsonnet Language Server Not Starting

The VS Code plugin checks for a new version of the language server on startup by checking the GitHub releases. If this check fails, the language server will not start. To alleviate this, disable the auto-update options in the extension.

TIP: Evaluate File on Save

The Jsonnet VS Code extension supports evaluating the Jsonnet file on save. This is useful for debugging and validating the Jsonnet code. This feature is accessible by the > Jsonnet: Evaluate File command or the button in the top right corner of the VS Code window.

Rate Limiting on GitHub

You might be met with a rate limiting error when using mise install. To solve this you need to create a GitHub token and place them within your environment as GITHUB_TOKEN. Do the following: