# How to Customize the Observability Skill for Your AWS Agent

> Customize the Observability skill for your AWS agent by editing markdown files and asset templates in the aws agent toolkit for aws repository. Changes take effect immediately without recompilation.

- Repository: [Amazon Web Services/agent-toolkit-for-aws](https://github.com/aws/agent-toolkit-for-aws)
- Tags: how-to-guide
- Published: 2026-06-28

---

**You can customize the Observability skill by editing the markdown reference files and asset templates in the `aws-agent-toolkit-for-aws` repository, with changes taking effect immediately at runtime without requiring recompilation.**

The Observability skill in the `aws/agent-toolkit-for-aws` repository provides AWS agents with guided capabilities for CloudWatch alarms, custom metrics, dashboards, and distributed tracing. Unlike compiled skills, this implementation loads its logic from plain markdown and configuration files, allowing you to customize thresholds, add new metrics, and modify infrastructure templates by editing the source files directly.

## Architecture of the Observability Skill

The skill is structured in four distinct layers that separate declarative configuration from execution:

- **Skill Manifest**: Defined in [`skills/core-skills/aws-observability/SKILL.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/SKILL.md) (lines 19-26), this file declares the skill name (`aws-observability`) and the routing table that maps user requests to specific reference documents.
- **Reference Documentation**: Located in `skills/core-skills/aws-observability/references/*.md`, these markdown files contain version-controlled guidance for alarms, metrics, dashboards, and tracing that the skill reads at runtime.
- **Asset Helpers**: Found in `skills/core-skills/aws-observability/assets/`, these include [`alarm-template.ts`](https://github.com/aws/agent-toolkit-for-aws/blob/main/alarm-template.ts) for CDK snippets and [`otel-config.yaml`](https://github.com/aws/agent-toolkit-for-aws/blob/main/otel-config.yaml) for ADOT collector configurations.
- **Execution Engine**: The Agent Core loads reference files on demand and executes AWS CLI commands through the MCP server, while the skill itself only guides users to edit files or run supplied snippets.

## How Customization Works

To customize the observability skill for your specific AWS environment, follow these steps:

1. **Identify the sub-area** you want to modify—alarms, metrics, dashboards, or tracing.
2. **Edit the corresponding reference file** in the `references/` directory or copy an asset helper into your own repository.
3. **Adjust values** such as alarm thresholds, metric dimensions, or CDK constructs to match your requirements.
4. **Invoke the skill again**—the Agent Toolkit loads files at runtime, so changes apply immediately without recompilation.

## Customizing Alarm Thresholds

The default alarm definitions live in [`skills/core-skills/aws-observability/references/alarms.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/references/alarms.md). To change a threshold, edit the markdown directly.

For example, to lower the CPU utilization alarm from 80% to 70%:

```markdown

# CPU Utilization Alarm

- **Metric:** AWS/EC2 – CPUUtilization
- **Threshold:** 70   <!-- changed from 80 -->
- **Period:** 300
- **EvaluationPeriods:** 2
- **ComparisonOperator:** GreaterThanThreshold

```

When you invoke the skill, it generates the corresponding AWS CLI command using your updated threshold:

```bash
aws cloudwatch put-metric-alarm \
  --alarm-name MyLambdaCPUHigh \
  --metric-name CPUUtilization \
  --namespace AWS/EC2 \
  --threshold 70 \
  --period 300 \
  --evaluation-periods 2 \
  --comparison-operator GreaterThanThreshold

```

## Adding Custom Metrics with EMF

To add application-specific metrics, edit [`skills/core-skills/aws-observability/references/metrics.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/references/metrics.md) with Embedded Metric Format (EMF) definitions.

Append a new custom metric definition:

```markdown

## Custom metric: MyAppRequests

- **Namespace:** MyApp/Performance
- **Dimensions:** Service=MyService
- **Unit:** Count
- **EMF example:**

```json
{
  "_aws": {
    "Timestamp": 1625247600000,
    "CloudWatchMetrics": [{
      "Namespace": "MyApp/Performance",
      "Dimensions": [["Service"]],
      "Metrics": [{"Name": "MyAppRequests"}]
    }]
  },
  "Service": "MyService",
  "MyAppRequests": 123
}

```

```

Then emit the metric from your application code:

```python
import json, time, boto3

client = boto3.client('cloudwatch')
payload = {
    "_aws": {
        "Timestamp": int(time.time() * 1000),
        "CloudWatchMetrics": [{
            "Namespace": "MyApp/Performance",
            "Dimensions": [["Service"]],
            "Metrics": [{"Name": "MyAppRequests"}]
        }]
    },
    "Service": "MyService",
    "MyAppRequests": 1
}
client.put_metric_data(
    Namespace='MyApp/Performance',
    MetricData=[{
        'MetricName': 'MyAppRequests',
        'Value': 1,
        'Dimensions': [{'Name': 'Service', 'Value': 'MyService'}],
        'StorageResolution': 60
    }]
)

```

## Extending CDK Infrastructure Templates

For Infrastructure as Code customization, copy [`skills/core-skills/aws-observability/assets/alarm-template.ts`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/assets/alarm-template.ts) into your CDK project and modify the constructs.

Add a latency alarm for your Application Load Balancer:

```typescript
import * as cw from 'aws-cdk-lib/aws-cloudwatch';
import * as cdk from 'aws-cdk-lib';

export class MyObservabilityStack extends cdk.Stack {
  constructor(scope: cdk.App, id: string) {
    super(scope, id);

    const latencyAlarm = new cw.Metric({
      namespace: 'AWS/ELB',
      metricName: 'Latency',
      dimensionsMap: { LoadBalancer: 'my-alb' },
      period: cdk.Duration.minutes(5),
    }).createAlarm(this, 'AlbLatencyHigh', {
      threshold: 0.2,          // seconds
      evaluationPeriods: 3,
    });
  }
}

```

Deploy with `cdk deploy`. The skill references this template when you request latency alarm setup guidance.

## Configuring the ADOT Collector

Customize distributed tracing by editing [`skills/core-skills/aws-observability/assets/otel-config.yaml`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/assets/otel-config.yaml).

Add a custom metric pipeline:

```yaml
receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch:

exporters:
  awscloudwatch:
    namespace: "MyApp/Observability"
    metric_descriptors:
      - name: "request_latency"
        unit: "Seconds"

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [awscloudwatch]

```

Restart the collector (`systemctl restart adot-collector`) to apply changes. The skill will incorporate this configuration into its tracing guidance.

## Summary

- The **Observability skill** loads configuration from plain markdown and YAML files at runtime, eliminating the need for recompilation.
- **Reference files** in `skills/core-skills/aws-observability/references/` control alarm thresholds, metric definitions, and dashboard layouts.
- **Asset helpers** in `skills/core-skills/aws-observability/assets/` provide customizable CDK TypeScript and ADOT YAML templates.
- Changes to [`alarms.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/alarms.md), [`metrics.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/metrics.md), [`alarm-template.ts`](https://github.com/aws/agent-toolkit-for-aws/blob/main/alarm-template.ts), or [`otel-config.yaml`](https://github.com/aws/agent-toolkit-for-aws/blob/main/otel-config.yaml) take effect immediately the next time the skill is invoked.
- The skill manifest at [`skills/core-skills/aws-observability/SKILL.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/SKILL.md) routes requests to these files without requiring code changes to the Agent Core.

## Frequently Asked Questions

### Do I need to rebuild the Agent Toolkit after editing the Observability skill files?

No. The Observability skill reads markdown and configuration files at runtime. Because the skill logic resides in `skills/core-skills/aws-observability/references/*.md` and `assets/` rather than compiled code, your edits are available immediately when the skill is next invoked.

### Can I use my own CDK project instead of the provided alarm templates?

Yes. Copy [`skills/core-skills/aws-observability/assets/alarm-template.ts`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/assets/alarm-template.ts) into your own repository and modify it to suit your infrastructure requirements. The skill will reference your customized version when providing guidance, or you can bypass the skill entirely and deploy your modified CDK stack directly using `cdk deploy`.

### How do I add support for custom business metrics not covered by AWS namespaces?

Edit [`skills/core-skills/aws-observability/references/metrics.md`](https://github.com/aws/agent-toolkit-for-aws/blob/main/skills/core-skills/aws-observability/references/metrics.md) to include your custom metric definitions using Embedded Metric Format (EMF). Define the namespace, dimensions, and units in the markdown, then implement the metric emission in your application code using the AWS SDK or CloudWatch Agent. The skill will include your custom metric in its guidance.

### What is the difference between the reference documentation and asset helpers?

**Reference documentation** (`references/*.md`) contains the narrative guidance and parameter definitions that the skill reads to generate commands and explanations. **Asset helpers** (`assets/`) contain executable code snippets and configuration files (TypeScript CDK constructs, YAML configs) that you can copy into your projects and deploy directly. The skill references both when answering user queries.