How to Customize the Observability Skill for Your AWS Agent

You can customize the Observability skill by editing the markdown reference files and asset templates in the aws-agent-toolkit-for-aws repository, with changes taking effect immediately at runtime without requiring recompilation.

The Observability skill in the aws/agent-toolkit-for-aws repository provides AWS agents with guided capabilities for CloudWatch alarms, custom metrics, dashboards, and distributed tracing. Unlike compiled skills, this implementation loads its logic from plain markdown and configuration files, allowing you to customize thresholds, add new metrics, and modify infrastructure templates by editing the source files directly.

Architecture of the Observability Skill

The skill is structured in four distinct layers that separate declarative configuration from execution:

• Skill Manifest: Defined in skills/core-skills/aws-observability/SKILL.md (lines 19-26), this file declares the skill name (aws-observability) and the routing table that maps user requests to specific reference documents.
• Reference Documentation: Located in skills/core-skills/aws-observability/references/*.md, these markdown files contain version-controlled guidance for alarms, metrics, dashboards, and tracing that the skill reads at runtime.
• Asset Helpers: Found in skills/core-skills/aws-observability/assets/, these include alarm-template.ts for CDK snippets and otel-config.yaml for ADOT collector configurations.
• Execution Engine: The Agent Core loads reference files on demand and executes AWS CLI commands through the MCP server, while the skill itself only guides users to edit files or run supplied snippets.

How Customization Works

To customize the observability skill for your specific AWS environment, follow these steps:

1. Identify the sub-area you want to modify—alarms, metrics, dashboards, or tracing.
2. Edit the corresponding reference file in the references/ directory or copy an asset helper into your own repository.
3. Adjust values such as alarm thresholds, metric dimensions, or CDK constructs to match your requirements.
4. Invoke the skill again—the Agent Toolkit loads files at runtime, so changes apply immediately without recompilation.

Customizing Alarm Thresholds

The default alarm definitions live in skills/core-skills/aws-observability/references/alarms.md. To change a threshold, edit the markdown directly.

For example, to lower the CPU utilization alarm from 80% to 70%:

# CPU Utilization Alarm

- **Metric:** AWS/EC2 – CPUUtilization
- **Threshold:** 70   <!-- changed from 80 -->
- **Period:** 300
- **EvaluationPeriods:** 2
- **ComparisonOperator:** GreaterThanThreshold

When you invoke the skill, it generates the corresponding AWS CLI command using your updated threshold:

aws cloudwatch put-metric-alarm \
  --alarm-name MyLambdaCPUHigh \
  --metric-name CPUUtilization \
  --namespace AWS/EC2 \
  --threshold 70 \
  --period 300 \
  --evaluation-periods 2 \
  --comparison-operator GreaterThanThreshold

Adding Custom Metrics with EMF

To add application-specific metrics, edit skills/core-skills/aws-observability/references/metrics.md with Embedded Metric Format (EMF) definitions.

Append a new custom metric definition:

## Custom metric: MyAppRequests

- **Namespace:** MyApp/Performance
- **Dimensions:** Service=MyService
- **Unit:** Count
- **EMF example:**

```json
{
  "_aws": {
    "Timestamp": 1625247600000,
    "CloudWatchMetrics": [{
      "Namespace": "MyApp/Performance",
      "Dimensions": [["Service"]],
      "Metrics": [{"Name": "MyAppRequests"}]
    }]
  },
  "Service": "MyService",
  "MyAppRequests": 123
}

Then emit the metric from your application code:

```python
import json, time, boto3

client = boto3.client('cloudwatch')
payload = {
    "_aws": {
        "Timestamp": int(time.time() * 1000),
        "CloudWatchMetrics": [{
            "Namespace": "MyApp/Performance",
            "Dimensions": [["Service"]],
            "Metrics": [{"Name": "MyAppRequests"}]
        }]
    },
    "Service": "MyService",
    "MyAppRequests": 1
}
client.put_metric_data(
    Namespace='MyApp/Performance',
    MetricData=[{
        'MetricName': 'MyAppRequests',
        'Value': 1,
        'Dimensions': [{'Name': 'Service', 'Value': 'MyService'}],
        'StorageResolution': 60
    }]
)

Extending CDK Infrastructure Templates

For Infrastructure as Code customization, copy skills/core-skills/aws-observability/assets/alarm-template.ts into your CDK project and modify the constructs.

Add a latency alarm for your Application Load Balancer:

import * as cw from 'aws-cdk-lib/aws-cloudwatch';
import * as cdk from 'aws-cdk-lib';

export class MyObservabilityStack extends cdk.Stack {
  constructor(scope: cdk.App, id: string) {
    super(scope, id);

    const latencyAlarm = new cw.Metric({
      namespace: 'AWS/ELB',
      metricName: 'Latency',
      dimensionsMap: { LoadBalancer: 'my-alb' },
      period: cdk.Duration.minutes(5),
    }).createAlarm(this, 'AlbLatencyHigh', {
      threshold: 0.2,          // seconds
      evaluationPeriods: 3,
    });
  }
}

Deploy with cdk deploy. The skill references this template when you request latency alarm setup guidance.

Configuring the ADOT Collector

Customize distributed tracing by editing skills/core-skills/aws-observability/assets/otel-config.yaml.

Add a custom metric pipeline:

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch:

exporters:
  awscloudwatch:
    namespace: "MyApp/Observability"
    metric_descriptors:
      - name: "request_latency"
        unit: "Seconds"

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [awscloudwatch]

Restart the collector (systemctl restart adot-collector) to apply changes. The skill will incorporate this configuration into its tracing guidance.

Summary

• The Observability skill loads configuration from plain markdown and YAML files at runtime, eliminating the need for recompilation.
• Reference files in skills/core-skills/aws-observability/references/ control alarm thresholds, metric definitions, and dashboard layouts.
• Asset helpers in skills/core-skills/aws-observability/assets/ provide customizable CDK TypeScript and ADOT YAML templates.
• Changes to alarms.md, metrics.md, alarm-template.ts, or otel-config.yaml take effect immediately the next time the skill is invoked.
• The skill manifest at skills/core-skills/aws-observability/SKILL.md routes requests to these files without requiring code changes to the Agent Core.

Frequently Asked Questions

Do I need to rebuild the Agent Toolkit after editing the Observability skill files?

No. The Observability skill reads markdown and configuration files at runtime. Because the skill logic resides in skills/core-skills/aws-observability/references/*.md and assets/ rather than compiled code, your edits are available immediately when the skill is next invoked.

Can I use my own CDK project instead of the provided alarm templates?

Yes. Copy skills/core-skills/aws-observability/assets/alarm-template.ts into your own repository and modify it to suit your infrastructure requirements. The skill will reference your customized version when providing guidance, or you can bypass the skill entirely and deploy your modified CDK stack directly using cdk deploy.

How do I add support for custom business metrics not covered by AWS namespaces?

Edit skills/core-skills/aws-observability/references/metrics.md to include your custom metric definitions using Embedded Metric Format (EMF). Define the namespace, dimensions, and units in the markdown, then implement the metric emission in your application code using the AWS SDK or CloudWatch Agent. The skill will include your custom metric in its guidance.

What is the difference between the reference documentation and asset helpers?

Reference documentation (references/*.md) contains the narrative guidance and parameter definitions that the skill reads to generate commands and explanations. Asset helpers (assets/) contain executable code snippets and configuration files (TypeScript CDK constructs, YAML configs) that you can copy into your projects and deploy directly. The skill references both when answering user queries.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}