Publishing Custom Metrics Using AWS Embedded Metrics Format (EMF): A Complete Guide

AWS Embedded Metrics Format (EMF) lets you publish custom CloudWatch metrics by embedding metric data directly inside structured log events, eliminating per-metric API calls and reducing costs.

The Agent Toolkit for AWS provides a comprehensive reference implementation that defines the EMF JSON schema, operational limits, and best-practice constraints for asynchronous metric publishing. This approach enables Lambda and container workloads to emit metrics via CloudWatch Logs without blocking on network requests.

How EMF Works in the Agent Toolkit

EMF operates by injecting a specialized metadata block into standard JSON log events. According to the Toolkit's reference documentation in skills/core-skills/aws-observability/references/metrics.md, the mechanism relies on four key components:

• Log Event – A JSON line written to CloudWatch Logs containing an _aws metadata block that describes one or more MetricDirectives (namespace, dimensions, and metric definitions).
• _aws.CloudWatchMetrics – An array of metric directives where each directive supports up to 100 metrics and 30 dimensions per set.
• Timestamp – Required field representing milliseconds since epoch. If omitted, CloudWatch uses log ingestion time, but explicit timestamps prevent clock-skew issues.
• Helper Libraries – Language-specific implementations like aws-embedded-metrics and Lambda Powertools that manage JSON serialization, dimension cardinality enforcement, and automatic flushing at ≤5-second intervals.

When CloudWatch Logs receives an EMF-formatted event, it parses the _aws block, extracts the metric datapoints, and surfaces them in your specified custom namespace. Parsing failures are automatically emitted as EMFParsingErrors and EMFValidationErrors metrics in the AWS/Logs namespace, enabling immediate troubleshooting visibility.

Architectural Flow

The metric publishing process follows this asynchronous pipeline:

1. Application code invokes a library method such as metrics.put_metric() or metrics.add_metric().
2. The library constructs the EMF JSON, injects the required _aws metadata block, and writes the structured line to stdout or a configured logger.
3. The CloudWatch Logs agent (or Lambda runtime) forwards the log line to the designated Log Group.
4. CloudWatch extracts the _aws payload and stores the metric datapoints in your custom metric namespace.
5. CloudWatch Alarms, dashboards, and Metric Insights query the published metrics using standard CloudWatch APIs.

Implementation Examples

The Agent Toolkit recommends specific libraries for Node.js and Python that handle the EMF JSON structure, enforce dimensional limits, and manage flush intervals automatically.

Node.js with aws-embedded-metrics

The aws-embedded-metrics library manages the _aws block construction and enforces the 100-metric limit per batch. In Lambda environments, flushing occurs automatically when the function exits; container workloads may call metrics.flush() explicitly.

// Install: npm install aws-embedded-metrics
const { metricScope, Unit } = require('aws-embedded-metrics');

exports.handler = metricScope(metrics => async (event) => {
  // Set low-cardinality dimensions (max 30 per set)
  metrics.setDimensions({ ServiceName: 'OrderService', Environment: 'Prod' });

  // Record custom metrics
  metrics.putMetric('Latency', 120, Unit.Milliseconds);
  metrics.putMetric('RequestCount', 1, Unit.Count);

  console.log('Order processed successfully');
});

Python with Lambda Powertools

Lambda Powertools automatically includes the required Timestamp field and emits the EMF payload when the Lambda function returns. This implementation is documented in the Toolkit's metrics reference alongside the Node.js approach.

# pip install aws-lambda-powertools

from aws_lambda_powertools import Metrics
from aws_lambda_powertools.metrics import MetricUnit

metrics = Metrics(namespace="MyService", service="order")

def lambda_handler(event, context):
    # Configure dimensions (respects 30 dimension limit)

    metrics.set_dimensions({"Environment": "Prod"})
    
    # Add metrics (enforces 100 metric limit per event)

    metrics.add_metric(name="Latency", unit=MetricUnit.Milliseconds, value=85)
    metrics.add_metric(name="RequestCount", unit=MetricUnit.Count, value=1)
    
    # Automatic EMF emission on function exit

    return {"statusCode": 200}

EMF Limits and Constraints

The reference file skills/core-skills/aws-observability/references/metrics.md specifies strict boundaries that prevent CloudWatch API throttling and ensure consistent parsing:

• Maximum 100 metrics per MetricDirective and 30 dimensions per DimensionSet.
• Dimension key length ≤ 250 characters; value length ≤ 1,024 characters.
• Flush interval must remain ≤ 5 seconds; longer intervals risk alarm evaluation delays.
• Cost structure incurs only log ingestion fees, with zero per-metric API charges.

Troubleshooting EMF Failures

When EMF events contain malformed JSON or exceed dimensional limits, CloudWatch Logs emits diagnostic metrics rather than failing silently. Monitor the AWS/Logs namespace for EMFParsingErrors (invalid JSON structure) and EMFValidationErrors (schema violations such as missing namespaces or invalid dimension types).

For detailed debugging guidance, consult skills/core-skills/aws-observability/references/troubleshooting.md in the Agent Toolkit repository. Containerized workloads utilizing the AWS Distro for OpenTelemetry (ADOT) can reference the collector configuration at plugins/aws-core/skills/aws-observability/assets/otel-config.yaml for EMF forwarding setup.

Summary

• AWS Embedded Metrics Format (EMF) publishes custom CloudWatch metrics by embedding them in structured log events, enabling asynchronous, zero-latency metric emission.
• Each EMF log event supports up to 100 metrics and 30 dimensions, with a required _aws metadata block containing the CloudWatchMetrics directive.
• Helper libraries like aws-embedded-metrics and Lambda Powertools automatically handle JSON serialization, dimensional constraints, and the ≤5-second flush interval.
• Parsing failures surface as EMFParsingErrors and EMFValidationErrors in the AWS/Logs namespace, providing immediate visibility into configuration issues.
• Cost optimization is achieved by eliminating PutMetricData API calls, paying only for CloudWatch Logs ingestion.

Frequently Asked Questions

What is the maximum number of metrics per EMF log event?

Each EMF log event supports a maximum of 100 metrics per MetricDirective, as defined in the Agent Toolkit's reference documentation. Additionally, you may define up to 30 dimensions per DimensionSet. Exceeding these limits results in EMFValidationErrors and rejected metric ingestion.

How does EMF differ from the CloudWatch PutMetricData API?

EMF embeds metric data within log events written to CloudWatch Logs, while PutMetricData requires synchronous HTTP API calls. According to the Toolkit's metrics reference, EMF is the default recommendation for Lambda and container workloads because it operates asynchronously, co-locates metrics with logs for correlation, and incurs only log ingestion costs rather than per-metric API charges.

Why are my EMF metrics not appearing in CloudWatch?

Missing metrics typically indicate parsing failures or validation errors. Check the AWS/Logs namespace for EMFParsingErrors (malformed JSON) or EMFValidationErrors (schema violations like missing namespaces or oversized dimension values). Ensure your log events include the required _aws metadata block with a valid Timestamp field in milliseconds since epoch.

What is the recommended flush interval for EMF metrics?

The Agent Toolkit specifies a maximum flush interval of 5 seconds. Exceeding this interval risks missing data points in CloudWatch Alarms, as alarm evaluation windows may complete before metrics arrive. Language-specific libraries like aws-embedded-metrics and Lambda Powertools automatically enforce this constraint.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}