How to List All AWS Glue Jobs Using boto3: A Complete Guide

Use the get_paginator('list_jobs') method from the boto3 Glue client to automatically handle pagination and retrieve every job name across your AWS account and region.

When managing AWS Glue ETL pipelines, you frequently need to inventory existing jobs programmatically. The aws/aws-sdk-js-v3 repository implements the underlying ListJobs API operation in the client-glue package, which boto3 mirrors through botocore to provide Python developers with identical functionality via the list_jobs() method and its corresponding paginator.

Understanding the ListJobs API Architecture

The AWS Glue service exposes the ListJobs operation to return job names within an account. In the JavaScript v3 SDK, this operation is defined in the Smithy model at codegen/sdk-codegen/aws-models/glue.json and implemented in clients/client-glue/src/commands/ListJobsCommand.ts. The SDK also auto-generates a paginator at clients/client-glue/src/pagination/ListJobsPaginator.ts to handle the NextToken pagination pattern.

boto3 consumes the same service definition through botocore, exposing list_jobs() as a client method that accepts a NextToken parameter and returns a response containing JobNames and a NextToken for subsequent pages.

Listing All Glue Jobs with boto3

Using the Built-in Paginator (Recommended)

The most robust approach uses boto3's paginator, which automatically manages the NextToken across multiple API calls:

import boto3

glue = boto3.client('glue')
paginator = glue.get_paginator('list_jobs')

all_job_names = [
    job_name
    for page in paginator.paginate()
    for job_name in page.get('JobNames', [])
]

print(f"Found {len(all_job_names)} Glue jobs")
print(all_job_names)

This implementation mirrors the pagination logic found in ListJobsPaginator.ts, iterating until no NextToken remains in the response.

Manual Pagination Without Paginator

For scenarios requiring explicit control over API calls, handle the NextToken manually:

import boto3

glue = boto3.client('glue')
next_token = None
all_job_names = []

while True:
    kwargs = {'NextToken': next_token} if next_token else {}
    response = glue.list_jobs(**kwargs)
    
    all_job_names.extend(response.get('JobNames', []))
    next_token = response.get('NextToken')
    
    if not next_token:
        break

print(f"Retrieved {len(all_job_names)} jobs")

This approach directly corresponds to the raw request/response handling implemented in ListJobsCommand.ts.

Retrieving Full Job Definitions

The list_jobs() operation returns only job names. To obtain complete job metadata—including script locations, default arguments, and connections—you must call batch_get_jobs() or get_job():

Fetching Complete Job Definitions

import boto3

glue = boto3.client('glue')
paginator = glue.get_paginator('list_jobs')

all_jobs = []
for page in paginator.paginate():
    job_names = page.get('JobNames', [])
    if job_names:
        # Batch retrieval is more efficient than individual get_job calls

        response = glue.batch_get_jobs(JobNames=job_names)
        all_jobs.extend(response.get('Jobs', []))

print(f"Retrieved full details for {len(all_jobs)} jobs")
for job in all_jobs:
    print(f"{job['Name']}: {job.get('Command', {}).get('ScriptLocation')}")

This pattern minimizes API calls by using batch_get_jobs (implemented in BatchGetJobsCommand.ts) rather than iterating with individual get_job requests.

Filtering and Searching Jobs

When working with numerous ETL pipelines, filter results client-side using Python's pattern matching:

import boto3
import fnmatch

glue = boto3.client('glue')
paginator = glue.get_paginator('list_jobs')

pattern = 'etl_*_daily'  # Match jobs starting with "etl_" and ending with "_daily"

filtered_jobs = [
    name for page in paginator.paginate()
    for name in page.get('JobNames', [])
    if fnmatch.fnmatch(name, pattern)
]

print(f"Found {len(filtered_jobs)} jobs matching '{pattern}':")
print(filtered_jobs)

This approach leverages the complete job inventory retrieved via the paginator while applying business-specific naming conventions to isolate relevant ETL workflows.

Summary

• Use the paginator: The get_paginator('list_jobs') method automatically handles the NextToken pagination pattern, ensuring you retrieve every job across large inventories without manual loop logic.
• List returns names only: The list_jobs() API returns only job names; call batch_get_jobs() to retrieve full job definitions including script locations, arguments, and configuration.
• Mirror of JS SDK: boto3 implements the same underlying service model as the aws-sdk-js-v3 repository, specifically the ListJobs operation defined in codegen/sdk-codegen/aws-models/glue.json and implemented in clients/client-glue/src/commands/ListJobsCommand.ts.
• Filter client-side: Since the API does not support server-side filtering by name pattern, use Python's fnmatch or list comprehensions to filter results after retrieval.

Frequently Asked Questions

How do I list Glue jobs across multiple AWS regions?

You must create separate boto3 clients for each region and invoke list_jobs() (or the paginator) for each client instance. The AWS Glue ListJobs API is regional, and there is no cross-region aggregation feature in the SDK.

Why does list_jobs only return job names instead of full metadata?

The ListJobs operation is optimized for inventory retrieval and returns only the JobNames list to minimize payload size and improve latency. This design follows the AWS API best practice of separating list operations from detailed get operations. To retrieve full job definitions—including script locations, default arguments, and connections—use the batch_get_jobs() method with the names retrieved from list_jobs().

What is the maximum number of jobs returned per list_jobs API call?

AWS Glue typically returns up to 100 job names per ListJobs API response, though this limit is subject to change by AWS. When the result set exceeds this limit, the response includes a NextToken that you must pass to subsequent calls. The boto3 paginator abstracts this pagination logic automatically, ensuring complete retrieval regardless of the per-page limit.

Can I filter Glue jobs by creation date or status using list_jobs?

No, the ListJobs API does not support server-side filtering by creation date, job status, or other metadata attributes. The operation accepts only the NextToken and MaxResults parameters. To filter by date or status, retrieve all job names using the paginator, then call batch_get_jobs() to fetch full metadata, and finally filter the results client-side using Python logic (e.g., checking the CreatedOn or LastModifiedOn fields in the job definitions).

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}