# AI Coding Agents for Ruby on Rails: Benchmarks and Best Practices

> What the Agents on Rails benchmarks reveal about AI-generated Rails code, and how to review, test, and secure coding agents in real applications.

- Canonical: https://haseebeqx.com/posts/ai-coding-agents-ruby-on-rails/
- Published: 2026-09-18


AI coding agents can inspect a Rails repository, edit several files, run tests, interpret failures, and revise their work. That makes them more useful than autocomplete, but it also makes their mistakes harder to see. A generated method can look reasonable while missing a migration, using the wrong Rails abstraction, introducing an N+1 query, or implementing only the visible part of a feature.

The Rails Foundation's [Agents on Rails benchmark](https://rubyonrails.org/ai) gives us useful evidence about that gap. In its first stage, the leading model solved 58 of 63 runs, or 92%, on small Rails tasks. In the feature-sized second stage, the leading result fell to 21 of 60 runs, or 35%.

That difference is more useful than any model ranking. It suggests that strong agents can often fix a localized Rails problem, but delivering a complete feature across models, controllers, migrations, jobs, views, and Hotwire remains unreliable.

This article explains what the benchmark measures, why Rails is a revealing environment for coding agents, and how to use those agents without allowing a green test or a confident summary to stand in for engineering judgment.



## What an AI coding agent does in a Rails application

A coding agent does more than produce a snippet. Given access to a repository and tools, it may:

- inspect models, controllers, routes, views, jobs, mailers, and tests;
- infer project conventions from nearby code;
- edit multiple application layers;
- generate and run migrations;
- execute focused tests and the complete test suite;
- react to failures and revise its implementation; and
- return a patch or pull request with a summary.

This work combines four different capabilities:

1. **Repository comprehension:** finding how this particular application is organized.
2. **Rails knowledge:** recognizing the framework API or convention that fits the problem.
3. **Feature implementation:** changing every required layer coherently.
4. **Verification:** demonstrating that the requested behavior works without regressions.

An agent may be good at one and weak at another. It can understand a ticket but fail to recall `generates_token_for`. It can know an Active Job API but enqueue at the wrong point in a transaction. It can make the visible page work while forgetting a data backfill or an authorization boundary.

Treating all of this as "code generation" hides the difficult parts.

## Why Rails is a useful test for coding agents

Rails gives an agent a predictable map. Models normally live in `app/models`, routes are centralized, migrations follow established naming patterns, and commands such as `bin/rails test` work across conventional applications. Associations connect class and table names, while generators provide standard starting points.

That predictability narrows the search space. An agent does not need to rediscover the application's basic structure before every change.

But Rails also makes superficial correctness easier to expose. A small declaration can control transaction timing, dependent deletion, validation, broadcasting, caching, or job retries. Two implementations can produce the same output in one test while differing substantially in security and maintenance cost.

For example, an agent might:

- build a custom token mechanism instead of using a signed Rails token;
- concatenate SQL instead of using a safe query API;
- issue a query inside a collection loop instead of preloading;
- broadcast without the correct record or tenant scope;
- enqueue a job before the surrounding transaction commits; or
- implement authentication from scratch when the current Rails generator already provides the foundation.

The Ruby can be valid and the happy path can pass while the implementation is still poor Rails code. Rails is therefore easier for an agent to navigate, but harder to imitate well without understanding the framework.

## What the Agents on Rails benchmark measures

The Rails Foundation and Evil Martians built Agents on Rails around real Rails applications rather than isolated code exercises. The [benchmark methodology](https://github.com/rails/ai-evals/blob/main/methodology.md), task definitions, reference patches, verification tests, and run artifacts are public.

Every model uses the same minimal agent and harness. Runs occur in a sandbox, and grading is deterministic rather than performed by another language model. Before verification, protected test surfaces are restored to their original state. A run must keep the application's own test suite green and pass hidden task-specific checks.

The published scores are snapshots, not permanent rankings. Model versions, prices, and capabilities will change. The useful evidence is how performance changes with the size and ambiguity of the work.

### Stage 1: small, focused Rails tasks

[Stage 1](https://rubyonrails.org/2026/8/13/agents-on-rails-the-first-benchmark-report) contains 21 tasks against Basecamp's Writebook application. Each model gets three attempts per task, producing 63 runs. The tasks cover small features, bug fixes, performance problems, security hardening, and a flaky test suite.

Each task is designed around a particular Rails API, but its description does not name that API. The verifier checks behavior, while Rails API recall is measured separately.

The leading result in the initial report was **58 successful runs out of 63, or 92%**. That is impressive, but it needs context: these were deliberately atomic tasks with a relatively clear boundary.

The more revealing result concerned framework knowledge. Initial Rails API recall ranged from **8% to 35%**. Agents often produced their own implementation when Rails already provided an answer.

The report found that:

- runs that recalled the intended Rails API passed 92% of the time;
- hand-written alternatives passed 87% of the time; and
- runs that encountered the API but chose another approach passed 64% of the time.

These figures show correlation within this task set, not proof that calling a Rails API always causes success. They still expose a maintenance problem: passing custom code becomes application code that the team must understand, test, and support.

An additional experiment supplied richer Rails documentation. API recall improved and agents hand-rolled less code, but the overall success score barely changed. Better context helped agents choose a more idiomatic implementation; it did not solve every reasoning problem.

### Stage 2: complete Rails features

[Stage 2](https://rubyonrails.org/2026/9/9/agents-on-rails-stage-2) contains 20 feature tickets against Fizzy, 37signals' kanban application. Again, each model receives three attempts, for 60 runs.

The tasks include card reactions, email-code sign-in, a Kamal deployment, an account-ID rollout and backfill, Japanese localization, and per-person email digests. A feature can cross models, controllers, views, jobs, mailers, migrations, and Hotwire behavior.

Each ticket also includes an implicit requirement that an experienced Rails developer should consider even when the ticket does not spell it out. Examples include an atomic cascade that cannot leave orphaned records, a cache key that varies by timezone, and a preload needed to prevent an N+1 query.

The leading model solved **21 of 60 runs, or 35%**. Other models ranged from 30% down to 0% in this snapshot.

The benchmark report describes a repeated pattern: agents implemented the happy path but missed edge cases, skipped migrations or the full suite, or stopped without a reliable definition of completeness. In the Japanese localization task, many runs translated the visible surfaces and saw green tests, but only one run met the verifier's complete requirement.

That is the practical limit to remember:

| Type of work | Leading result | What it tests |
|---|---:|---|
| Atomic Rails tasks | 58/63 runs, 92% | Localized fixes and framework knowledge |
| Feature-sized Rails tasks | 21/60 runs, 35% | Planning, integration, completeness, and edge cases |

The percentages should not be compared as if the stages were identical experiments; they use different applications and task corpora. The large decline nevertheless demonstrates that success on small patches does not establish reliable feature delivery.

## Passing tests is not the same as writing good Rails code

A test suite answers only the questions encoded in it. If it checks the visible happy path, an agent can optimize for that path and leave the rest unfinished.

Review AI-generated Rails code across at least three dimensions:

| Dimension | Question | Evidence |
|---|---|---|
| Behavioral correctness | Does the requested behavior work? | Focused, regression, and system tests |
| Rails quality | Does it use suitable framework APIs and local conventions? | Diff review, static analysis, and API review |
| Production readiness | Is it secure, operable, and safe to deploy? | Migration, performance, security, and operational checks |

A green suite is strongest on the first dimension. It says much less about the other two unless the project has deliberately encoded them.

### Framework reinvention

Generated code should be checked against the current Rails version. Models learn from years of public examples, so they may reproduce older patterns or invent facilities now supplied by Rails.

Look for custom implementations of:

- authentication and password-reset primitives;
- signed or expiring tokens;
- background-job continuation and enqueue timing;
- rate limiting;
- cache keys and broadcast scoping;
- query sanitization; and
- association lifecycle behavior.

Using a framework API is not automatically correct, but unnecessary custom code increases the surface the application owns.

### Database and performance mistakes

Agents can produce functionally correct queries that become expensive at realistic data volumes. Review:

- query count inside loops and serializers;
- missing preloads;
- `count`, `exists?`, and association calls repeated per record;
- indexes needed by new access patterns;
- cache keys that omit locale, viewer, timezone, or tenant context; and
- migrations that lock large tables or mix schema changes with long backfills.

Do not accept a performance claim without a repeatable measurement. I use a separate [before-and-after Rails benchmarking workflow](/posts/benchmarking-rails-performance-changes/) for changes that claim to make a request faster. For repeated database access, the patterns in [How to Fix N+1 Queries in Rails](/posts/how-to-fix-n-plus-one-queries-in-rails/) are useful review targets.

### Security mistakes

Rails defaults help only when generated code stays within them. Pay special attention to:

- authorization and tenant scope;
- dynamic SQL;
- unsafe HTML output;
- strong parameters;
- session and cookie behavior;
- secret handling and logging;
- destructive operations; and
- changes that disable Brakeman, dependency audits, or warnings.

Authentication, authorization, billing, privacy, and infrastructure changes should receive specialist review regardless of how confidently the agent reports completion.

## Best practices for using AI coding agents with Rails

The benchmark used a deliberately minimal agent to make model comparisons fair. A production workflow should supply more context and stronger controls. The objective is not to reproduce the benchmark; it is to make useful work safer.

### 1. Give the agent a bounded task

Agents are more dependable when the task has a clear operation and completion condition. Prefer:

> Add rate limiting to password-reset requests, preserve the existing response, and add tests for both accepted and rejected requests.

over:

> Improve authentication security.

For a larger feature, ask the agent to identify affected layers and assumptions before editing. A useful plan should mention routes, persistence, authorization, migrations, jobs, user-visible states, tests, and deployment concerns where applicable.

A plan does not guarantee a complete implementation, but it makes missing surfaces visible before they become a polished patch.

### 2. Document application-specific Rails conventions

Create a concise repository guide that tells an agent:

- the supported Ruby and Rails versions;
- setup, focused-test, and full-CI commands;
- the test framework and fixture or factory policy;
- authorization and tenant-scoping patterns;
- boundaries between controllers, models, jobs, and service objects;
- migration and backfill rules;
- style and security commands; and
- changes that always require human approval.

Point to representative files instead of pasting the entire repository into instructions. Local examples are often more useful than generic advice because they show how this application applies Rails conventions.

### 3. Use current, version-matched Rails documentation

The Stage 1 context experiment suggests that better Rails documentation can improve framework API selection even when it does not fix every reasoning failure.

In production, allow retrieval from trusted, version-matched sources where possible: the application's code, the correct Rails guides and API documentation, approved gem documentation, and architecture decisions. Do not let arbitrary internet access become a substitute for controlled context.

### 4. Build a layered verification loop

Run the fastest relevant check first, then broaden the evidence:

1. focused model, controller, job, or integration tests;
2. tests for neighboring behavior;
3. the complete regression suite;
4. system tests for browser and Hotwire interactions;
5. RuboCop and project-specific cops;
6. Brakeman and dependency audits;
7. migration and backfill checks; and
8. performance or query-count checks when the change affects data access.

Rails 8.1 provides `config/ci.rb` and `bin/ci`, which can give both developers and agents one conventional completion command. The [Rails 8.1 announcement](https://rubyonrails.org/2025/10/22/rails-8-1) demonstrates a CI definition combining setup, RuboCop, Bundler Audit, Importmap Audit, Brakeman, Rails tests, and seed validation.

A command such as `bin/ci` is more reliable than a natural-language instruction to "make sure everything is correct." The instruction guides behavior; the protected check enforces a minimum standard.

### 5. Do not let the agent control its verifier

An agent that can edit the tests, CI scripts, or security configuration used to judge its work can accidentally or intentionally make a broken change appear successful.

The open-source [`lemans` evaluation harness](https://github.com/rails/lemans) addresses this by restoring protected paths such as `test/`, `bin/`, and the test-environment configuration from a pre-agent snapshot before grading. Hidden checks are introduced only during verification.

The same principle belongs in normal development:

- run trusted checks outside the agent's writable workspace where feasible;
- separately review changes to tests and CI configuration;
- prevent an ordinary application patch from disabling required checks;
- capture commands, exit statuses, and logs rather than accepting "tests pass" as a claim; and
- require approval for changes to the policy that decides whether code can merge.

The worker should not own its judge.

### 6. Restrict credentials, commands, and network access

A coding agent with shell access is a privileged automation system. Repository text, issue descriptions, dependencies, or generated files can contain instructions that should not be trusted.

Use:

- an ephemeral sandbox;
- short-lived, least-privilege credentials;
- no production database or deployment access by default;
- network allowlists;
- approval for destructive commands;
- secret scanning and output redaction;
- branch protection and human merge approval; and
- complete tool and command logs.

`lemans` restricts benchmark egress to the model provider and filters known provider credentials from stored artifacts. That setup is intentionally stricter than most development environments, but it demonstrates the controls a capable agent may require.

### 7. Require evidence in the final report

Ask the agent to report:

- files changed and why;
- tests and checks run, with outcomes;
- migrations, backfills, and rollback implications;
- security and privacy considerations;
- Rails APIs considered;
- assumptions made about ambiguous requirements; and
- anything that still needs human verification.

This report is not proof. It is a review index. Compare it with the actual diff and logs, and treat omissions as reasons to investigate.

## Which Rails tasks are suitable for coding agents?

Current evidence supports a graduated approach rather than full autonomy.

### Good starting points

- repository exploration and explanation;
- a small bug with a reproducible failing test;
- conventional CRUD changes;
- focused test generation for understood behavior;
- repetitive updates across consistent files;
- documentation and pull-request summaries; and
- small refactors protected by a fast suite.

### Tasks that need close review

- multi-layer features;
- migrations and data backfills;
- jobs involving retries, idempotency, or transaction timing;
- query and cache changes;
- Hotwire behavior requiring browser verification;
- authentication and authorization; and
- broad changes in legacy applications with inconsistent conventions.

### Tasks that should retain explicit approval gates

- production deployment;
- access to production data;
- destructive schema or data operations;
- billing and privacy behavior;
- credential or infrastructure changes; and
- modifications to CI, tests, or security policy.

The boundary should depend on measured performance in your application, not a public leaderboard alone.

## How to evaluate a coding agent on your own Rails application

Public benchmarks tell you what happened on Writebook and Fizzy under a fixed harness. They do not tell you how an agent will perform in a legacy monolith, with your gems, conventions, tests, and deployment constraints.

Build a small private evaluation set from historical or synthetic tasks:

- localized bug fixes;
- N+1 and query-performance issues;
- migrations and backfills;
- jobs with transaction boundaries;
- authorization and tenant isolation;
- mailers and timezone behavior;
- Hotwire interactions; and
- complete feature tickets.

Run each task more than once because agent output varies. Preserve the repository revision, prompt, model settings, patch, trajectory, test logs, duration, and cost.

Score separate outcomes instead of reducing quality to one pass bit:

1. functional behavior;
2. regression safety;
3. Rails API and convention use;
4. security;
5. query and performance behavior;
6. migration and deployment safety;
7. human review or repair time; and
8. cost and latency.

Keep some tasks private and rotate them. The Agents on Rails corpus is open, which makes it inspectable but also creates a growing risk that future models have encountered the tasks or solutions during training.

The metric that matters is not generated lines or attempted tickets. It is reviewed, secure, maintainable changes merged per engineer-hour, adjusted for defects that appear later.

## What the benchmarks do not prove

The Agents on Rails project provides unusually inspectable evidence, but its limits matter:

- it is not a controlled comparison between Rails and other frameworks;
- results can change with a different agent, prompt, toolset, or retrieval system;
- three attempts per task leave sampling uncertainty;
- passing hidden tests does not guarantee production readiness;
- public tasks may eventually contaminate model training data;
- network-isolated model knowledge differs from a production workflow with trusted documentation; and
- results on two conventional applications do not automatically generalize to a legacy codebase.

Do not turn "Rails gives agents a map" into the stronger claim that Rails has been proven to be the best framework for AI coding. That experiment has not been run here.

## Conclusion

AI coding agents are already useful in Ruby on Rails development, particularly for repository discovery, focused fixes, repetitive changes, and tasks with strong deterministic feedback. The Agents on Rails benchmark also shows why that usefulness should not be confused with autonomous feature delivery.

The leading result fell from 92% of runs on atomic tasks to 35% on feature-sized work. Agents often implemented the visible path while missing implicit requirements, framework APIs, migrations, edge cases, or complete integration. Even when behavior passed, many solutions recreated capabilities Rails already supplied.

Rails conventions help agents navigate a project and give teams a common structure for review. Rails' semantic depth then makes good verification essential. Give agents bounded work, current framework context, fast tests, protected CI, narrow permissions, and human review proportional to risk.

Use an agent to accelerate engineering work, not to remove the evidence and judgment that make the work safe.

## Sources

- [Agents on Rails](https://rubyonrails.org/ai)
- [Agents on Rails: the first benchmark report](https://rubyonrails.org/2026/8/13/agents-on-rails-the-first-benchmark-report)
- [Agents on Rails: Stage 2. Can a model ship a feature?](https://rubyonrails.org/2026/9/9/agents-on-rails-stage-2)
- [Agents on Rails: `lemans` goes open source](https://rubyonrails.org/2026/8/24/agents-on-rails-lemans)
- [`rails/ai-evals` methodology](https://github.com/rails/ai-evals/blob/main/methodology.md)
- [`rails/lemans`](https://github.com/rails/lemans)
- [Rails 8.1: Job continuations, structured events, local CI](https://rubyonrails.org/2025/10/22/rails-8-1)