# How to Benchmark a Rails Performance Fix > A practical workflow for turning a slow Rails request into a repeatable before-and-after benchmark with results that developers and stakeholders can trust. - Canonical: https://haseebeqx.com/posts/benchmarking-rails-performance-changes/ - Published: 2026-09-18 When I optimize a slow Rails endpoint, I create a small `profile_compare` script for that specific workload. The script does six things: 1. runs the slow request against a known baseline revision; 2. warms the application before collecting results; 3. records request duration and the metrics related to the suspected problem; 4. runs the same workload against the proposed fix; 5. verifies that both responses still contain the expected data; and 6. produces before-and-after tables for review. This gives me a fast feedback loop while testing fixes and a defensible result to share with maintainers or stakeholders. I do not reuse one universal benchmarking script: each performance problem needs a workload and measurements that reflect its cause. At a high level, the script looks like this: ```ruby # Pseudocode: the helpers depend on the application. baseline = resolve_commit(BASELINE_REF) candidate = resolve_commit(CANDIDATE_REF) before = run_profile(revision: baseline, workload: :products_index) after = run_profile(revision: candidate, workload: :products_index) validate_comparable!(before, after) write_markdown_report(before:, after:, baseline:, candidate:) ``` The important part is not this wrapper. It is how `run_profile` creates representative data, warms each revision, measures the operation, and checks its result. The snippets below are intentionally pseudocode rather than reusable project scripts. ## What the benchmark should produce Here is the kind of result this workflow produced for the [Gumroad Products API](/case-studies/gumroad-products-api-n-plus-one/): | Metric | Before mean | After mean | Change | |---|---:|---:|---:| | Request duration | 162.36 ms | 96.70 ms | **−40.4%** | | SQL duration | 29.30 ms | 8.98 ms | **−69.4%** | | Allocated objects | 58,396 | 44,762 | **−23.3%** | | SQL events | 103 | 23 | **−77.7%** | | Per-product target queries | 70 | 0 | **−100%** | The request timing communicates the user-facing improvement. The query counts explain why it improved. Both are needed: timing alone is noisy, while a lower query count does not necessarily mean the complete request became faster. The rest of this article explains how to build a comparison that can produce results like these without publishing the project-specific scripts themselves. ## 1. Start with an observed performance issue Performance work should begin with evidence that a real operation is not performing well enough. The signal might come from: - an APM alert or a regression in endpoint latency; - timeouts, failed requests, or exhausted workers; - user or support complaints about a slow workflow; - a request missing an agreed response-time target or service-level objective; or - a background job taking longer than its processing window. Capture the context before opening the profiler. Which route or job is affected? Is the problem typical latency, a high percentile such as p95, or occasional failures? Does it affect every account or only customers with a particular data volume? Did it begin after a deployment? Turn that observation into a narrow statement. For example: > APM shows that the products endpoint exceeds the agreed response time for sellers with large, fully populated catalogs. This gives the investigation a boundary and a success condition. It also prevents optimizing code that looks inefficient but is not responsible for the reported problem. I capture that information in a short benchmark brief before writing the comparison: ```yaml # Pseudocode benchmark brief source: APM latency alert operation: GET /api/v2/products observed: p95 exceeds the agreed target for large catalogs affected_shape: seller with populated products and variants success: lower request duration without changing the response ``` The brief is small, but it keeps the benchmark connected to the original issue. ## 2. Define the exact workload Translate the observed issue into one repeatable operation, not “the Rails application.” Good benchmark targets include: - `GET /api/v2/products` for a seller with ten populated products; - an admin page rendering 50 users; - a search request returning ten work packages; or - a background job processing a batch of 100 records. Use the same route, authentication, parameters, permissions, and data volume that exposed the problem. If the production complaint concerns large accounts, benchmarking an account with two empty records will exercise a different workload. The shape of the data matters as much as its size. To reproduce a serialization N+1, each parent record may need distinct associated records. Reusing one product, user, or variant across the entire fixture can let Active Record’s association cache hide repeated lookups. Before measuring anything, make the benchmark verify its setup. For an API response, that might mean checking: - the request succeeded; - the expected number of records was returned; - required nested fields are present; and - values affected by the optimization remain correct. Otherwise, an error response or empty result can appear to be a major performance improvement. A workload definition might look like this: ```ruby # Pseudocode: setup is outside the measured block. seller = create_seller 10.times do product = create_product(seller: seller) create_distinct_price_file_thumbnail_and_variant(product) end workload = -> { authenticated_get("/api/v2/products", as: seller) } validate = ->(response) { assert response.success? assert response.json["products"].size == 10 assert response.json["products"].all? { |product| product["variants"].any? } } ``` The validation runs for both revisions. The timer should cover the request, not fixture creation or response assertions. ## 3. Profile before choosing metrics The benchmark script should come after the first profiling pass. Use an ordinary Rack Mini Profiler result to inspect overall server duration, SQL duration, and executed queries. Use a flamegraph when you need to find the expensive call path through a controller, serializer, view, or service. If one query is slow, inspect its execution plan. If Ruby work dominates, investigate CPU time or allocations instead of assuming the database is responsible. The profiler should lead to a specific hypothesis: > Serializing each product separately loads prices, files, thumbnails, and variants. Or: > Resolving a work package’s type variant repeatedly queries the same project/type combination. Once the cause is specific, the benchmark can measure it directly. Do not use flamegraph requests as the final timing runs. Sampling introduces overhead, and Rack Mini Profiler does not provide its normal SQL timings during a flamegraph request. Use flamegraphs to find the path; use ordinary repeated requests to score the change. ## 4. Measure the result and the mechanism I normally collect metrics at three levels. ### Overall result Measure the operation the user waits for: - complete request duration; - service duration; or - job duration. This is the number stakeholders care about most. ### Supporting cost Add a metric for the resource identified during profiling: - SQL duration for database-heavy work; - allocated objects for object-heavy serialization; - cached versus non-cached SQL events; or - duration of a specific processing phase. Not every benchmark needs every metric. A focused report is easier to trust than a table full of unrelated numbers. ### Targeted cause Count the work the proposed fix is intended to remove: - total SQL events; - duplicate query fingerprints; - queries against selected tables; - per-record association lookups; or - calls to an expensive service. These counts help distinguish a real improvement from timing variance. In the [OpenProject work-packages case](/case-studies/openproject-work-packages-type-variant-n-plus-one/), median server duration fell by 55.7%, but the clearest evidence was that two query patterns fell from 381 executions each to zero. Normalize SQL before counting duplicate fingerprints so differences in IDs and literals do not make the same query shape look unique. Also separate cached SQL events when the query cache is involved. A cached query still represents repeated application work even though Rails avoided another database round trip. The measurement part of `run_profile` can be sketched like this: ```ruby # Pseudocode: use a monotonic clock in the real implementation. def measure(workload, validate:) sql_events = [] allocations_before = allocated_objects response = nil duration = nil subscribe_to("sql.active_record", record_into: sql_events) do duration = elapsed_time { response = workload.call } end validate.call(response) sql_events.reject!(&:schema_query?) fingerprints = sql_events.map { |event| normalize_sql(event.sql) } { duration_ms: duration.in_milliseconds, sql_duration_ms: sql_events.sum(&:duration_ms), sql_events: sql_events.size, duplicate_fingerprints: fingerprints.size - fingerprints.uniq.size, target_queries: sql_events.count { |event| target_query?(event.sql) }, allocations: allocated_objects - allocations_before } end ``` `target_query?` is deliberately specific to the hypothesis. For a product serialization issue, it may count price, file, thumbnail, and variant lookups. For a CPU-bound service, it might be replaced with a phase timer or call counter. ## 5. Compare the correct Git revisions A before-and-after result is only useful when the revisions differ by the change being evaluated. Choose the baseline based on the question: | Question | Baseline | |---|---| | Will this release improve production? | The currently deployed commit | | Did this optimization work? | The commit immediately before the optimization | | Does this pull request improve upstream? | The upstream commit on which the change is based | | Which proposed fix is best? | One pinned baseline used for every candidate | Do not rely only on a moving name such as `main` or `dev`. Resolve it to a commit SHA before the run and include that SHA in the report. If the branch contains unrelated work, comparing it with the latest `main` measures all of those differences, not just the optimization. ### Worktree, not subtree A Git **worktree** is useful here. It creates a second checkout of the same repository, allowing the baseline to remain in a temporary directory while the candidate stays in the main checkout. The comparison can run both revisions without repeatedly changing the developer’s current branch. This is different from a Git subtree, which is a way to include another repository inside a repository. I use a temporary detached worktree when the application can run from two checkout paths. The baseline and candidate can share infrastructure where appropriate, but each revision still needs its own application preparation and warmups. ```ruby # Pseudocode for the worktree-based orchestration. baseline_sha = git_resolve(BASELINE_REF) baseline_dir = create_detached_worktree(baseline_sha) begin before = prepare_warm_and_measure(checkout: baseline_dir) reset_database_and_caches after = prepare_warm_and_measure(checkout: current_checkout) ensure remove_worktree(baseline_dir) end ``` The benchmark workload should live outside the candidate change or be copied to a temporary location so the same code can exercise both revisions—even when the baseline does not contain the comparison script. Some Docker development environments bind-mount one fixed repository path. In that case, a second worktree may not be practical. The safer alternative is to: 1. record the current branch and commit; 2. stash tracked and untracked changes; 3. switch to the detached baseline; 4. restart and measure it; 5. restore the candidate and working tree; 6. restart and measure again; and 7. restore everything on errors or interrupts. The Gumroad comparisons use temporary worktrees. The OpenProject comparisons switch revisions because the running Compose environment is tied to the repository path. The implementation differs, but the goal is the same: run an identical workload against two clearly identified code states. ## 6. Control the environment Rails benchmarks are easily distorted by boot work, lazy initialization, cache state, and fixture differences. ### Warm both revisions For each revision: 1. start or restart the application; 2. wait for Rails and its dependencies to become ready; 3. execute several complete warmup operations; 4. discard those results; and 5. collect multiple measured runs. A complete warmup means making the same request or invoking the same operation as the benchmark. A health check may warm the database connection, but it will not necessarily load the serializer, policy, template, and application code used by the target endpoint. My case-study scripts commonly use three full warmups followed by five measured runs per revision. That is enough to expose obvious variance while keeping the script useful during development. More runs may be appropriate when the workload is short or the environment is noisy. ```ruby # Pseudocode: call this independently for before and after. def warm_and_measure(workload, validate:, warmups: 3, runs: 5) warmups.times { validate.call(workload.call) } # discarded runs.times.map { measure(workload, validate:) } # retained in full end ``` If cold-cache behavior is the actual problem, draw a different boundary: ```ruby # Pseudocode for an intentionally cold-cache experiment. runs.times.map do reset_relevant_cache measure(workload, validate:) end ``` Do not mix these two experiments. Design the cold benchmark separately, reset the relevant state before every run, and never compare a cold baseline with a warm candidate. ### Keep data equivalent Fixture creation, authentication setup, and cleanup should normally happen outside the timed operation. Both revisions should receive equivalent records in the same state. A disposable environment can recreate the database between revisions and build the same fixtures again. When using an existing dataset, keep the request read-only or restore mutations before the next run. Redis, search indexes, and other supporting services may also need to be reset when they affect the workload. ### Match dependencies to the revision A worktree shares Git objects, not installed dependencies. Reusing installed gems or JavaScript packages is safe only when both revisions expect compatible lockfiles. Assets and database migrations also need to match the checked-out revision. These details are less interesting than the final percentage, but they determine whether that percentage means anything. ## 7. Use the comparison to test fixes The first fix is not always the best fix. A repeatable script makes it inexpensive to test alternatives against the same baseline. For a serialization N+1, I might compare: - expanding the preload tree; - adding scoped associations that match serializer filters; - replacing per-record checks with one grouped query; - passing precomputed values into the serializer; or - changing the serializer to reuse an already-loaded association. After each attempt, I read the metrics in this order: 1. Did the targeted repeated work disappear? 2. Did total SQL or allocation cost improve? 3. Did complete request duration improve? 4. Did the response validations still pass? This order makes the result easier to interpret. If duration improves but the target query count is unchanged, the difference may be noise. If the repeated queries disappear but allocations rise sharply, the preload may be loading too much data. If query count becomes constant while local duration changes only slightly, the fix may still improve scaling—but the report should describe that bounded query count rather than claim a large latency reduction. Running the workload at two collection sizes can make scaling behavior visible. In the [Gumroad upsells benchmark](/case-studies/gumroad-upsells-n-plus-one/), the optimized query count remained one while the previous implementation issued one query per product. The larger catalog showed why the structural change mattered. ## 8. Generate a report for review The comparison should keep every measured run, then summarize them using a method chosen in advance. I usually include: - baseline and comparison commit SHAs; - route or operation; - Rails environment; - input size and important fixture details; - warmup and measured-run counts; - per-run before and after tables; and - mean and median comparisons. Median is useful when occasional pauses create outliers. Mean communicates the average of the observed runs. With a small sample, retaining both prevents the summary from hiding unstable results. The reporting step can remain simple: ```ruby # Pseudocode: never discard the individual rows. report.metadata( baseline: baseline_sha, candidate: candidate_sha, workload: "10 populated products", warmups: 3, runs: 5 ) report.run_table("Before", before_rows) report.run_table("After", after_rows) report.comparison_table( summarize(before_rows, using: [:mean, :median]), summarize(after_rows, using: [:mean, :median]) ) ``` The generated Markdown is an artifact of the run, not a hand-edited success story. If one run is an outlier, it remains visible in the table. The final explanation should separate three things: - **Measurement:** request duration fell by 40.4%, and SQL events fell from 103 to 23. - **Cause:** all 70 per-product target queries were eliminated. - **Limitation:** timings came from a local controlled environment, not production traffic. That gives a stakeholder a clear outcome while giving a reviewer enough technical evidence to understand why the result should hold. ## What this benchmark can and cannot prove A local comparison can show that one code revision performs a defined workload better than another and that the intended repeated work was removed. It does not prove production throughput, tail latency under concurrency, infrastructure savings, or behavior across every customer dataset. Those require load testing or production telemetry. For performance work I therefore use both: 1. a controlled comparison to validate the code change and its mechanism; and 2. production monitoring to confirm the effect under real traffic after rollout. The benchmark turns “this optimization should help” into a measured engineering result. The profile finds the expensive path, the comparison guides the implementation, and the generated report makes the outcome understandable to the people deciding whether the change is worth shipping. If your team has a slow Rails endpoint but needs more than a list of recommendations, this is the workflow I use in a focused [Rails performance optimization engagement](/services/rails-performance-optimization/): reproduce the problem, measure a baseline, implement the fix, and deliver a verified before-and-after result.