Scott Bartell

Different Ways to Set Attributes in ActiveRecord (Rails 7)

2022-04-12T00:00:00+00:00

Rails 7 has a rich API that allows you to update your ActiveRecord objects in several ways. Some methods have slightly different behavior which can sometimes result in unexpected consequences, so it’s important to understand their differences.

Note: This cheat sheet is for Rails 7. See cheat sheets for other versions here:

Rails 3 (external link)
Rails 4 (external link)
Rails 5
Rails 6

Here’s a cheat sheet highlighting the differences between all the methods available for setting attributes in Rails 7:

Cheat Sheet

Method	Uses Default Accessor	Saves to Database	Runs Validations	Runs Callbacks	Updates `updated_at/on`	Respects Readonly
`attribute=`	Yes	No	n/a	n/a	n/a	n/a
`attributes=`	Yes	No	n/a	n/a	n/a	n/a
`assign_attributes`	Yes	No	n/a	n/a	n/a	n/a
`write_attribute`	No	No	n/a	n/a	n/a	n/a
`[]=`	No	No	n/a	n/a	n/a	n/a
`update`	Yes	Yes	Yes	Yes	Yes	Yes
`update_attribute`	Yes	Yes	No	Yes	Yes	Yes
`update_column`	Yes	Yes	No	No	No	Yes
`update_columns`	Yes	Yes	No	No	No	Yes
`User.update`	Yes	Yes	Yes	Yes	Yes	Yes
`User.update_all`	No	Yes	No	No	No	No
`User.upsert`	No	Yes	No	No	No	No
`User.upsert_all`	No	Yes	No	No	Yes/No¹	No
`User.insert`	No	Yes	No	No	No	No
`User.insert_all`	No	Yes	No	No	Yes/No¹	No

Timestamps will be updated depending on the model’s record_timestamps config or the value of the record_timestamps argument passed into these methods. ↩ ↩²

Exploring Ruby’s `clamp` and `minmax` methods

2021-09-29T00:00:00+00:00

I came across an interesting problem recently where I was given a value and an array of “buckets” and I basically needed to return which bucket the value was closest to.

This made me extremely excited at the prospect of having a practical use case for Ruby’s Comparable#clamp and Enumerable#minmax!

In a simplified example I ended up playing around with an implementation similar to this:

buckets = [1, 2, 3, 4]
0.clamp(*buckets.min_max)
=> 1

Super cool stuff! Let’s dig a bit deeper into both clamp and minimax!

Comparable#clamp

Ruby’s Comparable#clamp “clamps” a value within a provided minimum and maximum value.

(1.2).clamp(1.3, 1.5)
=> 1.3

(1.6).clamp(1.3, 1.5)
=> 1.5

(1.4).clamp(1.3, 1.5)
=> 1.4

('a').clamp('b', 'd')
=> "b"

As of Ruby 2.7, clamp can also take a range:

6.clamp(3..5)
=> 5

1.clamp(2..)
=> 2

2.clamp(..1)
=> 1

Enumerable#minmax

There’s also Enumerable#minmax which gets both the minimum and maximum value in a given enumerable:

[1,2,3].minmax
=> [1, 3]

%w[b c a].minmax
=> ["a", "c"]

Finding and Fixing Missing Indexes in a Rails app

2021-02-24T00:00:00+00:00

When looking at the “Most time consuming” database operations for our application in New Relic and I came across something that looked odd to me.

Looking at the list I noticed two operations that I wouldn’t have expected to be among the most time consuming: OrderCreditTransaction find and Discount find.

In this post we’re going to dig into OrderCreditTransaction find find the problem and fix it.

What does “Most time consuming” mean?

“Most time consuming” tells you how much total time was spent doing an operation in a minute. This is equivalent to the average throughput (number of calls/minute) multiplied by the average response time.

For example if User find has an average response time of 4.71 ms and an average throughput of 743 cpm then it would consume: 4.71 ms * 743 cpm = 3.5 seconds

So this means that out of all database operations our application is spending the second most time in OrderCreditTransaction find. And, in order for that to be the case, it’s either being called a ton (high cpm), has a slow response time, or some combination.

Let’s dig in and find out.

Determining the problem

First let’s take a look at the OrderCreditTransaction find database operation by itself. Upon inspecting this operation you’ll notice it has relatively low throughput (~6.34 cpm) and a pretty slow avg response time (~196 ms).

This seems to indicate that the problem is likely related to response time and not throughput.

Let’s scroll down and see what some of the slow queries are…

Interesting! It looks like some simple SELECT queries filtering on receipt_id are taking up to almost 400ms! That seems like we could be a missing index… Let’s verify by clicking on one of those “Slow queries” and looking at an EXPLAIN.

If we look at line 4 of the Explain plan we can see that Postgres is doing a Parallel Sequential Scan on the order_credit_transactions table; this means that before this query can return it must look at every single record in the table (which at the time of this writing is 3,320,909 records). This seems to confirm our missing index theory!

Now, let’s take a look at our codebase and verify that an index is missing.

First off, let’s take a look at the order_credit_transactions table in schema.rb:

create_table "order_credit_transactions", force: :cascade do |t|
  t.integer  "order_id"
  t.integer  "receipt_id"
  t.datetime "created_at",                     null: false
  t.datetime "updated_at",                     null: false
end

Aha! It looks like there is no index on the receipt_id column!

Ideally, at this point (before we add any new indexes) we would want to confirm the behavior we saw from the EXPLAIN in New Relic on our local machine so that we can understand how the query plan will change once we introduce the new index (and confirm that our change will actually fix the problem).

We can do this by firing up a rails console locally and running something like this:

OrderCreditTransaction.where(receipt_id: 1).explain

This will run an EXPLAIN on your local database for the generated query.

However, there is a problem with this. Running this locally may give us a meaningfully different result that what we saw in production - this is because Postgres dynamically creates query plans depending on several factors, some of which are different between our local environment and production (most notably the number of records in the table).

To address this we could take some time to first copy some of the relevant data from production to our local machine and then run our EXPLAIN. But, in this case, we had a very high degree of certainty that this was the issue so we decided that the effort involved in doing that was not worth it.

Fixing the problem

Now, let’s add the missing index and make sure that we do it concurrently to ensure there is minimal to no performance impact to the table while the index is being built.

Rails doesn’t provide a way to cleanly generate a migration that adds an index to an existing column concurrently, so this needs to be a three step process.

First we generate an empty migration, then we manually edit the migration as outlined below, and finally we run the migration, commit and deploy.

$ bin/rails g migration AddIndexToOrderCreditTransactions

class AddIndexToOrderCreditTransactions < ActiveRecord::Migration[5.0]
  disable_ddl_transaction!

  def change
    add_index :order_credit_transactions, :receipt_id, algorithm: :concurrently
  end
end

$ bin/rails db:migrate

Verifying our fix

After the code gets merged and we run the migrations on staging and production let’s make sure we actually fixed the problem.

Woah! Looks like it worked. After deployment (around 11am) the average query time dropped from an average response time of ~196 ms to ~1 ms. That is a 99.49% decrease in response time!

Finally, let’s take a look a couple days after deployment to make sure that everything still looks good.

It still looks good!

Also, it’s no longer one of the most time consuming queries and it looks like it’s going to continue to fall down on that list!

RSpec Rails Tip: `have_http_status` matcher

2021-02-16T00:00:00+00:00

Sometimes you need to test the status code of a response in a controller, request, or feature spec. With the rspec-rails gem this can be done cleanly using the have_http_status matcher.

have_http_status can take three different arguments:

numeric code (200, 404, etc)
status name as defined in Rack::Utils::SYMBOL_TO_STATUS_CODE (:ok, :not_found, etc)
generic status type (:success, :missing, :redirect, or :error)

This means that these are actually different:

it { is_expected.to have_http_status(:ok) }
it { is_expected.to have_http_status(:success) }

Because :ok, as defined in Rack::Utils::SYMBOL_TO_STATUS_CODE, only matches status 200 while :success is a generic status type that matches all “successful” status types.

According to the RSpec source code, the generic status types map to the following status codes:

case expected
when :error, :server_error
  "5xx"
when :success, :successful
  "2xx"
when :missing, :not_found
  "404"
when :redirect
  "3xx"
end

Capybara Tip: Current Path Matcher

2021-02-11T00:00:00+00:00

Sometimes within a Capybara test you need to verify that you are on a particular page. The obvious way to test for this is by making expectations against current_path or current_url.

The problem with this approach is that the previous action that was taken (such as a button click, or some fancy JS / History.pushState()) may not have completed when your expectation runs against current_path.

One way to address this is to add an expectation that has waiting logic built into it before the expectation against current_path:

expect(page).to have_content('Login')
expect(current_path).to eq('/login') 

An even better way to do this is to use have_current_path which has waiting/retry logic built in so you don’t need to remember to always add an extra expectation with waiting above it.

expect(page).to have_current_path('/login')

Capybara Tip: Disabling CSS Animations

2020-07-06T00:00:00+00:00

You can disable CSS animations and transitions globally by setting the Capybara.disable_animation option.

Stick this in your rails_helper.rb:

Capybara.disable_animation = true 

Instead of a boolean, you can set this option to a CSS selector and it will only disable animations that match that selector:

Capybara.disable_animation = '#with_animation a'

For a better understanding of how this works take a look at the tests for Capybara.disable_animation in the Capybara source code.

Building a Sumo Logic Dashboard for a Rails App on Heroku

2020-02-12T00:00:00+00:00

Note: The dashboard and graphs outlined in this article are simplified examples of what I have used successfully in a production environment in the past. You should modify the dashboard and queries to fit your application’s specific needs.

You can download the JSON config to import this dashboard for your own application.

Overview

Our goal is to create a Sumo Logic dashboard that we can use to help us identify trends and problems with the performance of our Heroku Ruby on Rails application.

To do this we are going to focus on a few key areas:

Rails Application (Web Dynos)
- Web dyno load¹
- Web dyno memory utilization
Heroku Postgres
- Postgres load²
Worker Dynos and Sidekiq Jobs
- Sidekiq worker error rate
- Sidekiq worker failures by job

In order to build dashboards for these different areas, we first need to consider what kinds of logs Heroku makes available.

Heroku runtime dyno performance logs (e.g. CPU, memory usage) are available if you enable the Heroku Labs log-runtime-metrics feature. $ heroku labs:enable log-runtime-metrics
Heroku Postgres logs are available by default for Heroku Postgres Standard and Premium Tier databases.
Sidekiq logs are available by default as standard application logs from worker dynos.

Sumo Logic Graph Implementations

Some things to note with the implementations below:

All queries timeslice with buckets: This is done so that the time period can be changed on the dashboard and the graphs auto adjust their precision accordingly.
All queries omit application specific filters: This is done so that the dashboard can be applied to any application with a SumoLogic Dashboard Filter.

Heroku Web Dyno Load

This graph is designed to help provide insight into the CPU usage for you application. It uses Heroku’s CPU load average calculation which, in my experience, is mostly useful to identify relative spikes in load.

Graph

Logs

$ heroku logs --dyno web --source heroku --app my-app | grep load_avg | head -1
2020-01-28T23:59:31.883407+00:00 heroku[web.1]: source=web.1 dyno=heroku.93034875.4635d2a8-08be-4bf5-93c8-aed05a2fda5c sample#load_avg_1m=0.00 sample#load_avg_5m=0.00 sample#load_avg_15m=0.01

Query

"source=web."
| parse "sample#load_avg_1m=* " as load_avg_1m
| timeslice 60 buckets
| avg(load_avg_1m) as load_avg_1m by _timeslice

Heroku Web Dyno Memory Utilization

This graph is designed to help identify your application’s utilization of the available memory.

Graph

Logs

$ heroku logs --dyno web --source heroku --app my-app | grep memory | head -1
2020-01-29T00:02:49.243705+00:00 heroku[web.1]: source=web.1 dyno=heroku.93034875.4635d2a8-08be-4bf5-93c8-aed05a2fda5c sample#memory_total=338.52MB sample#memory_rss=252.74MB sample#memory_cache=85.79MB sample#memory_swap=0.00MB sample#memory_pgpgin=5641245pages sample#memory_pgpgout=5554583pages sample#memory_quota=512.00MB

Query

"source=web."
| parse "sample#memory_rss=*MB" as memory_rss
| parse "sample#memory_quota=*MB" as memory_quota
| timeslice 60 buckets
| max(memory_rss) as max_memory_rss, max(memory_quota) as max_memory_quota by _timeslice
| max_memory_rss / max_memory_quota * 100 as memory_utilization
| fields _timeslice, memory_utilization

In this query we divide the Resident Memory (memory_rss) by the Memory Quota (memory_quota) to get a maximum memory utilization rate for our dynos. At the end of the query we do a fields _timeslice, memory_utilization so that memory_rss and memory_quota don’t show up in our graph.

Resident Memory (memory_rss): The portion of the dyno’s memory (megabytes) held in RAM.
Memory Quota (memory_quota): The resident memory (memory_rss) value (megabytes) at which Heroku triggers an R14 error.

Heroku Postgres Load

This graph is designed to help identify if your Postgres database is CPU-bound.

Graph

Logs

$ heroku logs --dyno heroku-postgres --app my-app | head -1
2020-01-29T00:36:10+00:00 app[heroku-postgres]: source=HEROKU_POSTGRESQL_RED addon=postgresql-aerodynamic-12345 sample#current_transaction=1012345 sample#db_size=3743972487bytes sample#tables=10 sample#active-connections=8 sample#waiting-connections=0 sample#index-cache-hit-rate=0.99682 sample#table-cache-hit-rate=0.84557 sample#load-avg-1m=0 sample#load-avg-5m=0 sample#load-avg-15m=0 sample#read-iops=0 sample#write-iops=0.12397 sample#tmp-disk-used=33849344 sample#tmp-disk-available=72944943104 sample#memory-total=4044932kB sample#memory-free=78976kB sample#memory-cached=3700368kB sample#memory-postgres=23292kB

Query

"heroku-postgres"
| parse "source=* " as db_name
| parse "sample#load-avg-1m=* " as load_avg_1m
| (load_avg_1m / 1) * 100 as utilization_rate
| timeslice 60 buckets
| max(utilization_rate) as max_utilization_rate by _timeslice

Sidekiq Error Rate

This graph is designed to give you insight into your frequency of failed Sidekiq jobs over time.

Graph

Logs

$ heroku logs --source app --dyno worker --app my-app | grep MyExampleJob
2020-01-26T09:02:44.353201+00:00 app[worker.1]: pid=4 tid=6bjk class=MyExampleJob jid=b027792cdb4a453576377c69 elapsed=10.576 INFO: done
2020-01-26T09:02:33.777529+00:00 app[worker.1]: pid=4 tid=6bjk class=MyExampleJob jid=b027792cdb4a453576377c69 INFO: start
2020-01-26T09:02:07.485427+00:00 app[worker.1]: pid=4 tid=5si0 class=MyExampleJob jid=b027792cdb4a453576377c69 elapsed=44.526 INFO: fail
2020-01-26T09:01:22.959446+00:00 app[worker.1]: pid=4 tid=5si0 class=MyExampleJob jid=b027792cdb4a453576377c69 INFO: start

Query

"host app worker."
| parse "elapsed=* INFO: *" as run_time, status
| if (status = "fail", 100, 0) as unsuccessful
| timeslice 60 buckets
| avg(unsuccessful) as error_rate by _timeslice

In this query we filter for the Sidekiq logs that represent a job being successful or failing. From those logs, we assign the successful messages a value of 100 and assign the failure messages a value of 0. Finally, we take the average of all of those logs to get an error rate out of 100%.

Sidekiq Job Failure Count by Class

This table is designed to work with the Error Rate graph to help quickly identify which jobs are contributing to the error rate.

Graph

Logs

$ heroku logs --source app --dyno worker --app my-app | grep MyExampleJob
2020-01-26T09:02:44.353201+00:00 app[worker.1]: pid=4 tid=6bjk class=MyExampleJob jid=b027792cdb4a453576377c69 elapsed=10.576 INFO: done
2020-01-26T09:02:33.777529+00:00 app[worker.1]: pid=4 tid=6bjk class=MyExampleJob jid=b027792cdb4a453576377c69 INFO: start
2020-01-26T09:02:07.485427+00:00 app[worker.1]: pid=4 tid=5si0 class=MyExampleJob jid=b027792cdb4a453576377c69 elapsed=44.526 INFO: fail
2020-01-26T09:01:22.959446+00:00 app[worker.1]: pid=4 tid=5si0 class=MyExampleJob jid=b027792cdb4a453576377c69 INFO: start

Query

"host app worker."
| parse "class=* " as job_name
| parse "elapsed=* INFO: *" as run_time, status
| where status = "fail"
| count by job_name
| order by _count

CPU load average is a very complicated calculation that I’ve found to only be useful relative to itself overtime. The calculation uses an exponentially damped moving average of the number of runnable tasks from the previous 30 minutes. Read more here. ↩
Postgres load average is calculated differently than how Heroku CPU load average for dynos. It is the average system load (for a 1, 5, or 15 min period) divided by the number of available CPUs. ↩

Different Ways to Set Attributes in ActiveRecord (Rails 6)

2020-01-30T00:00:00+00:00

Rails 6 has a rich API that allows you to update your ActiveRecord objects in several different ways. Some methods have slightly different behavior which can sometimes result in unexpected consequences so it’s important to understand their differences.

Note: This article was written for Rails 6. See cheat sheets for other versions here:

Rails 3 (external link)
Rails 4 (external link)
Rails 5
Rails 7

Here’s a cheat sheet highlighting the differences between all the methods available for setting attributes in Rails 6:

Cheat Sheet

Method	Uses Default Accessor	Saves to Database	Runs Validations	Runs Callbacks	Updates `updated_at/on`	Respects Readonly
`attribute=`	Yes	No	n/a	n/a	n/a	n/a
`attributes=`	Yes	No	n/a	n/a	n/a	n/a
`assign_attributes`	Yes	No	n/a	n/a	n/a	n/a
`write_attribute`	No	No	n/a	n/a	n/a	n/a
`[]=`	No	No	n/a	n/a	n/a	n/a
`update`	Yes	Yes	Yes	Yes	Yes	Yes
`update_attribute`	Yes	Yes	No	Yes	Yes	Yes
`update_attributes`¹	Yes	Yes	Yes	Yes	Yes	Yes
`update_column`	Yes	Yes	No	No	No	Yes
`update_columns`	Yes	Yes	No	No	No	Yes
`User.update`	Yes	Yes	Yes	Yes	Yes	Yes
`User.update_all`	No	Yes	No	No	No	No
`User.upsert`	No	Yes	No	No	No	No
`User.upsert_all`	No	Yes	No	No	No	No
`User.insert`	No	Yes	No	No	No	No
`User.insert_all`	No	Yes	No	No	No	No

update_attributes has been deprecated in favor of update in Rails 6. ↩

Automating Rails Database Migrations on Heroku

2020-01-22T00:00:00+00:00

One problem that teams tend to run into with Rails apps on Heroku is forgetting to run database migrations before the code that depends on them is released.

Luckily, there is a way to automate the running of database migrations on Heroku and to guarantee that they are always run before the dependent code is released: the Heroku Release Phase.

Heroku Release Phase

The Heroku Release Phase allows you run certain tasks after your app is built but before the app is released.

Specifying release phase tasks

To specify the tasks to run during release phase, define a release process type in your app’s Procfile.

In our case we want to run database migrations directly in our release task so we do the following:

web: bundle exec puma -C config/puma.rb
release: bundle exec rake db:migrate

That’s all you need to do. The next time you push code Heroku will see that you have a release task configured and automatically run bundle exec rake db:migrate after the app is built.

Notable Implications

In my experience this flow works really well. It helps guarantee that we never forget to run migrations, makes it so that we don’t have to worry about code being released before a migration is run, and couples the running of the database migrations with the releasing of code (giving us an audit trail for when migrations were run).

That being said, this does introduce some complexity and some new cases that we will need to consider.

1. If the Release Phase fails the app will not be deployed

If the Release Phase ever fails (in our case, if the migration fails) the app will not be released. This means that, in order to avoid the database ending up in a partially migrated state, we need to ensure all database migrations are wrapped in transactions. For Rails apps this typically isn’t a problem because by default ActiveRecord wraps all migrations in a transaction. However, if we ever want to take advantage of something like Postgres’s CONCURRENTLY option for CREATE INDEX, or run some sort of large data backfill outside of a transaction, you might need to disable transactions.

2. The Release Phase runs on Review apps

If you are using Heroku Review Apps you should consider what will happen when the release task runs in that environment.

For example, if you have configured Review Apps to share the same database as your staging environment, when a Review App is deployed the release task will run which will run database migrations in the Review App code against the staging database.

One potential solution to this is to have the release task look for an environment variable that is configured differently for Review Apps:

Create a new file in bin/release-tasks:

#!/usr/bin/env bash

if [ "$DISABLE_RAILS_DB_MIGRATIONS" = "true" ]; then
  echo "SKIPPING DATABASE MIGRATIONS"
else
  bundle exec rake db:migrate
fi

Update Procfile to call the new task:

...
release: bin/release-tasks

3. Database Migrations will always run before the new code is released

All database migrations will run before the new code is shipped. This means that certain schema changes (such as column renames) may result in exceptions and/or downtime.

In order to avoid this you must ensure all of your schema changes/database migrations support both the new code as well as the code that is running before the new code is released.

Different Ways to Set Attributes in ActiveRecord (Rails 5)

2019-07-15T00:00:00+00:00

Rails 5 has a rich API that allows you to update your ActiveRecord objects in several different ways. Some methods have slightly different behavior which can sometimes result in unexpected consequences so it’s important to understand their differences.

Over the years I came across a really helpful cheat sheet outlining the various methods available for Rails 4 so I figured I’d put together something similar for Rails 5!

Note: This article was written for Rails 5. See cheat sheets for other versions here:

Rails 3 (external link)
Rails 4 (external link)
Rails 6
Rails 7

Below is a cheat sheet with in-depth information for Rails 5:

Cheat Sheet

Method	Uses Default Accessor	Saves to Database	Runs Validations	Runs Callbacks	Updates `updated_at/on`	Respects Readonly
`attribute=`	Yes	No	n/a	n/a	n/a	n/a
`attributes=`	Yes	No	n/a	n/a	n/a	n/a
`assign_attributes`	Yes	No	n/a	n/a	n/a	n/a
`write_attribute`	No	No	n/a	n/a	n/a	n/a
`[]=`	No	No	n/a	n/a	n/a	n/a
`update`	Yes	Yes	Yes	Yes	Yes	Yes
`update_attribute`	Yes	Yes	No	Yes	Yes	Yes
`update_attributes`	Yes	Yes	Yes	Yes	Yes	Yes
`update_column`	Yes	Yes	No	No	No	Yes
`update_columns`	Yes	Yes	No	No	No	Yes
`User.update`	Yes	Yes	Yes	Yes	Yes	Yes
`User.update_all`	No	Yes	No	No	No	No