JSON Masterclass

Why use JSON in a Rails app?

JSON is a robust and well-supported data storage and transfer format. It's very useful when transferring data between frontend and backend, or between services.

It's also useful for storing data in the database when you don't know the structure of the data in advance, or you want to change the structure frequently and don't want to have to migrate the database every time.

Used correctly it can be helpful in creating isomorphism between the frontend and backend, and in creating a more flexible and maintainable app.

What you'll build

In this pattern we'll be learning the ins and outs of using JSON in a Rails app.

We start with learning about the different ways to store JSON in the database and how to query it.

Then we'll look at how to validate JSON data and how to serialize and deserialize it.

Finally we'll look at how to transfer JSON data to the view.

App setup

We start with the pattern_base1 base app which can also be created with the following commands:

APP_NAME=json-masterclass
rails new $APP_NAME -d postgresql -c tailwind --skip-jbuilder --skip-test
cd $APP_NAME
git add -A && git commit -m "rails new $APP_NAME -d postgresql --skip-jbuilder --skip-test"

We use the default test setup

rails db:create

rails g scaffold app data:jsonb --no-helper --no-test-framework
rails db:migrate

Data layer

Database choices

First off if you want to store JSON data in your Rails app you'll really want to be using PostgreSQL as your database. It has the best and most established JSON support of all the major databases. See the Postgres JSON type reference, the Postgres JSON Functions and Operators reference and the Rails PostgreSQL Guide for more information on Rails support.

MySQL JSON is supported in Rails despite not being well documented, see the MySQL JSON Data Type reference, and the Mysql JSON Functions reference.

SQLite also has JSON support, see SQLite JSON1 reference. The plugin is compiled into the database by default and the functions and operators are designed to be compatible with PostgreSQL and MySQL.

Creating columns

To create a column on an existing table:

rails g migration AddDataToApps data:jsonb

Like most data types the default will be nil so add a default to the column. This can be anything that serializes to JSON. Arrays, hashes, or anything that responds to to_json:

class AddDataToApps < ActiveRecord::Migration[7.1]
  def change
    add_column :apps, :data, :jsonb, default: {}
  end
end

JSON vs JSONB

The json type stores the json as text in the database. It will be serialized and deserialized, but cannot be queried or indexed. By contrast the jsonb type is stored as a binary format which means it can be queried and indexed by internal properties of the JSON object. Point updates can be made and in general updates are faster. jsonb is the modern recommended choice, you would only choose the older json type for legacy reasons, or for databases which don't support jsonb.

Querying JSON

JSON in Postgres supports a number of functions and operators. The operators are generally most useful, they all extract data from the JSON object and return it to the query. Here are some examples:

-> Get JSON object field as JSON

Here we're comparing the JSON object in the database to a JSON object in the query.

App.where(%[data->'profile' = ?], {"name":"Jon"}.to_json)

Here we're using virtual attributes with SELECT to extract the JSON object from the database and return it as a Ruby object.

class App < ActiveRecord::Base
  attribute :profile, :jsonb, default: {}
end

App.select("data->'profile' AS profile").first
# #<App:0x000000011d177be0 id: nil, profile: {"name"=>"Jon"}>

->> Get JSON object field as text

Here' we're extracting the text value of a JSON object field and comparing it to a string in the query. You can see we use the -> operator to get to the node and then the ->> operator to extract the text value.

App.where(%[data->'profile'->>'name' = ?], "Jon")

#> Get JSON object at path as object

The #> operator is is another way of getting at JSON object fields by using a path. Here we're getting the profile object from the JSON object and then getting the name field from that object.

Note we have to convert the string to JSON in order to compare it to the JSON object that #> returns.

Note also that '{profile,name}' is in fact an array in SQL.

App.where(%[data#>'{profile,name}' = ?], "Jon".to_json)

Here's another way to extract the profile object

App.select("data#>'{profile}' AS profile").first

#>> Get JSON object at path as text

This is the same as #> but returns the text value of the node.

App.where(%[data#>>'{profile,name}' = ?], "Jon")

- Delete key/value pair or string element from JSON object

Here we delete the whole profile attribute from the JSON object.

App.where(%[data#>>'{profile,name}' = ?], "Jon").update_all("data = data - 'profile'")

#- Delete key/value pairs at path from JSON object

Here we just delete the name attribute from the profile object.

App.where(%[data#>>'{profile,name}' = ?], "Jon").update_all("data = data #- '{profile,name}'")

Scopes

You can use the above operators to create scopes on your model. For example:

scope :with_country, ->(country) { where("data->>'country' = ?", country) }

There are many more functions for specific use cases but these will get you far.

Point updates

Updating a specific deeply nested node in your JSON object can be useful, but isn't supported natively. Postgres does include a json_set function which includes a create_if_missing option, but it only works if all the previous nodes in the path exist.

To create a deep node with all the intermediate nodes we're going to need to add a custom SQL function to our app.

First we're going to add the fx gem which allows us to migrate SQL functions in the same way that we migrate data in our app. Then we'll generate the function migration like so:

rails generate fx:function jsonb_set_deep

This will generate a migration file and a SQL function file. Edit it as follows

-- db/functions/jsonb_set_deep_v01.sql

CREATE OR REPLACE FUNCTION jsonb_set_deep(target jsonb, path text[], val jsonb)
  RETURNS jsonb AS $$
    DECLARE
      k text;
      p text[];
    BEGIN
      IF (path = '{}') THEN
        RETURN val;
      ELSE
        IF (target IS NULL) THEN
          target = '{}'::jsonb;
        END IF;

        FOREACH k IN ARRAY path LOOP
          p := p || k;
          IF (target #> p IS NULL) THEN
            target := jsonb_set(target, p, '{}'::jsonb);
          ELSE
            target := jsonb_set(target, p, target #> p);
          END IF;
        END LOOP;

        -- Set the value like normal.
        RETURN jsonb_set(target, path, val);
      END IF;
    END;
  $$ LANGUAGE plpgsql;

Then we add a helper method in the model:

# app/models/app.rb

class App < ActiveRecord::Base
  def set_json(attr, path, value)
    self.class.where(id: id).update_all(["#{attr} = jsonb_set_deep(#{attr}, '{#{path.join(', ')}}', ?)", value.to_json])
  end
end

And use it like so:

a.set_json(:data, [:level1, :level2, :level3], "Value")

Deleting and renaming

# app/models/app.rb

def delete_json_attr(path)
  self.class.where(id: id).update_all(["data = data #- '{#{path.join(', ')}}'"])
end

def rename_json_key(path, new_key)
  new_path = path.dup.tap { |p| p.pop; p.push(new_key) }
  where_clause = "data #> '{#{path[0..-2].join(', ')}}' ? '#{path.last}'"
  update_clause = "data = jsonb_set(data #- '{#{path.join(', ')}}', '{#{new_path.join(', ')}}', data #> '{#{path.join(', ')}}')"

  self.class.where(id: id).where(where_clause).update_all(update_clause)
end

Paths

A path to any point in the JSON is an array of strings, numbers or any other value that can be used as a key in JSON.

Model layer

JSON Schema

An easy way to validate your JSON data is to use the json-schema gem which uses JSON Schemas to validate JSON.

bundle add activerecord_json_validator

# app/models/app.rb

class App < ActiveRecord::Base
  APP_DATA_JSON_SCHEMA = Rails.root.join('config/schemas/app_data.json_schema')

  validates :data, presence: true, json: { schema: APP_DATA_JSON_SCHEMA }
end

// config/schemas/app_data.json
{
  "type": "object",
  "$schema": "http://json-schema.org/draft-04/schema#",
  "properties": {
    "city": { "type": "string" },
    "country": { "type": "string" },
    "profile": {
      "type": "object",
      "properties": {
        "name": { "type": "string" },
        "age": { "type": "integer" }
      },
      "required": ["name"]
    }
  },
  "required": ["country"]
}

Serialization and deserialization

By default Rails will deserialize the binary JSON blob or string stored in the database into a Ruby object which can be used immediately. Likewise changes made to that Ruby object will be serialized and saved to the database along with any other changes made to the model.

In many cases that Hash or Array is sufficient, but as the complexity of the app and its dependency on the JSON data grows you may want to start adding functions that operate on the data. At this point you probably want to replace the default deserialization and deserialize to an object.

This can be done in vanilla Rails like so:

# app/models/app.rb

serialize :data, coder: Datum

delegate :user, :table, :user_table, to: :data

Now let's define our serialization target. This is a plain old Ruby object that defines the dump and load methods. Here we also include some ActiveModel modules to make the definition a little cleaner.

# app/models/datum.rb

class Datum
  include ActiveModel::Attributes
  include ActiveModel::AttributeAssignment
  include ActiveModel::Serializers::JSON

  attribute :user, :string, default: ''
  attribute :table, :string, default: ''

  def user_table
    "#{user}_#{table}"
  end

  def self.dump(value)
    value
  end

  def self.load(value)
    return new if value.blank?

    new.tap do |datum|
      datum.assign_attributes value
    end
  end
end

Other serialization options

If you have a lot of attributes or you need array types the above can get a little verbose. In that case you can use the attr_json gem or the json_attribute gem. Both of these gems give more control over the serialization and deserialization process. I've personally used attr_json and it's worked well for me.

to_json vs as_json

All objects in Rails have both of these functions. While to_json converts an object into a JSON string. as_json converts an object into the ruby version of the JSON object. Generally to_json should be left alone but you can override the default as_json on a Rails object like so:

# app/models/app.rb

def as_json(options = {})
  super({
    include: %i[associations],
    methods: %i[users]
  }.reverse_merge(options))
end

This shows you how to add associations and other methods to the returned hash, and allow further options (:only and :except) to be passed through

Note these are associations and methods on the ActiveRecord object. If you have a JSON data column and you want a nested JSON column in the output you want to create a function for it on the serialization target.

Controller and view layers

Once your JSON data is in the database and you've deserialized it into an object you'll probably want to make it available to the view. There are a number of ways to do this:

API data transfer

The easiest way to transfer JSON to the view is to respond to the JSON format in your controller:

# app/controllers/apps_controller.rb

class AppsController < ApplicationController
  def show
    respond_to do |format|
      format.html
      format.json { render json: @app }
    end
  end
end

Then you can fetch the JSON data from the server using JavaScript and update the view. This is a common pattern in React and other JavaScript frameworks:

// app/javascript/controllers/application.js

document.addEventListener("turbo:load", e => {
  fetch('/apps/4.json')
    .then(response => response.json())
    .then(data => {
      console.log(data)
    })
})

Jbuilder

If your json needs are more complex you can use the Jbuilder gem to create a view for your JSON data. Jbuilder is included by default in most Rails apps but we created this one without it, so let's add it to the gemfile first:

bundle add jbuilder

Then add a view for your JSON data:

# app/views/apps/show.json.jbuilder

json.array! @apps do |app|
  json.id app.id
  json.data app.data
end

And fetch it on the frontend:

// app/javascript/controllers/application.js

document.addEventListener("turbo:load", e => {
  fetch('/apps.json')
    .then(response => response.json())
    .then(data => {
      console.log(data)
    })
})

No need to change the controller as Jbuilder will automatically render the view if it exists and the URL ends in .json.

The great thing about JBuilder is that it allows you to assemble a more complex JSON object from multiple view partials.

Gon

If you wan't your object globally available you can use the gon gem. This gem allows you to set a global variable in your view which can be accessed in your JavaScript.

bundle add gon

<% # app/views/layouts/application.html.erb %>

<head>
  <title>some title</title>
  <%= Gon::Base.render_data %>
  ...
</head>

class AppsController < ApplicationController
  before_action :set_gon, only: %i[ show edit ]

  private

  def set_gon
    gon.push({ app: @app.as_json })
  end
end

Then anywhere in your JavaScript you can access the gon variable:

console.log(gon.app)

Stimulus controllers

If you're using Stimulus you can define a value of type Object on the controller and make it available in the controller like so:

<% # app/views/apps/show.html.erb %>

<div class="mx-auto md:w-2/3 w-full flex" data-controller="app" data-app-data-value="<%= @app.data.to_json %>">
</div>

// app/javascript/controllers/app_controller.js

import { Controller } from "@hotwired/stimulus"

export default class extends Controller {
  static values = { data: Object }

  connect() {
    console.log(this.dataValue)
  }
}

Stimulus automatically deserializes the JSON string into a JavaScript object.

 JSON in React

If you follow our React on Rails guide you'll be able to drop your JSON as props into your React components like so:

<%= react_component('Post', props.as_json, camelize_props: true) %>

// app/javascript/components/Post.js

import React from 'react'

export default function Post({ title, published, publishedAt }) {
  return (
    <div>
      <h1>{title}</h1>
      <p>{published ? 'Published' : 'Draft'}</p>
      <p>{publishedAt}</p>
    </div>
  )
}

As you can see we've used the camelize_props option to convert the snake_case keys to camelCase. If you want to do this globally

# config/initializers/react_on_rails.rb

Rails.application.config.react.camelize_props = true

Key transformation

In Ruby the convention is to use snake_case for variable names and in JSON the convention is to use camelCase. Most linters will complain if you use the wrong case and it will make bugs more likely. So you'll probably want to transform your JSON keys to camelCase on the way out of the server. and then back to snake_case on the way in.

Let's start by adding a camelized_json function to our ApplicationRecord class. This is the class that all our models inherit from so the function will be available to all of them. You could of course add it only to the models that need it.

# app/models/application_record.rb

class ApplicationRecord < ActiveRecord::Base
  primary_abstract_class

  KEYS_FOR_VALUE_INFLECTION = %w[type].freeze

  def as_camelized_json(options = {})
    as_json(options)
      .deep_dup
      .deep_transform_values_with_key! { |k, v| KEYS_FOR_VALUE_INFLECTION.include?(k) ? v.camelize(:lower) : v }
      .deep_transform_keys { |k| k.to_s.camelize(:lower) }
  end
end

Note we also allow for a list of values that can be inflected as well. This is useful for string constants.

We can then use it in our controller like so:

class AppsController < ApplicationController
  def show
    respond_to do |format|
      format.html
      format.json { render json: @app.as_camelized_json }
    end
  end
end

To do it on the way back:

# app/controllers/application_controller.rb

class ApplicationController < ActionController::Base
  before_action :underscore_params

  private

  def underscore_params
    params.deep_transform_keys! { |k| k.is_a?(String) ? k.underscore : k }
    params.deep_transform_values_with_key! { |k, v| ApplicationRecord::KEYS_FOR_VALUE_INFLECTION.include?(k) ? v.underscore : v }
  end
end

Depending on your needs you can do this globally as above or just on specific controllers.

In JBuilder you can use the key_format! and deep_format_keys function to do the same thing:

json.key_format! camelize: :lower
json.deep_format_keys true

Or to apply it across the entire app:

# config/initializers/jbuilder.rb

Jbuilder.key_format camelize: :lower
Jbuilder.deep_format_keys true

Another option to solve this problem is the olive_branch gem.

JSON Editor

If you want the ability to edit your JSON in a form you can use the vanilla-jsoneditor package. It offers formatting, validation, syntax highlighting and other features. It's great for admin areas.

Add the package to your app:

yarn add vanilla-jsoneditor
# or
bin/importmap pin vanilla-jsoneditor

Add a Stimulus controller wrapper:

// app/javascript/admin/json_editor_controller.js

import { Controller } from '@hotwired/stimulus'

import { JSONEditor } from 'vanilla-jsoneditor'

export default class extends Controller {
  static targets = ["textarea", "editor"]

  connect() {
    let content = { text: this.textareaTarget.value }

    const target = this.editorTarget
    const onChange = (updatedContent, previousContent, { contentErrors, patchResult }) => {
      this.textareaTarget.value = updatedContent.text
      content = updatedContent
    }

    const editor = new JSONEditor({ target, props: { content, mode: 'text', onChange }})
  }
}

Then in your views

<% # app/views/apps/_form.html.erb %>

<div data-controller="json-editor">
  <%= form.label :settings %>
  <%= form.text_area :settings, value: form.object.settings.to_json, data: { 'json-editor-target': 'textarea' }, class: "hidden" %>
  <div data-json-editor-target="editor" style="width: 400px; height: 400px;"></div>
</div>

Testing

Fixtures

If your testing your JSON app there's a good chance you'll want to load fixtures into your suite. You can do this by adding a test helper:

# spec/support/helpers.rb

module Helpers
  def json_data(filename)
    file_content = file_fixture("json_data/#{filename}.json").read
    JSON.parse(file_content, symbolize_names: true)
  end
end

Then create your fixture:

// spec/fixtures/files/json_data/app_data.json
{
  "name": "Jon",
  "age": 40
}

Load it in a factory:

# spec/factories/apps.rb

FactoryBot.define do
  factory :app do
    data { json_data :app_data }
  end
end

And use it in your tests:


RSpec.describe App, type: :model do
  let(:app) { create :app }

  it 'has data' do
    expect(app.data).to eq({ name: 'Jon', age: 40 })
  end
end

TODO