One of these days, a friend came to talk to me about an Ecto problem that could be solved using metaprogramming. He was using the insert_all/3 function with the on_conflict option so it would become a upsert command.

Photo by Melanie Karrer on Unsplash

The problem is that it was necessary to update only the entries that were newer than the current one. To do that, we need a custom query to check the timestamp, but when we use a custom query, we lose the ability to use the :replace_all option to replace all the fields. According to the doc:

:on_conflict - It may be one of :raise (the default), :nothing, :replace_all, {:replace_all_except, fields}, {:replace, fields}, a keyword list of update instructions or an Ecto.Query query for updates.

If we want to replace all the fields and still use the Ecto.Query, we will need to update all the fields manually, and this is error-prone.

def update_user(users) do
  Repo.insert_all(
    User,
    users,
    on_conflict: on_conflict_query(),
    conflict_target: :event_id
  )
end

def on_conflict_query do
  User
  |> where([u], f.updated_at < fragment("EXCLUDED.updated_at"))
  |> update([u],
    set: [
      first_name: fragment("EXCLUDED.first_name"),
      last_name: fragment("EXCLUDED.last_name"),
      phone_number: fragment("EXCLUDED.phone_number"),
      gender: fragment("EXCLUDED.gender"),
      date_of_birth: fragment("EXCLUDED.date_of_birth"),
      city: fragment("EXCLUDED.city"),
      state_code: fragment("EXCLUDED.state_code"),
      email: fragment("EXCLUDED.email"),
      event_id: fragment("EXCLUDED.event_id"),
      updated_at: fragment("EXCLUDED.updated_at")
    ]
  )
end

Let’s see how we can use metaprogramming to solve this.

You can skip to the final solution here if you want to.

Metaprogramming to the rescue

Where do I start?

Elixir is a AST based programming language. Meaning that when our code is compiled, its source is transformed into a tree structure, this structure is exposed in a form that can be represented in Elixir’s own data structure.

Metaprogramming is the ability to write code using code, so we can extend the language to dynamically change the code, and we do it by manipulating Elixir’s AST.

The quote

The quote macro is responsible to get the representation of any expression. It returns a tuple with three elements:

1. An atom or another tuple in the same representation;
2. A keyword list of values that represents Elixir metadata about the function;
3. The arguments to the function call.

iex(1)> quote do
...(1)>   1 + 1
...(1)> end
{:+, [context: Elixir, import: Kernel], [1, 1]}

In this example, the tuple represents a function call to the Kernel arithmetic addition function, passing two arguments, 1 and 1.

The unquote

The unquote macro is responsible for injecting an AST expression into the AST. In the “Metaprogramming Elixir” book, Chris McCord says:

You can think of quote/unquote as string interpolation for code. If you were building up a string and needed to inject the value of a variable into that string, you would interpolate it. The same goes when constructing an AST. We use quote to begin generating an AST and unquote to inject values from an outside context. This allows the outside bound variables, expression and block, to be injected directly into our if ! transformation.

– Chapter 1. The Language of Macros. Metaprogramming Elixir: Write Less Code, Get More Done (and Have Fun!), by Chris McCord, The Pragmatic Bookshelf, 2015.

iex> number = 1
1

iex> "this is an interpolation of #{number}"
"this is an interpolation of 1"

iex> quote do
...>   unquote(number) + 1
...> end
{:+, [context: Elixir, import: Kernel], [1, 1]}

We can see that when we use the unquote macro, it injects the number value into the AST, just like it does when we are interpolating a string.

iex> number = 1
1

iex> "this is an interpolation of number"
"this is an interpolation of 1"

iex> quote do
...>   number + 1
...> end
{:+, [context: Elixir, import: Kernel], [{:number, [], Elixir}, 1]}

When we don’t use the unquote macro it includes :number into the AST, instead of the value that we want.

iex> Macro.to_string(quote(do: number + 1))
"number + 1"

iex> Macro.to_string(quote(do: unquote(number) + 1))
"1 + 1"

Macros

According to the documentation Macros are compile-time constructs that are invoked with Elixir’s AST as input and a superset of Elixir’s AST as output. Once the construct finishes its processing, the response will then be injected back into the application.

defmodule MyModule do
  defmacro increment(number) do
    quote(do: unquote(number) + 1)
  end
end

iex> MyModule.increment(1)
2

Extending Ecto’s on_conflict

Back to our problem, the update/3 function expects a set operator with a keyword list with field-value pairs as values.

User
|> update([u],
  set: [
    first_name: fragment("EXCLUDED.first_name")
  ]
)

We need to write a macro that will take a list of fields and build a fragment passing the field’s name.

For every schema that we create on Ecto, we will have a schema function that can be used for runtime introspection of the schema. We will use it to fetch all non-virtual fields from the schema.

User.__schema__(:fields)

Now that we have a list of all the fields we want to update, we will use quote and unquote to dynamically build a fragment with Postgres EXCLUDED table to reference what’s being updated.

:fields
|> User.__schema__()
|> Enum.map(fn f ->
  {f, quote(do: fragment(unquote("EXCLUDED.#{f}")))}
end)

After that, we will pass the values we built for the set operator. Instead of unquote this time, we will use unquote_splicing because we are passing a list of expressions, instead of one expression.

defmacro custom_on_conflict_update_replace_all(queryable) do
  values =
    :fields
    |> User.__schema__()
    |> Enum.map(fn f ->
      {f, quote(do: fragment(unquote("EXCLUDED.#{f}")))}
    end)

  quote(do: Ecto.Query.update(unquote(queryable), [u], set: [unquote_splicing(values)]))
end

We are almost done now, we just need to call the macro on the on_conflict_query/0.

def on_conflict_query do
  User
  |> where([u], f.updated_at < fragment("EXCLUDED.updated_at"))
  |> custom_on_conflict_update_replace_all()
end

The final solution:

def update_user(users) do
  Repo.insert_all(
    User,
    users,
    on_conflict: on_conflict_query(),
    conflict_target: :event_id
  )
end

defmacro custom_on_conflict_update_replace_all(queryable) do
  values =
    :fields
    |> User.__schema__()
    |> Enum.map(fn f ->
      {f, quote(do: fragment(unquote("EXCLUDED.#{f}")))}
    end)

  quote(do: Ecto.Query.update(unquote(queryable), [u], set: [unquote_splicing(values)]))
end

def on_conflict_query do
  User
  |> where([u], f.updated_at < fragment("EXCLUDED.updated_at"))
  |> custom_on_conflict_update_replace_all()
end

Now, every time we make a change on the User schema we don’t need to bother to come back and change this function.

I hope it helps!