Description

We received a security report saying that it was possible to confirm records with a blank confirmation_token parameter. In order words, hitting the following URL /users/confirmation?confirmation_token= would successfully confirm a user instead of showing a validation error – e.g. Confirmation token can't be blank.

This only happens if there are records with a blank confirmation token in the database. This is because of the way the method find_first_by_auth_conditions works (which is similar to ActiveRecord’s #find_by). It is important to mention that we haven’t found a case where Devise sets confirmation_token to an empty string.

In summary, before version 4.7.1, this is what would happen after hitting the confirmation endpoint with a blank confirmation_token:

Started GET "/users/confirmation?confirmation_token=" for ::1 at 2019-09-05 10:24:29 -0300
Processing by Devise::ConfirmationsController#show as HTML
  Parameters: {"confirmation_token"=>""}
  User Load (1.0ms)  SELECT  "users".* FROM "users" WHERE "users"."confirmation_token" = $1 ORDER BY "users"."id" ASC LIMIT $2  [["confirmation_token", ""], ["LIMIT", 1]]
Code language: PHP (php)

Notice that the SQL query is trying to find users with confirmation_token = "".

It’s also worth mentioning that only unconfirmed records – i.e. with confirmed_at: nil in the database – would be confirmed in this case. For already confirmed users, a validation error (Email was already confirmed, please try signing in) would be displayed.

Possible implications

Considering that there are records with an empty confirmation_token in the database, a request sending a blank parameter would confirm the first record found in the database. This means that someone’s account would be confirmed by mistake.

A more sensible scenario is for applications that automatically sign in accounts after confirmation. That would not only confirm someone’s account but also give an attacker access to it. Although this feature is not included in Devise, we know that some applications might have it.

For already confirmed records – i.e. with a confirmed_at date in the database – the validation error would be displayed, but their email would be leaked to the end user inside the form’s input.

Solution

The solution was to validate whether the confirmation_token is empty before doing any query in the database. So now, when the same endpoint is hit, nothing happens:

Processing by Devise::ConfirmationsController#show as HTML
  Parameters: {"confirmation_token"=>""}
Completed 200 OK in 371ms (Views: 339.0ms | ActiveRecord: 5.9ms)
Code language: PHP (php)

And the end-user will see the validation error on the screen.

See the pull request with the solution for more information.

Actions you might want to take

Aside from updating Devise, you might also want to check whether you have records in that state in your application. You can do that with a query like this one:

irb(main):001:0> User.where(confirmation_token: "")
  User Load (0.6ms)  SELECT  "users".* FROM "users" WHERE "users"."confirmation_token" = $1 LIMIT $2  [["confirmation_token", ""], ["LIMIT", 11]]
=> #<ActiveRecord::Relation []>
Code language: PHP (php)

If you get results out of this query, you might want to nullify them to avoid the confirmation by mistake:

irb(main):002:0> User.where(confirmation_token: "").update(confirmation_token: nil)
  User Load (0.4ms)  SELECT "users".* FROM "users" WHERE "users"."confirmation_token" = $1  [["confirmation_token", ""]]
   (0.2ms)  BEGIN
  User Update (0.7ms)  UPDATE "users" SET "confirmation_token" = $1, "updated_at" = $2 WHERE "users"."id" = $3  [["confirmation_token", nil], ["updated_at", "2019-09-05 14:03:20.857488"], ["id", 1]]
   (1.2ms)  COMMIT
Code language: JavaScript (javascript)

Causes

Although we received a report where someone worked in an application that had records with a blank confirmation token in the database, no code or use case was found inside Devise that would make records end up in that state.

If you find a case where this happens, please contact us at opensource@plataformatec.com.br and we’ll look at it.

Finally, we want to thank Anthony Mangano for reporting this issue and helping with the solution.

The post Improve confirmation token validation in Devise (CVE-2019-16109) first appeared on Plataformatec Blog.

But don’t be upset, it turns out you might not need to override Devise’s SessionsController or monkey patch some of its internals. In this article, you’ll learn how to create a token-based authentication for a JSON API by relying on Warden’s features.

Disclaimers

Warden? Huh?

This article will focus on how to include custom Warden strategies in a Rails application that uses Devise. If you want to know more about Warden strategies, I gave a talk last year at RailsConf that explains them in more details.

Show me the code!

The first part of this article will show how to set up a Rails application using Devise. If you want to skip to the token authentication part, click here.

Setup

Create a new Rails application (this example uses Postgres as the database to take advantage of UUIDs to generate the access tokens):

rails new devise-token-based-auth --database=postgresql

Add the devise gem to your Gemfile:

gem 'devise'

Now run the Devise generators:

rails generate devise:install
rails generate devise User

We are going to need a column to store the api_token. For this, we’ll use the pgcrypto extension’s gen_random_uuid() function.

First, create a migration to enable the pgcrypto extension:

rails generate migration enable_pgcrypto_extension

Now edit the migration to call the #enable_extension method:

class EnablePgcryptoExtension < ActiveRecord::Migration[5.2]
  def change
    enable_extension 'pgcrypto'
  end
end

The database is now able to use pgcrypto‘s functions. Now we can create a migration to add the api_token column:

rails generate migration add_api_token_to_users

Now edit the migration like the one below. Notice the default is set to gen_random_uuid():

class AddApiTokenToUsers < ActiveRecord::Migration[5.2]
  def change
    add_column :users, :api_token, :string, default: -> { 'gen_random_uuid()' }
    add_index :users, :api_token, unique: true
  end
end

Don’t forget to create the database and run the migrations:

rails db:create db:migrate

The next step is to create a user using rails console and grab its api_token:

rails console
Running via Spring preloader in process 60784
Loading development environment (Rails 5.2.2)
irb(main):001:0> user = User.create!(email: 'bruce@wayne.com', password: '123123')
=> #
irb(main):002:0> user.reload.api_token
=> "a4839b85-4c96-4f22-96f1-c2568e5d6a7f"

It’s time to create the Warden strategy now!

The Api Token Strategy

Create a file app/strategies/api_token_strategy.rb with the following content:

class ApiTokenStrategy < Warden::Strategies::Base
  def valid?
    api_token.present?
  end

  def authenticate!
    user = User.find_by(api_token: api_token)

    if user
      success!(user)
    else
      fail!('Invalid email or password')
    end
  end

  private

  def api_token
    env['HTTP_AUTHORIZATION'].to_s.remove('Bearer ')
  end
end

In short, the strategy tries to find a user for the token sent in the Authorization header. If it does, it signs the user in. Otherwise, it returns an error. If you are not familiar with the success! and fail! methods, watch the talk on the start of the blog post to get a sense on how Warden works.

Warden needs to know about this strategy. Create a file config/initializers/warden.rb with the following code:

Warden::Strategies.add(:api_token, ApiTokenStrategy)

This allows Warden to recognise that it should call the ApiTokenStrategy when it receives the :api_token symbol.

Authenticating a user

Now it’s time to use the strategy. Create an UsersController that renders the current_user in JSON:

class UsersController < ApplicationController
  def show
    render json: current_user.to_json
  end
end

Don’t forget to add a route for this controller action. Open config/routes.rb in your editor and include the following:

Rails.application.routes.draw do
  devise_for :users
  resource :user, only: :show
end

To require authentication in the controller, the method #authenticate! should be called passing the desired strategy as a parameter:

class UsersController < ApplicationController
  def show
    warden.authenticate!(:api_token)
    render json: current_user.to_json
  end
end

You can see that this works using a simple curl request:

curl http://localhost:3000/user -H 'Authorization: Bearer a4839b85-4c96-4f22-96f1-c2568e5d6a7f'

{"id":3,"email":"bruce@wayne.com","created_at":"2018-12-26T13:45:37.473Z","updated_at":"2018-12-26T13:45:37.473Z","api_token":"a4839b85-4c96-4f22-96f1-c2568e5d6a7f"}

It is also possible to define :api_token as a default strategy so that it’s called when no strategy is passed as a parameter. Add the following code in the config/initializers/devise.rb file:

Devise.setup do |config|
   # The secret key used by Devise. Devise uses this key to generate...
   config.warden do |manager|
     manager.default_strategies(scope: :user).unshift :api_token
   end

# ==> Mountable engine configurations...
end

This will add the :api_token strategy in the first position, followed by Devise’s default strategies (:rememberable and :database_authenticatable).

Now it’s possible to use Devise’s #authenticate_user! helper, and the :api_token will still be used:

class UsersController < ApplicationController
  before_action :authenticate_user!

  def show
    render json: current_user.to_json
  end
end

Summary

And… we’re done! The focus here was to show how to include custom Warden strategies in a Rails application. The example was straightforward but you can follow this structure to create custom authentication logic to suit your application’s needs.

The entire application used in this article can be found in GitHub.

The post Custom authentication methods with Devise first appeared on Plataformatec Blog.

A flexibilidade de poder armazenar os dados sem se preocupar com a estrutura das tabelas parece interessante, porém qual o impacto disso na performance?

A verdade é o que os dados do tipo JSON são armazenados da maneira como foram inseridos – isto é, em texto – o que deixa as consultas um pouco mais lentas já que os dados precisam ser parseados novamente. Porém, existe uma outra variação: o JSONB. Esse formato processa o texto no momento da inserção e o armazena em formato binário. Isso faz com que a inserção demore um pouco mais, em troca de consultas mais rápidas. Ah, e o JSONB também suporta índices!

Nesse artigo você vai encontrar uma comparação dos tipos de índices disponíveis para JSONB no Postgres, um benchmark e como usá-los em aplicações Rails.

Setup

Para testar a performance de índices, é importante ter um banco populado com uma quantidade razoável de registros. Abaixo você encontrará as etapas necessárias para criar o banco de dados utilizado nesse benchmark.
Para acompanhar as etapas à seguir, precisaremos do Postgres devidamente instalado e rodando.

Iremos importar cerca de 600 mil registros na estrutura abaixo:

{

        "customer_id": "ATVPDKIKX0DER",
        "product": {
            "category": "Arts & Photography",
            "group": "Book",
            "id": "1854103040",
            "sales_rank": 72019,
            "similar_ids": [
                "1854102664",
                "0893815381",
                "0893816493",
                "3037664959",
                "089381296X"
            ],
            "subcategory": "Art",
            "title": "The Age of Innocence"
        },
        "review": {
            "date": "1995-08-10",
            "helpful_votes": 5,
            "rating": 5,
            "votes": 12
        }
   }

Para importar esses dados, faça o download do arquivo e siga as instruções abaixo:

1. Criar banco de dados

❯ psql

CREATE DATABASE store;

2. Criar tabela

Vamos criar a tabela reviews com um único campo (content) para armazenar o conteúdo de cada registro:

❯ psql store

CREATE TABLE reviews(content jsonb);

3. Importar os dados

Por fim, vamos importar os dados do JSON que acabamos de baixar utilizando o comando copy dentro do console do Postgres:

copy reviews (content) FROM 'customer_reviews_nested_1998.json'

B-tree

O índice do formato B-tree funciona somente com uma chave do documento e para um operador específico: isto é, o operador usado na definição do índice deverá ser o mesmo utilizado na consulta:

CREATE INDEX ON reviews((content #>> '{product,category}'));

Para que o índice seja utilizado, devemos fazer a consulta com o mesmo operador:

SELECT COUNT(*) FROM reviews WHERE content #>> '{product,category}' = 'Arts & Photography';

Se utilizarmos outros operadores o índice não será aplicado, e consequentemente a busca ficará mais lenta:

SELECT COUNT(*) FROM reviews WHERE content->'product'->>'category' = 'Arts & Photography';

O índice para a consulta acima seria o seguinte:

CREATE INDEX ON reviews((content->'product'->>'category'));

Se for necessário responder perguntas como “quais as reviews tem rating 4 ou mais” ou “quais reviews não são de livros” – operadores >, =, <, >=, <= e != – esse é o índice mais indicado.

Porém fica uma observação aqui: nos testes feitos nesse benchmark, os índices para esses operadores acabaram não valendo a pena, o tempo de execução foi o mesmo. Mas pode ser que esse não seja sempre o caso, por isso é importante fazer benchmarks no seu ambiente para ter uma escolha mais assertiva.

Prós:

• Melhor performance na consulta;
• Mais rápido para gerar;
• Utiliza menos espaço em disco;
• Suporta os operadores >, =, <, >=, <= e !=.

Contras:

• Somente um operador de consulta;
• Se for necessário buscar por várias chaves do documento, será preciso criar um índice para cada uma.

Hash

Apenas uma menção honrosa. Sua funcionalidade é muito similar ao B-tree, porém o próprio banco já lança um warning ao criá-lo. O warning informa que o Hash não suport algo chamado WAL e, portanto, seu uso é desencorajado:

CREATE INDEX ON reviews USING HASH((content #>> '{product,category}'));
WARNING:  hash indexes are not WAL-logged and their use is discouraged

O Write-Ahead Logging (WAL) é um padrão que garante a integridade dos dados. De forma resumida, ele garante que as alterações nos arquivos do banco de dados só sejam salvas depois que essas mudanças forem salvas em um log. Isso permite que, se houver um crash no banco, seja possível aplicar as mudanças perdidas com base no log.
Caso queira saber mais sobre WAL, recomendo ler a documentação de Postgres.

Por não suportar o WAL, o Hash acaba sendo um pouco mais rápido que o B-tree, porém ele não foi incluído no benchmark por conta de seu uso ser desencorajado.

Atualização: a partir do Postgres 10, os índices de tipo Hash passaram a suportar o WAL. Portanto, pode valer a pena considerar essa opção na hora de escolher qual índice utilizar.

GIN (General inverted Index)

A partir do Postgres 9.4, podemos utilizar esse novo formato de índice para os documentos. Esse índice pode ser usado em todo o documento (com qualquer chave).

CREATE INDEX on reviews USING GIN(content);

Nesse caso podemos realizar a consulta dessa forma:

SELECT COUNT(*) FROM reviews WHERE content @> '{"product": {"category": "Arts & Photography"}}';

Também é possível utilizar os operadores ?, ?| e ?& – que são usados para saber se o documento contém alguma chave.

GIN (jsonb_path_ops)

Normalmente o operador mais usado é o @> , para saber se o documento “contém o pedaço” que for passado como argumento.
Se na sua aplicação for possível utilizar apenas esse operador, é interessante usar a variação jsonb_path_ops:

CREATE INDEX ON reviews USING GIN(content jsonb_path_ops);

Por suportar somente esse operador, o índice fica um pouco menor e as consultas mais performáticas.

Prós:

• Funciona com qualquer chave do documento;
• Um índice serve para consultas por qualquer chave.

Contras:

• Ocupa mais espaço em disco;
• Demora mais tempo para ser gerado;
• Consulta é mais lenta;
• Não permite utilizar os operadores >, =, <, >=, <= e !=.

Benchmarks

Os testes foram realizados em um MacBook Pro 13’ Early 2015 com 8GB de RAM, Core i5 2.7GHz e 120GB SSD.
O banco possuí 589.859 registros.

Criação do índice

Índice	Tempo (ms)
Btree	1207.831
GIN	20676.872
GIN (jsonb_path_ops)	8880.895

Consulta

Índice	Tempo (ms)
Btree	2.661
GIN	13.643
GIN (jsonb_path_ops)	3.985

Com o resultado do benchmark, podemos concluir que: se a ideia for utilizar operadores de comparação simples (>, =, <, >=, <= e !=) ou apenas um chave do documento, a melhor opção é o B-tree.
Se as consultas necessitarem de maior flexibilidade – buscar por várias chaves do documento – a melhor opção é o GIN. E se você utilizar somente o operador @>, vale a pena utilizar a variação jsonb_path_ops do GIN.

Uso em aplicações Rails

Os índices acima contém expressões de SQL, então como podemos criar esses tipos de índice no Rails? Bom, se sua aplicação estiver usando Rails 4 ou anterior, o jeito é usar o método execute que aceita um SQL literal como argumento:

class AddIndex < ActiveRecord::Migration
  def up
    execute 'CREATE INDEX index_reviews_on_content ON reviews USING GIN(content jsonb_path_ops)'
  end

  def down
    remove_index :reviews, name: 'index_reviews_on_content'
  end
end

Por conta disso também é preciso especificar como a migration pode ser revertida – ou seja, implementar o método down.

A boa notícia é que, a partir do Rails 5 podemos utilizar expressões SQL na criação dos índices:

class AddIndex < ActiveRecord::Migration[5.0]
  def up
    add_index :reviews, 'content jsonb_path_ops', using: :gin, name: 'index_reviews_on_content'
  end

  def down
    remove_index :reviews, name: 'index_reviews_on_content'
  end
end

Em um próximo post, vamos mostrar alguns exemplos de como fazer queries e updates utilizando Rails e JSONB. Fique ligado!

Links de referência

https://www.postgresql.org/docs/9.5/static/functions-json.html

https://www.postgresql.org/docs/9.5/static/datatype-json.html

https://blog.heapanalytics.com/when-to-avoid-jsonb-in-a-postgresql-schema/

http://schinckel.net/2014/05/25/querying-json-in-postgres/

PostgreSQL internals: JSONB type and its indexes

https://blog.2ndquadrant.com/jsonb-type-performance-postgresql-9-4/

https://stackoverflow.com/questions/36075918/postgresql-index-on-json

http://blog.bigbinary.com/2016/07/20/rails-5-adds-support-for-expression-indexes-for-postgresql.html

The post Índices para JSONB no Postgres first appeared on Plataformatec Blog.

In some period of time, we’re working on an address autocomplete feature using Google places. Suddenly, we receive an error report complaining about an undefined object in Javascript. Something like this:

Uncaught TypeError: Cannot read property 'maps' of undefined

A developer looked at the stacktrace, then, found the problematic line. The line was something like this:

const autocomplete = new google.maps.places.Autocomplete(input, options);

After a moment, the developer opened a Pull Request like this:

// Sometimes google is undefined
if (google) {
const autocomplete = new google.maps.places.Autocomplete(input, options);
}

Can you say if it’s a good solution? I can say it’s an answer. Sometimes this kind of code can be a solution. However, when I see a solution like this, too easy, it sets my spidey sense off. Let’s see why it’s not good.

That code initializes the Autocomplete widget. It means, when google is undefined, the address suggestion is not working. The if might prevent an error report, but it won’t fix the problem. This version now is worse than the previous one. Why? Now, users still can’t have the address suggestion and the team will not be aware of the problem. The code is hiding the error, not fixing the users’ problem.

You might think it was a junior mistake, but I can tell you, I could see many developers with many years of experience doing that. Everybody can make a mistake, that’s why we have a process of reviewing other people’s code to prevent errors like this.

A bad context can make a good professional produce poor results. If the leader presses the team to make the errors reports go down fast, mistakes like that become more common. People start finding the easiest way of stopping that error of being reported.

“People with targets […] will probably meet the targets – even if they have to destroy the enterprise to do it.” – Deming pic.twitter.com/EJZLIZ50zT

— Lindsay Holmwood (@auxesis) February 5, 2017

The target should not be reducing the errors reported. The goal should be fixing the users’ problems. For every bug reported by a bug track tool, ask yourself: Which problem is it generating for users? Then, make that problem your target to fix, not the error reported. This mindset that generates better solutions, a better product, a better codebase.

If you are curious to know a good solution for the example problem, I’ll explain to you. That Autocomplete class is provided by a third party script included in the HTML document. The problem was a race condition. Our code was executing faster than the script being loaded and available. A good fix for that would be only executing that code after the Google’s script initialization. Thankfully, you can pass a function to be executed as an argument to Google’s script inclusion. Example:

<script src="https://maps.googleapis.com/maps/api/js?key=YOUR_API_KEY&libraries=places&callback=initAutocomplete" async defer></script>

We passed the functioninitAutocomplete to initialize our address suggestion widget. It’s a good answer that fixes the user’s problem. It does not hide the error.

Every time you see a solution for an error, ask yourself: is that really fixing the problem? Try to figure out how that error impacts the end users. Remember, is better having errors reported than invisible problems. Avoid defensive code that hides errors instead of fixing them.

The post Stop hiding the error and start fixing the problem first appeared on Plataformatec Blog.

You can try the new Rails version using the published guides and report any problem you may find on Rails issue tracker.

Why update?

I think it is an important question you may need to answer to your team members. My favorite reasons are: security, bug fixes, and features.

Rails has supported versions to receive security fix releases. If you are in a version not supported, you may be vulnerable. You should read the security policies and check if your app is using the supported versions.

The framework also has maintenance policies that you should be aware of. Performance improvements and bug fixes you may miss if your app is too old. You need to code by yourself and do workarounds to have the same benefits, a code that would be more reliable being inside the framework.

We usually hear developers complaining about applications that use old versions of the framework. The reason is new versions of the tool usually bring improvements to make the developer’s life easier. In the developer’s perspective, it’s demotivating knowing there’s a robust, elegant and productive way to do things and they are not able to use it.

Keeping your Rails up to date will help your code base health and also can be a factor of motivation for your team.

What I should do?

We have been maintaining and evolving several old Rails apps in different contexts for years, and we have seen some practices that make it easier to keep your app updated:

• strict use of gems
• avoid monkey patches
• keep a good test coverage

Gems

Adding new dependencies using RubyGems and bundler is awesome, but overusing gems can not only slow your app down but slow you with the amount of work you need to update your Rails version. Some gems are coupled to the framework when you update it. These gems may break. When they break, you need to update them together.

I recommend you considering these points before putting a new gem in your Gemfile:

• Is this gem adding great value to your project? Some gems increase your app security, solve complex problems and reduce a lot of worktimes. Other gems add less value and can easily be replaced with your own implementation. Sometimes, it’s worth doing your own implementation than depending on a third­party code.

• Is this gem well maintained? It’s good checking if the commits are recent, the project is well documented, the changelog contains helpful messages between releases, the maintainers often close issues and answer them respectfully. These are good signs that the gem you are adding won’t give you trouble in the future.

Adding a gem to your project is adding a coupling. Software coupling has its downsides, for example, the ripple effect of changes. The downside of more work to update your dependencies will be worth it if you accurately added them. If you want to learn more about downsides of dependencies, you can read this blog post about Ruby dependencies and this one about Node.

Avoid monkey patches

Monkey patches are often used to change the behavior of the code you don’t own. For example, rewriting or adding Ruby standard library classes or methods to fit your application needs. Careless monkey patches can bring you serious problems while updating Rails.

Adding code that changes the behavior of gem classes can result in hidden bugs and turn any upgrade into a painful experience. It’s often uncertain how many objects are depending on your change and predicting all the effects of the monkey patched method. For example, in a new version of the gem, your monkey patch can change a method that was updated or doesn’t even exist anymore.

We have seen some monkey patches that are a gem fix or enhancement. These changes may benefit others users of the gem and were hidden in application code. You may get in touch with gem maintainers and propose your changes. If your contribution is accepted, your fix will have a solid and proper place to be.

If your behavior is specific for your app, you should consider extending the gem and applying your wanted behavior using the conventional OO practices. For example, you can create an adapter class, subclass or composition. You may also take a look at Ruby Refinements. Using this structure, you can create scoped changes that must be used explicitly. Explicit code reduces costs to maintain the app and the effort of updating to a new Rails version.

After checking all previous solutions and you still think that a monkey patch better suits your needs, you should know there are organized ways of adding them. Regardless of coding, I recommend you to add a great documentation to keep it sane for your teammates and your future self. In this documentation, you should describe why it is necessary, when it can be removed and how it can be removed. It will help a better understanding of the app and provide useful information while updating your Rails app.

Keep a good test coverage

We all need to pay special attention to the tests. Some changes while updating Rails will be required and having a trustful test suite will help you discover problems before deploying your app to your end users.

Having 100% of test coverage is no guarantee of testing the correct behavior. I would say the team should have a test coverage that they feel confident about.

Some symptoms that your application doesn’t have good coverage is when recurrent errors are found while the application is running and all your automated tests are passing. You may use a tool like simplecov to check your coverage rate, having a very low coverage is a bad sign.

Adding proper tests is the only solution for low test coverage. If you are in this situation, you should start adding tests focused on the most used and important features. Having a good test coverage is essential to evolve because its main purpose is providing fast feedback that previously working features are still working.

Conclusion

Keeping the application’s Rails version updated is a responsibility that should not be ignored since it brings improvements for your team and security for your users.

I hope these tips help you keep your code easy to update.

Do you have more tips? Please leave comments below and tell us about your experience to keep your application up to date.

The post Keeping your Ruby on Rails app easy to update first appeared on Plataformatec Blog.

Kasper and I worked a lot changing the underlying implementation of the sanitize helper to give Rails developers a more robust, faster and secure solution to sanitize user input.

This new implementation should be fully backward compatible, with no changes to the API, which should make the update easier.

You can see more information about the previous and the new implementation on this talk I presented in a Brazillian conference this year (the slides are in English).

Now, I’ll let Kasper share his words with you.

Scrubbing Rails Free of HTML-scanner

Everyone, at least one time, has already needed to use the sanitize method to scrub some pesky HTML away.

<%= sanitize @article.body %>

If you were to run this on Rails 4.1 (and before) this would take advantage of the html-scanner, a vendored library inside Rails, for the sanitization. Since the summer of 2013 I have been working to destroy that notion by wiping the traces of html-scanner throughout Rails. Before you become concerned of my mental health, I didn’t do this unwarranted. I’m one of the Google Summer of Code students working on Ruby on Rails. My project proposal was to kick html-scanner to the curb (technical term) and grab a hold of Loofah instead. Why did the old library need replacing, though?

The out washed HTML-scanner

html-scanner has been with us for a long time now. The copyright notice in the library clocks it in at 2006, when Assaf Arkin created it. This library relies on Regular Expressions to recognize HTML (and XML) elements. This made the code more brittle. It was easier to introduce errors via complex Regular Expressions, which also gave it a higher potential for security issues.

The Rails Team wanted something more robust and faster, so we picked Loofah. Loofah uses Nokogiri for parsing, which provides a Ruby interface to either a C or Java parser depending on the Ruby implementation you use. This means Loofah is fast. It’s up to 60 to 100% faster than html-scanner on larger documents and fragments.

I started by taking a look at the SanitizeHelper in Action View, which consists of four methods and some settings. The four methods of the are sanitize, sanitize_css, strip_tags and strip_links.

Let’s take a look at the sanitize method.

Comparing with the old implementation, sanitize still uses the WhiteListSanitizer class to do it’s HTML stripping. However, since Action View was pulled out of Action Pack and both needed to use this functionality, we’ve extracted this to it’s own gem.

Developers meet Rails::Html::WhiteListSanitizer

When you use sanitize, you’re really using WhiteListSanitizer‘s sanitize method. Let me show you the new version.

def sanitize(html, options = {})
  return nil unless html
  return html if html.empty?

No surprises here.

  loofah_fragment = Loofah.fragment(html)

The first trace of Loofah. A fragment is a part of a document, but without a DOCTYPE declaration and html and body tags. A piece of a document essentially. Internally Nokogiri creates a document and pulls the parsed html out of the body tag, leaving us with a fragment.

  if scrubber = options[:scrubber]
    # No duck typing, Loofah ensures subclass of Loofah::Scrubber
    loofah_fragment.scrub!(scrubber)

You can pass your own Scrubber to sanitize! Giving you the power to choose if and how elements are sanitized. As the comment alludes, any scrubber has to be either a subclass of Loofah::Scrubber or it can wrap a block. I’ll show an example later.

  elsif allowed_tags(options) || allowed_attributes(options)
    @permit_scrubber.tags = allowed_tags(options)
    @permit_scrubber.attributes = allowed_attributes(options)
    loofah_fragment.scrub!(@permit_scrubber)

We have been very keen on maintaining backwards compatibility throughout this project, so you can still supply Enumerables of tags and attributes to sanitize. That’s what the PermitScrubber used here handles. It manages these options and makes them work independently. If you pass one it’ll use the standard behavior for the other. See the documentation on what the standard behavior is.
You can also set the allowed tags and attributes on the class level. Like this:

Rails::Html::Sanitizer.allowed_tags = Set.new %w(for your health)

That’s simply what allowed_tags and allowed_attributes methods are there for. They’ll return the tags or attributes from the options and fallback to the class level setting if any.

  else
    remove_xpaths(loofah_fragment, XPATHS_TO_REMOVE)
    loofah_fragment.scrub!(:strip)
  end

The StripScrubber built in to Loofah will strip the tags but leave the contents of elements. Which is usually what we want. We use remove_xpaths to remove elements along with their subtrees in the few instances where we don’t. If you have trouble with the syntax above, they’re XPath Selectors.

  loofah_fragment.to_s
end

Lastly we’ll take the elements and extract the remaining markup with to_s. Internally Nokogiri will call either to_xml or to_html depending on the kind of document or fragment you have.

Rub, buff or clean it off, however you like

So there you have it. I could go through how the other sanitizers work, but they’re not that complex. So go code spelunking in the source.

If this was the first time you’ve seen a Loofah::Scrubber, be sure to check out the source for PermitScrubber and see an example of how to implement one. You can also subclass PermitScrubber and get the sanitization you need without worrying about the implementation details of stripping elements and scrubbing attributes. Take a look at TargetScrubber – the weird PermitScrubber – and how it uses that to get scrubbing fast.

Before I scrub off though, I promised you an example of a custom scrubber. I’ll use the option that wraps a block here, but you could easily create a subclass of Loofah::Scrubber (in a helper maybe?) and override scrub(node). So here goes:

<%= sanitize @article.body,
  scrubber: Loofah::Scrubber.new { |node| node.name = "script" } %>

The code above changes all the HTML tags included in the article body to be a tag <script>.

<sarcasm>
If you’re going to introduce bugs, why not make everything a potential risk of running arbitrary code?
</sarcasm>

The post The new HTML sanitizer in Rails 4.2 first appeared on Plataformatec Blog.