Posts tagged "rails"

The article below was originally written by Kasper Timm Hansen (@kaspth on github & twitter) about his work during the Google Summer of Code 2013.

Kasper and I worked a lot changing the underlying implementation of the sanitize helper to give Rails developers a more robust, faster and secure solution to sanitize user input.

This new implementation should be fully backward compatible, with no changes to the API, which should make the update easier.

You can see more information about the previous and the new implementation on this talk I presented in a Brazillian conference this year (the slides are in English).

Now, I’ll let Kasper share his words with you.

Scrubbing Rails Free of HTML-scanner

Everyone, at least one time, has already needed to use the sanitize method to scrub some pesky HTML away.

<%= sanitize @article.body %>

If you were to run this on Rails 4.1 (and before) this would take advantage of the html-scanner, a vendored library inside Rails, for the sanitization. Since the summer of 2013 I have been working to destroy that notion by wiping the traces of html-scanner throughout Rails. Before you become concerned of my mental health, I didn’t do this unwarranted. I’m one of the Google Summer of Code students working on Ruby on Rails. My project proposal was to kick html-scanner to the curb (technical term) and grab a hold of Loofah instead. Why did the old library need replacing, though?

The out washed HTML-scanner

html-scanner has been with us for a long time now. The copyright notice in the library clocks it in at 2006, when Assaf Arkin created it. This library relies on Regular Expressions to recognize HTML (and XML) elements. This made the code more brittle. It was easier to introduce errors via complex Regular Expressions, which also gave it a higher potential for security issues.

The Rails Team wanted something more robust and faster, so we picked Loofah. Loofah uses Nokogiri for parsing, which provides a Ruby interface to either a C or Java parser depending on the Ruby implementation you use. This means Loofah is fast. It’s up to 60 to 100% faster than html-scanner on larger documents and fragments.

I started by taking a look at the SanitizeHelper in Action View, which consists of four methods and some settings. The four methods of the are sanitize, sanitize_css, strip_tags and strip_links.

Let’s take a look at the sanitize method.

Comparing with the old implementation, sanitize still uses the WhiteListSanitizer class to do it’s HTML stripping. However, since Action View was pulled out of Action Pack and both needed to use this functionality, we’ve extracted this to it’s own gem.

Developers meet Rails::Html::WhiteListSanitizer

When you use sanitize, you’re really using WhiteListSanitizer‘s sanitize method. Let me show you the new version.

def sanitize(html, options = {})
  return nil unless html
  return html if html.empty?

No surprises here.

  loofah_fragment = Loofah.fragment(html)

The first trace of Loofah. A fragment is a part of a document, but without a DOCTYPE declaration and html and body tags. A piece of a document essentially. Internally Nokogiri creates a document and pulls the parsed html out of the body tag, leaving us with a fragment.

  if scrubber = options[:scrubber]
    # No duck typing, Loofah ensures subclass of Loofah::Scrubber
    loofah_fragment.scrub!(scrubber)

You can pass your own Scrubber to sanitize! Giving you the power to choose if and how elements are sanitized. As the comment alludes, any scrubber has to be either a subclass of Loofah::Scrubber or it can wrap a block. I’ll show an example later.

  elsif allowed_tags(options) || allowed_attributes(options)
    @permit_scrubber.tags = allowed_tags(options)
    @permit_scrubber.attributes = allowed_attributes(options)
    loofah_fragment.scrub!(@permit_scrubber)

We have been very keen on maintaining backwards compatibility throughout this project, so you can still supply Enumerables of tags and attributes to sanitize. That’s what the PermitScrubber used here handles. It manages these options and makes them work independently. If you pass one it’ll use the standard behavior for the other. See the documentation on what the standard behavior is.
You can also set the allowed tags and attributes on the class level. Like this:

Rails::Html::Sanitizer.allowed_tags = Set.new %w(for your health)

That’s simply what allowed_tags and allowed_attributes methods are there for. They’ll return the tags or attributes from the options and fallback to the class level setting if any.

  else
    remove_xpaths(loofah_fragment, XPATHS_TO_REMOVE)
    loofah_fragment.scrub!(:strip)
  end

The StripScrubber built in to Loofah will strip the tags but leave the contents of elements. Which is usually what we want. We use remove_xpaths to remove elements along with their subtrees in the few instances where we don’t. If you have trouble with the syntax above, they’re XPath Selectors.

  loofah_fragment.to_s
end

Lastly we’ll take the elements and extract the remaining markup with to_s. Internally Nokogiri will call either to_xml or to_html depending on the kind of document or fragment you have.

Rub, buff or clean it off, however you like

So there you have it. I could go through how the other sanitizers work, but they’re not that complex. So go code spelunking in the source.

If this was the first time you’ve seen a Loofah::Scrubber, be sure to check out the source for PermitScrubber and see an example of how to implement one. You can also subclass PermitScrubber and get the sanitization you need without worrying about the implementation details of stripping elements and scrubbing attributes. Take a look at TargetScrubber – the weird PermitScrubber – and how it uses that to get scrubbing fast.

Before I scrub off though, I promised you an example of a custom scrubber. I’ll use the option that wraps a block here, but you could easily create a subclass of Loofah::Scrubber (in a helper maybe?) and override scrub(node). So here goes:

<%= sanitize @article.body,
  scrubber: Loofah::Scrubber.new { |node| node.name = "script" } %>

The code above changes all the HTML tags included in the article body to be a tag <script>.

<sarcasm>
If you’re going to introduce bugs, why not make everything a potential risk of running arbitrary code?
</sarcasm>


We just released Simple Form 3.1.0.rc1 with support to Bootstrap 3.

To make it possible, we leveled up the Wrapper API to make it more extensible and to allow developers to directly configure it instead of relying on global state. After such improvements, it was very easy to change the Simple Form configuration to work with Bootstrap 3. I’ll talk more about this in a future post.

Integrating Bootstrap with Simple Form should be as easy as it was before. We are working on the documentation before the final release but you can find examples about how to integrate Simple Form with Bootstrap in our sample application. The app source code is available too.

Besides this main feature, Simple Form’s new 3.1.0 comes with a lot of enhancements. You can find the whole list of changes in the CHANGELOG file.

I hope you are excited about this release as much as we are. Please try out the release candidate and if you find an issue, report at the Simple Form issues tracker.

We plan to release the final version in a month or so, and we’ll write a new blog post with more details about Simple Form 3.1.

Acknowledgments

We, at Plataformatec, have worked hard to get this release out, with Lauro Caetano and Rafael França (it’s me!) working together on this final sprint. Also kudos to Gonzalo Rodríguez-Baltanás Díaz and the whole community for helping us to improve the sample application.


Rails 4.1 was just released this week and I already had a great experience trying out the release candidates on my latest project, so I decided to write a bit about my favorites features on this release and some things I have learned by using them so far.

1) secrets.yml

Placing your configuration in a YAML file isn’t exactly a revolutionary feature, but the usage of the config/secrets.yml file that comes with Rails 4.1 holds a more important idea: the promise of a default approach for environment aware custom configuration on Rails applications. Over the years the community created several ways to manage such configuration so every app out there deals with this differently, but now we can use the Rails default as a standard just like we do with the app folder or the routing patterns, taking the configuration madness outside the list of things to worry about when working with Rails. So instead of dealing with multiple YAML files or constants left out inside initializers, we can go with the secrets.yml as the default for our apps.

Remember that you can place any kind of configuration – not just secrets like tokens or passwords – that need to be handled differently through your application environments, like API Endpoints or S3 bucket names. And for any gem maintainers out there, you can make your gem read these settings from the secrets.yml automagically through an initializer block and maybe remove a configuration step from the gem setup. Adding this to Devise on this pull request was easier than I expected and I suggest you to try it out on your gems as well.

If you want to try to organize your configuration through the secrets.yml without having to update to Rails 4.1 right now, Andrew White backported this feature on the rails-secrets gem for Rails 4.0 apps.

So, if you are dealing with some configuration mess or aren’t using something like dotenv for your application, I strongly suggest that you try to migrate your config to use the secrets.yml file and see how it goes for your application.

2) Action Pack Variants

Variants are proving to be a great solution to render device specific views when mixed with any device detection solution like the useragent or browser gems, which you integrate quickly with just a before_action block:

class ApplicationController < ActionController::Base
  before_action :set_variant
 
  private
 
  def set_variant
    if browser.tablet?
      request.variant = :tablet
    elsif browser.mobile?
      request.variant = :mobile
    else
      request.variant = :desktop
    end
  end
end

Even though the main examples are dealing with User Agent sniffing, this feature can be used in any context where you want to have more control of which views are rendered by your application, like:

  • A/B Testing different partials based on the user cookies.
  • API versioning for your Jbuilder templates.
  • Maintaining current and redesigned views for the same controller.
  • Authorization aware views, like users/index.html+admin.erb or products/show.html+guest.erb.

In the end, Variants are just a way for you to have more control over how your views will be used by the app, helping you to remove boilerplate logic from your code and letting the framework handle it through a more elegant solution.

3) The improved cookies serializer

The changes on how Rails serializes cookies are a great improvement when it comes to security and stability of web apps. Before this, any object placed in the cookies Hash would be serialized (and deserialized) through Marshal.dump and Marshal.load, which could possibly lead to remote code execution if an attacker got hold on your application secret.

Now this serializer is configurable through the config.action_dispatch.cookies_serializer configuration option, and new apps will ship with a smarter default: a JSON serializer that won’t recreate complex objects besides Strings, Integers and other JSON data types. And for a smooth upgrade, you can use the :hybrid serializer that will convert your existing marshalled cookies into JSON cookies, so this upgrade can be transparent for your application users.

This upgrade highlights a possible bad practice in our applications where we end up placing more complex objects in the session that can’t be completely restored by the JSON serializer, when we should be using more simple structures for the data stored in cookies. Thanks to a related issue reported on the Devise issue tracker we could simplify the gem code a bit, so instead of serializing Time objects we could work with numbers.

So, when updating your application to use the :hybrid serializer, don’t forget to do a double check of whatever kind of data the app stores in your users cookies and look for possible backwards incompatibility. And if you want to take a closer look on how this was implemented, be sure to check the related issues and pull requests on the Rails repo: #12881, #13692 and #13945.

Keeping up to date with the latest Rails changes

Following the activity on the Rails repository over GitHub helped a lot to understand better these features and the rationale behind their implementations, but going through all the commits and discussions on Issues and Pull Requests would demand a lot of your time. If you want some of the inside scoop but don’t have that much time to go through the Rails activity over the week, Godfrey Chan has put up a weekly digest about Rails named This Week in Rails. I suggest that you subscribe to the list and even check some of the previous editions on the archive page.

Try it yourself!

Take some time and upgrade one of your Rails 4 apps and try out some of the new features! I bet that some of them will help you improve your codebase or make your coworkers life a bit easier, and we are eager to hear from your experience with the 4.1 release.


We are glad to announce that Devise 3.1.0.rc is out. On this version, we have focused on some security enhancements regarding our defaults and the deprecation of TokenAuthenticatable. This blog post explains the rationale behind those changes and how to upgrade.

Devise 3.1.0.rc runs on both Rails 3.2 and Rails 4.0. There is a TL;DR for upgrading at the end of this post.

Note: We have yanked 3.1.0.rc and released to 3.1.0.rc2 which fixes some regressions. Thanks everyone for trying out the release candidates!

Do not sign the user in after confirmation

In previous Devise versions, the user was automatically signed in after confirmation. This meant that anyone that could access the confirmation e-mail could sign into someone’s account by simply clicking the link.

Automatically signing the user in could also be harmful in the e-mail reconfirmation workflow. Imagine that a user decides to change his e-mail address and, while doing so, he makes a typo on the new e-mail address. An e-mail will be sent to another address which, with the token in hands, would be able to sign in into that account.

If the user corrects the e-mail straight away, no harm will be done. But if not, someone else could sign into that account and the user would not know that it happened.

For this reason, Devise 3.1 no longer signs the user automatically in after confirmation. You can temporarily bring the old behavior back after upgrading by setting the following in your config/initializers/devise.rb:

config.allow_insecure_sign_in_after_confirmation = true

This option will be available only temporarily to aid migration.

Thanks to Andri Möll for reporting this issue.

Do not confirm account after password reset

In previous Devise versions, resetting the password automatically confirmed user accounts. This worked fine in previous Devise versions which confirmed the e-mail just on sign up, so the e-mail both confirmation and password reset tokens would be sent to were guaranteed to be the same. With the addition of reconfirmable, this setup change and Devise will no longer confirm the account after password reset.

Thanks to Andri Möll for reporting this issue and working with us on a fix.

CSRF on sign in

Devise’s sign in page was vulnerable to CSRF attacks when used with the rememberable feature. Note that the CSRF vulnerability is restricted only to the sign in page, allowing an attacker to sign the user in an account controlled by the attacker. This vulnerability does not allow the attacker to access or change a user account in any way.

This issue is fixed on Devise 3.1.0 as well as 3.0.2 and 2.2.6. Users on previous Devise versions can patch their application by simply defining the following in their ApplicationController:

def handle_unverified_request
  super
  Devise.mappings.each_key do |key|
    cookies.delete "remember_#{key}_token"
  end
end

Thanks to Kevin Dew for reporting this issue and working with us on a fix.

Store digested tokens in the database

In previous versions, Devise stored the tokens for confirmation, reset password and unlock directly in the database. This meant that somebody with read access to the database could use such tokens to sign in as someone else by, for example, resetting their password.

In Devise 3.1, we store an encrypted token in the database and the actual token is sent only via e-mail to the user. This means that:

  • Devise now requires a config.secret_key configuration. As soon as you boot your application under Devise 3.1, you will get an error with information about how to proceed;
  • Every time the user asks a token to be resent, a new token will be generated;
  • The Devise mailer now receives one extra token argument on each method. If you have customized the Devise mailer, you will have to update it. All mailers views also need to be updated to use @token, as shown here, instead of getting the token directly from the resource;
  • Any previously stored token in the database will no longer work unless you set config.allow_insecure_token_lookup = true in your Devise initializer. We recomend users upgrading to set this option on production only for a couple days, allowing users that just requested a token to get their job done.

Thanks to Stephen Touset for reporting this issue and working with us on a solution.

Token Authenticatable

Jay Feldblum also wrote to us to let us know that our tokens lookup are also vulnerable to timing attacks. Although we haven’t heard of any exploit via timing attacks on database tokens, there is a lot of research happening in this area and some attacks have been successful over the local network. For this reason, we have decided to protect applications using Devise from now on.

By digesting the confirmation, reset password and unlock tokens, as described in the previous section, we automatically protected those tokens from timing attacks.

However, we cannot digest the authentication token provided by TokenAuthenticatable, as they are often part of APIs where the token is used many times. Since the usage of the authenticatable token can vary considerably in between applications, each requiring different safety guarantees, we have decided to remove TokenAuthenticatable from Devise, allowing users to pick the best option. This gist describes two of the available solutions.

Thanks to Jay Feldblum for reporting this issue and working with us on a solution.

TL;DR for upgrading

As soon as you update Devise, you will get a warning asking you to set your config.secret_key. By upgrading Devise, your previous confirmation, reset and unlock tokens in the database will no longer work unless you set the following option to true in your Devise initializer:

config.allow_insecure_token_lookup = true

It is recommended to leave this option on just for a couple days, just to allow recently generated tokens by your application to be consumed by users. TokenAuthenticable has not been affected by those changes, however it has been deprecated and you will have to move to your own token authentication mechanisms.

Furthermore, the Devise mailer now receives an extra token argument on each method. If you have customized the Devise mailer, you will have to update it. All mailers views also need to be updated to use @token, as shown here, instead of getting the token directly from the resource.

With those changes, we hope to provide an even more secure authentication solution for Rails developers, while maintaining the flexibility expected from Devise.

One more thing, we’re writing a free ebook about Devise!

If you want to know more about Devise, we’re writing a free ebook about it.
Fill in the form below so that we can send you updates with new chapters and beta releases.


Devise has been reported to be vulnerable to CSRF token fixation attacks.

The attack can only be exploited if the attacker can set the target session, either by subdomain cookies (similar to described here) or by fixation over the same Wi-Fi network. If the user knows the CSRF token, cross-site forgery requests can be made. More information can be found here.

Note Devise is not vulnerable to session fixation attacks (i.e. the user cannot steal another user session by fixating the session id).

Releases

Devise 3.0.1 and 2.2.5 have been released with fixes for the attack.

If you can’t upgrade, you must protect your Devise application by adding the next three lines to a Rails initializer:

Warden::Manager.after_authentication do |record, warden, options|
  warden.request.session.try(:delete, :_csrf_token)
end

Notice the code above and the updated Devise versions will clean up the CSRF Token after any authentication (sign in, sign up, reset password, etc). So if you are using AJAX for such features, you will need to fetch a new CSRF token from the server.

Acknowledgements

We want to thank Egor Homakov for reporting the issue and working with us on a fix.

Here at Plataformatec we use Github Pull Requests a lot for code review and this usually yields tons of constructive comments and excellent discussions from time to time. One of the recent topics was about whether we should use scopes or class methods throughout the project to be consistent. It’s also not hard to find discussions about it all over the internet. The classic comment usually boils down to “there is no difference between them” or “it is a matter of taste”. I tend to agree with both sentences, but I’d like to show some slight differences that exist between both.

Defining a scope

First of all, lets get a better understanding about how scopes are used. In Rails 3 you can define a scope in two ways:

class Post < ActiveRecord::Base
  scope :published, where(status: 'published')
  scope :draft, -> { where(status: 'draft') } 
end

The main difference between both usages is that the :published condition is evaluated when the class is first loaded, whereas the :draft one is lazy evaluated when it is called. Because of that, in Rails 4 the first way is going to be deprecated which means you will always need to declare scopes with a callable object as argument. This is to avoid issues when trying to declare a scope with some sort of Time argument:

class Post < ActiveRecord::Base
  scope :published_last_week, where('published_at >= ?', 1.week.ago)
end

Because this won’t work as expected: 1.week.ago will be evaluated when the class is loaded, not every time the scope is called.

Scopes are just class methods

Internally Active Record converts a scope into a class method. Conceptually, its simplified implementation in Rails master looks something like this:

def self.scope(name, body)
  singleton_class.send(:define_method, name, &body)
end

Which ends up as a class method with the given name and body, like this:

def self.published
  where(status: 'published')
end

And I think that’s why most people think: “Why should I use a scope if it is just syntax sugar for a class method?”. So here are some interesting examples for you to think about.

Scopes are always chainable

Lets use the following scenario: users will be able to filter posts by statuses, ordering by most recent updated ones. Simple enough, lets write scopes for that:

class Post < ActiveRecord::Base
  scope :by_status, -> status { where(status: status) }
  scope :recent, -> { order("posts.updated_at DESC") }
end

And we can call them freely like this:

Post.by_status('published').recent
# SELECT "posts".* FROM "posts" WHERE "posts"."status" = 'published' 
#   ORDER BY posts.updated_at DESC

Or with a user provided param:

Post.by_status(params[:status]).recent
# SELECT "posts".* FROM "posts" WHERE "posts"."status" = 'published' 
#   ORDER BY posts.updated_at DESC

So far, so good. Now lets move them to class methods, just for the sake of comparing:

class Post < ActiveRecord::Base
  def self.by_status(status)
    where(status: status)
  end
 
  def self.recent
    order("posts.updated_at DESC")
  end
end

Besides using a few extra lines, no big improvements. But now what happens if the :status parameter is nil or blank?

Post.by_status(nil).recent
# SELECT "posts".* FROM "posts" WHERE "posts"."status" IS NULL 
#   ORDER BY posts.updated_at DESC
 
Post.by_status('').recent
# SELECT "posts".* FROM "posts" WHERE "posts"."status" = '' 
#   ORDER BY posts.updated_at DESC

Oooops, I don’t think we wanted to allow these queries, did we? With scopes, we can easily fix that by adding a presence condition to our scope:

scope :by_status, -> status { where(status: status) if status.present? }

There we go:

Post.by_status(nil).recent
# SELECT "posts".* FROM "posts" ORDER BY posts.updated_at DESC
 
Post.by_status('').recent
# SELECT "posts".* FROM "posts" ORDER BY posts.updated_at DESC

Awesome. Now lets try to do the same with our beloved class method:

class Post < ActiveRecord::Base
  def self.by_status(status)
    where(status: status) if status.present?
  end
end

Running this:

Post.by_status('').recent
NoMethodError: undefined method `recent' for nil:NilClass

And :bomb:. The difference is that a scope will always return a relation, whereas our simple class method implementation will not. The class method should look like this instead:

def self.by_status(status)
  if status.present?
    where(status: status)
  else
    all
  end
end

Notice that I’m returning all for the nil/blank case, which in Rails 4 returns a relation (it previously returned the Array of items from the database). In Rails 3.2.x, you should use scoped there instead. And there we go:

Post.by_status('').recent
# SELECT "posts".* FROM "posts" ORDER BY posts.updated_at DESC

So the advice here is: never return nil from a class method that should work like a scope, otherwise you’re breaking the chainability condition implied by scopes, that always return a relation.

Scopes are extensible

Lets get pagination as our next example and I’m going to use the kaminari gem as basis. The most important thing you need to do when paginating a collection is to tell which page you want to fetch:

Post.page(2)

After doing that you might want to say how many records per page you want:

Post.page(2).per(15)

And you may to know the total number of pages, or whether you are in the first or last page:

posts = Post.page(2)
posts.total_pages # => 2
posts.first_page? # => false
posts.last_page?  # => true

This all makes sense when we call things in this order, but it doesn’t make any sense to call these methods in a collection that is not paginated, does it? When you write scopes, you can add specific extensions that will only be available in your object if that scope is called. In case of kaminari, it only adds the page scope to your Active Record models, and relies on the scope extensions feature to add all other functionality when page is called. Conceptually, the code would look like this:

scope :page, -> num { # some limit + offset logic here for pagination } do
  def per(num)
    # more logic here
  end
 
  def total_pages
    # some more here
  end
 
  def first_page?
    # and a bit more
  end
 
  def last_page?
    # and so on
  end
end

Scope extensions is a powerful and flexible technique to have in our toolchain. But of course, we can always go wild and get all that with class methods too:

def self.page(num)
  scope = # some limit + offset logic here for pagination
  scope.extend PaginationExtensions
  scope
end
 
module PaginationExtensions
  def per(num)
    # more logic here
  end
 
  def total_pages
    # some more here
  end
 
  def first_page?
    # and a bit more
  end
 
  def last_page?
    # and so on
  end
end

It is a bit more verbose than using a scope, but it yields the same results. And the advice here is: pick what works better for you but make sure you know what the framework provides before reinventing the wheel.

Wrapping up

I personally tend to use scopes when the logic is very small, for simple where/order clauses, and class methods when it involves a bit more complexity, but whether it receives an argument or not doesn’t really matter much to me. I also tend to rely more on scopes when doing extensions like showed here, since it’s a feature that Active Record already gives us for free.

I think it’s important to clarify the main differences between scopes and class methods, so that you can pick the right tool for the job™, or the tool that makes you more comfortable. Whether you use one or another, I don’t think it really matters, as long as you write them clear and consistently throughout your application.

Do you have any thought about using scopes vs class methods? Make sure to leave a comment below telling us what you think, we’d love to hear.