Written by When David Chelimsky was visiting São Paulo in last April, we invited him to go out for some coffee, beers and brazilian appetizers. We had a great time and we talked about different topics like OO, programming languages, authoring books and, as expected, about testing.
One of the topics in our testing discussion was the current confusion in rspec-rails request specs when using Capybara. There is an open issue for this in rspec-rails’ issues tracker and discussing it personally allowed us to talk about some possible solutions, which could take months in internet time. 🙂
rspec-rails is a gem that wraps Rails testing behaviors into RSpec’s example groups. For example, the controller example group is based on ActionController::TestCase::Behavior. There are also example groups for views, helpers and so forth, but for now we are interested in the request example group, which is as a wrapper for ActionDispatch::Integration::Runner. The Rails’ integration runner is built on top of rack-test, a great small gem that adds support to methods like get, post, put and delete and handle the rack request and response objects.
This setup with the request example group running on top of Rails’ Integration Runner works fine until you add Capybara to your application (which is always a good idea). The issue is that Capybara by default includes its DSL in the same request example group and that’s when the confusion starts.
Capybara, being an acceptance test framework, does not expose low-level details like a request or response object. In order to access a web page using Capybara, the developer needs to use the method visit (instead of get). To read the accessed page body, the developer must use page instead of manipulating the response.
However, since both Capybara DSL and Rails’ Integration Runner are included in the same example group, both methods visit and get are available! Not only that, even if I visit a web page using Capybara’s visit, I can still access the request and response object that comes from Rails, except that they will be blank since Capybara uses a completely different stack to access the application.
This confusion not only happens inside each test but it also leads to a poor testing suite. I have seen many, many files inside spec/requests that mixes both syntaxes.
Talking to David, I have expressed a possible solution to this problem based on how we have been building applications at Plataformatec. First of all, we start by having two directories: spec/requests and spec/acceptance. Since both are supported by Capybara, this (mostly) works out of the box.
Everything you want to test from the user/browser perspective goes under spec/acceptance. So if you want to test that by filling the body and the title fields and pressing the button “Publish” publishes a new blog post, you will test that under acceptance (protip: we usually have subdirectories inside spec/acceptance based on the application roles, like spec/acceptance/guest, spec/acceptance/admin, etc).
Everything under spec/requests applies to the inner working of your application. Is it returning the proper http headers? Is this route streaming the correct JSON response? Also, since APIs are not part of the user/browser perspective, they are also tested under spec/requests and not under spec/acceptance.
This separation of concerns already helps solving the confusion above. Under spec/acceptance, you should use only Capybara helpers. Inside spec/requests, you are using Rails provided tools. However, this does not solve the underlying problem that both helpers are still included in spec/requests.
Therefore, while this blog post means to provide some guidance for those that run into such problems, we also would like to propose a solution that we discussed with David. The solution goes like this:
1) We change RSpec to no longer generate spec/requests, but both spec/api and spec/features (I have proposed spec/acceptance but David pointed out those are not strictly speaking acceptance tests). The Capybara DSL (visit, page and friends) should not be included in spec/api under any circumstance.
2) We change Capybara to include by default its DSL and RSpec matchers under spec/features and change the feature method to rely on the type :features instead of :requests.
The proposal suggests the addition of two new directories instead of changing the behavior of existing ones in order to be backwards compatible while ensuring a safe and more semantic future for everyone else. David asked me to outline our conversation in a blog post, so we can get some awareness and feedback before undergoing such changes. So, what do you think?
I use `spec/acceptance/` as well. I think I kind of understand David’s intention when preferring api/ and feature/, but I’d love a clarification of why not acceptance/.
I’m +1 with this.
That’s great! So far I’ve been just implementing this behavior in every app in which I need acceptance tests with rspec. Not that’s a lot of work, but definitely something confusing and error prone if you don’t know what’s going on.
sabereent: I’ve been generally opposed to “acceptance” because I think it’s understood differently by too many people. The naming in rspec-rails focuses on the things that are being spec’d (models, controllers, views, etc), so I’d be more inclined to use a name like “application” or “system” than “acceptance”, which describes a process rather than a thing.
I agree with this, we’ve been putting both API and our other request specs into /requests and it doesn’t feel right. I like the idea of enforcing the user of get/post/put/etc… in the api specs
So if I understand well, both Capybara DSL and Rails’ Integration Runner will still be available under spec/features. Not ideal right?
It would be great if RSpec could detect Capybara presence and so only adds its DSL (get/post/…) under spec/api and lets spec/features with Capybara DSL only.
Deny. Just capybara will be available under spec/features. 🙂
At the moment most of our team write a lot of acceptance tests and use them as rough view tests as well. Placed in spec/requests of course 😉 It’s easier for new colleagues to get into BDD this way.
We of course struggle with testing APIs, Workers, Authentication and Authorization and Rack Apps that integrate into the process.
What I lack the most at some times is guidance. I simple don’t know how to test this. What I and many others end up with is grabbing random ideas off forums, blogs and stackoverflow and see what happens. And if I then teach my fuzzy solutions to my colleagues the resulting code is even worse.
I absolutely agree with the folder separation, but I would also like to see some guidelines on how to properly test POST requests, APIs, RackApps,…
Imagine putting something like the Rails guides together, that covers more than just one tool.
Without writing hundreds of lines of test code and in different levels of modularity. Speccing every little bit for itself is nice, until you find that the whole thing together doesn’t do what it should.
Renaming some folders and including less test methods is one thing, it’ll provide structure, but it won’t solve the problems many of us have.
I also use spec/acceptance directory, which is default when using Steak (https://github.com/cavalle/steak).
I like the idea to have spec/features.
Directory spec/api will be there for testing requests (like authentication => spec/api/authentication_spec.rb).
But where to put tests for API (ex. http://myapp/api/v1/users.json)? In spec/api/v1/users_spec.rb?
I you use subdirectories inside spec/acceptance based on the application roles, how do you solve shared test for more than one role?
> But where to put tests for API (ex. http://myapp/api/v1/users.json)? In spec/api/v1/users_spec.rb?
Yes, I would place them like that.
> I you use subdirectories inside spec/acceptance based on the application roles, how do you solve shared test for more than one role?
Put on the “lowest” role (i.e. the one with less permissions, it is usually expected that an admin can do everything the other roles can plus some extras) or create a new one: editors_and_collaborators.
> Put on the “lowest” role (i.e. the one with less permissions, it is
usually expected that an admin can do everything the other roles can
plus some extras) or create a new one: editors_and_collaborators.
Maybe shared examples could solve this better than editors_and_colaborators:
https://www.relishapp.com/rspec/rspec-core/v/2-10/docs/example-groups/shared-examples
It is hard to generalize but having a set of “acceptance” tests running under different roles seems very, very expensive.
It is fine to use shared examples at unit testing level to ensure some basic behaviors, but using it for acceptance tests seems an abuse of the feature.
How do you envision this working? rspec-rails’ example groups all include XXX::TestCase::Behavior from Rails, so I think we’d need something in Rails that provides RSpec all of the helpers it needs minus Rack::Test. Are you thinking there’s nothing else we need from Rails for a Capy-specific type of group?
> Are you thinking there’s nothing else we need from Rails for a Capy-specific type of group?
Exactly. I cannot think of anything Rails would need to do. In my opinion, all we need from RSpec Rails is to generate a spec/features directory on install and document its purpose. Capybara would then register the appropriate hooks, as it does for spec/acceptance.
I agree this is a blurry area, that’s why we just need to talk about each gem responsibility, because today’s issue comes exactly from Capybara and RSpec giving different roles to the same example group (i.e. spec/requests).
+1 for spec/features
> Everything under spec/requests applies to the inner working of your application. Is it returning the proper http headers? Is this route streaming the correct JSON response?
I don’t get this. Isn’t this what controller tests are for?
+1
+1
Controller tests make less sense in Rails 3 stack. For example, you cannot assert the proper HTTP headers in some cases because they are being set in a middleware, not in the controller. So while you can assert **some** of the behavior in the controller, I personally prefer it to go through my middleware stack. I reserve the (few) controller tests for fail fast tests, often using stubs to isolate the controller. In my talk to Chelimsky, we both noted this is a common subject of confusion to newcomers, but that’s another discussion.
—