Paginating special queries

Posted by Andreas on Saturday, July 21, 2007 at 19:07 (CEST)

The will_paginate plugin for Rails is one of the best pagination plugins I know of and I like the ease of using it. Instead of using Model.find, simply use Model.paginate(:page => x) and you’ll get back a collection that behaves like an array and can be displayed in views like any other collection. Additionally the returned collection has some new methods (like page_count and total_pages), which are useful to display pagination links. Installing the will_paginate plugin adds the pagination class method to all models; it even works with automatic finders: e.g. you can use Model.paginate_by_attr, which will paginate queries for Model.find_by_attr.

However, pagination can become tricky if you want to paginate collections returned by special queries…

Pagination works fine when using the find method or an automatic finder method directly with a model or through an association. Consider the following models:

1 class Group < ActiveRecord::Base
2   has_many :users
3 end
4 class User < ActiveRecord::Base
5   belongs_to :group
6 end

All of the following would work perfectly:

1 @active_users = User.paginate_by_status('active', :page => x)
2 @active_users = @group.users.paginate_by_status('active', :page => x)
3 @active_moderators = @group.users.paginate(:conditions => ['status = ? AND role = ?', 'active', 'moderator'], :page => x)

However, you’ll probably want to avoid doing queries with that many SQL conditions in a controller. Special queries like the above should stay out of a controller and better be moved into the model.

Method 1: Additional has_many associations

Easiest way and what first comes into mind is to add an extra has_many association to the model. Like described in The Rails Way: Association Proxies (Case Three):

1 class Group < ActiveRecord::Base
2   has_many :users
3   has_many :active_moderators, :class_name => 'User', :conditions => ['status = ? AND role = ?', 'active', 'moderator']
4 end

Now you can easily paginate a list of active moderators of a group:

1 @active_moderators = @group.active_moderators.paginate(:page => x)

This also works if you pass additional options, e.g. to get a sorted list:

1 @active_moderators = @group.active_moderators.paginate(:order => 'username', :page => x)

The above method works well, but has some drawbacks:

  1. You can only use the special query through the association. You cannot e.g. get a global list of all active moderators.
  2. The code (the SQL conditions) which determines the filter to apply on the User collection is now kept in the Group model whereas it would better fit into the User model itself.
  3. If you have multiple models that have has_many associations with the User model, you need to duplicate the extra has_many associations into every other model. That’s error-prone and not DRY.

Method 2: Custom finder methods

Another method to refactor a special query to the model, would be a custom class method (a custom finder) for the model:

1 class User < ActiveRecord::Base
2   def self.find_active_moderators (options = {})
3     with_scope(:find => { :conditions => ['status = ? AND role = ?', 'active', 'moderator']})
4       find(:all, options)
5     end
6   end
7 end

Using with_scope ensures, that the special conditions are applied while still allowing us to pass additional options like :order (or :limit and :offset, which are used by the pagination plugin). Now, all code to query a special user collection stays within the User model and can be used globally or through associated models:

1 @active_moderators = User.find_active_moderators(:order => 'username')
2 @active_group_moderators = @group.users.find_active_moderators(:order => 'username')

The will_paginate plugin is smart enough to also use custom finder methods. Using User.paginate_active_moderators will internally use User.find_active_moderators to fetch the collection (where :limit and :offset options are applied to retrieve the collection for the given page):

1 @active_moderators = @group.users.paginate_active_moderators(:order => 'username', :page => x)

Cool, isn’t it? But wait! There’s still one problem: pagination relies on getting a correct count of the total entries to display. This is done by querying the database for the number of records. With a custom finder method, the will_paginate plugin isn’t able to get the correct number of records to display, because the custom finder method encapsulates the conditions. Will_paginate therefore takes the number of total records instead, which results in a view that shows the correct entries, but displays additional empty pages (Ticket #66).

The only way I found to work around this, was to pass the correct number of records to the pagination plugin. Therefore, for every custom finder method, I also created a custom counter method:

1 class User < ActiveRecord::Base
2   def self.count_active_moderators (options = {})
3     with_scope(:find => { :conditions => ['status = ? AND role = ?', 'active', 'moderator']})
4       count(:all, options)
5     end
6   end
7 end

Unfortunately, this results in code duplication again (custom finder and count methods do use the same with_scope statement, but call different methods within the block. This could be refactored to a separate method, e.g. with_active_moderators). Furthermore, will_paginate does not automatically use a custom counter method, so you have to add the :total_entries option to paginate calls in the controller:

1 @active_moderators = @group.users.paginate_active_moderators(:order => 'username', :page => x, :total_entries => @group.users.count_active_moderators)

Hopefully some day, the will_paginate plugin will come up with an easier solution. Maybe by automatically using a custom count method, or – even better – by somehow automatically counting the records within the right scope.

Another method could be to use the scope_out plugin, which automatically creates model class methods for a given scope. However, I didn’t look into this plugin yet and don’t know if it works well with will_paginate.

Update (2007-08-02): The will_paginate plugin has changed a bit when it comes to custom finder methods. It now calls find_whatever when using paginate_whatever (formerly it used to call find_all_whatever). I changed the above text accordingly.

Update (2007-08-03): I checked out the scope_out plugin today, and liked it a lot. However, using will_paginate with scope_out suffers from the same problem I described above, since behind the scenes, scope_out creates custom finder methods like the ones I described above (although it simplifies it by providing a single method call to create finder, scope and calculation methods).

Update (2007-10-20): The best solution I found is to use the HasFinder plugin, which combines the features of scope_out and scoped_proxy and works well with will_paginate.

11 comments

Gravatar
Mislav wrote about 6 hours later:

Nice writeup. But you didn’t think outside the box when you went to use with_scope! Hop to ticket #66 to see what I meant.

Your writing style is really good, keep em coming, but please – put some more contrast on the design. When I’m reading your blog I feed blind :O
-


Gravatar
Andreas wrote about 23 hours later:

Thanks, Mislav. Indeed you’re right that having a custom finder and a custom count method would not be DRY, because you’d use with_scope exactly the same in both methods. I mentioned that above and suggested to refactor the with_scope parts into an own method named with_active_moderators or so. However in the end I didn’t like that, because generating 2 more methods just to get a pagination on a custom finder doesn’t feel right.

In Ticket #66, you also mentioned a way, that didn’t come into my mind at first – to define a custom paginate method:

1 class User < ActiveRecord::Base
                  2 def self.paginate_active_moderators (options = {})
                  3     with_scope(:find => { :conditions => ['status = ? AND role = ?', 'active', 'moderator']})
                  4       paginate(options)
                  5     end
                  6   end
                  7 end

This would indeed solve the problem — however, only if you need just the paginated finder. If you also need to access the special query without pagination (e.g. from other models), you’d need the custom finder method again, which would again lead to code duplication (same with_scope in find_all_active_moderators and paginate_active_moderators).

Furthermore, I think, a pagination plugin should add the feature of pagination and be as unobstrusive as possible. That’s what I like about will_paginate the most – you can develop models like you did before and get the pagination on top. The models will even continue to work if you completely remove the will_paginate plugin again.

It would be great if this would also work for special queries – just write your finder methods as usual and get pagination automatically. Well, it’s almost like that already. The only thing that’s missing for will_paginate would be to retrieve the correct record count. That doesn’t seem to hard, if you consider that (in theory) it only has to let the custom finder build the SQL conditions string like usual and use it with count (‘select count(*)’) instead of find (‘select *’) to get the record count. However I don’t know if this would be possible the way AR works without chaning AR itself…

As for the colors… Are you referring to the code snippets or the site at all? The code snippets indeed don’t have a good contrast; especially keywords and symbols are hard to read. I think, I’ll have a look at the stylesheet this evening…

Gravatar
Justin wrote 11 days later:
1 class User  { :conditions => [status = ? AND role = ?’, active’, moderator’]})
                  2       find_or_paginate(options)
                  3     end
                  4   end
                  5 end
Gravatar
Adam Cohen wrote about 1 month later:

Great post Andreas! I’ve also run into the same problem that you have regarding complex custom finders in the model, and I finally came up with the following interim solution:

1 # list of tickets for the event, paginated if page param is given
                  2   def tickets(args = { })
                  3     conditions = { :conditions => ['line_item_id = line_items.id and product_id = ?', self], :include => :line_item, :order => 'last_name'}
if args.key?(:page) TicketInfo.paginate(:all, conditions.merge(:page => args[:page])) else TicketInfo.find(:all, conditions) end end

so using the above, I can either call @event.tickets to perform a regular find on the model, or I can use @event.tickets(:page => params[:page]) within my controller to get a paginated list.

Of course I’d also love to see a standardized solution from will_paginate to allow this type of behaviour, but until that happens, I’ll keep using the above.

Also, great work on the plugin Mislav, my only complaint is the issue described here! (oh yeah, and adding ajax link_to_remote support would be nice as well ;)

Gravatar
Mislav wrote about 1 month later:

You complain about scope_out not being supported. But what about this: http://plugins.require.errtheblog.com/changesets/326

Gravatar
Adam Cohen wrote about 1 month later:

Mislav, I apologize, I should’ve looked through the changesets before posting my message. Great work, thanks!

Gravatar
Nick Kallen wrote 2 months later:

You guys should check out my plugin has_finder:

http://www.pivotalblabs.com/articles/2007/09/02/hasfinder-its-now-easier-than-ever-to-create-complex-re-usable-sql-queries

It allows you to do:

class Article {:published => true} has_finder :popular, :conditions => … end Article.published.paginate(:page => 1) Article.published.popular.count Article.popular.find(:first) Article.popular.find(:all, :conditions => {…})

It’s available as a gem!

Gravatar
Nick Kallen wrote 2 months later:

Er, not sure what happened to my markup, but check out the link you’ll get the idea!

Gravatar
Maximus wrote 5 months later:

I would like to see a continuation of the topic

Gravatar
Andreas wrote 5 months later:

@Maximus: My personal solution for some time now is to use the will_paginate and the HasFinder plugin, which seem to work well together. I posted a summary a while ago: Paginating special queries with HasFinder and will_paginate

Gravatar
Forexman wrote 10 months later:

Hi. This is really interesting post. Thank You! I have just subscribed to Your rss!

Best regards

Comments are closed