How do I display all this data?
Making decisions on whether to paginate should start with considering your user.
A coworker came across an excellent article by Ben Nadel called The User Experience of Pagination that got us wanting to dive deeper into the subject of pagination and really take a practical look at determining how you present data. Like Nadel says in his article, "(Pagination) is there to solve an access problem for the user." Nadel shared the example of a long list of attorneys he set to display ten per page without really considering why he'd set it up that way. When questioned about his reasoning for doing so, he couldn't give a good answer. Let's look at some of the essential things to consider when making this decision.
The alluring draw of infinite scroll
The day has finally come: we can scroll endlessly and seem to never run out of data to consume. Nom nom nom. Bandwidth increases and advancements in web development now give us the option to implement infinite scrolling into our websites. Awesome, let's do it! Hold on a minute, though. Google still pages its search results, and Amazon.com only gives us 27 products before I have to click that "Next page" button. Have they dropped the ball on this, perhaps focusing too much on drone package delivery and autonomous cars, or do they know something we don't?
Consider your user
Etsy decided it would join the trend of infinite scrolling on search results in 2012, but they started seeing drops on product clicks after implementation. As a young company who tries hard to take care of their sellers, this was bad news. They even received backlash from their shoppers who said they didn't like it. Why was this? Etsy engineer Dan McKinley admitted, "we should have done a better job of understanding the people using our website". One possible reason for the failed implementation is the fact that it takes away your user's orientation. With paging, they have a series of checkpoints that are giving them a mental marker of where certain things are. When you take that away, they can end up wandering deep into search results without realizing the things they pass up will be more difficult to navigate back to. People like being in control, but infinite scrolling can end up making them feel like they're floating aimlessly through your site. It certainly is a nice fit, though, in image-based searches like Google images, as well as on Facebook and Twitter.
(Etsy has reverted to its paged results after seeing problems with infinite scrolling)
Let your users help out
Instead of trying to determine what the user wants during development, set results up in a way that will allow them to easily customize data in a way they want to see it. For large sets of data, having pre-set "view 25, 50, 100, or view all results" options can get your user what they want quickly. While simplification has its strengths, you also want to equip users with the tools they need.
Don't forget about SEO
Kristi Kellogg at Bruce Clay, Inc, a leading SEO company, says in regards to infinite scrolling,
Yikes. Google, however, was helpful in letting us know that you can implement in a way that will allow all content to be crawled.
No universal solution
Unfortunately, there is no universal solution on pagination! While some developers prefer the standard implementation of things like pagination, others must use the latest trend. In this case, the solution needs to be determined through careful consideration of your user's intent, which starts in the development and continues after launch by way of user feedback.