Is TV pilot testing ready for the masses?

The much anticipated debuts of the new Fall line-up of TV shows are airing within the next few weeks, which means it’s time to see if all that focus group testing holds true to form.  It’s an exciting yet nerve wracking time of the year for media executives and researchers because we know all too well that TV pilots only have two options–one, to swim, or two, to sink.

That’s a lot riding on a small sample group of viewers sitting in a room with dials in their hands. But what if the networks market tested their TV pilots publicly online instead of in private viewing rooms? No, this isn’t some hair-brained idea. In fact, as pointed out in this Entertainment Weekly article, Amazon and its production house Amazon Studios have been doing exactly that by first airing their “fledgling” pilots on their site and asking viewers to rate and comment on them; then using that feedback to decide which pilots get picked up and which get the axe.

Slidermetrix uses a hand- or swipe-controlled slider to capture continuous, in-the-moment feedback.

Using an on-screen slider, Slidermetrix allows for online “dial testing” following a similar method to the dial testing that networks typically conduct in more controlled offline focus groups.


So, does Amazon’s approach represent the wave of the future or simply a marketing gimmick?  Well, there’s plenty of evidence to show that Amazon might be on to something. First off, today’s viewers are no longer parking it on the couch to watch all of their favorite programs on their CRTs. They’re watching on-the-go—on their laptops at coffee shops and at the office, or on their tablets and smartphones, just about anywhere. Watching longer format programming online is quickly becoming the new norm, so in many ways, testing these programs online offers a much more true-to-life viewer experience than testing them in a controlled, offline environment such as a focus group room. Second, the Amazon approach could save the networks big bucks. Cost will always be a factor and any way that networks can limit their upfront investment and risk with new pilots is going to get their attention. And third, allowing viewers to be active participants in the process can help build a loyal following both for the network and for a new series.

What’s the biggest downside to Amazon’s approach? Well, for one, replicating the type of feedback that media execs are used to getting from the offline dial groups. The good news there is that new technology is closing the gap, allowing researchers to do online what for years could only be done in controlled focus group settings. Case-in-point, dial testing, which has been a standard research method used by the networks for new pilot testing. But recent advances in online research technology is now allowing for dial testing to be conducted online. And new tools like Slidermetrix make it easy for a company like Amazon to dial test clips, trailers and even full episodes on a public website. As example, we took a few trailers from some of the hottest fall pilots and set them up in Slidermetrix for you to view and dial test. Give it a try and then comeback and comment on if you’re on board with this new open approach to TV pilot testing.