I don't give star ratings for reviews. I used to. I never will again if I can help it. A simple Yes or No will do the trick for those seeking a rating, along with some discussion of what the thing is about and why it is about it. But star ratings mislead readers, distort the function of commentary, and are all but meaningless. They don't even do much towards their intended goal, providing gradations of recommendation for a given work.

I've been working on a version of this essay, on and off, for about a year. Recently, news dropped that Netflix was abandoning its user-feedback star rating system for one based on a simple thumbs-up/thumbs-down. The rationale for this move echoed a lot of things I'd long thought about why star ratings — or numbers, or letter grades — are more trouble than they're worth.

Thumbs up, thumbs down

Once or twice 'pon a time, I wrote anime and manga reviews for a couple of different venues that used either number or star ratings as part of their article formats. I'd already accrued a bad taste in my mouth about such mechanisms by the time I walked in, but in neither case was I in a position to demand an entirely new paradigm for how to format and present reviews. So I gritted my teeth and picked a number, or assigned some stars, and each time I did, I regretted it. Yes, even for the stuff I gave top marks.

My main reasons for this regret encompass two main points.

1. Scores provide consumer recommendations, rather than promote intelligent criticism

Most review outlets are consumer guides. They are addressed to a prospective buyer or viewer, to help them navigate the ever-growing jungle of content out there, and provide them with a way to figure out what's worth bothering with.

This is a valuable and important function. What I don't approve of is using star ratings as a substitute for intelligent, nuanced criticism. They shouldn't be used to summarize something where you need more than a summary. It's handy for people in a hurry, sure, but if the whole trade turns into nothing but catering to people who want to be told what to do, what's left?

There's a lot of discussion to be had around any entertainment that has nothing to do with just whether or not it's worth seeing — in big part because everyone has totally different standards for how their time is best spent.

2. Scores are meaningless beyond a certain number of gradations

To me, this is the real crux of the argument. Beyond a "yes" or a "no", scores lose any informative value very quickly, and soon become meaningless anyway.

At one point I tried to take the standard zero-to-four-star rating system and apply to it some kind of criterion for how the stars were to be used. What I ended up with was something that went like this:

★★★★ A classic of its kind (subject to the revisions of history; check back in five years)
★★★½ Way above average, with only some provisional issues
★★★ Above average, perhaps not for everyone but worth checking out if the concept is intriguing
★★½ Intriguing concept, but flawed delivery
★★ Decent concept, flawed delivery
★½ Flawed at heart, but at least it isn't mean-spirited
★ Fundamentally flawed
½★ Fundamentally flawed, and a laughingstock
no stars Fundamentally flawed, and reprehensible to boot

That's nine gradations. If you're looking for a star rating for something in the first place, wouldn't any information for you beyond go/don't-go just get screened out anyway? Everything beyond that ends up just being noise, or being something to discuss in a far less confining way.

In situations where I had to assign a zero to ten rating, I found another wrinkle. Almost never did anyone assign anything below a seven anyway — which meant that we were now effectively dealing with four gradations and not ten. Or, if we went by half-point gradations, we were back to seven or eight grades all over again, just presented on a different scale. The numbers were once again rendered all but meaningless.

I ran into still another variant of this when for one outlet I was asked to supply different grades for the presentation of the material as well as the material itself — e.g., picture and sound quality for DVDs. Save for the hard-core audiophiles in the audience, there's no real reason to bother with such discussions anymore, except maybe when they deviate grossly from the norm. And again, those ratings rarely dipped below a certain range anyway.

Mistakes in a given edition of something are worth calling out. The recent Ghost In The Shell: Stand Alone Complex Blu-ray Disc reissues are getting roasted good and hard for including hard-of-hearing subtitles in lieu of a proper English sub track. That's worth calling out, as it's an awful decision. (Maybe it was a mastering error?) But the vast majority of the time, for the vast majority of viewers, there isn't going to be anything to talk about in that department. One's time and energy are better spent elsewhere.

Keeping score

Some sites do exist that are explicitly about rating the quality of a piece of media in a given incarnation. Blu-ray.com, for instance, assigns grades for picture, sound, extras, and so on. That's fine, especially since fans want to know they're not buying something where a crucial part of the presentation isn't up to snuff. But nobody putting together a blog of their own should feel obliged to include those kinds of discussions for fear of looking like they're not a full-service outfit.

Some sites are also forced to use scoring as a way to improve SEO. Game reviews, for instance, tend to be very score-heavy, especially since they're used to provide a Metacritic score for a game. Without a score, a game simply doesn't show up if people search for it. That matters doubly more for indie creations that would otherwise have little to no chance of finding an audience. In this case, a scoring mechanism can be used as shorthand for a critical assessment, a way to distill a single insight — it's good, it's not — from the whole discussion.

But even in cases like that, it's wise to not let the review score tail wag the review dog. The minute you bring the star or letter grades into the picture as a process, everything tends to live that much more in its shadow. Discussions that don't lend themselves readily to such things, or even to a go/don't-go recommendation, become harder to have. This stuff needs to happen in a different space bound by different rules.

I should point out I am not talking about reviews I've submitted for sites like Goodreads, where I have no control over the format, and where I mostly use that service as a kind of social bookmarking site (in this case, in the classic sense of the term "bookmark"). I'm talking about something where the format for the review has the potential to be entirely open-ended, and where you have no obligation to assign stars or letter grades or engage in other kinds of grade-school folderol.

What about Rotten Tomatoes? They're an interesting case, as they apply some degree of subjective analysis to render reviews Fresh or Rotten — again, a simple Yes or No, which is fine. Then the site produces a percentage from that. Thing is, I'm not sure how useful that is as a gauge of something's quality; what you're seeing is critical consensus, which is not always the same thing. Critical consensus is at least as much of its moment in time as audience taste. Many movies we now enshrine as classics or milestones were received quite tepidly, or negatively when they first appeared. (Note that I don't believe those things are due to gradated ratings, but I do think gradated ratings can exacerbate them.)

If we must have gradations at all, the most I could ever see being needed is three. Music magazine SPIN used this approach: green light (go for it!), yellow light (eh, maybe), red light (naah). In all the chasms and cracks between those three ratings falls whatever discussion you could make, and the vast majority of even the most nuanced and difficult-to-summarize discussions probably fall into one of those three buckets. But anything more granular than that is fiddling.

Some of the movie sites I frequent these days never started with gradated ratings, and don't ever seem to intend to use them. Birth.Movies.Death, for instance, treats every review like an essay. You want to know what the deal is, set aside a few minutes and read. That makes the process of dealing with this stuff less a consumer relationship, and more a conversant one, one where the audience and the creators and the critics are all part of a process of interchange. Sounds like a far better deal to me.



About the Author

Serdar Yegulalp (@genjipress) () is Editor-in-Chief of Ganriki.org. He has written about anime professionally as the Anime Guide for Anime.About.com, and as a contributor to Advanced Media Network, but has also been exploring the subject on his own since 1998.
Comment Policy: Comments are moderated for politeness and relevance. Be considerate, be on-topic. Disagree agreeably. No pirate links.