Metacritic has been in the news again lately, this time because some apparent insights into the site’s “weighing” system. Data recently revealed in a study from Adams Greenwood-Ericsen of Full Sail University made some pretty damning claims concerning how the site weights review scores for their “Metascore.” And while this does reignite a discussion about the site’s influence, I think we need to go beyond what this study says on the surface.
If you take a look at Metacritic itself, the site explains the system in a broad sense. “Metascore is a weighed average in that we assign more importance, or weight, to some critics and publications than others, based on their quality and overall stature.” That’s it. They don’t provide any inkling into what this actually means, just that there is some set of internal mathematics which determines how valuable a reviewer’s score is. The study in question aimed to provide these statistics.
Greenwood-Ericson’s study made some bold assertions, claiming that reviews for the highest weighed sites were worth upwards of six times that of the lowest. These numbers are even more surprising when you look at the names in certain sections, with Giant bomb standing out in the second to lowest category. Destructoid and Joystiq only fall in the middle, with Yahoo Games sitting high in the top category. It really paints a striking picture.
Of course, Metacritic has responded to the study via Facebook, claiming it is “wildly, wholly inaccurate.” The post goes on to claim that far fewer tiers exist, with many sites places in “comically” wrong positions. But, as could be expected, the site won’t release any hint at its actual process.
And herein lies my main problem with this system. Even before the release of the study, Metacritic openly admitted to using a weighing system, but provided no insight into the system itself. Because of this, certain questions remain open. Regardless of how accurate the study’s findings may be, it shines a light on the fact that we still don’t know who’s words are valued more and why.
One could argue that this needs to be a secret, as it’s basically the site’s key to profit. If everyone knew the internal workings, anyone could make a new Metacritic style site to compete. But I don’t give a shit about their business model. I care about the value that’s being placed on games critics. Metacritic has become a mythological figure on high, doling out value as it sees fit. I find that terrifying.
In response to this, complete transparency would help quell most of my fears about the site. Without this, they will remain a faceless gatekeeper to the world of games reviews and criticism. If the internal logic of the site was even slightly more open, the perspectives for critics would certainly be helpful. We as gamers would also have a straightforward view of what these numbers really mean. More importantly, it would be much easier to realize that they are just that: numbers. Ratings and Metascores could hopefully work towards being surface layer references, and not the mark of a game’s success. I hope this eventually happens, but I have my doubts that Metacritic will act.
As it stands, the site has long since become a system that publishers game for PR purposes. Having your publication present on Metacritic can often be a determining factor for receiving review copies, forcing everyone to put on the little song and dance necessary to get in. That, or be forced into irrelevance. Frustration only begins to express how this makes me feel, especially when we have no idea how much our words will even be worth if accepted. And it’s all because we still feel this need to place so much value on a number score; the keystone on which Metacritic is built.
We as a community need to stop fetishizing number scores. It leads us down a road that reduces all meaningful discussion and critique down to a single character. I’m not saying they need to go away completely, but the fact that this study is even newsworthy shows we’re still way too far down that road. The power Metacritic still wields shows that we need to change if real games criticism is ever going to exist.
Examples of this exist with sites like The Penny Arcade Report, and video series’ like Zero Punctuation, but this type of discourse is still in the minority. The current focus creates a system that’s not fair to anyone. It’s not fair to the lesser known writer who’s words become worthless. It’s not fair to the developer who has to worry about achieving some absurdly abstract goal. It’s not fair to the PR person who’s stuck in the middle of the resulting circus. The study shouldn’t matter because Metacritic scores shouldn’t matter. The site is a nifty tool, and that’s all it should be.