During my tenure as Editor of Realskiers.com, one of the benefits of membership has been a long-form ski review, accompanied by test scores complied by our small volunteer army of testers. Long-form reviews are still a principal component of the Realskiers’ membership bundle of benefits, but the data will no longer be a part of the presentation for the simple reason the paucity of entries results in data that is more detrimental than beneficial.
Now for the not so simple reasons. The Realskiers Test Team has always been primarily comprised of shop personnel, from owners down to seasonal salespeople. Once upon a time, this produced hundreds of what were then paper test cards, often with 20 or more entries per model. The test cards are digital now, which creates its own set of challenges – pencils don’t suddenly go haywire for no apparent reason, but apps do – but the much greater problem is that shops are already understaffed; sending a manager and a coterie of testers off to test skis for 2-3 days is not in the cards.
There are also fewer demo events to attend, in part because suppliers are imposing early order deadlines, drastically shrinking the window between the first appearance of demo skis and the drop-dead date for submitting an order. And that’s assuming the weather cooperates, but last year at Mammoth, it was in no mood to trifle. The event lasted all of one day before being weathered-out.
With fewer testers getting fewer opportunities to test, getting any kind of feedback has become a season-long chase. The end results are wildly imbalanced, and so few in number that if you start tossing out anomalies, pretty soon you don’t have many numbers to work with. It seems apparent to this ski test maven – I’ve been at this game since the winter of 1986/87 – that the numbers can’t be trusted. Nor is there some formula by which they can be “fixed.” When numbers beg for fixing, you know their value to the end user is damaged beyond repair.
To put the present circumstances in perspective, allow me to cite my friend Bob Lefsetz – whose copious output you can read at (https://www.lefsetz.com/), a prolific social commentator – and passionate skier – who happens to be the most influential blogger in the contemporary music omniverse, on the frailties of data, to wit: “Not all data is created equal.” To which he trenchantly adds, “But even worse, all data analysis is not created equally.” Lefsetz is pontificating about today’s political punditry, but his observations pertain to any realm where numbers posture as the distillate of reality.
Despite its air of substance, data can distort reality even before it’s entered. Any tests that use a 5-point scale will always end up cramming nearly the entire field into a de facto 3-point range, a phenomenon I first observed when reviewing skis for Snow Country Magazine. When I started with a 5-point scale, the better testers were often putting an “X” on the line between enumerated boxes, to create some separation in a field that might extend to 20 models. I realized then that if I really wanted to identify the best in any category – in a market with far more brands than we have today – I needed a 10-point scale. This allowed me to then rank the results in order of finish, helping drive consumer interest in our start-up publication.
The reason I published our detailed test results then and have done likewise for Realskiers subscribers to the present day, is that people place a disproportionate trust in data, as if the numbers being published weren’t made by the same people who wrote the text. My point is that numbers aren’t inherently truthful, despite being composed of numerals instead of letters.
From my perspective, the biggest issue with data and its interpretation is that it’s an arid way to describe the ski experience. It potentially still has some value when comparing one ski to another, but whether a ski earned a 3.7 versus a 4.0 for edge grip still doesn’t tell you much about how it feels to ski on it. Even this limited attribute goes out the window when one ski result has 3 entries behind it and another has 30. There’s no way to equalize the imbalance in the number of entries, period. Not to mention that treating all results equally is based on the generous assumption that all skiers are equally qualified to submit entries in the first place.
It’s reached the point that numbers are so problematic they are as likely to distort as to describe a ski’s performance envelope. Testing skis on snow is still a valid exercise, particularly when asking a specific question about a specific feature, such as a change to a single core component. For testing to have any validity beyond this limited scope, there has to some rationale to the composition of the test team, just as there has to be an educated, common understanding of each test criterion. These requirements beg for a measure of control over the data collection process, control that I can’t exert from where I sit.
One unequivocal benefit to sharing ski test data with my Dear Readers: doing so provided a system for ordering each category’s reviews based on merit rather than the alphabet. Despite the absence of published data, I will continue to divide each genre into Power and Finesse camps and rank them according to my estimate of how they measure up against their peers.
If there’s a redeeming aspect to the demise of dependable data, it’s that its absence places the emphasis of any review squarely on the narrative, where it has always belonged. The qualities that real skiers seek in a new ski can’t be conveyed by numbers, and aren’t so easily fashioned into readable prose, either, as a romp through almost any ski website will sadly make plain.
The goal of any ski review is, or ought to be, to help the prospective ski buyer sort through a maze of options and end up on the best choice for him or her. The average price for a quality ski/binding set-up will easily exceed $1,000 this season, so it pays to do a little research. There is no better place on earth to do this research than Realskiers.com, if for no other reason than our site has one asset no other can simulate, imitate or copy: me. One of the privileges of membership is that I answer all queries personally, and I stay with you until we get your problem solved.
I look forward to hearing from you.
Related Articles
The Making of a Skier, Part IX: The ASTM, Carl Ettlinger and I
One of the many hats I wore as North American binding product manager for Salomon in the early 1980’s was that of delegate to the American Society for Testing and Materials (ASTM). I believe the first meeting of F8.14 – the sub-committee on ski safety – that I attended was in Pennsylvania. I was flying under the wings of Salomon’s seer of all standards and patents, Gilbert Delouche, and the binding product manager for the North American zone at that time (and my mentor), Joe Campisi.
I was a babe in the woods, but I soon caught on to the game under Delouche’s patience guidance. I recall a debate on the binding specification then being batted around in the technical committee chaired by Carl Ettlinger. Ettlinger wanted language that would require any release/retention setting of 10 or above to be “visually distinctive” from the rest of the scale.
In Memorium, Carl Ettlinger
Carl was a giant of a man whose outsized voice roiled every conversation like a burst dam and whose expansive vision reached across the mixed milieus of research, journalism, risk management and education. I knew him when he was at the peak of his powers, as he explained to me when I interviewed him for a “where are they now?” profile in Skiing History. He was able to conduct long-term research on injury patterns as well as analyze the particulars of the current binding market, turn around and package this knowledge into articles for Skiing and Skiing Trade News, followed up by a workshop tour that would bring enlightenment to the grassroots level. No one but Carl could have pulled this off, and Lord knows no one has had the requisite talent, energy and will power since.
But time and tide wait for no man, and Carl’s finely spun web of influence was eventually plucked apart. The loss of his pivotal positions in the press allowed him to slip from public view before we, the skiers of the world, realized we hadn’t taken the time to thank him.
We have the time to thank him now.
So thanks, Carl, for being first and foremost a teacher, for teaching is at the heart of the evangel’s mission.
Thanks for being so damn stubborn. Your insistence on improving skier safety wore through a wall of resistance as tough as Vermont marble.
Thanks for having a heart as big as that melon-sized head of yours. The fuel to your tireless mind was a caring heart that tried to embrace the world.
Thanks for all the stories once the Mount Gay flowed. Who knew we would have won the Vietnam War if only his superiors had listened? I can’t remember exactly how – he wasn’t the only one drinking Mount Gay – but I recall the light in his eyes as he relayed his twisted tales, taking us down successive rabbit-holes of digression that I lost track of at the seventh level.
That’s what I remember most vividly about my many interactions with Carl: his brain so teemed with thoughts he rushed to get them out in a verbal jailbreak that would travel around the cosmos until returning, many lost minutes later, to the subject that had inspired them. That was Carl: too many words for one sentence, too many tasks to tend to and all of it, every erg of his endless energy, devoted to a cause he never ceased to serve.
Fare thee well, Carl Ettlinger. The world misses you already for it will never see another quite like you, whose every moment seemed larger than life itself.
I raise my glass to you, old friend. Mount Gay, of course.
Jackson Hogen
June 23, 2020
Why This Buyer’s Guide?
Don’t read the 2021 Masterfit Buyer’s Guide in Partnership with Realskiers.com for its 62 ski reviews. I should know. I wrote or edited all of them.
Not that the ski reviews aren’t worth the read. But ski reviews on the web are as common as rice, while the Buyer’s Guide contains something no other publication, whether in digital, print or video format, can claim: the most respected, thorough and dependable boot reviews in the world.
This isn’t mere puffery. The Masterfit Boot Test is so well regarded by the supplier community that nearly every brand not only sends its following year’s line-up in four men’s sizes plus three for women, it also dispatches its top designers and/or product managers to a distant North American site for most of the test’s five-day duration.
