I think that the biggest problem that sites such as this one face is the fact that the people who come to these sites are very knowledgeable, at times even more so than the people running the site.
Take Ace's Hardware for example.....the expertise of many of the site's most passionate visitors is amazingly high. If Johan and Brian (site operators) were to put out an article that isn't well thought out, well reasoned, and provide hard data to back up their claims, they would be roasted to no end. To their credit, they put out some of best review articles I have ever seen on the net. There is always an abundance of benchmarking, design analysis, and testing methodology available in the reviews. Thus, even if you don't agree with their results or final opinion, you at least have a better understanding of how they got to their final conclusion and can more easily ask questions or voice criticisms. It makes for much more substantial debates.
What is not so easily tolerated, nor should it be, are "reviews" where a bunch of opinions are voiced based on benchmarks that are either A) absent, B) supposedly found elsewhere but never referenced, or C) just poorly done (scary how many sites still think Quake 3 is the end all/be all benchmark). Worse yet is when sites go out of their way to use only those benchmarks (or a large majority of them) that make their chosen favorite shine.
It even needs to be taken a step farther.....you cannot just throw out a bunch of benchmarks and simply declare a winner. It is important that the reviewer makes the end user understand to the best of his/her ability why a benchmark between two equally clocked processors would favor one over the other (say a 1.4GHz Athlon vs. a 1.4 GHz P4). This goes a long way in helping the reader decide which processor is better for what they need it for. Are they using a lot of SSE apps? Is memory latency critical to their applications? Do they require more bandwidth? A stronger FPU?
Possibly the most difficult part of this is the fact that sites are not always able to get comparable hardware. This is where the methodology used becomes very important. When doing a review with benchmarks, it is important that the process be kept as standardized as possible. Of course, the process will differ is you're doing a processor review vs. a chipset review vs. a motherboard roundup, etc. Things like processor steppings, BIOS levels, driver info should always be part and parcel of a review. One of the best known hardware sites out there, Tom's Hardware, was recently blasted (and rightly so) for a roundup they did of motherboards based on the Ali Magik chipset. Why you ask? Well, the BIOS revisions they used for the different boards were old compared to what was currently available from the manufacturers at the time. As anyone who has kept tabs on the Ali based boards knows, BIOS revisions have made a world of difference in the performance and stability of the chipset.
So, what to do? Well for one, if you don't have the hardware at hand, DO NOT DO A REVIEW!!!!! Or at the very least don't call it such. Call it an opinion piece, call it an editorial. Without having tested the hardware, it isn't a review, it's a guess. And if you do have the hardware, make sure your methodology is well thought out, your testing thorough, and your conclusions based at the evidence in hand, not some wild speculation.
Also, there will be those who still aren't fond of the testing you do. That's unavoidable as there will always be someone's favorite benchmark that wasn't included in the test. If you've done your homework, you should be readily able to explain why you didn't do said test. Perhaps it's just something you hadn't thought of doing. The readers aren't perfect, but then again, neither are the reviewers. There is always room for improvement.
Want to enjoy fewer advertisements and more features? Click here to become a Hardware Analysis registered user.
Hey guys. Thanks for taking the time to write. Although I feel you got a little off the topic of my piece (I'm not talking about any one site, or even a review site at all. Im talking about the way people react to new ideas), I agree with what you said. In a nutshell, if you don't know what you're talking about, don't pretend you do. Absolutely.
But that was precisely my point. My point is that, too often, people dismiss something as being written by someone who doesnt know what he/she is talking about, simply because THEY DON'T AGREE.
The message was not 'You have to listen to everyone, even if he's full of crap'. The message was 'Think very, very, VERY carefully before you accuse someone of being full of crap'. I'm not saying you have to listen to crap, I'm inviting us all to be more careful about what we declare as crap.
Anyway, thanks again for commenting. Take care.
By the same token, sites have to be careful about posting stuff that is easily labled as "crap".
If two sites provided two different conclusions about a product, who are you more likely to believe; A) the site with conclusions derived from benchmarks? or B) the site with nothing but the reviewers personal feelings as to the product with nothing to back it up?
By the same token if those two sites both have benchmarks available yet still arrive at different conclusions, the questions change.
Why did they get different benchmark results?
Why did they use/fail to use certain benchmarks?
Do the conclusions they arrive at seem to mesh with the benchmark results?
Give me hard data, tell me what the environment is, tell me what you think.....I'll tell you if I agree with you.
Give me speculations, offer obscure references with no links or direct quotes.....I'll tell you it's crap. By the same token, if the response to the article fails to provide links, direct quotes, or personally run benchmarks, I'll tell you that's crap as well.
Again, I agree completely. You're absolutely right.
But try to think on the bigger scale. Im not just talking about websites and reviews. I just used that as an example. Perhaps it's my fault for not making that clearer - I'll be more careful next time. Live and learn. The article was speaking far more generally about human behavior. If I could prove to you right now that the world IS flat, would you believe me? Even if I had solid scientific evidence?
Probably not. Most people wouldnt. They'd try to discredit me. I'd probably do the exact same thing (please don't think Im trying to say I'm above anyone here...that wasn't the point). My point was that, for a LOT of people, it doesn't matter if I have good, solid evidence, or if I'm just shouting it out with no real reason. And that's not the way it should be.
You are right Dan, there are always people who will be convinced of something despite all evidence to the contrary. How many great thinkers/scientists of the past had been ridiculed, imprisoned, or put to death simply because what they had discovered through scientific and/or mathematical experimentation flew in the face of what was generally accepted as true?
The problem comes when we allow ourselves and others with actualy numbers to back our claim to be drowned out by the howling masses who are too lazy/stubborn/fanatical to accept that what they think is true may not be after all.
I came here by way of 3D.net I read the newsand all i wanted to make a post but not being a member i had to join to so i did... just to post this
you know there are always people who will take the other side no matter what, you can talk till your blue in the face it will not change them no matter what. Now i think you can chose to worry about this or you can chose to go on and keep writing things base on what you find thru test the choice is yours. No one in history hasw ever changed the mass so why not be content to write for those who seek the truth. We are not the minority WE ARE THE MAJORITY who seek the truth....
I will come back to read you reviews ....Reny abrego