Stupid things you hear from the DBT/Null test crowd....

steveinaz

Did anyone expect this thread to go in any other direction than it did? You only learn something when you are willing and open to learning. If your entire position is a defensive one, why bother?

Rock on, do what you do. I know I will.

DarqueKnight

DarqueKnight wrote: »

Specifically, how do you conduct an audio equipment trial (procedurally) and what performance metrics do you evaluate?

Syndil wrote: »

I do a simple XY comparison with the help of a friend.

Metrics? I am not using laboratory gear when judging audio gear, I am using my perception and my brain. It is a subjective comparison, so the only "metric" I have to determine is which one I like the sound of better. I may occasionally make notes about the differing characteristics of the sound if the comparison is close, but ultimately I prefer the deciding factor to be which sound is able to draw out the most emotion while I listen.

The exchange above perfectly illustrates the basic difference between serious listeners and blind test for stereo adherents: the first group is interested in, and knowledgeable of, stereophony. The second group is not.

Notice that the deciding factor in Syndil's evaluation is which component can "draw out" the most emotion. Emotional thrills are an important part of the entertainment experience. However, emotional thrills are not unique to stereophony. Emotional thrills come from every other entertainment activity (movies, television, sex, food, etc.). If your primary performance measurement for a stereo system is emotional evocation, you might as well be listening in mono.

Typical of blind test for stereo adherents, Syndil did not mention sound stage dimensions, three-dimensionality, image weight, holography or any of the other unique primary performance characteristics of stereo sound. He stated that he is primarily interested in the evocation of an emotional response. Therefore, trying to explain to a such a person that there is no need to hide equipment when you can perfectly well hear that one component produces a wider and deeper sound stage, one component has a more stable image localization, or one component produces more tactile sensation against your body, is an exercise in futility. These are foreign concepts to them. Syndil even goes so far as to scoff at the advisement of the people who invented home stereo systems. This is not surprising, considering that the early pioneers intended for their invention to be evaluated primarily on the basis of stereophonic performance. In other words, the primary metric of a stereo system is the quality of the three dimensional aural illusion it creates.

As I said a couple of years ago about a well known blind cable test:

DarqueKnight wrote: »

This test shares the same heritage as all the other ABX audio tests I have read:

They all use evaluative methods that display an appalling ignorance of the way stereophonic audio systems work.

If I'm asked about my stereophonic performance metrics, I'm going to talk about how wide and deep and tall the sound stage is. I'm going to talk about being able to turn my head to the left or right and "look" at a sound image and not have the sound stage collapse. I'm going to talk about being able to "look behind" the vocalist in the center and focus on the drummer behind her. I'm going to talk about the bass sensations coming through the floor, armrests and my body. I'm going to talk about vocal and instrumental clarity and detail. I'm going to talk about the layering and liquidity of sound as it flows from the sound stage and into my ears. I'm going to talk about the "airiness" of high notes and the "swirling" sound patterns of thick synthesizer notes...and so on and so forth. This kind of descriptive talk sounds crazy to most blind test for stereo adherents...because they've never experienced it. They just want to know if there is a "difference". Never mind the nature of the difference...that's not important.

If I were evaluating gear mainly on my "feelings", then such things as physical appearance, cost, and brand name might be influential in a gear selection decision. I can understand the need for blinding in that case...if your mind and ears are not mature enough to be trusted...if you are fearful of, or susceptible to, being tricked. Decisions based on emotion are highly biased.

If gear is being evaluated on the standard parameters of a complex stereophonic presentation, rather than feelings, then you can just document the stereophonic presentation of each component...if you know how. It is not a trivial undertaking.

Is it "wrong" to hide the identity of components when doing a stereophonic evaluation? Of course not. Just like it is not "wrong" to use bicycle training wheels after a child has learned to keep balance. But why would you?

The question is often asked "if an expert listener can tell a difference while knowing the identity of components, why can't they do the same if the component's identities are hidden?" The answer is they can. But why would they? What is the value of adding the additional complexity of a blind test to an already properly set up stereo evaluation? The answer is there is none. Serious listeners are not interested in providing confidence in evaluation to the naysayers and other inexperienced non-serious listeners. They are interested in providing reference data to themselves, to other properly trained and experienced serious listeners and to inexperienced serious listeners who have an interest in learning appropriate stereophonic evaluation techniques.

What About Television?

The naysayer claims that blinding is ABSOLUTELY REQUIRED to eliminate bias in stereo component selection. However when we look at how televisions are evaluated by serious viewers, these tests are non-blind and use experienced and trained evaluators. There is no controversy over the need for blind tests. Here are some links to well known television "shoot-outs":

cnet.com-how-we-test

2010-value-electronics-flat-panel-shootout

Panasonic Wins 2010 HDTV Shootout

sharp-elite-wins-2011-value-electronics-hdtv-shootout

2009-tweaktv-value-electronics-hdtv-shoot-out.html

The common thread through these televisions is the utilization of evaluators trained in television performance metrics. Hence, this quote from CNET, taken from the first link above:

"The most important piece of test equipment is a trained, expert eye. Test patterns and the latest gear are no substitute for a knowledgeable, keen-eyed evaluator with a background in reviewing similar types of TVs. CNET's TV reviewers, David Katzmaier and Ty Pendlebury, have extensive experience reviewing and calibrating displays, and performs all measurements and tests themselves."

In these tests, trained evaluators place competing televisions side-by-side and measure their performance. There is no attempt to hide brands, even though it could easily be done by framing each set with poster board. Among videophiles (serious, trained viewers) there is no demand for blind tests to avoid bias and being tricked.

Television evaluators seem to have discovered a way to eliminate the effect of visual bias in their evaluations: EYE TRAINING!!!

Interestingly, blind tests are sometimes used in television comparisons: when the evaluator is a random consumer. The link below is an example:

LG TV Consumer Blind Test

Now, if we are fair, we must ask ourselves, "if blind testing is not required for serious evaluation of televisions, why would it be required for serious evaluation of stereo systems?" Is the ear more susceptible to bias and trickery than the eye? No, we know that the eye is the primary human sense organ and that it is highly susceptible to being "tricked". That is why eyewitness testimony is considered unreliable by the justice system.

In the video world, serious viewers are advised to adopt a set of performance evaluation practices that will assure fair, accurate and competent trials. Videophiles are advised to train themselves on picture quality settings and to invest in inexpensive video calibration software. Similar to this, music lovers/serious listeners/audiophiles were advised near the beginning of the availability of home stereo systems to " become sophisticated in the art of sound localization" and to "play the same records many times and thus become familiar with the more subtle artistic and technical effects of stereo sound"

I have been accused of having some "agenda" against blind testing. As I have said many times, blind testing has its application for some types of audio, but not for stereo. My "agenda" is assisting serious listeners in getting the highest performance and stereophonic listening pleasure from their stereo systems.

DarqueKnight

steveinaz wrote: »

Did anyone expect this thread to go in any other direction than it did? You only learn something when you are willing and open to learning. If your entire position is a defensive one, why bother?

There are many newcomers to our hobby who had/have questions about these topics. An example of this is evhudsons' comment bellow.

I agree that nothing can be done with a closed, defensive mind. However, not every one who asks these types of questions is closed-minded. Some people have serious interest in stereo, yet are confused by all the conflicting information available on the Internet. Some things in audio are counter-intuitive and require research, reasoning, and sometimes guidance, to properly work out.

Many people were very patient and generous in answering my questions and providing guidance when I entered this hobby 27 years ago. Shouldn't I be equally patient and generous with whatever knowledge and experience I have to share?

These types of threads typically run into the thousands of views. That means many more people are reading the thread than posting in the thread. Some are reading for entertainment, but some are reading for learning...as evidenced by the many PM's and emails I receive expressing appreciation for having helped them understand some stereophonic concept they were struggling with.

evhudsons wrote: »

Just speaking as an enthralled, listening, and trying to understand, but grateful for your thoughts noobie. Without folks like H9, DarqueKnight, Nspindle, and Trey/vrsomething, and many others I would not have the audio joy I have at home. My wife thanks you all too.

Syndil

DarqueKnight wrote: »

Is it "wrong" to hide the identity of components when doing a stereophonic evaluation? Of course not. Just like it is not "wrong" to use bicycle training wheels after a child has learned to keep balance. But why would you?

Finally, amidst the ad hominem attack on my experience (based on incorrect assumptions), bold but rubbish claims, and a completely irrelevant tangent about TV testing, an actual answer. Was that so hard? I've come to expect and tolerate the holier-than-thou attitude from self-proclaimed audiophiles, but really, it didn't need to take this long to get to this point.

So we agree that there is nothing wrong with blind testing; a properly conducted blind test of stereophonic equipment will produce the same results as an unblinded test--if the unblinded test is free of bias. Correct?

sucks2beme

Once again, blind testing (not DBT) is ok, but really requires people who know how to listen.
My wife just enjoys the music, where I often find myself picking out instruments and following
them along through the piece. Same with vocals. That wasn't always the case. My old gear
couldn't render them well, and it wasn't something I ever listened for. Now I can hear the
components of music. Same with many vocals. I can hear the other vocalists and the effect it
has on the song. This becomes more pronounced using my Stax headphone set up. Without room
effects, the image suffers somewhat, but this effect of hearing all the pieces of the music
is amazing. It was a long journey to get there. I test a lot of stuff. Sometimes I do a sanity
(or according to the wife, an insanity check) by pulling a lot of my gear out of the closet and
running through everything to make sure the stuff in the system is the better sounding stuff.
In the case of headphones, they all have a different sound, so rotating them around gives
a different view of the music.

steveinaz

Raife, I applaud your patience and knowledge on these subjects.

BlueFox

Just what is the point of a 'blind' test? It amazes me how much time and effort can be devoted to an irrelevant topic. The question is, does a new addition to the stereo sound better, the same, or worse than the original. If so then keep it. If not then return it. Hiding the gear is not going to change the sound. Is it really that hard to grasp this concept?

DarqueKnight

Syndil wrote: »

Finally, amidst the ad hominem attack on my experience (based on incorrect assumptions), bold but rubbish claims, and a completely irrelevant tangent about TV testing, an actual answer. Was that so hard? I've come to expect and tolerate the holier-than-thou attitude from self-proclaimed audiophiles, but really, it didn't need to take this long to get to this point.

So we agree that there is nothing wrong with blind testing; a properly conducted blind test of stereophonic equipment will produce the same results as an unblinded test--if the unblinded test is free of bias. Correct?

I find your mindset to be similar to that of people in the ante bellum South who could, at any time, spout off 1000 justifications for slavery.

heiney9

Syndil wrote: »

So we agree that there is nothing wrong with blind testing; a properly conducted blind test of stereophonic equipment will produce the same results as an unblinded test--if the unblinded test is free of bias. Correct?

Love how you flip-flop. I bet you change colors when it suits you too.

H9

DarqueKnight

BlueFox wrote: »

Just what is the point of a 'blind' test? It amazes me how much time and effort can be devoted to an irrelevant topic. The question is, does a new addition to the stereo sound better, the same, or worse than the original. If so then keep it. If not then return it. Hiding the gear is not going to change the sound. Is it really that hard to grasp this concept?

It is not hard to grasp that concept. It is hard to abandon one's religious views. You must realize that asking a blind test for stereo cultist to accept ear training and standardization of evaluation procedure is like asking a Catholic to become a Hindu.

The point of a blind test is to test the tester. In other words, it is not enough for some people that a trained evaluator comes to an evaluative result using proper procedure. Some people need to "test the tester" in order to be assured that evaluator bias did not play a role in the results.

For example, the discussion in this thread regarding the audibility of changes in power cords:

1. First the naysaysers claim that there are no scientific studies that show a performance difference between power cords.
2. When a scientific study is done showing different power cords have different effects on reducing power line noise, the naysayers claim the effect is inaudible, yet they produce no scientific substantiation for their claim.
3. Scientific evidence is provided which includes measurements of noise reduction provided by power cable changes that show the noise reduction is much greater than the threshold of audibility established by a prior peer-reviewed scientific study.
4. Naysayers then claim the the results will not be acceptable until they are "proven" with a blind test. In other words, even though it was quantitatively proven that there was a noise improvement with a higher quality power cord and that the improvement was well above the threshold of audibility, the tester needed to be tested in order to "prove" that he heard what he heard.

Another consideration is that blind testing frequently leads to no perceived difference in audio components. This is very comforting to individuals who want to believe that their $200 CD player performs every bit as well as a $2000 player.

Syndil

DarqueKnight wrote: »

I find your mindset to be similar to that of people in the ante bellum South who could, at any time, spout off 1000 justifications for slavery.

What? Nevermind. Are we in agreement about this or not:

Syndil wrote: »

So we agree that there is nothing wrong with blind testing; a properly conducted blind test of stereophonic equipment will produce the same results as an unblinded test--if the unblinded test is free of bias. Correct?

heiney9 wrote: »

Love how you flip-flop. I bet you change colors when it suits you too.

On which point did I flip flop??

DarqueKnight

heiney9 wrote: »

Love how you flip-flop. I bet you change colors when it suits you too.

Are you suggesting a career in politics? :cheesygrin: :cheesygrin: :cheesygrin:

heiney9

You and others have stated the only way for a proper test to be done is blind, now that DK has acquiesced that blind testing can be done if one chooses, but it's not necessary. You agree that you can obtain the exact same results with both methods. Until now you have never stated that, inferred that, or ever once eluded to that. Your position and others is that blind testing has to be done to eliminate the bias. Now you have flip-flopped saying they will both give the same results (with the caveat the sighted test has the bias removed).

I'm sure now you will twist something you've said around to point to the fact that all along you were in favor of sighted testing being the same as blind testing as long as the bias is removed or some such string of thoughts and points you never brought up specifically.

If I have misinterpreted your stance on the issue please point me to the previous posts where you stated they (blind and sighted tests in stereophonic audio analysis) can achieve the same results. There is a lot of white noise in this thread and I might have missed it, or glanced over it.

H9

P.s. When I say to the previous posts, I mean a direct quote or link to the verbage used at the time. Not your reinterpretation or paraphrasing to cover your a$$.

DarqueKnight

Syndil wrote: »

Are we in agreement about this or not:?

This statement and post #175 seem to indicate that you act like you won't be able to enjoy your emotion driven evaluative process unless I agree with you. What you do with your gear is no concern of mine. As I clearly stated, I am interested in exchanging ideas and collaborating with other serious listeners interested in stereophony.

I have laid out my evaluative process in detail. If you choose to take from that something diametrically opposite to what I wrote, that is fine. I think that the content of my prior posts is quite clear for those who evaluate stereo systems with other metrics other than emotion.

As for you, I think it is great that you have found an evaluative procedure that works for you. I also think it is great that you are able to rationalize away the need for stereophonic ear training and proper procedure...and that you only need your "feelings" to evaluate gear. I would still think it was great if you did listening tests standing on your head wearing a blindfold, ear muffs and a ski mask.

DarqueKnight

heiney9 wrote: »

You and others have stated the only way for a proper test to be done is blind, now that DK has acquiesced that blind testing can be done if one chooses, but it's not necessary.

This is not acquiescensce or some "new revelation" on my part. I specifically stated in post #146 that I sometimes use blind testing:

DarqueKnight wrote: »

Items 1-10 comprise just a partial list of the performance parameters I evaluate. Some people may feel the need "blind" themselves. I do not. The people who invented stereo intended for the individual to learn to evaluate stereo equipment on their on...without the need for assistance from second, third, or more parties in setting up blind trials. Either the sound stage is wider/deeper/taller or it is not. Either I hear more details and clarity or I do not.

Sometimes I will blind trial another trusted set of trained ears out of curiosity to see if the other person hears the same things at the same intensity as me. Sometimes I will blind trial a set of untrained ears...particularly if the differences I perceive are subtle.

heiney9 wrote: »

You agree that you can obtain the exact same results with both methods. Until now you have never stated that, inferred that, or ever once eluded to that.

He is desperate to receive acceptance for his emotionally based stereo evaluation procedure.

I have consistently held the position that, for the untrained listener, blinding can lead to inaccurate results and that for the trained listener, blinding is unneccessary. If it is not required for trained listeners, why insist that it must ALWAYS be done? This is from post #138 in this thread where I discuss the use of blind trials in my journal paper case study:

DarqueKnight wrote: »

I was advised by the journal's review panel, which was composed of sensory science experts, that the blind nature of the trial was unnecessary. I was further advised that blinding with minimally trained subjects could induce guessing as the subject might focus on getting the right answer rather than focus on the sensory stimuli presented. More information on that journal publication can be read here.

Notice how he dismissed as "irrelevant" the evidence that blind trials are not used in television. Why is this irrelevant? If bias can be in hearing can't it also be in seeing? I would think even moreso since humans are much more influenced by what they see than what they hear!!!

Surely someone can explain why blinding for audio is required but blinding for television is not.

heiney9

I am sorry DK, "acquiescensce" was not the proper term. I was surprised to see him "jump" at that concept even though all along you (and I know I didn't) ever say blind testing isn't or can't be done. Speaking strictly for myself, I have ALWAYS maintained for me and my experiences it doesn't have to be a blind test for me to figure out which one I prefer.

H9

DarqueKnight

heiney9 wrote: »

I was surprised to see him "jump" at that concept even though all along you (and I know I didn't) ever say blind testing isn't or can't be done.

Desperation makes people do desperate things. If you dangle a cobra in front of a drowning man he will grab it!

DarqueKnight

Deleted duplicate post.

heiney9

DarqueKnight wrote: »

Notice how he dismissed as "irrelevant" the evidence that blind trials are not used in television. Why is this irrelevant? If bias can be in hearing can't it also be in seeing? I would think even moreso since humans are much more influenced by what they see than what they hear!!!

Surely someone can explain why blinding for audio is required but blinding for television is not.

Of course it was dismissed by him. It once again proves that a trained observer can make unbiased observations without resulting to sighted biases. He had to dismiss it because embracing it would create a serious conflict to his current argument and POV.

H9

heiney9

DarqueKnight wrote: »

Surely someone can explain why blinding for audio is required but blinding for television is not.

Yes, I would like to understand this as well. They are so similar one can't help but think there is a very strong commonality.

H9

DarqueKnight

heiney9 wrote: »

Of course it was dismissed by him. It once again proves that a trained observer can make unbiased observations without resulting to sighted biases. He had to dismiss it because embracing it would create a serious conflict to his current argument and POV.

Nooooooooo! It couldn't be that. There has to be some good, solid, scientifically and quantitatively justified reason why television evaluators can be trained to overcome sighted bias but stereo evaluators cannot.

Oh, the sheer and utterly abysmal hypocrisy:

Syndil wrote: »

I think it's all about you both not being able to address my concerns directly. I see a lot of dodging and dismissal, whereas I am being brief and to the point.

Were you brief and to the point in posts #145 and #175? Is it wrong for others to fully develop their points, along with substantiating evidence (that you always summarily dismiss), yet it is ok for you to write dissertations pleading over and over again for people to "answer" questions previously answered in detail?

DarqueKnight

Uh oh...I see Ken is reading...I hope we aren't about to get shut down....not before we find out why blind testing is good for the goose (audio) but not for the gander (video).

C'mon Ken!

heiney9

I honestly think has been one of the more civil discussions. No one resorting to name calling, etc. Just basically having some passionate banter about audio.

H9

DarqueKnight

....and video.

BlueFox

DarqueKnight wrote: »

Uh oh...I see Ken is reading...I hope we aren't about to get shut down....not before we find out why blind testing is good for the goose (audio) but not for the gander (video).

C'mon Ken!

As long as the Chili stories do not start appearing then the thread should be okay.

headrott

DarqueKnight wrote: »

Surely someone can explain why blinding for audio is required but blinding for television is not.

Of course it's because you cannot do the test if you can't see the TV! You don't need your eyes to listen to audio gear, but you do need your eyes to watch TV!:razz:

Greg

DMara

I think blind testing is a good method to identify who's the real deal and who's not. In this audio hobby, there are plenty of folks who are naive enough to think that their setup is the best as well as there are quite a few folks who are bribed/paid by manufacturers to give a certain review/opinion. Of course blind testing isn't necessary for true audiophiles who always try to look for and conquer a higher summit, or even for normal listeners like myself who want to taste various specs to find out what we really like, which signature really pleases our ears. Therefore, for me, I didn't learn anything from this thread that I can say it's useful to me.

steveinaz

I believe there are people that can make eating a banana split a complex activity.

SCompRacer

steveinaz wrote: »

I believe there are people that can make eating a banana split a complex activity.

Just imagine a thread on making one.:eek::twisted::cheesygrin:

George Grand

SCompRacer wrote: »

Just imagine a thread on making one.:eek::twisted::cheesygrin:

Everybody knows you just hit it with an axe. Not much discussion there.