Blind cable testing..... Guess the outcome lol

DarkHorror

F1nut wrote: »

Post #174 based on reality, not your fantasy world.

So you can't, nothing you said there refutes anything I said.

heiney9

teekay0007 wrote: »

Wow, Joe, thanks for thinking I'm cool - very kind of you! :cool: :cheesygrin:

But no, I'm no tough guy, internet or otherwise. Sure, I enjoy the occasional Dirty Harry or John Wayne movie, but I don't think that qualifies me as such. :loneranger:

In regard to H9 and DK - I highly value their opinions and have learned A TON from each of them on this forum. Their contributions here, to date, far outweigh the cumulative contibution I will make here in my lifetime, manyfold. However, on the topic of this thread, I disagree with them and am willing to tell them so. I think I've done so without any namecalling or insulting them personally - well.....maybe a tiny jab here and there, but nothing egregious.

I do find it quite comical that you would take a shot at DarkHorror's reading abilities while in your next breath excusing H9 for the very same offense. :rolleyes:

Perhaps you could explain in detail why you disagree and what methods and processes you have engaged in to support you opinion. I am curious to know what experiences you have had related to audio. Like the gear you've owned, speakers you've owned, cables you have compared to determine you can't discern a difference. Or if you are one of those who reads topics on the internet and decides that's how it is.

H9

F1nut

DarkHorror wrote: »

So you can't, nothing you said there refutes anything I said.

Sure it does, it's just that you don't want to accept reality.

teekay0007

F1nut wrote: »

Post #174 based on reality, not your fantasy world.

F1, Post #174 is a Classic! It has "Scientific Study" and "Irrefutable Evidence" written all over it. :exclaim:

Was it Monoprice and Monster cables you and your pal were talking about? :cheesygrin:

DarkHorror

teekay0007 wrote: »

F1, Post #174 is a Classic! It has "Scientific Study" and "Irrefutable Evidence" written all over it. :exclaim:

Was it Monoprice and Monster cables you and your pal were talking about? :cheesygrin:

LOL really "Irrefutable Evidence"? I hope I am detecting sarcasm?

But what was said does NOT refute anything I have said. The results you got may have very well been real, there very well could have been a difference that you both heard. But that doesn't have anything to do with what I said.

DarkHorror

F1nut wrote: »

Sure it does, it's just that you don't want to accept reality.

You guys aren't even debating what I am talking about, I don't know how you can miss what I say by sooooo much.

heiney9

You haven't said anything that makes sense let alone any concrete ideas. I have now asked you on 3 seperate occasions to site where you got your information from, you know the 100% and 50% thing. Did you just make that up? If not, where did you get it from? Certainly you have some scientific study or something from a reliable outside source in the field to make such a claim.

H9

teekay0007

heiney9 wrote: »

Perhaps you could explain in detail why you disagree and what methods and processes you have engaged in to support you opinion. I am curious to know what experiences you have had related to audio. Like the gear you've owned, speakers you've owned, cables you have compared to determine you can't discern a difference. Or if you are one of those who reads topics on the internet and decides that's how it is.

H9

Wow, H9, you need to just back away from the keyboard for a few minutes - at least until it stops smoking! :evil:


Hey, I'm glad you like your MIT cables. I hope they give you many years of listening enjoyment. However, your fervent defense of unblinded speaker cables comparison studies and disdain for blinded ones reeks of someone who is trying with all they've got to convince the whole world, and themselves, that they made a good purchase! Good luck in your efforts.

heiney9

If you have actually compared MIT cables to your current cables and didn't notice a difference then I respect that and the effort you put into it. But, if you just say there is no difference without ever hearing them then it's you who needs to get away from the keyboard. Because that is absolutely absurd to make that claim w/o ever having listened.

DarkHorror

heiney9 wrote: »

You haven't said anything that makes sense let alone any concrete ideas. I have now asked you on 3 seperate occasions to site where you got your information from, you know the 100% and 50% thing. Did you just make that up? If not, where did you get it from? Certainly you have some scientific study or something from a reliable outside source in the field to make such a claim.

H9

It's simple logic, if there is a difference you should be able to tell the difference if you know what you are listening to or not thus you should get them all correctly. If there isn't a difference and you try to tell which is which. You will simply be randomly choosing one of the two. Thus you will be right 50% of the time.

If i gave you two cups one with milk and one with some other white liquid that tastes nothing like milk. Then you tasted them, you would be able to tell 100% of the time which is milk and which isn't. Now if I gave you milk and something that tasted exactly the same as milk and asked you to choose which is which. You would be correct 50% of the time.

heiney9

If both taste exactly like milk, I couldn't tell the difference. So how does it become 50/50? What a stupid analogy.

Do you have some sort of learning disability? I am being serious.

H9

DarkHorror

heiney9 wrote: »

If both taste exactly like milk, I couldn't tell the difference. So how does it become 50/50? What a stupid analogy.

Do you have some sort of learning disability? I am being serious.

H9

LMAO, you have to chose one or the other... it's a 50% chance you choose correctly. I am not sure how you are having such a hard time with this.

Really you are trying to ask if I have a learning disability while you can't even understand most of what I have said...

DarqueKnight

BlueFox wrote: »

You forgot "Internet Bully", which was said in some other thread. Or maybe it was this one.

Thanks man, I had forgotten that one. It was in another thread. Ok, the revised list is:

I don't "deserve" to have any of my questions answered pertaining to the scientific principle of the "debiased consumer" or the scientific justification for blind testing in stereo, nor should any of my comments or views be taken seriously because I am:

1. Silly,
2. Stupid,
3. Arrogant,
4. Hypocritical,
5. Unreasonable.
6. An Internet bullly
7. Untrained in Human Subject Evaluation.
8. Exhibiting the symptoms of Asperger's Syndrome...even my system showcase shows hints of it,

DarkHorror wrote: »

What we have now is two arguments which boil down to...

H9: I don't believe blind tests are valid.

DK: I don't think a trained evaluator needs to do a blind test.

This is simplistic and wrong. Your comment would have been correctly stated as:

"H9: I don't believe blind tests are valid for the evaluation of stereophonic audio equipment.

DK: There is no scientific evidence that a person trained in stereophonic evaluation needs to do a blind test."

DMara wrote: »

This statement shows more than just about jelousy, it's actually about the incompetence of the speaker!

He is not incompetent. His has a co-worker with Asperger's syndrome. This qualifies him to diagnose the condition in someone else simply by reading their posts on an Internet forum and by looking at pictures of their audio equipment. Furthermore, his (claimed and unsubstantiated) vast experience in human subject trials means that we should take his word for whatever he says.

unc2701 wrote: »

DK-Finally, and this is not meant as an insult: I think you have Asperger's. I work with a guy that has it and your posts are absolutely full of markers... even your system showcase pics have some hints of it. Again, this is not an insult, this is a purely objective observation on how you react and interact on this forum from someone that knows the condition.

unc2701 wrote: »

I've worked on hundreds of clinical trials; I work with human subject data every day. Evaluator bias is huge, even in relatively objective measures.

As for NCC1701 being a jealous a$$hole, well, I've never met him so I will defer judgement on that until I have more substantial evidence.

unc2701 wrote: »

Well, given that many people report hearing differences in cables, but there have been no blinded trials that have shown this, the empirical evidence would support his statement.

unc2701 wrote: »

1)DK's evaluation methodology creates a profile of a cable's characteristics: soundstage size, instrument placement, etc

4)The evaluator should be able to take his notes and with reasonable accuracy identify which of the two cables that they are listening to in a blinded manner.

This would prove without doubt that the evaluator can actually is observing the different characteristics of the cables.

My understanding is that blind tests fall into the class of tests called "null tests" or "null hypothesis tests". The problem with null tests is that a null hypothesis can never be proved. This is from the Wikipedia Wikipedia article on Null Hypothesis

"The practice of science involves formulating and testing hypotheses, assertions that are capable of being proven false using a test of observed data. The null hypothesis typically corresponds to a general or default position. For example, the null hypothesis might be that there is no relationship between two measured phenomena or that a potential treatment has no effect.

It is important to understand that the null hypothesis can never be proven. A set of data can only reject a null hypothesis or fail to reject it. For example, if comparison of two groups (e.g.: treatment, no treatment) reveals no statistically significant difference between the two, it does not mean that there is no difference in reality. It only means that there is not enough evidence to reject the null hypothesis (in other words, the experiment fails to reject the null hypothesis)."

Therefore, a blind test can only tell if there is or is not enough evidence to reject the assertion that there is no difference between two things. If a blind test indicates there is no difference between two things, it does not mean there is no difference in reality, it only means that sufficient evidence was not detected to indicate a difference.

For someone to say that they do not believe a difference exists between two things because it has never been proven by a blind test does not make sense to me. By definition, a difference test can only indicate if there was or was not a detectable difference between two things. It cannot prove or disprove whether there was a difference in reality. Furthermore, in cases where a difference is reliably detected, null tests do not provide descriptive information on the nature of the difference.

Question 1: What is the scientific validity in someone demanding proof that a difference exists by using a test that, by definition, cannot provide proof, but can only indicate whether there is enough evidence to reject the hypothesis that there is not difference?

Question 2: Considering blind tests within the context of stereophonic audio evaluation, what valuable information would it provide to the consumer to merely state that a difference exists between two components?

It has been suggested that a trained evaluator could "certify" sighted results by comparing sound profiles made during blind sessions to those made during sighted sessions. If the two results agree, this would "certify" that the evaluator actually heard what he/she said they heard.

The problem with this proposition is that the practice of creating sound profiles, even with the identities of components hidden, is not a true blind test. It is descriptive analysis. If a trained evaluator is able to accurately describe the sound of a component while looking at it, they would certainly be able to describe the sound of the same component if it were hidden.

As for detecting a difference, I have stated that I sometimes do not become aware of a difference until I review my listening notes. This is because I do not focus on detecting a difference. I focus on accurately describing everything in the stereophonic presentation.

Question 3: If someone is trained to accurately describe the sound of something, what scientific evidence is there to support the theory that such sonic evaluation will be more or less accurate by the evaluator seeing the object?

Hypothetical Case Study

Consider a proposed methodology for determining the physical attractiveness of women based on:

1. Texture and length of hair.
2. Facial symmetry (shape of face and relative sizes of eyes, nose, lips).
3. Body shape.
4. Body size
5. Smoothness of complexion.
6. Pleasantness of the sound of voice.

In summary, women in this case are evaluated on their shape, size and sound.

Two women are evaluated by sight and by talking to them. At the conclusion of the trial, Ethan Winer and a few enforcers from the Blind Test Inquisition knock the door down and demand that the test be done blind in order to ensure "scientific validity".

The evaluator informs the Inquisition that he can do even better than what they ask. Rather than the evaluator repeating the test, he brought in Stevie Wonder from the next room and had him evaluate the women using the same set of criteria.

Mr. Wonder was able to accurately describe each woman, in fine detail, by repeatedly running his fingers over each woman's hair, face and body, and by listening to them speak. His results agreed with those of the evaluator's sighted results. Mr. Wonder said he "looks for" big ****, big butts, long hair, smooth skin, long legs, a pleasant midtone feminine voice, long hair curly hair, full lips, large eyes, oval shaped faces, a nose in proportion to the size of eyes and lips, and a medium size "hourglass" figure...and he does not need eyes to assess any of that!

The Inquisition members bitterly cursed the women, the evaluator and Mr. Wonder and hastily left, even after being offered the opportunity to do a "blind trial" with each woman.

---

DarkHorror, you still here? I thought you had resigned from the discussion? Since you are still around, please take a crack at questions 1-3. Thanks.

Maybe we should have Stevie Wonder do a series of cable tests too.

Yeah, that would be super!~DK

F1nut

DarkHorror wrote: »

You guys aren't even debating what I am talking about, I don't know how you can miss what I say by sooooo much.

You think blind tests are the answer. My post proves they are not. I get how you missed that by sooooo much. If only you would get it too.

DarqueKnight

heiney9 wrote: »

If both taste exactly like milk, I couldn't tell the difference. So how does it become 50/50? What a stupid analogy.

Do you have some sort of learning disability? I am being serious.

The condition is more likely that of being "punch drunk".

punch-drunk:
adj.
1. Showing signs of brain damage caused by repeated blows to the head. Used especially of a boxer.
2. Behaving in a bewildered, confused, or dazed manner.

audiocr381ve

They let Stevie touch their ****?!?!?! :eek:

pearsall001

Holy Cow!!! You guys actually take all this stuff seriously...now that's funny!

transmaster

Let's see y'all nay say a Papal Bull from a Speaker God. Kenneth Kantor :razz:

http://www.kenkantor.com/publications/audio_fetishes/fetish_part_02.pdf

pearsall001

transmaster wrote: »

Let's see y'all nay say a Papal Bull from a Speaker God. Kenneth Kantor :razz:

http://www.kenkantor.com/publications/audio_fetishes/fetish_part_02.pdf

Hysterical!!!! You know the age old saying...if the shoe fits, wear it...well it sure fits a lot of guys around here! Priceless!!!

DarqueKnight

transmaster wrote: »

Let's see y'all nay say a Papal Bull from a Speaker God. Kenneth Kantor :razz:

http://www.kenkantor.com/publications/audio_fetishes/fetish_part_02.pdf

No need. I posted an extensive treatment of the subject here: A-Historical-Overview-of-Stereophonic-Blind-Testing

audiocr381ve wrote: »

They let Stevie touch their ****?!?!?! :eek:

Stevie touched EVERYTHING!

nwohlford

DarqueKnight wrote: »

My understanding is that blind tests fall into the class of tests called "null tests" or "null hypothesis tests". The problem with null tests is that a null hypothesis can never be proved.

Therefore, a blind test can only tell if there is or is not enough evidence to reject the assertion that there is no difference between two things. If a blind test indicates there is no difference between two things, it does not mean there is no difference in reality, it only means that sufficient evidence was not detected to indicate a difference.

For someone to say that they do not believe a difference exists between two things because it has never been proven by a blind test does not make sense to me. By definition, a difference test can only indicate if there was or was not a detectable difference between two things. It cannot prove or disprove whether there was a difference in reality. Furthermore, in cases where a difference is reliably detected, null tests do not provide descriptive information on the nature of the difference.

Question 1: What is the scientific validity in someone demanding proof that a difference exists by using a test that, by definition, cannot provide proof, but can only indicate whether there is enough evidence to reject the hypothesis that there is not difference?

Yes, a statistical test can never show that two things are identical, but this is often not of concern. In any situation, there is a level of minimum level of difference below which you would not care. I think that we would all agree that if 50.1% preferred one item and 49.9% preferred the other (the true preferences, not the preferences found in a test), there would effectively be no difference between the items. Thus you can construct a 95% (or other significance level) confidence interval, and if the limits of the confidence interval are within what you would consider an insignificant difference, then you can say that there is likely not a meaningful difference.

If a test is properly powered (sometimes very hard to be certain), then failing to prove a difference (i.e., failing to reject the null) says that it is unlikely that there is a meaningful difference.

As for your statement that the test just shows there is a difference, this is why the estimate and confidence intervals are preferred over p-values in summarizing results.

DarqueKnight

nwohlford wrote: »

Yes, a statistical test can never show that two things are identical, but this is often not of concern.

Is it, or should it be, of concern within the context of evaluating stereo equipment?

Are you aware that the overwhelming majority of stereo blind tests show that there was no difference (i.e. items sounded identical) in the evaluated items?

nwohlford wrote: »

I think that we would all agree that if 50.1% preferred one item and 49.9% preferred the other (the true preferences, not the preferences found in a test), there would effectively be no difference between the items. Thus you can construct a 95% (or other significance level) confidence interval, and if the limits of the confidence interval are within what you would consider an insignificant difference, then you can say that there is likely not a meaningful difference.

How does knowing the preference of a group of people provide meaningful information to someone considering a piece of stereo equipment?

Wouldn't a consumer need meaningful information related to performance in order to provide some indication of the potential performance gains? How can a difference test or preference test provide such information?

nwohlford wrote: »

If a test is properly powered (sometimes very hard to be certain), then failing to prove a difference (i.e., failing to reject the null) says that it is unlikely that there is a meaningful difference.

What would you consider a properly powered test pertinent to stereophonic equipment selection and how easy would it be to establish confidence in the results?

nwohlford wrote: »

As for your statement that the test just shows there is a difference, this is why the estimate and confidence intervals are preferred over p-values in summarizing results.

My understanding is that probability (p) values are not applicable to null tests as it would be like assigning probability to proving a negative.

Estimate and confidence intervals are driven by subject preference. Again, within the context of information useful to stereophonic equipment evaluation, how valuable is the knowledge of what someone else prefers?

unc2701

The first post you respond to was a formulaic layout of assumptions to reach a logical conclusion. You clearly didn't recognize it as such.

DarqueKnight wrote: »

The methodology has been shown to produce very similar results with different people trained in the same methodology, using different gear, but the same piece of music....

Why would you say it is worthless for two audiophiles to document how the same piece of music sounds on two different systems? ....

I didn't say different people; I said the same person.
-You're not very good at reading what people write
This repeatability is called intra-rater reliability. It's a basic measure of the quality of an evaluation methodology, scale, etc. If you can't get the same person to achieve the same results with in a reasonable level of variability, it's worthless. Inter-rater reliability is trickier, but a good process will have a high level of repeatability across raters. The remainder of your paragraph is ascribing a vast amount of things I did NOT say off of one basic sentence.
-You're not familiar with standard measures of assessing the quality of an evaluation methodology
-You have a habit of vastly misunderstanding other's statements and their intentions.

DarqueKnight wrote: »

Well, if two cables have the same sound profile but one costs $100 and the other costs $500, do you consider that worthless information? Really? If so, you must not be an audiophile...

Let's look at my statement:
"There exist two cables which have different profiles. If this is not true, the evaluation is worthless."
Let's negate it:
There do not exist two cables which have different profiles. Which would mean all cables have the same profile. So, an evaluation can have an extremely high reliability (see above), but be worthless if it always returns the same value. This statement is important in the sequence of statements, because you have to rule out the possibility that the reliability is driven by simply giving the same value over and over.
-You are unable to correctly negate a simple statement.
-You again ascribe meaning to a simple statement where none existed

DarqueKnight wrote: »

Absolutely. I addressed this in my previous post. Perhaps you, with all of your quantitative analysis training and insight, can enlighten us as to why a sighted quantitative measurement needs to be validated by a blinded trial. For example, if a preamp costing $10,000, like my current Pass Labs X0.2 ...

Why? Bias.

Yes. It happens all the time and your brain wouldn't even know it happened. But here's the question: Why have you not done this? Do your unbiased, golden ears shrivel up when don't know which cable you're listening to? You moan about the cable naysayers continually, but if 1,2,3 are true, someone, somewhere, should be able to demonstrate this in a well controlled, blinded manner. Telling the cable naysayers about the size of your soundstage and citing papers is unconvincing. If there is no reason why your methodology cannot be applied in a blinded manner, then DO IT. Put up or shut up.

Additionally, you use "sighted" as the opposite of "blinded" and appear to be using it in the sense of vision (for example when you spoke of the sound that one of your power cords makes with plugging it in). This is not a correct usage and is not a mistake that someone with experience in human subject data would make.

DarqueKnight wrote: »

Rather than laughing at us, why not help us by pointing to research which invalidates the economic concept of the "debiased" consumer? Since the concept is so laughable and since you are such a highly trained and experienced researcher, it should be a trivial matter for you to find contravening research.

Anyone else read that paper? Cause it has absolutely no bearing on this topic. I'm a little baffled why no one has bothered to call you out on it, but whatever.

DarqueKnight wrote: »

I am glad you offered this example, as it provides more insights into your character and motivation.

Here is an example of some "trick" being "pulled" on researchers. Please explain how an exercise, whose stated goal is to "trick" researchers, is pertinent to an evaluative method for stereo.

What do Cartier boxes have to do with audio? You brought it up. But how it matters is that humans are biased and they will respond with their preconceptions. EVEN WHEN DEALING WITH PURELY OBJECTIVE MEASURES. This example is a clear counterpoint to your musings.

DarqueKnight wrote: »

Please clarify:

If these researchers are actually trained professionals, why would they be "tricked" by such an experiment? Are they trained in conducting water density experiments? If so, shouldn't they know about calibrating measurement devices to a standard? Shouldn't they know about the possible experimental variables such as "crap in the water"? Shouldn't a professional trained in water density calculations be able to get the "volume just right"?

Are they getting close to 1 because they are lying or because the equipment is inaccurate?

Trained in what? Were they trained in the knowledge, procedures and processes applicable to the experiment? I advocate training in sound localization and developing a descriptive lexicon in order to document and communicate what is heard. I do not advocate setting people up with uncalibrated measurement devices and other deceptions.

It is so sad that you think an exercise purposely designed to trick people with uncalibrated measurement devices is analogous to and honest exercise with calibrated measurement devices.

Ok, this is where it really becomes clear that you've never worked with human subject data. This is not some kind of thought exercise- it's been done over and over again. I participated in one at Bowman Gray Med school with a bunch of lab techs, MD's, Biochemists, pretty much people that work in the lab all day, and yes, they were qualified in using scales and graduated cylinders. This experiment will work on any large group of people aware that the density of water is 1.

As for purposely tricking people?.... Really? You are aware that nearly all drug trials use a placebo? Those WICKED, WICKED researchers, tricking people with their sugar pills! You could take nearly any experiment done on humans and find some way to characterize it as a "trick".

-Again, it could not be more clear that you know nothing about working with human subjects.
-Again, you have a habit of vastly misunderstanding other's statements and their intentions.

DarqueKnight wrote: »

Which is more lame:

1. An analogy showing that a quantitative measurement will remain the same whether it is done sighted or blind.

2. An analogy showing an exercise purposely designed to trick people, which by description of the participants in the exercise, should not be tricked as they should know how to conduct such and experiment and to avoid inherent pitfalls.

1) is clearly more lame. It is an analogy which exists purely in your mind full of false statements as shown by an actual, repeatable experiment. 2) Is not an analogy, and those people aren't "tricked"- they're simply victims of human nature.

I also find it quite curious that you completely miss this distinction between analogy and actual experimentation. Furthermore, your reaction with the word "tricked" is very odd.

DarqueKnight wrote: »

Correct me if I misunderstand
....

A stereophonic sound stage is a psychoacoustic phenomenon, an illusion, generated by a person's mind.

I agree with all the above. Where I'm on the fence in this whole debate is whether it matters if an actual difference exists or not. As you say, it's a psychoacoustic phenomenon. So if the cable tastes like a brazillian supermodel's.... toes... and that makes you think it sounds better: Does it matter that it is, in truth, no different?

DarqueKnight wrote: »

I realize this a difficult concept to grasp...

And this is where we have a failure in communication. Everything you lay out would lead one to believe that with your methodology should work quite well in a blinded environment, you complain about the cable naysayers, dig up piles of articles... but why not just do the test? As you note in a later post, failure doesn't actually mean that it didn't work, it just means you were unable to detect a difference. If that happens, then you can cite your pile of articles for reasons why it didn't.

DarqueKnight wrote: »

Are the clinical trials and human subject data similar to the types of stimuli and data generated by stereophonic audio systems?

Depends on the trial. But you have a human reporting how they are perceiving their environment in both subjective and objective domains.

DarqueKnight wrote: »

The bias we are concerned with in stereo trials is visual bias. Is this the same type of bias you deal with?

Again, the misunderstanding of blinding and vision. But to answer your question, yes. Pill size, shape and color impact both efficacy and AEs. Additionally, I'm working with a client on a number of visual scales.

Oh, and there's this:

That just cannot be unseen.

DarqueKnight wrote: »

What you have worked on is irrelevant. Either you have some credible information to share or you do not.

What one has worked on is quite relevant when one makes sweeping generalizations. You claim human bias and preconceptions can simply be erased based on what appears to be little or no experience.

unc2701

Hokay, wrapping up one or two things (BTW, how do you people do this all day?):
1)I wasn't making a diagnosis or suggesting that was possible. That'd take a clinician and an evaluation against the DSM definition. I was stating my opinion based on some unique patterns I've observed in you both.
2)You took this out of context: "Well, given that many people report hearing differences in cables, but there have been no blinded trials that have shown this, the empirical evidence would support his statement." That was in reference to someone else's remarks that a sighted and double blind test would produce different results.

nwohlford

DarqueKnight wrote: »

Is it, or should it be, of concern within the context of evaluating stereo equipment?

Are you aware that the overwhelming majority of stereo blind tests show that there was no difference (i.e. items sounded identical) in the evaluated items?



How does knowing the preference of a group of people provide meaningful information to someone considering a piece of stereo equipment?

Wouldn't a consumer need meaningful information related to performance in order to provide some indication of the potential performance gains? How can a difference test or preference test provide such information?



What would you consider a properly powered test pertinent to stereophonic equipment selection and how easy would it be to establish confidence in the results?



My understanding is that probability (p) values are not applicable to null tests as it would be like assigning probability to proving a negative.

Estimate and confidence intervals are driven by subject preference. Again, within the context of information useful to stereophonic equipment evaluation, how valuable is the knowledge of what someone else prefers?

I was just trying to point out that your previous concerns were not really justified. The main issue to address is that blind testing does not restrict in any way the testing you can perform. You just need to propose a hypothesis and develop a study to test the hypothesis while making sure that the study can provide you with meaningful data that can be analyzed to assess the validity of that hypothesis. Once you start working out the study details, you can address the questions you have above, since these are questions of study design and not questions involving blinding.

Most amateur audio trials such as that one that started this thread are very poorly designed and doomed from the start. A well designed study takes a lot of thought and expertise (and often money), and the analysis methods should always be taken into account when designing a study. Variables such as anticipated effect size and variability should be well understood when designing the study.

DarqueKnight

unc2701 wrote: »

The first post you respond to was a formulaic layout of assumptions to reach a logical conclusion. You clearly didn't recognize it as such.

unc2701 wrote: »

1)DK's evaluation methodology creates a profile of a cable's characteristics: soundstage size, instrument placement, etc
2)This methodology produces very similar results if the same piece of music is evaluated by the same trained person using the same gear. If this is not true, the evaluation method worthless.
3)There exist two cables which have different profiles. If this is not true, the evaluation is worthless.
4)The evaluator should be able to take his notes and with reasonable accuracy identify which of the two cables that they are listening to in a blinded manner.

This would prove without doubt that the evaluator can actually is observing the different characteristics of the cables.

I didn't say different people; I said the same person.
-You're not very good at reading what people write
This repeatability is called intra-rater reliability. It's a basic measure of the quality of an evaluation methodology, scale, etc. If you can't get the same person to achieve the same results with in a reasonable level of variability, it's worthless. Inter-rater reliability is trickier, but a good process will have a high level of repeatability across raters. The remainder of your paragraph is ascribing a vast amount of things I did NOT say off of one basic sentence.
-You're not familiar with standard measures of assessing the quality of an evaluation methodology
-You have a habit of vastly misunderstanding other's statements and their intentions.

I agreed with your points 1, 2 and 4. With regard to my response to point 2, I was providing clarification that the methodology is extensible to different people with different systems.

With regard to repeatability, this quote is from my 2010 stereo evaluation paper:

"With regard to audio equipment evaluation and selection, training in descriptive sensory methods is proposed as a method to produce quantifiable and repeatable evaluative results (Murray et al. 2001; Meilgaard et al. 2007). References in the sensory science literature indicate that descriptive tests, rather than discrimination tests, are indicated when differences between the sound of two products is being evaluated (Junker et al. 2001; Siekierski et al. 2001; Evin and Siekierski 2002; Civille and Seltsam 2003; Susini et al. 2001; Susini et al. 2004; Meilgaard et al. 2007). Audio equipment evaluations using the prevailing discrimination methods have been shown to produce conflicting and inaccurate results (Masters 1987; Atkinson and Hammond 1989; Banks and Krajicek 1989; Serinus 2004)."

unc2701 wrote: »

Let's look at my statement:
"There exist two cables which have different profiles. If this is not true, the evaluation is worthless."
Let's negate it:
There do not exist two cables which have different profiles. Which would mean all cables have the same profile. So, an evaluation can have an extremely high reliability (see above), but be worthless if it always returns the same value. This statement is important in the sequence of statements, because you have to rule out the possibility that the reliability is driven by simply giving the same value over and over.
-You are unable to correctly negate a simple statement.
-You again ascribe meaning to a simple statement where none existed

Again, if all cables sound alike, yet have very different prices, this is valuable information.

unc2701 wrote: »

Why? Bias.

Yes. It happens all the time and your brain wouldn't even know it happened. But here's the question: Why have you not done this? Do your unbiased, golden ears shrivel up when don't know which cable you're listening to? You moan about the cable naysayers continually, but if 1,2,3 are true, someone, somewhere, should be able to demonstrate this in a well controlled, blinded manner. Telling the cable naysayers about the size of your soundstage and citing papers is unconvincing. If there is no reason why your methodology cannot be applied in a blinded manner, then DO IT. Put up or shut up.

Anticipating the controversial nature of the methodology paper I published in 2010, the original submission included three case studies that included blind trials. The published version of the paper only contained only one blind trial case study using a minimally trained evaluator. The case study is covered on pages 810-814 of the paper. I was advised to edit out the other two case studies for the sake of brevity. It was suggested that the other case studies could be included in a follow-up paper.

Obviously you have never properly investigated the methodology you have been railing against for nearly two years.

"Condemnation without investigation is the height of ignorance"--Albert Einstein

unc2701 wrote: »

Additionally, you use "sighted" as the opposite of "blinded" and appear to be using it in the sense of vision (for example when you spoke of the sound that one of your power cords makes with plugging it in). This is not a correct usage and is not a mistake that someone with experience in human subject data would make.

I have said numerous times in this thread that "blinding" refers to hiding the knowledge of price, brand and appearance. I said this in posts 64, 80, 98, 113, 115, 144, 164 of this thread.

This is from post #80:

DarqueKnight wrote: »

The stated reason for the requirement for blind stereo tests is to remove bias induced by knowledge of brand, price and appearance. However, many scientific studies from the fields of economics, psychology and sociology prove that people can overcome the effects of an extremely strong bias through training, education and therapy.

This is from post #113:

DarqueKnight wrote: »

But we aren't talking about ALL biases are we? We are talking about visual bias due to knowledge of brand, price and appearance.

unc2701 wrote: »

Anyone else read that paper? Cause it has absolutely no bearing on this topic. I'm a little baffled why no one has bothered to call you out on it, but whatever.

Obviously you did not read "that paper" because I actually provided links to two economics papers which discussed the abatement of consumer bias through training and education.

If a position is held that stereo tests must be done blind due to bias induced by knowledge of brand, price and appearance and if credible economic studies show that such bias, or the effect of it, can be eliminated through training and education, why would they have no bearing on this topic?

unc2701 wrote: »

As for purposely tricking people?.... Really? You are aware that nearly all drug trials use a placebo? Those WICKED, WICKED researchers, tricking people with their sugar pills! You could take nearly any experiment done on humans and find some way to characterize it as a "trick".

You specifically said:

unc2701 wrote: »

DK-There's a well known experiment pulled on researchers with some regularity:

The term "pulled on" has negative connotations and is synonymous with deception, as in "pulling a fast one" or "pulling one over on".

If you overheard someone saying they were going to pull something on you, would eagerly anticipate that they were planning something good for you?

DarqueKnight

nwohlford wrote: »

I was just trying to point out that your previous concerns were not really justified.

I was just trying to point out that blind testing is not required for every evaluative situation.

nwohlford wrote: »

The main issue to address is that blind testing does not restrict in any way the testing you can perform.

I agree, but again, just because blind testing is not restrictive in a particular situation, it does not mean it is required. If it is not needed, why do it?

In the hypothetical case study in post #224, test results would have been the same whether it was performed blind or sighted. It could be argued that knowledge of each woman's age, economic status, educational level, family background, race or color could influence the perceived attractiveness of each woman. However, even if an evaluator had preferences with regard to age, economic status, educational level, family background, race or color, and possessed this knowledge for each woman, they would still be bound by the stated evaluative criteria. For example, if the less attractive woman was a billionaire, and the evaluator had a preference for a wealthy woman, he could not assign higher points to the less attractive woman if she had a stick figure body and bad acne while the selection criteria called for a curvy, proportionate body and a clear complexion.

I was interested in reading your views on what you would consider to be a well designed blind test for stereo, but I will not press the matter.

drumminman

What tempest in a teapot.

While some here may quibble about DK's evaluation methods one thing that's inarguable is in the final analysis he gets very positive results! Not necessarily with the first attempt, but he problem solves, and creates and tests hypotheses until he usually achieves his goal. He also documents his failures, which is extremely useful.

And if he gets consistently good results, doesn't that mean he's pretty good at evaluating?

I've done several mods he has initiated and documented and have never been less than satisfied. I'm happy to (somewhat ignorantly) ride on the coat tails of his knowledge of all things SDA.

gp4jesus

Lots of valid info. Unfortunatly too much "mud flung" along the way!

239 posts and counting.

"Why can't we just get along?"
Rodney King

tonyb

gp4jesus wrote: »

Lots of valid info. Unfortunatly too much "mud flung" along the way!

Thats the result of asking those to put up or shut up, which when they can't, the mud flies. Grade school behavior at best.