A Historical Overview of Stereophonic Blind Testing

DarqueKnight

Introduction

The application of blind and double-blind tests is thought by a small, but vocal, minority in the audio community to be the supreme evaluation standard for detecting audible differences in audio systems. It is true that some types of audio systems are well suited for blind and double-blind A/B or A/B/X type tests. A/B and A/B/X tests are useful in scenarios when the two audio signals being compared are simple in nature. For example, telephone company engineers have routinely used, and continue to use, A/B and A/B/X tests to evaluate improvements in voice circuit quality. [1] [2] [3] [4] However, we must realize and understand that a test that is suitable for one type of audio system might not be suitable for another. It is worth noting that the same company (the Bell Telephone System) that was responsible for the invention and implementation of telephone service was the same company that was responsible for the invention and implementation of home stereophonic audio systems. [5] [6] [7] It is even more interesting to note that while A/B and A/B/X tests were found to be appropriate for evaluating voice quality improvements on bandwidth-limited telephone circuits, subjective, non-blind listening tests based on careful listening, evaluator training and realistic home listening conditions were the scientific standards for the evaluation of stereophonic audio systems. [5] [6] [8] [9] [10] [11] [12] [13] [14]

It should not be too difficult to understand that a testing methodology that is appropriate for evaluating simple band-limited monophonic signals would most probably not be appropriate for evaluating complex stereophonic signals that cover the full range of human hearing and which are designed to convey aural, spatial and tactile information. Telephone systems are audio systems, but they are audio systems which are primarily designed to convey clear voice communication. Stereophonic systems are audio systems, but they are audio systems which are designed to convey a weighty, complex, realistic illusion of a three-dimensional music concert performance.

Origins Of Blind Stereophonic Audio Testing

A paper published by Jon Boley and Michael Lester in the proceedings of the 127th Convention of the Audio Engineering Society, October 2009, stated:

"ABX tests have been around for decades and provide a simple, intuitive means to determine if there is an audible difference between two signals."

Within the audio engineering community, the ABX methodology has become the standard psychoacoustic test for determining if an audible difference exists between two signals." [15]

The first statement is true if the signals are very simple in nature, especially if they are monophonic signals. The second statement is questionable since both founders of the ABX audio testing religion wrote ten-year follow-up papers lamenting the widespread unacceptance of ABX testing by audio engineers and the audio press. [19] [20]

Ethan Winer, at his "Audio Myths, Artifact Audibility and Comb Filtering" workshop presented at the 127th Convention of the Audio Engineering Society in October 2009 stated:

"Double blind tests are the gold standard in every field of science."

and

"It amazes me when some people claim that double blind testing is not valid for assessing audio gear." [16]

What is truly amazing is that some people would stray so far from the scientifically valid subjective listening evaluation procedures developed at Bell Telephone Laboratories and other electronics firms that participated in the invention and early development of home stereophonic systems (e.g. General Electric, Radio Corporation of America, etc.).

Another amazing feature of the Winer presentation is that he included a staged purse-snatching demonstation (at time 9:56) to illustrate the unreliability of short term visual memory. None of the audience members could accurately identify the "purse-snatcher", even though some were sure that they could. The purpetrator was only in the room for 10 seconds. Mr. Winer later contradicts himself (at time 27:50) by advocating the use of an audio evaluation test that uses short term aural memory.

As far as I have been able to determine, the seminal papers in the application of ABX methodology to stereophonic systems are a paper presented by Stanley ****, Ph.D. and Dr. John Vandekooy Ph.D. in 1980 to the 65th Convention of the Audio Engineering Society in London and a paper presented by David Clark in 1981 to the 69th Convention of the Audio Engineering Society in Los Angeles.

Drs. **** and Vanderkooy stated:

"In order for subjective tests to be meaningful to others, the following should be observed...The test must be blind or preferably double-blind. To implement such tests we advocate the use of A/B switchboxes." [17]

Mr. Clark stated:

"Listening tests used to evaluate audio equipment can seldom be considered scientific tests".

and

"A system for practical implementation of double-blind audiobility tests is described. The controller is a self-contained unit, designed to provide setup and operational convenience while giving the user maximum sensitivity to detect differences." [18]

The contoller that Mr. Clark mentioned was an "ABX Comparator" system that he and some associates were marketing through the "ABX Company".

It is curious to note that neither of these seminal papers present a discussion of how the proposed ABX methodology relates to the evaluation of the primary performance metrics of stereophonic sound systems, such as:

1. Optimization of sound stage width,
2. Optimization of sound stage depth,
3. Stable stereo image placement,
4. Clarity,
5. Detail,
6. Dynamics (dynamic range),
7. Tactile impact,
8. Sonic realism.

Whereas the founders of stereophonic audio systems emphasized listener education and ear training with music, Mr. Clark proposed a different training paradign for increasing the resolution sensitivity of listeners ([18, p. 332]):

"Great improvements in resolution can be achieved if the listener knows what to listen for. Sensitizing tests can use pink noise, sine waves, or pulses as appropriate to hear a difference. Sometimes an artificially enhanced distortion can be produced by reducing feedback or connecting multiple devices in series for distortion buildup. The listener is then more able to hear the difference in music."

Really?

Ten years after writing this, Mr. Clark, in a paper presented to the 91st Audio Engineering Society Convention in 1991 stated:

"Ten years ago [in 1982] the present author presented a paper to the AES on double-blind testing using the A/B/X technique. For the next five years, a device to conveniently implement this test was commercially available. It was thought by the author and his associates that general use of this system would resolve "The Great Debate" of whether or not small differences in audio components were audible." [19]

In the same paper, Mr. Clark stated:

"It becomes an ethical and perhaps legal question when it is claimed that improved sound quality is delivered despite failure of tests to prove it.

This would be less of an issue if the number of engineers who dismiss double-blind test results were small, but this is not the case. As Chairman of an AES Workshop on Esoteric Audio in 1988, I asked, by a show of hands, who in the audience believed that different gain-matched amplifiers of modern design sound different from each other. It was stated that all would measure good in conventional tests and all were operated below clipping or other gross distortion levels. Approximately 70% of the audience indicated they believed the amplifiers would likely sound different. This is an amazing response from members of an engineering society which failed to support the claim." [19]

Ten years after his seminal double-blind audio testing advocacy paper was published, Dr. **** presented a paper to the AES 8th International Conference in 1991 in which he stated:

"It is now ten years since my initial involvement in the controversy surrounding double-blind subjective testing in audio and twelve years since this subject first hit the headlines with the Quad power amplifier comparison challenge in England.

A lot of water has passed under the bridge in the intervening years, but our hopes of a decade ago, that the validity of the method would be generally accepted by the audio press and adopted wherever feasible, have not been realized." [20]

At the same 1990 AES conference where Dr. **** lamented the lack of widespread acceptance of blind testing in stereophonic system evaluation, Tom Nousaine, in a paper entitled "The Great Debate: Is Anyone Winning?", was decidely more upbeat:

"This paper simply presents a compilation of the twenty two blind and double blind listening tests [from 1978 to 1990] of power amplifiers for which numerical results have been published. There is a rather large collection of data which contains some surprising information and ultimately confirms that one side of the debate seems to have a commanding lead."

DarqueKnight

Mr. Nousaine concluded:

"Many factors can contribute to the subjective enjoyment of a given amplifier but sound quality differences are not among them.

This does not suggest that amplifiers are perfect and they will never be found to sound different. It does suggest to purchasers of today's audio amplifiers that as long as the product in question meets basic traditional measured performance standards, has enough output capability, and adequate quality of construction, it will be sonically indistinguishable from all others meeting those criteria." [21]

It is a shame that Mr. Nousaine and some others view the pursuit of stereophonic audio (stereophony) as some sort of contest to be won rather than an attempt to recreate a realistic reproduction of the live concert experience in the listener's home.

At the same 1991 AES conference where Mr. Clark lamented the lack of widespread acceptance of blind testing in stereophonic system evaluation and the demise of his company which was offering ABX Comparator devices, Tom Nousaine, in a paper entitled "Can You Trust Your Ears?", was decidely more upbeat:

"In assembling a summary of 22 blind listening tests published between 1978 and 1990 we discovered that subjects consistently reported preferences or differences in sound quality when given two identical alternatives."

and

"The evidence clearly demonstrates a bias in listeners who have an interest in audio to report differences which do not exist." [22]

Whereas the inventor of sterephonic home audio systems, Dr. Harvey Fletcher, and the other early researchers in the field advocated the use of comfortable, stress-free listening environments similar to a typical consumer's home, Mr. Nousaine preferred a different evaluative environment. The "test design" section of the paper stated:

"Our basic test procedure requirements included:

1. Listeners with a strong interest in audio sound quality ("Audiophiles")

2. Listeners with no audio background ("Consumers")

3. Single or Double Blind presentations

4. "Preference Scoring (Prefer A, B or Neither?")

5. Reasonably large sample size (30 by category and 300 overall)

6. Controlled introduction of loudness differences (1 dB)

7. "Purchase-like" conditions/scientific controls"
* Short musical selections
* Single listeners or small groups
* Written scores
* A/B Presentations with no repeat
* Maximum 10 trials/15 minute sessions (Preserve listener freshness)" [22]

In blind and double-blind telephone voice quality trials, the tests are administered under conditions representative of the way consumers actually use telephones. In blind and double-blind pharmaceutical trials, medicine is administered under conditions representative of the way patients would actually use it. However, when we see blind and double-blind trials applied to stereophonic audio systems, they are consistently used in a manner that is highly unrealistic and detrimental to an accurate stereophonic presentation. Why would a purportedly serious audio evaluation study seek to replicate "Purchase-like" listening conditions? This is not optimal, reasonable, realistic or scientific for a study which purports to be a critical analysis of a listener's ability to discern sound quality differences in stereophonic sound systems. "Purchase-like" listening conditions would be more appropriate for a marketing study. Why not replicate typical "home-like" listening conditions? That is what the scientists did in [9], [10], [11], [12], [13], [14], [23], [24] and [25].

Why would a purportedly serious audio evaluation study only allow very short musical selections? This was actually answered in the paper:

"Subjects also consistently requested shorter evaluation intervals. Sixty seconds seemed too long and some became impatient with 30 seconds. Subjects were paid a small stipend for participation." [22]

Really? Thirty seconds was "too long" to listen to a musical selection during a critical listening session conducted as part of a "scientific study"? I wonder if any of the "audiophiles" preferred the 30 second music snippets. One might also wonder if the participants were just there to collect a check as quickly as possible.

Why would a purportedly serious audio evaluation study use a "small group" listening arrangement? The optimal sound quality of stereophonic audio systems is only found in the stereo sweet spot, which can only be occupied by one person at a time. This is a fundamental design aspect of stereophonic audio systems which must never be violated in critical listening situations unless the intent is to acquire performance data for off-axis (out of the sweet spot) listening.

The extremely short listening intervals (30-60 seconds) and group listening sessions provide some insight into the contributing factors toward different listening impressions from identical pieces of equipment.

Mr. Nousaine provides some statistical analysis of his test results. One of the references in Mr. Nousaine's "Can You Trust Your Ears?" study is "Introduction to Probability and Statistics" (1967) by William Mendenhall. Mr. Nousaine cites Mendenhall's chapter 11. However, chapter 9 provides some important insight:

"The reader will note that we have employed two different statistical tests to test the same hypothesis. Is it not peculiar that the t test, which utilizes more information (the actual sample measurements) than the binomial test, fails to supply sufficient evidence for rejection of the hypothesis u1 = u2?

The explanation of this seeming inconsistency is quite simple. The t test described in Section 9.3 is not the proper statistical test to be used for our example." [Emphasis mine.]

Comparisons Of A/B/X and Blind Test Procedures To Basic Sterephonic Principles

Dr. Harvey Fletcher said that his motivation for inventing sterophonic sound systems (initially called auditory perspective systems) was to provide an exciting home concert experience:

"This symposium describes principles and apparatus involved in the reproduction of music in large halls, the reproduction being of a character that may give even greater emotional thrills to music lovers than those experienced from the original music." [5]

Therefore, high quality stereophonic sound systems are based on the following principles:

1. Realistic and accurate reproduction of the sonic and tactile sensations of the live concert experience,
2. Stable and realistic sonic imaging,
3. Sonic images distributed throughout a three-dimensional sound stage that is analogous to the way real instruments and singers are distributed throughout an actual concert stage,
4. Lifelike instrumental and vocal clarity,
5. Lifelike instrumental and vocal detail,
6. Listener education and training,
7. Emotional thrills.

In the 1981 seminal paper on blind testing for stereophonic systems, Drs. **** and Vanderkooy appear to mock the founding design principles of stereophonic sound systems:

"The last half-dozen years have seen a remarkable proliferation in the number of adjectives used to describe the alleged audible qualities of audio components by the audio press, both above and below ground-words such as depth, air, graininess and liquidity spring to mind. The differences supposedly characterized by these emotive epithets are generally discovered during the course of a listening test, in which the component under test is either heard in isolation, or else is compared with other components of the same type." [17]

Seemingly unknown to Drs. **** and Vanderkooy, much of the basic descriptive vocabulary used by the audio press came directly from peer-reviewed scientific journal papers written by the inventor of stereophonic sound and subsequent researchers in the field. For example, Steinberg and Snow devote considerable discussion to stereophonic depth perception in [9]. T. Somerville in [12] devotes considerable discussion to concert hall and home listening room acoustics. This is the paper in which the term "sound stage" was coined. Somerville discusses reverberation effects (which later came to be called "air" and "ambiance"), sound stage width and depth and "aesthetic presentation". One of the most profound comments made by Somerville was:

"A listener in a concert hall, because of the binaural characteristics of hearing, is able to distinguish between the various sections of the orchestra, and, in particular, he can pick out the solo part. In this, hearing is also assisted by sight." [12] [Emphasis mine.]

DarqueKnight

After over fifty years of scientific brilliance in the field of stereophonic sound, some people came in with "better" ideas:

1. According to **** and Vanderkooy, It is better to blindfold or otherwise visually handicap music listeners. This directly conflicts with the fact that sound image localization is one of the basic principles of stereophonic reproduction. Somerville and other scientists found that hearing (sound localization and sonic depth perception) is assisted by sight. [12, p. 205]

2. According to Clark, the best ear training is gained from listening to pink noise, sine waves, pulses and artificially enhanced distortion. According to Nousaine and other blind-listening test proponents, critical listening sessions are best conducted when music is listened to in 30 to 60 second snippets. This directly conflicts with the fact that spatially correct reproduction of a live concert experience is one of the basic principles of stereophonic reproduction. Gaining evaluative expertise by listening to actual musical performances was the preferred training regimen of the founding scientists working on stereophonic systems. Stereophonic sound systems were invented for music lovers, not test signal lovers.

3. According to Clark, Nousaine, and other A/B/X and blind listening test proponents, sitting off axis and far outside the stereo sweet spot and/or listening in a group environment is a proper method of evaluating stereophonic sound systems. This is absurd and gives the impression that blind listening proponents are desperate to prove a point: That audiophiles are a delusional lot and that the perceived differences in audio components are largely imaginary.

The highly regarded (among blind audio testing proponents) A/B/X test is not scientifically applicable to stereophonic systems. According to the worldwide standard basic textbook on sensory evaluation techniques [26], the A/B/X test, which is usually referred to in the scientific literature by its proper name: duo-trio balanced reference test, is appropriate under the following conditions:

"Duo-Trio Test - Scope and Application

The duo-trio test (ISO 2004a) is statistically less efficient than the triangle test because the chance of obtaining a correct result by guessing is 1 in 2. On the other hand, the test is simple and easily understood.

Use this method when the test objective is to determine whether a sensory difference exists between two samples. This method is particularly useful in situations

1. To determine whether product differences result from a change in ingredients, processing, packaging, or storage.

2. To determine whether an overall difference exists, where no specific attributes can be identified as having been affected.

The duo-trio test has general application whenever more than 15, and preferably more than 30, test subjects are available. As a general rule, the minimum is 16 subjects, but for less than 28, the beta error is high. Discrimination is much improved if 32, 40, or a larger number [of subjects] can be employed."

[Beta error is a statistical error in testing when it is concluded that something is negative when it is actually positive. Beta error is often referred to as "false negative".]

"Two forms of the test exist: the constant reference mode, in which the same sample, usually drawn from regular production, is always the reference, and the balanced reference mode [ABX], in which both of the samples being compared are used at random as the reference.

Use the constant reference mode with trained subjects whenever a product well known to them can be used as the reference.

Use the balanced reference [ABX] mode if both samples are unknown or if untrained subjects are used." [26]

Conclusion

We now have nearly thirty years of documented experience in the application of blind and A/B/X testing to stereophonic audio systems. The A/B/X test setup arrangements and test results have been consistently absurd and consistently statistically similar to guessing. One would think that, after all these years of "all amplifiers sound alike", A/B/X and blind listening test proponents would begin to question the validity of their testing methodology. I don't expect this to ever happen because ridiculing audiophiles is so much fun.

The literature promoting these tests display a profound lack of understanding of how stereophonic systems work and a profound lack of understanding of how the human senses work. Some senses (sight/hearing and taste/smell) are closely interrelated. For a given sensory exercise, more than one sense may be employed. The primary sense for a given stimulus is typically enhanced by the sensory contributions of secondary senses. Compromising one secondary sense can adversely affect the perception of the primary sense. [27] [28] [29] [30] [31] During food consumption, flavor (taste) is the primary stimulus, but the perception of flavor is enhanced by the appearance (sight), aroma (smell), texture (touch), and sound (hearing) of the food. Soggy cereal is every bit as nutritious and tastes the same as crunchy cereal, but most people won't eat a bowl of soggy cereal if the texture is expected to be crunchy. They might perceive it as tasting bad, even though the chemical composition that affects taste receptors is the same. This is an example of an impairment in the sense of touch (secondary sense) affecting the sense of taste (primary sense) during food consumption.

The literal meaning of "stereophonic" is "solid sound" (from the Greek "stereos" for "solid" and "phone" for "sound"). [32] Solid things can be seen and felt. Therefore, "stereophonic" audio systems produce "solid sound" or sound which can be heard, seen and felt. The live concert experience is a sensory exercise which employs the senses of hearing, sight and touch. These same senses are employed in the home stereo experience. For stereophonic listening, the sense of hearing is primary, but the senses of sight and touch play important secondary roles in the creation of the stereophonic illusion of "solid sound". Compromising a person's sense of sight and/or touch, especially in the short term, is stressful and can have detrimental affects on sound localization ability. This stress can exert undue influence on listening evaluation results.

Stereophonic music reproduction is designed according to the principles of sound localization, long term sonic memory of actual musical events, and the reception of tactile sensations from the sound stage. Blind audio testing, which includes visually obscuring all or part of the sound stage, rapid switching of musical selections and off-axis and group seating, impairs the listener's ability to localize sounds (seeing), to internalize and evaluate aural cues (hearing) and to receive correct stereophonic tactile information (touching). Any stereophonic audio system testing methodology which compromises and hinders the processes of human sensory perception is useless.

ABX and blind testing proponents say that they want to apply a scientifically rigorous testing methodology to stereophonic audio in order to determine if the claimed differences in audio components actually exist. However, they ignore decades of scientifically and mathematically rigorous subjective listening techniques that were developed by the inventor and subsequent researchers in the field of stereophonic sound.

DarqueKnight

References

[1] Snow, W. B., "Audible Frequency Ranges of Music, Speech and Noise", Journal of the Acoustical Society of America, Vol. 3, Issue 1A, July 1931, pp. 155-166.
[2] Farhid, M. and Tinati, M.A., "Robust voice conversion systems using MFDWC", Proceedings of the 2008 IEEE International Symposium on Telecommunications, Tehran, Iran, August 2009, pp. 778-781.
[3] Kun, Liu Jianping and Zhang, Yonghong Yan, "High Quality Voice Conversion through Phoneme-Based Linear Mapping Functions with STRAIGHT for Mandarin", Proceedings of the IEEE Fourth International Conference on Fuzzy Systems and Knowledge Discovery, Haikou, China, August 2007, pp. 410-414.
Chinese Acad. of Sci., Beijing
[4] Cheng-Yuan Lin and Jang, J.-S.R. "New Refinement Schemes for Voice Conversion", Proceedings of the IEEE 2003 International Conference on Multimedia and Expo, Baltimore, MD, July 2003, pp. 25-728.
[5] Fletcher, Harvey, "Symposium on Wire Transmission of Symphonic Music and Its Reproduction in Auditory Perspective-Basic Requirements", Bell System Technical Journal, Vol. 13, 1934, pp. 239-244.
[6] Fletcher, Harvey, "Hearing, The Determining Factor for High-Fidelity Transmission", Proceedings of the I.R.E., Columbus, OH, June 1942, pp. 266-277.
[7] Hilliard, John K., "The History of Stereophonic Sound Reproduction", Proceedings of the Institute of Radio Engineers, Vol. 50, No. 5, May 1962, pp. 776-780.
[8] Fletcher, Harvey, “Hearing Aids and Deafness”, Bell Laboratories Record, Vol. 5, No. 2, October 1927, p. 33.
[9] Steinberg, J. C., and Snow, W. B., "Symposium on Wire Transmission of Symphonic Music and Its Reproduction in Auditory Perspective-Physical Factors", Bell System Technical Journal, Vol. 13, 1934, pp. 245-258.
[10] Harvey, F. K. and Schroeder, M. R., "Subjective Evaluation of Factors Affecting Two-Channel Stereophony", Journal of The Audio Engineering Society, Vol. 9, No. 1, January 1961, pp. 19-28.
[11] Moir, J. and Leslie, J. A., "The Stereophonic Reproduction of Speech and Music", Journal of the British Institution of Radio Engineers, London, September 1951, pp. 360-366.
[12] Somerville, T., "Survey of Stereophony", Proceedings of the Institution of Electrical Engineers, Convention on Stereophonic Sound Recording, Reproduction and Broadcasting, London, March 1959, pp. 201-208.
[13] Moore, H. B., "Listener Ratings of Stereophonic Systems", Institute of Radio Engineers Transactions on Audio, September-October 1960.
[14] Beaubein, W. H. and Moore, H. B., "Perception of Stereophonic Effect as a Function of Frequency", Journal of the Audio Engineering Society, Vol. 8, No. 2, April 1960, pp. 76-86.
[15] Boley, Jon and Lester, Michael, "Statistical Analysis of ABX Results Using Signal Detection Theory", Proceedings of the 127th Convention of the Audio Engineering Society, October 2009, New York, NY.
[16] Winer, Ethan, "Audio Myths, Artifact Audibility and Comb Filtering" workshop presented at the 127th Convention of the Audio Engineering Society in October 2009, New York, NY. http://www.youtube.com/watch?v=BYTlN6wjcvQ, YouTube video accessed 8/3/2010.
[17] ****, S. P. and Vanderkooy, J., "The Great Debate: Subjective Evaluation", Journal of the Audio Engineering Society, Vol. 29, No. 7/8, July/August 1981, pp. 482-491.
[18] Clark, D., "High-Resolution Subjective Testing Using a Double-Blind Comparator", Journal of the Audio Engineering Society, Vol. 30, No. 5, May 1982, pp. 330-338.
[19] Clark, David, "Ten Years Of A/B/X Testing", 91st Audio Engineering Society Convention, New York, NY, October 1991.
[20] ****, S., "The Great Debate-Some Reflections Ten Years Later", Audio Engineering Society 8th International Conference, Washington, D.C., May 1990, pp. 121-123.
[21] Nousaine, Tom, "The Great Debate: Is Anyone Winning?", Audio Engineering Society 8th International Conference, Washington, D.C., May 1990, pp. 117-119.
[22] Nousaine, Tom, "Can You Trust Your Ears?", 91st Audio Engineering Society Convention, New York, NY, October 1991.
[23] Schjonneberg, K. and Olson, F., "Listening Test Methods and Evaluation", Journal of the Audio Engineering Society, Vol. 9, No. 1, January 1961, pp. 29-36.
[24] Olsen, Harry F., "Stereophonic Sound Reproduction in the Home", Journal of the Audio Engineering Society, Vol. 6, No. 2, April 1958, pp. 80-90.
[25] Snow, W. B., "Auditory Perspective", Bell Laboratories Technical Journal, Vol. 12, No. 7, March 1934, pp. 194-198.
[26] Meilgaard, Morten, Civille, Gail Vance, Carr, B. Thomas, "Sensory Evaluation Techniques", 4th Ed., CRC Press, Boca Raton, FL, 2007, pp. 72-73.
[27] Hull, J., "Touching the Rock: An Experience of Blindness", Vintage Publishing, London, 1992.
[28] Montagu, A., "Touching: The Human Significance of the Skin", Columbia University Press, New York, London, 1971.
[29] Wright, D., "Deafness: A Personal Account", Faber, London, 1990.
[30] Yuan, Yi-Fu, "Sight and Picture", Geographical Review, Vol. 69, pp. 413-422, 1979.
[31] Rodaway, Paul, "Sensuous Geographies: Body, Sense and Place", Routledge, London, New York, 1994.
[32] World Book Dictionary, 1973 ed. Vol. 2, p. 2033.

hearingimpared

Holy research Bat Man!

LessisNevermore

Very well done, and amen!

quadzilla

Very well done indeed. But for me this is not a matter of faith, therefore, no amen. I make a change in my set up and sometimes it's better, sometimes worse, and sometimes I don't hear any difference.

hearingimpared

I just read this through again and all I can say is thank you DK for doing the research and providing us with your insight.

DarqueKnight

This is the sort of thing I like to do while waiting to change planes.;)

wayne3burk

i got through about 3/4s of the first page...

obviously i suffer from lack of in depth understanding or Adult onset ADD.

either way....


I think if you were to do double blind testing on simple or complex stereo systems.... you would only be able to deal with one variable at a time. -- the more complex the system, the more variables.

speakers for example come to mind. The most obvious difference when doing double blind testing with speakers is the DB efficiency comes to mind. So if you wanted to test two pairs of speakers with different dB efficiency ratings you'd have to have to identical amps set up and the volume adjusted accordingly so that your results are not biased towards the speaker pair that was audibly louder.

conversely if you're testing preamps all other factors would have to be identical, tone controls set to flat, same speakers, amp, input source, etc.

and on and on and on -- then try to do double blind testing of HTRs - or AV preamps, with all their diverse parameters of features. It begins to be more of a feature set / cost benefit / affordability comparison.

I think if you enjoy your music and can listen for hours without ear fatigue then you're system is just fine - whether your double blind or not.

j allen

I've got to say, DarqueKnight, I consistently look forward to your posts. They're always well researched, thought out, and informative. Keep it up!

DarqueKnight

Dr. Stanley P. **** (Ph.D. in Mathematics) is a professor at the University of Waterloo, Waterloo, Ontario, Canada. He holds joint appointments in the departments of Applied Math and Physics and is a member of the Audio Research Group at the University of Waterloo. He is a fellow (highest member grade) of the Audio Engineering Society and is currently an associate technical editor of the Audio Engineering Society Journal. Dr. **** was awarded the AES Silver Medal in 1993 and the AES Publication Award in 1994. His paper entitled "The Great Debate: Subjective Evaluation" (1981), which introduced the concept of the application of ABX and blind testing to stereophonic audio, was deemed a significant contribution to the field of audio engineering and he was elected to "AES Fellow" as a result.

Dr. John Vanderkooy (Ph.D. in Physics) is a retired (distinguished professor emeritus) professor of Physics and adjunct professor at the University of Waterloo. He is a member of the Audio Research Group at the University of Waterloo. Dr. Vanderkooy is currently the editor of the Audio Engineering Society Journal. Dr. Vanderkooy was awarded the AES Silver Medal in 1993 and the AES Publication Awards in 1992 and 1994. He is a co-author of "The Great Debate: Subjective Evaluation" (1981).

Mr. David L. Clark (B.S. in Electrical Engineering) is the owner and president of DLC Design, an independent Audio R&D, Test and Evaluation center with over 25 years serving the automotive audio industry. He is a Fellow of the Audio Engineering Society and he is Chair of the Technical Committee for Loudspeakers and Headphones. Mr. Clark was elected to AES Fellow status in 1985 and was awarded the AES Board of Governors Award in 1991. He is the author of one of two seminal papers on the application of ABX double-blind testing to stereophonic audio: "High-Resolution Subjective Testing Using a Double-Blind Comparator" (1982).

Dr. Harvey Fletcher (Ph.D. in Physics) (d. 7-23-1981, age 96) was a former director of research at Bell Telephone Laboratories, where he invented, among other things, the electronic hearing aid, the electronic audiometer and stereophonic two channel home audio systems. He was also the first dean of the Brigham Young University College of Engineering. Dr. Fletcher was awarded honorary membership in the Audio Engineering Society in 1952 and received the AES's highest award, the Gold Medal, in 1958. He was also an honorary member of the American Ontological Society. Dr. Fletcher made some important contributions to cinematic sound and was given the Progress Medal Award by the Society of Motion Picture and Television Engineers in 1949.

Dr. Fletcher's was a founder, fellow and first president of the American Acoustical Society. His other professional society memberships included the Institute of Electrical and Electronics Engineers (I.E.E.E.), the American Society for Hard of Hearing (president), the American Physical Society (president), the American Association for the Advancement of Science (vice-president), the National Academy of Sciences and the National Hearing Division Committee of Medical Sciences.

In a paper authored by Dr. Fletcher and published in Physics Today magazine after his death, he revealed that he was a major collaborator with his doctoral adviser, Dr. Robert Millikan, on the oil drop experiment that accurately measured the charge of an electron. In an agreement that both men took to their graves, Dr. Millikan took sole credit for the oil drop experiment and Dr. Fletcher took sole credit for less significant, but related work. The oil drop experiment was a significant contributing factor to Dr. Millikan being awarded the 1923 Nobel prize in physics. Dr. Fletcher said that he had no hard feelings toward Dr. Millikan for the incident:

"People have frequently asked me if I had bad feelings toward Millikan for not letting me be a joint author with him on this first paper which really led to his getting the Nobel prize. My answer has always been no. It is obvious that I was disappointed on that first paper as I had done considerable work on it and had expected to be a joint author. But Professor Millikan was very good to me while I was at Chicago. It was through his influence that I got into the graduate school. He also found remunerative jobs for me to defray all my personal and school expenses for the last two years. Above this was the friendship created by working intimately together for more than two years. This lasted throughout our lifetime. Remember when we worked together he was not the famous Millikan that he later became. When he wrote his memoirs shortly before he died, he had probably forgotten some of these early experiences."

______________________________________


Audio Engineering Society Award Designations:

* The AES Gold Medal Award (formerly The John H. Potts Award) is given in recognition of outstanding achievements, sustained over a period of years, in the field of Audio Engineering.

* The Silver Medal Award (formerly the Emile Berliner Award) is given in recognition of outstanding development or achievement in the field of audio engineering.

* The Bronze Medal Award (formerly the AES Award) is given annually to a person who has helped significantly in the advancement of the Society.

* The Distinguished Service Medal Award is given in recognition of extraordinary service to the Society over a period of years.

* The Publications Award is given each year, alternately, to an author under 35 years of age and to an author regardless of age for the outstanding paper published in the Journal of the Audio Engineering Society during the previous two years.

* An Honorary Member is a person of outstanding repute and eminence in the science of audio engineering or its allied arts.

* Citations are given in recognition of services or accomplishments that do not fit into any other category.

* The Board of Governors Award is given for outstanding contributions to the Audio Engineering Society.

* The Fellowship Award is given to a member who had rendered conspicuous service or is recognized to have made a valuable contribution to the advancement in or dissemination of knowledge of audio engineering or in the promotion of its application in practice.

______________________________________________

References

[1] AES Awards Page: http://www.aes.org/awards/
[2] AES Journal Masthead Page: http://www.aes.org/journal/masthead.cfm
[3] University of Waterloo Website: http://www.science.uwaterloo.ca
[4] DLC Design Website: http://www.dlcdesignaudio.com
[5] Fletcher, Harvey, "My Work with Millikan on the Oil-drop Experiment", Physics Today, June 1982, pp. 43-47.
[6] In Memory Of Harvey Fletcher Webpage: http://www.et.byu.edu/~tom/family/Harvey_Fletcher/harvey_fletcher.html
[7] Father of Stereophonic Sound Webpage: http://www.byhigh.org/History/Fletcher/DrHarvey.html

hearingimpared

DarqueKnight wrote: »

This is the sort of thing I like to do while waiting to change planes.;)

:eek::eek::eek::eek: It would take me a month to do what you did in probably a couple of hours!:o

285exp

There appear to be no lengths to which some of you will not go to justify your conceit that you can indeed see the Emperors New Clothes.

Face

And there will always be Luddites.

hearingimpared

285exp wrote: »

There appear to be no lengths to which some of you will not go to justify your conceit that you can indeed see the Emperors New Clothes.

Conceit!?! Jackass remark. What has conceit have to do with listening to gear. Hit the road!

vc69

Click the pic


Thanks for the article DK.

hearingimpared

Hey Kev, that clown has been around for 6 years and I'm willing to bet 75% of his 43 posts are for trolling.

DarqueKnight

285exp wrote: »

There appear to be no lengths to which some of you will not go to justify your conceit that you can indeed see the Emperors New Clothes.

Ouch! That's gonna leave a mark.:p

Seriously, I think conceit is thinking that a proposed stereophonic testing method is "better", although it goes contrary to the design of stereo systems, and when said method is in no way similar to, or an improvement of, the test and evaluation methods developed by the geniuses who designed home stereo systems.

quadzilla

hearingimpared wrote: »

Hey Kev, that clown has been around for 6 years and I'm willing to bet 75% of his 43 posts are for trolling.

A quick check of his posting history seems to indicate it's closer to 100%. And generally, all his argumentation is from the position of personal incredulity or else calls for "scientific testing" to generate valid data, except for those that are flat out flames. I think it's safe to assume he's firmly in the AB/ABX testing is the only valid testing methodology camp. Just put him on ignore and call it a work around for noise in the system.

DarqueKnight

I can't imagine why any stereo hobbiest would not be interested in a comparative analysis of stereophonic testing methods. We would certainly want to go all the way back to the "mother church" of stereophonic music systems, Bell Telephone Laboratories, and read the writings of the inventor and early developers.

The stereophonic science papers of Bell Laboratories scientists, such as Dr. Harvey Fletcher, Mr. W. B. Snow, Mr. J. C. Steinberg, Dr. M. R. Schroeder, and Mr. F. K. Harvey would seem to be required reading for anyone wishing to do a serious study in the development of alternative evaluation methods in stereophonic sound systems.

Recently, I have began to look into some of the early papers on the application of psychoacoustics to home stereo systems. Two interesting and informative papers I have just read are:

1. "What is Psychoacoustics", by Dr. Gerard G. Harris of Bell Telephone Laboratories. This paper was published in the The Journal of the Audio Engineering Society, Vol. 9, No. 1, January 1961, pp. 2-6, with abstract as follows:

"What we can hear depends first of all on what the ear hears. It also depends on what the brain does with what the ear hears. On the one hand, there has been considerable study of the mechanical-electrical operation of the ear. Generally speaking, the ear acts as a complex time frequency analyzer. On the other hand, there are extensive investigations as to what one hears subjectively; that is, the psychological aspects of hearing. Psychoacoustics includes and attempts to relate these studies. Why high pitch sounds are more easily masked by low pitch sounds than vice versa and how we can pick out a sound in space are discussed in order to illustrate the psychoacoustical viewpoint."

2. "Psychoacoustics Applied to Stereophonic Reproduction Systems", by Dr. Peter C. Goldmark and Mr. John M. Hollywood of CBS Laboratories. This paper was published in the The Journal of the Audio Engineering Society, Vol. 7, No. 2, April 1959, pp. 72-74, with abstract as follows:

"In an attempt to develop less cumbersome stereo speaker systems for home use, subjective listening tests were undertaken. A good two-speaker system system was used as reference. Using crossover networks and delaying signals to speaker reduced the "precedence effect" and the listener's ability to assign particular frequencies to a particular speaker. A simple application of the principles to home use is illustrated."

At the time that Dr. Goldmark co-authored the paper above, he was President of CBS Laboratories. The long-playing record was developed by the research team lead by Dr. Goldmark at CBS Laboratories as well as the first high fidelity compact phonograph, the Colombia 360. Dr. Goldmark was a fellow of the American Institute of Electrical Engineers (now the I.E.E.E.), the Institute of Radio Engineers, the Society of Television and Motion Picture Engineers, and the British Television Society. In 1945 Dr. Goldmark was awarded a medal by the Television Broadcasters Association for his color television pioneering work.

Home stereo systems, and the subjective listening tests for them, were developed by scientists of the highest caliber. The peer-reviewed scientific literature on subjective listening test methods for home stereo is extensive and spans decades up to the present time.

Ironically, Fifty-four years after the invention of the home stereo, some other scientists came in proposing blindfolds, curtains, ABX switchboxes and off-axis seating as tools to facilitate "scientific" testing of home stereo systems. Indeed, one such scientist, who was looking to sell some ABX switchboxes, wrote:

"Listening tests used to evaluate audio equipment can seldom be considered scientific tests."*

Wow.

*Clark, D., "High-Resolution Subjective Testing Using a Double-Blind Comparator", Journal of the Audio Engineering Society, Vol. 30, No. 5, May 1982, pp. 330-338.

audiobilly

DK, I think I am in love. Are you Married?

Of course, just kidding. For those who chose to take my lose wording so seriously.

Your inputs, on all fronts, are greatly appreciated. PLEASE do not stop. I love it!! A TRUE learning experience.

One can merely speculate, without true knowledge and true experience.

Thank you Knight of the Darque-ness!

hearingimpared

Raife, so why did it go awry? If these highly acclaimed scientists who pretty much invented Stereophonic and the LP had the subjective scientific method of discerning the difference in a piece of music on different equipment, who & why did these people start with the blindfold and ABX method?

I'm going to take a guess and say that it all started in the '60s/'70s when preamp and amplifiers were taken to the next level to reproduce music as well as the realization that ICs & speaker wire made a big difference in how the music sounded with various gear. The Ohm's law ONLY advocates then set out to disprove that there is a difference in how gear affects the sound of music.

quadzilla

Or it's simply a group that simply over-analyzed the issue and came to the conclusion that wire makes no difference. I've seen numerous assertions that, for example, skin effect makes no difference at audio frequencies because it's primarily an issue at the MHz level and that the capacitance and impedance of a cable doesn't factor because the distances are too short to make any difference.

I can relate to this since that's actually the position I came from. I used rat shack speaker wire and ICs for a very long time and always thought they were fine. This lasted until I needed to get some new wire and wanted a larger gauges due to the fact that some of my runs were going to be in the > 35 ft range. Picked up the cheapest Monster speaker cable I could find and did notice a difference over the rat shack stuff, even on the shorter runs that were the same length of the rat shack cable I'd been using. Wasn't expecting it. Wasn't opposed to hearing a difference. Just didn't expect to. That's when I realized that cables can make a difference.

I think in the case of at least some others, they set out to "prove" that cables don't make a difference, and so they design their tests and select the data that backs up that assertion, while calling everything else "snake oil" as though that were a magic incantation that makes anything that contradicts that viewpoint simply disappear. Granted, there is a lot of genuine snake oil out there (and wouldn't that be a great name for a line of cables and gear?), but just because some make false claims does not, ad infinitum, prove that all other claims are equally false. This is argumentation from the position of false equivalence. That, combined with an AB/ABX testing methodology which seems designed to produce failure, and you get what amounts to a peanut gallery heckling those that have started to get a clue that just because we can't explain and/or measure something right now, doesn't mean that we never will be able to.

cheif pontiac

I'm lost. I knew I should have received a PHD in something. Where is the remote control and beer? LOL. If only I could have made it through Calc 1 after 3 tries. Interesting stuff. Maybe now that I have an interest in stereo I will see if I can wrap my mind around some technical stuff. I do know I like it when In turn my volume knob up.

DarqueKnight

hearingimpared wrote: »

...so why did it go awry?

For music lovers, for whom home stereo was invented, nothing went awry. Things have gotten progressively better with regard to recording and playback equipment and with analog and digital recording media (although many would say digital playback was a big step backward ).

For the objectivists, things did go awry. I would refer you to the 10-year follow-up papers published by the founders of the ABX religion. For the vast majority of experienced stereophonic music lovers, blind testing is a non-issue because they know what they hear.

hearingimpared wrote: »

If these highly acclaimed scientists who pretty much invented Stereophonic and the LP had the subjective scientific method of discerning the difference in a piece of music on different equipment, who & why did these people start with the blindfold and ABX method?

Objectivist naysayers are not unique to audio. Such fringe elements are common in technology based hobbies. In photography, there is a very vocal objectivist fringe who, rather than talking about the pictures they take and offering experience-based advice for taking better pictures, they do nothing but look for flaws in pro-level Nikon and Canon cameras and lenses in an effort to "prove" that such expensive, high-end cameras and lenses are wastes of money and that "consumer grade" lenses from Tokina, Tamron, Sigma, etc. are every bit as good. They seem to (think that they) know more than the thousands of professional photographers over many decades who have built their careers and reputations using professional Nikon and Canon cameras and lenses.

There are various reasons why people start or join naysayer audio cults. Just a few reasons are:

1. They tried something and it didn't work, therefore they stereotype every type of higher performance audio gear as snake oil.

2. They really have a poor understanding of how stereo works.

3. They have a poor understanding of human sensory perception.

4. Audiophile culture is poorly understood by the masses and audiophiles make convenient targets for ridicule.

5. They want to be known as an "expert" in something. Especially on the Internet where credentials can easily be made up.

6. They learn one or a few basic electronics and/or acoustics rules and think that everything in the field can be characterized by those few simple rules.

7. They love to argue, particularly on the Internet where anonymity greatly diminishes the risks of personal confrontation.

8. Some realize they are spewing garbage, but the risk of getting "caught" is minimal so they do it anyway..because ridiculing audiophiles is so much fun.

9. They are envious.

10. They are jealous.

11. They are cheap.

12. Their wives/girlfriends/significant others won't allow them to have a nice stereo.

....and so on and so forth.

quadzilla

On a somewhat tangential note, this guy wants you to know that unless you embrace only the latest technology in audio gear while eschewing all previous generations of gear, then you must be a snob who wants audiophilia to die.

http://cts.vresp.com/c/?HomeTheaterReview.co/8bed0b7119/fdf83ca92b/6a6a0d4d4f

There's also a poke in there about how all formats really sound the same anyway, except that digital is always better.

hearingimpared

quadzilla wrote: »

On a somewhat tangential note, this guy wants you to know that unless you embrace only the latest technology in audio gear while eschewing all previous generations of gear, then you must be a snob who wants audiophilia to die.

http://cts.vresp.com/c/?HomeTheaterReview.co/8bed0b7119/fdf83ca92b/6a6a0d4d4f

There's also a poke in there about how all formats really sound the same anyway, except that digital is always better.

Allow me to translate - vinyl is and remains dead, even if Millenials like to retro-shop at Amoeba Records for old, used records. Used records do not an HD format make or are a meaningful business, nor are Millennial teeny-boppers buying Goldmund turntables and fancy phono stages for their iPod-based playback systems.

b. Cancel your subscriptions to publications that preach retro technologies and ancient ways of doing things. It's **** to suggest that vinyl is booming, but there are print mags in the audiophile world that say it's true.

IF this guy had any credibility at all it went down the toilet with these statements.

Used records do not an HD format make or are a meaningful business, nor are Millennial teeny-boppers buying Goldmund turntables and fancy phono stages for their iPod-based playback systems. The new blood must be embraced, but they also need to be sold to with products that they understand, and that means something a little more high definition than a crappy old LP or a 30 year old standard definition Compact Disc.

UGH! I have some LPs that are more than 30 years old that put new HD releases to shame.

My reaction to this guys article is that I smell BS!

DarqueKnight

hearingimpared wrote: »

IF this guy had any credibility at all it went down the toilet with these statements.

His credibility went down the toilet when he made this statement:

"Personally, I don't wish it [audiophilia] on anybody because audiophiles get off more on audio gear than the music itself, but the 'philes for nearly a generation have built a cottage industry around high-dollar, esoteric audio equipment designed to take the listener as close as they can get to the master tape as physically possible."

He obviously does not know the difference between audio snobs, audio fools and audiophiles.

He obviously he is just another Internet "expert" whose expertise is based on parroting what he read from other Internet "experts" rather than conducting actual, careful research (and gaining personal experience) in order to achieve a true understanding.

He obviously does not understand that media type (digital or analog) does not guarantee a great recording. Each media type has their strengths and weaknesses and there exists trash and treasure on both.

He obviously has never listened to a properly setup high end vinyl rig. Too bad for him. I hate all the ritualistic rigmarole associated with vinyl, but the sound I get more than compensates me for the inconvenience.:)

So, what else is new among audio "experts"...on the Internet.:D

hearingimpared

DarqueKnight wrote: »

His credibility went down the toilet when he made this statement:



He obviously does not know the difference between audio snobs, audio fools and audiophiles.

He obviously he is just another Internet "expert" whose expertise is based on parroting what he read from other Internet "experts" rather than conducting actual, careful research (and gaining personal experience) in order to achieve a true understanding.

He obviously does not understand that media type (digital or analog) does not guarantee a great recording. Each media type has their strengths and weaknesses and there exists trash and treasure on both.

He obviously has never listened to a properly setup high end vinyl rig. Too bad for him. I hate all the ritualistic rigmarole associated with vinyl, but the sound I get more than compensates me for the inconvenience.:)

So, what else is new among audio "experts"...on the Internet.:D

Everything you said, I agree with and guess what the bolded part, I use to love the ritualistic rigamarole associated with vinyl but since I've gotten older especially these past two years, it's become old hat but like you said the results in sound more than compensate me for the aversion to the ritual.

arny

DarqueKnight wrote: »

ABX and blind testing proponents say that they want to apply a scientifically rigorous testing methodology to stereophonic audio in order to determine if the claimed differences in audio components actually exist. However, they ignore decades of scientifically and mathematically rigorous subjective listening techniques that were developed by the inventor and subsequent researchers in the field of stereophonic sound.

The conclusion that "They (meaning DBT advocates) ignore decades of scientifically and mathematically rigorous subjective listening techniques that were developed by the inventor and subsequent researchers in the field of stereophonic sound" is not warranted by the evidence presented above, and the evidence presented above is woefully incomplete and even contradicts the author's conclusion.

I think that I will just back up a few lines above the above summary to show where this paper goes awry:

"According to **** and Vanderkooy, It is better to blindfold or otherwise visually handicap music listeners."

Nonsense. A fabrication. I have known **** and Vanderkooy personally for decades, have sat with them in DBTs whose results they subsequently published in the JAES, and they think no such thing, and engage in no such practices.

"According to Clark, the best ear training is gained from listening to pink noise, sine waves, pulses and artificially enhanced distortion."

Again nonsense and a complete fabrication. Clark's (Clark is a decades-long personal friend, one time business partner, AES Fellow and past AES national officer, a highly -respected and internationally known audio expert) primary tool for listener training is a collection of music from regular commercial soruces, much highly appreciated by other listening test advocates both sighted and blind, that is known as the LTT. Those who are intersted in more details can start here: http://www.dlcdesignaudio.com/

Unfortunately, that's just the tip of the iceberg when it comes to the misinformation and grotesque errors that fill the previous several posts by DarqueKnight, AKA RAIFE F. SMITH II .