Further Thoughts On ABX Testing Of Stereophonic Audio Systems
DarqueKnight
Posts: 6,765
Introduction
Recently, some members of the audio engineering community wondered if the sensory evaluation techniques used in the food and beverage industry have some application to audio. The attached flyer from the Audio Engineering Society's 38th International Conference on Sound Quality Evaluation (June 2010) states:
"One can ask whether methods used widely in other areas of sensory evaluation, such as food and beverage, are applicable to audio engineering, or whether we can learn from the work going on in picture quality analysis."
Further Study On ABX Theory
While doing further study on the ABX test, I came upon a textbook written by Dr. Harry T. Lawless, a professor of food science at Cornell University and Dr. Hildegarde Heymann, a professor of viticulture and enology at the University of California at Davis. Dr. Lawless is also the co-editor of the Journal of Sensory Studies.
In section 4.2.7, entitled "The ABX Discrimination Task", of Lawless and Heymann's textbook, "Sensory Evaluation of Foods: Principles and Practices" (2nd edition), they write:
"This test has been widely used as a forced choice measure of discrimination in psychological studies, for example, in discrimination of speech sounds and in measuring auditory thresholds (Macmillan et al., 1977; Pierce and Gilbert, 1958). Several signal detection models (see Chapter 5) are available to predict performance using this test (Macmillan and Creelman, 1991). The method has been rarely if ever applied to food testing, although some sensory scientists have been aware of it (Frijters et al., 1980). Huang and Lawless (1998) did not see any advantages to the use of this test over more standard discrimination tests." (p. 88)
One sentence of the passage above bears further investigation:
"The method has been rarely if ever applied to food testing, although some sensory scientists have been aware of it (Frijters et al., 1980)."
In the same section, Lawless and Heymann address the question of why the ABX test has rarely been used in food testing:
"As in other general tests of overall difference (triangle, duotrio) the nature of the difference is not specified and this presents a challenge to the panelists to discover relevant dimensions of sensory difference and not be swayed by apparent but random differences. As foods are multi-dimensional, random variation in irrelevant dimensions can act as a false signal to the panelists and draw their attention to sensory features that are not consistent sources of difference (Ennis and Mullen, 1986)."
One sentence of the passage above bears further investigation:
"As foods are multi-dimensional, random variation in irrelevant dimensions can act as a false signal to the panelists and draw their attention to sensory features that are not consistent sources of difference (Ennis and Mullen, 1986)."
Certainly, like foods, a stereophonic sound stage is multi-dimensional and we could substitute the words "stereophonic sound stages" into the passage above with no loss in accuracy:
"As stereophonic sound stages are multi-dimensional, random variation in irrelevant dimensions can act as a false signal to the panelists and draw their attention to sensory features that are not consistent sources of difference (Ennis and Mullen, 1986)."
Relevant sensory dimensions in stereophonic sound are things such as clarity, detail, image location, physical proportions of the sound stage and tactile sensation. Irrelevant sensory dimensions are things such as minor variations in sound level and the physical appearance of equipment.
One may ask, since physical appearance of equipment is an irrelevant sensory dimension, should not the appearance, and by extension, the cost and brand name, of equipment be hidden from subjects? No. People can train themselves to ignore appearance, brand name and cost when evaluating the sound performance of audio equipment. If this were not true, reviewers in the audio press would always pick the most attractive and costly audio equipment from the most prestigious manufacturers. Also, if this were not true, the term "giant killer" would not exist. Some reviewers, and many audiophiles, seem to relish the finding of a modestly priced, "Plain Jane" piece of gear whose performance equals or exceeds alternatives which are much better looking and whose prices are multiples of the Plain Jane gear. Certainly, snobbery exists in audio as it does in every field of human endeavor, but we should not project the failings of some to the entire audio enthusiast community. In my case, I have previously stated on this forum that I do not like silver-faced electronics and oak veneer. Yet, my two channel system is composed entirely of silver-faced amplification components (which are discreetly hidden behind a black metal mesh equipment cabinet door
). My loudspeakers are finished in loathsome oak veneer.
An appropriate question would be whether the ABX test used by food scientists is the same ABX test used by audio scientists. The answer is yes. Although the same term can have different meanings in different scientific disciplines, the attached table from Lawless and Heymann's "Sensory Evaluation of Food" textbook (table 4.1, page 81) shows that the ABX test they discuss is the same ABX test used in monophonic and stereophonic audio evaluations.
Conclusion
According to Lawless and Heymann, food scientists "rarely, if ever" use the ABX test in food testing because it is not a good tool for the evaluation of multi-dimensional stimuli.
In spite of this, some audio scientists enthusiastically believe the ABX test to be the "gold standard" for stereophonic system evaluation. One reason for this may be the fact that, as Lawless and Heymann noted, the ABX test is widely used in tests of discrimination of speech sounds and in measuring auditory thresholds. As I noted in my study of stereophonic blind testing, the ABX test is also widely used in the telecommunications industry used for voice circuit quality evaluation.
Some audio scientists may have thought that, since the ABX test was successfully used in evaluating other types of audio, it would be appropriate to use it for stereophonic audio evaluation. It may also be that some audio scientists discovered a reasonable explanation as to why the multi-dimensional nature of stereophonic audio makes it a good candidate for ABX application but the multi-dimensional nature of food makes it a poor candidate. There is much empirical evidence to support the former. I am still searching for the latter.
It is certainly clear that some audio scientists believe that stereophonic audio is exempt from the following two rules:
1. A thing must be tested in a manner accurately representative of the way it will be used.
2. Sensory evaluations must be conducted in a manner that does not prevent or diminish relevant sensory stimuli from reaching the subject.
References
1. Lawless, H. T. and Heymann H., "Sensory Evaluation of Food: Principles and Practices", pp. 79-100, 2010.
2. Ennis, D. M. and Mullen, K., "Theoretical Aspects of Sensory Discrimination", Chemical Senses, Vol. 11, No. 4, pp. 513522, 1986.
3. Link: A Historical Overview of Stereophonic Blind Testing
4. Link: A Survey Of Early Stereophonic System Subjective Evaluation
Recently, some members of the audio engineering community wondered if the sensory evaluation techniques used in the food and beverage industry have some application to audio. The attached flyer from the Audio Engineering Society's 38th International Conference on Sound Quality Evaluation (June 2010) states:
"One can ask whether methods used widely in other areas of sensory evaluation, such as food and beverage, are applicable to audio engineering, or whether we can learn from the work going on in picture quality analysis."
Further Study On ABX Theory
While doing further study on the ABX test, I came upon a textbook written by Dr. Harry T. Lawless, a professor of food science at Cornell University and Dr. Hildegarde Heymann, a professor of viticulture and enology at the University of California at Davis. Dr. Lawless is also the co-editor of the Journal of Sensory Studies.
In section 4.2.7, entitled "The ABX Discrimination Task", of Lawless and Heymann's textbook, "Sensory Evaluation of Foods: Principles and Practices" (2nd edition), they write:
"This test has been widely used as a forced choice measure of discrimination in psychological studies, for example, in discrimination of speech sounds and in measuring auditory thresholds (Macmillan et al., 1977; Pierce and Gilbert, 1958). Several signal detection models (see Chapter 5) are available to predict performance using this test (Macmillan and Creelman, 1991). The method has been rarely if ever applied to food testing, although some sensory scientists have been aware of it (Frijters et al., 1980). Huang and Lawless (1998) did not see any advantages to the use of this test over more standard discrimination tests." (p. 88)
One sentence of the passage above bears further investigation:
"The method has been rarely if ever applied to food testing, although some sensory scientists have been aware of it (Frijters et al., 1980)."
In the same section, Lawless and Heymann address the question of why the ABX test has rarely been used in food testing:
"As in other general tests of overall difference (triangle, duotrio) the nature of the difference is not specified and this presents a challenge to the panelists to discover relevant dimensions of sensory difference and not be swayed by apparent but random differences. As foods are multi-dimensional, random variation in irrelevant dimensions can act as a false signal to the panelists and draw their attention to sensory features that are not consistent sources of difference (Ennis and Mullen, 1986)."
One sentence of the passage above bears further investigation:
"As foods are multi-dimensional, random variation in irrelevant dimensions can act as a false signal to the panelists and draw their attention to sensory features that are not consistent sources of difference (Ennis and Mullen, 1986)."
Certainly, like foods, a stereophonic sound stage is multi-dimensional and we could substitute the words "stereophonic sound stages" into the passage above with no loss in accuracy:
"As stereophonic sound stages are multi-dimensional, random variation in irrelevant dimensions can act as a false signal to the panelists and draw their attention to sensory features that are not consistent sources of difference (Ennis and Mullen, 1986)."
Relevant sensory dimensions in stereophonic sound are things such as clarity, detail, image location, physical proportions of the sound stage and tactile sensation. Irrelevant sensory dimensions are things such as minor variations in sound level and the physical appearance of equipment.
One may ask, since physical appearance of equipment is an irrelevant sensory dimension, should not the appearance, and by extension, the cost and brand name, of equipment be hidden from subjects? No. People can train themselves to ignore appearance, brand name and cost when evaluating the sound performance of audio equipment. If this were not true, reviewers in the audio press would always pick the most attractive and costly audio equipment from the most prestigious manufacturers. Also, if this were not true, the term "giant killer" would not exist. Some reviewers, and many audiophiles, seem to relish the finding of a modestly priced, "Plain Jane" piece of gear whose performance equals or exceeds alternatives which are much better looking and whose prices are multiples of the Plain Jane gear. Certainly, snobbery exists in audio as it does in every field of human endeavor, but we should not project the failings of some to the entire audio enthusiast community. In my case, I have previously stated on this forum that I do not like silver-faced electronics and oak veneer. Yet, my two channel system is composed entirely of silver-faced amplification components (which are discreetly hidden behind a black metal mesh equipment cabinet door
). My loudspeakers are finished in loathsome oak veneer.An appropriate question would be whether the ABX test used by food scientists is the same ABX test used by audio scientists. The answer is yes. Although the same term can have different meanings in different scientific disciplines, the attached table from Lawless and Heymann's "Sensory Evaluation of Food" textbook (table 4.1, page 81) shows that the ABX test they discuss is the same ABX test used in monophonic and stereophonic audio evaluations.
Conclusion
According to Lawless and Heymann, food scientists "rarely, if ever" use the ABX test in food testing because it is not a good tool for the evaluation of multi-dimensional stimuli.
In spite of this, some audio scientists enthusiastically believe the ABX test to be the "gold standard" for stereophonic system evaluation. One reason for this may be the fact that, as Lawless and Heymann noted, the ABX test is widely used in tests of discrimination of speech sounds and in measuring auditory thresholds. As I noted in my study of stereophonic blind testing, the ABX test is also widely used in the telecommunications industry used for voice circuit quality evaluation.
Some audio scientists may have thought that, since the ABX test was successfully used in evaluating other types of audio, it would be appropriate to use it for stereophonic audio evaluation. It may also be that some audio scientists discovered a reasonable explanation as to why the multi-dimensional nature of stereophonic audio makes it a good candidate for ABX application but the multi-dimensional nature of food makes it a poor candidate. There is much empirical evidence to support the former. I am still searching for the latter.
It is certainly clear that some audio scientists believe that stereophonic audio is exempt from the following two rules:
1. A thing must be tested in a manner accurately representative of the way it will be used.
2. Sensory evaluations must be conducted in a manner that does not prevent or diminish relevant sensory stimuli from reaching the subject.
References
1. Lawless, H. T. and Heymann H., "Sensory Evaluation of Food: Principles and Practices", pp. 79-100, 2010.
2. Ennis, D. M. and Mullen, K., "Theoretical Aspects of Sensory Discrimination", Chemical Senses, Vol. 11, No. 4, pp. 513522, 1986.
3. Link: A Historical Overview of Stereophonic Blind Testing
4. Link: A Survey Of Early Stereophonic System Subjective Evaluation
Proud and loyal citizen of the Digital Domain and Solid State Country!
Post edited by DarqueKnight on
Comments
-
While I enjoy reading your posts, I have to admit I am not quite certain exactly what an “ABX test” means. I have always thought of the A/B test as a listener being asked to tell which item, A or B, is different from the control. What is the ‘X’ part in ABX testing?
However, that question is really irrelevant. To me, A/B testing implies an immediate, obvious difference. Is it Coke or Pepsi? But audio differences can be subtle. It is as if one day you notice a difference in attitude with your wife, girlfriend, friend, dog, cat, etc. You know something has changed, but to anybody else everything is the same. The only way you detect that change is by being intimately familiar with the subject (so to speak). How do you measure that? The same is with audio. Unless you are really familiar with the gear, any subtle changes are going to be impossible to immediately detect, but they do exist.
Being an engineer I expect everything in audio is measureable. Of course, if you do not know what to measure then it can’t be measured, but that does not mean it does not exists.
I recently purchased some Shunyata power cords, and they provided literature about how they developed test equipment to measure the performance of power cords. Now that they can measure the performance, they have the ability to tweak cables to change the results. Theoretically, this means they can now objectively make cables that perform better and better, but I suspect that unless you are intimately familiar with the sound an A/B test would result in no differences detected.
Anyway, enough Christmas break beer induced rambling. Thank you for efforts to bring a sense of honesty to this subject.Lumin X1 file player, Westminster Labs interconnect cable
Sony XA-5400ES SACD; Pass XP-22 pre; X600.5 amps
Magico S5 MKII Mcast Rose speakers; SPOD spikes
Shunyata Triton v3/Typhon QR on source, Denali 2000 (2) on amps
Shunyata Sigma XLR analog ICs, Sigma speaker cables
Shunyata Sigma HC (2), Sigma Analog, Sigma Digital, Z Anaconda (3) power cables
Mapleshade Samson V.3 four shelf solid maple rack, Micropoint brass footers
Three 20 amp circuits. -
Sorry DK, I fell asleep halfway through reading it. Can't you sum it up in a paragraph or 2? HT SYSTEM-
Sony 850c 4k
Pioneer elite vhx 21
Sony 4k BRP
SVS SB-2000
Polk Sig. 20's
Polk FX500 surrounds
Cables-
Acoustic zen Satori speaker cables
Acoustic zen Matrix 2 IC's
Wireworld eclipse 7 ic's
Audio metallurgy ga-o digital cable
Kitchen
Sonos zp90
Grant Fidelity tube dac
B&k 1420
lsi 9's -
While I enjoy reading your posts, I have to admit I am not quite certain exactly what an ABX test means. I have always thought of the A/B test as a listener being asked to tell which item, A or B, is different from the control. What is the X part in ABX testing?
In an A/B test, you compare A and B and note which has the greatest amount of a specific attribute.
In an ABX test, you match an unknown sample, X, to one of two references A and B. One of the references is the same as X. That is why ABX is called a "matching to sample" test.Proud and loyal citizen of the Digital Domain and Solid State Country! -
....Political Correctness'.........defined
"A doctrine fostered by a delusional, illogical minority and rabidly promoted by an unscrupulous mainstream media, which holds forth the proposition that it is entirely possible to pick up a t-u-r-d by the clean end."
President of Club Polk -
Oh Emm Gee.
Really? Raife, phone home already, we miss you!
Cheers,
RussCheck your lips at the door woman. Shake your hips like battleships. Yeah, all the white girls trip when I sing at Sunday service. -
I will enthusiastically concede that another thread on toilet functions would have been more appropriate, but we all have our vices. Proud and loyal citizen of the Digital Domain and Solid State Country!
-
Simpletons Raife. I, and others do appreciate you making us think (and think hard), just saying I miss the SDA shrine posts, the *simple mod / tweak*, the recommended recording, etcetera.
Just come back to earth every now and then, so joe 6 pack can partake, that's all. You're a writer, don't forget the audio nursery rhyme every now and again.
Mix it up, there, 3 words nutshelled it.
Cheers,
RussCheck your lips at the door woman. Shake your hips like battleships. Yeah, all the white girls trip when I sing at Sunday service. -
Noted. Some of that is in the works.Proud and loyal citizen of the Digital Domain and Solid State Country!
-
You the man.Check your lips at the door woman. Shake your hips like battleships. Yeah, all the white girls trip when I sing at Sunday service.
-
I think the M theory (emphasis of string theory) can explain how ABX testing works. Everything will fall into place with a string.Trying out Different Audio Cables is a Religious Affair. You don't discuss it with anyone. :redface::biggrin:
-
Dude, I dig your passion.Source: Bluesound Node 2i - Preamp/DAC: Benchmark DAC2 DX - Amp: Parasound Halo A21 - Speakers: MartinLogan Motion 60XTi - Shop Rig: Yamaha A-S501 Integrated - Shop Spkrs: Elac Debut 2.0 B5.2
-
In simple terms he is saying that using abx for audio evaluation is moronic.
For those who want to ABX knock yourself out.
I have much better things to do.
RT1 -
DarqueKnight wrote: »Introduction
"As foods are multi-dimensional, random variation in irrelevant dimensions can act as a false signal to the panelists and draw their attention to sensory features that are not consistent sources of difference (Ennis and Mullen, 1986)."
I would have to ask what they mean by this uncontrollable variable. As I read this there is random variability of some undefined x/y/z that the panelist will evaluate on that they should not.
While I agree one's seasons grapes will make a style of wine taste one way one year and another year different due the uncontrolled variability of nature. I can understand why ABX may not work in that instance. It's not what they are after.
I don't agree that audio is subject to that same wide swing. When I gain match my pre out and level match my speakers there are only a few things I can't control for: Atmosphere, listener head position. But most likely fall into a fudge margin that won't affect the statistical output. When I play something 5 times over it is going to measure the same +/-.
Are you going to have a chance to present at AES? Are there any other testing methodologies that AES currently subscribes to?
BTW my wife pulled your paper for me and I had a chance to sit down a read it about a month ago. -
Explains why food loses its taste when I eat during critical listening sessions...More later,
Tour...
Vox Copuli
Better to remain silent and be thought a fool, than to open your mouth and remove all doubt. - Old English Proverb
"Death doesn't come with a Uhaul." - Dennis Gardner
"It's easy to get lost in price vs performance vs ego vs illusion." - doro
"There is a certain entertainment value in ripping the occaisonal (sic) buttmunch..." - TroyD -
Thanks DarqueKnight I also love reading your stuff but put me in the "simpleton" group on this one!
Keep em coming!Analog Source: Rega P3-24 Exact 2 w/GT delrin platter & Neo TT-PSU Digital Source: Lumin T2 w/Roon (NUC) DAC: Denafrips Pontus II Phono Preamp: Rega Aria MK3 Preamp: Rogue RP-7 Amp: Pass X150.8 Speakers: Joseph Audio Perspective 2, Audio Physic Tempo Plus Cables: Morrow M4 ICs & Audio Art SC-5 ePlus, Shunyata PCs Misc: Shunyata Hydra Delta D6, VTI rack, GIK acoustic panels -
And Bruce appears out of nowhere, like Kane in Kung Fu....Check your lips at the door woman. Shake your hips like battleships. Yeah, all the white girls trip when I sing at Sunday service.
-
Sorry DK, I fell asleep halfway through reading it. Can't you sum it up in a paragraph or 2?
Sure.
Food scientists do not use the ABX test on food because food has too many variables. Some of those variables are irrelevant to the sensory analysis of food and can distract a subject toward erroneous conclusions. The ABX test does not provide a process for differentiating between relevant and irrelevant differences nor does it provide a process for characterizing differences in fine detail.
Similar to food, stereophonic audio has many variables, some of which are irrelevant to the sensory analysis of stereophonic audio and can distract a subject toward erroneous conclusions. The ABX test does not provide a process for differentiating between relevant and irrelevant differences.
How's that.I think the M theory (emphasis of string theory) can explain how ABX testing works. Everything will fall into place with a string.
I understand how ABX testing works. I am trying to understand why we should apply it to stereophonic audio.If ABX testing isn't suitable based on your argument above, what then is the best test to use?
Descriptive analysis.Repeatability of results is a good measure of a test protocol. Are you saying that ABX tests are not repeatable yielding statistically relevent results?
ABX test results are repeatable, but the results do not appear to be statistically relevant to stereophonic audio because of the 50/50 chance of guessing a correct or incorrect answer and due to the fact that, similar to food and beverage, there is simply too much sensory information of varying intensities and types which precludes reducing a subject's choice to a simple difference between reference A and reference B.Are there statistics to support the weakness of ABX testing or prove the superiority of another method of testing?
The 50% chance of guessing a correct answer should provide insight on the statistical weakness of this test when applied to something as complex as a stereophonic sound field.A table comparing the differences and similarities of sensory characteristics between foods and stereophonic sound would be beneficial.
Great idea. For all we know, someone may have already done this. If not, are you volunteering?One common trait of being "multi-dimensional" may be outweighed by several uncommon traits netween these two. ABX testing may be beneficial in assessing those characteristics. Did the authors you cite explore this?
I do not see a need to explore this.
I don't think you are grasping what Lawless and Heymann wanted to convey. ABX, and other difference tests, are suitable for finding overall differences. It is not suitable for delineating the contributing factors and attributes of that difference. A difference may be detected in an ABX test that is unrelated to a relevant attribute of the thing being tested. See my second response to Jinjuku for an example of this.reeltrouble1 wrote: »In simple terms he is saying that using abx for audio evaluation is moronic.
No, I am not saying that at all. I am saying that the ABX test does not appear to be a good test for stereophonic audio, and that the users of this test have not adequately justified its use in light of the complexities of stereophonic sound fields and the simplicities of difference tests.I would have to ask what they mean by this uncontrollable variable. As I read this there is random variability of some undefined x/y/z that the panelist will evaluate on that they should not.
While I agree one's seasons grapes will make a style of wine taste one way one year and another year different due the uncontrolled variability of nature. I can understand why ABX may not work in that instance. It's not what they are after.
Lawless and Heymann did not use the term uncontrollable variable. They used the term "random variation in irrelevant dimensions ". The key word being "irrelevant".I don't agree that audio is subject to that same wide swing. When I gain match my pre out and level match my speakers there are only a few things I can't control for: Atmosphere, listener head position. But most likely fall into a fudge margin that won't affect the statistical output. When I play something 5 times over it is going to measure the same +/-.
Your example illustrates the fallacy in relying too much on absolute measurements.
You cannot control for atmosphere and listener head position. You also cannot control the amount of noise inherent in the physical material of an electronic component. You can level match two amplifiers in absolute terms so that they measure the exact same number of decibels on a sound level meter. However, the amplifier with the lower noise content will sound apparently louder because, although the amps are matched in absolute loudness, the lower noise amp is providing a higher amplitude of music to the subject. In this case, electrical noise content is an irrelevant dimension because we do not want to listen to noise, we want to listen to music. The subject has been told that both amps are playing at the same level, yet they sound different. The subject may say that the lower noise amp sounds better due to the fact that it makes a bigger presentation at his ears, although in terms of relevant stereophonic attributes, the higher noise amp may provide better image weight and better sound stage width, depth and height.Are you going to have a chance to present at AES?
I'm sure I could If I wanted to. I'm not sure if I want to. The AES accepts just about anything audio-related at their conventions and, unlike IEEE conferences, the AES does not accept any responsibility for the content of AES conference (convention) papers.
The following disclaimer is appended to every AES convention paper in my possession from the late 1990's and earlier:
"This preprint has been reproduced from the author's advance manuscript, without editing, corrections or consideration by the Review Board. The AES takes no responsibility for the contents."
The AES convention papers I have read dating from the early 2000's and onward all display the following disclaimer:
"The papers at this Convention have been selected on the basis of a submitted abstract and extended precis that have been peer reviewed by at least two qualified anonymous reviewers. This convention paper has been reproduced from the author's advance manuscript, without editing, corrections, or consideration by the Review Board."
If I wanted to present at a technical conference, I am thinking a conference with a more rigorous peer review process, like an IEEE Signals and Systems Society conference or a Society of Sensory Professionals conference, would be more appropriate. The AES disclaimers seem to relegate their conference papers to mere author opinion.
Most electrical engineers who are active in the audio engineering field prefer to publish in the peer-reviewed journal "IEEE Transactions on Signals and Systems". This journal was formerly named "IRE Transactions on Audio".Are there any other testing methodologies that AES currently subscribes to?
Of course there are, from subjective to objective and every variation in between. The AES is an international body of audio technical professionals. The flyer I attached above indicates that there is wide variety of interest in various testing methodologies for audio.
The Journal of the AES has audio evaluation papers going back to the 1970's which report the application of sensory analysis techniques to audio. However, many of those papers used monophonic audio systems or unrealistic stereophonic system arrangements.
I will share with you one paper that I found fascinating. The title of the paper is "Psychoacoustic Detection Threshold of Transient Intermodulation Distortion, by M. Petri-Larmi, M. Otala and J. Lammasniemi. It was published in the March 1980 issue of the Journal of the Audio Engineering Society. The abstract is as follows:
"The audible threshold of transient intermodulation distortion (TIM) was determined for the six most sensitive subjects of the previously reported test series of 68 listeners. Improved equipment, carefully controlled listening environment, a digital TIM generator, and five recorded stereophonic music samples were used. The results show that in certain types of passages of music, a trained and sensitive listener can reliably detect extremely low values of distortion. Low distortion values were perceived only as changes in sound character, and not as distortion."
Some excerpts:
"The thresholds reported are for experienced, well-rehearsed subjects, and are therefore believed to be conservative. Further improvements in equipment and records may, however, yield lower thresholds in the future."
"The test records were selected on the basis of the experience gained from the previous investigation to represent technically excellent and audibly "clean" recordings."
"The subject was allowed to play the record for 15-30 minutes."
"Comprehensive auditory tests were performed for all the subjects."
"At the end of the learning period the subject was asked to indicate the distortion level he thought he could perceive in a blind A/B test."BTW my wife pulled your paper for me and I had a chance to sit down a read it about a month ago.
OK...and...?Proud and loyal citizen of the Digital Domain and Solid State Country! -
Explains why food loses its taste when I eat during critical listening sessions...
LOL, well, the senses are interrelated.
Welcome back!!Proud and loyal citizen of the Digital Domain and Solid State Country! -
DarqueKnight wrote: »
You cannot control for atmosphere and listener head position. You also cannot control the amount of noise inherent in the physical material of an electronic component. You can level match two amplifiers in absolute terms so that they measure the exact same number of decibels on a sound level meter. However, the amplifier with the lower noise content will sound apparently louder because, although the amps are matched in absolute loudness, the lower noise amp is providing a higher amplitude of music to the subject. In this case, electrical noise content is an irrelevant dimension because we do not want to listen to noise, we want to listen to music.
And this difference can't be picked out A/B/X? But you are listening to the noise. You're asking to throw the baby out with the bath water (?). It is amp A and amp B et al. The discrimination is in the unadulterated experience for the average user. I personally want equipment that will make a positive, clearly noticeable, impact for the average Joe. I just don't believe in treating the enjoyment of audio like training for a marathon.
As far as the reading of your paper:
I still haven't made up my mind. In part because I personally want to have a speaker that is un-questionably up to the task of being revealing of source warts and all. That is the Mission Statements that I'm am going to start putting together this weekend. That part is all on my side of the court.
Since we are talking variability. Do you see any funding source for you to get a sizable N together for a longitudinal study. I am curious as to how reproducible results are given your postulated methodology. -
DarqueKnight wrote: »Your example illustrates the fallacy in relying too much on absolute measurements.
You cannot control for atmosphere and listener head position. You also cannot control the amount of noise inherent in the physical material of an electronic component. You can level match two amplifiers in absolute terms so that they measure the exact same number of decibels on a sound level meter. However, the amplifier with the lower noise content will sound apparently louder because, although the amps are matched in absolute loudness, the lower noise amp is providing a higher amplitude of music to the subject. In this case, electrical noise content is an irrelevant dimension because we do not want to listen to noise, we want to listen to music. The subject has been told that both amps are playing at the same level, yet they sound different. The subject may say that the lower noise amp sounds better due to the fact that it makes a bigger presentation at his ears, although in terms of relevant stereophonic attributes, the higher noise amp may provide better image weight and better sound stage width, depth and height.
I vividly recall that one of the first things an author of an ABX audio test mentioned was the matching of sound output levels. On the surface, this seems like an excellent way to assure equivalence of results. However, sound level is an irrelevant dimension. All that needs to be done is to set the music at a comfortable listening level. The relevant dimensions are where the images are placed in the sound field and the sonic characteristics of those images.
A listener untrained in stereophonic attribute evaluation could easily come to an erroneous conclusion after receiving the assurance of sound level matching.
Revisiting the example of the two level matched amplifiers with two different noise characteristics, a subject untrained in stereophonic attribute evaluation may be more impressed by the apparently louder sounding amplifier, even though the noisier amplifier offers more stereophonic attribute performance.
Let's assume that the noise content of the two amplifiers was equal and the music sounded equally as loud to a subject. Since the amplifiers are different, they will have different noise characteristics even though the absolute level of noise is equal. Amp A's noise characteristic may only diminish bass detail and articulation. Amp B's noise characteristic may only diminish image weight. Aside from those two differences, the amps are sonically indistinguishable.
An untrained subject would probably detect no difference between the two amplifiers because they would not be looking for things like bass articulation and image weight. Have you read many ABX tests where the subjects were asked to evaluate things relevant to stereophonic performance such as bass articulation, image weight, image placement, clarity and detail? Typically, subjects in ABX audio tests are just asked to match an unknown sample to one reference or the other. What if the subject does not know how to properly evaluate all the sonic attributes of the sample? What if some irrelevant attribute (like more apparent loudness) draws the subject's attention away from more relevant attributes like image weight and image placement?Proud and loyal citizen of the Digital Domain and Solid State Country! -
And this difference can't be picked out A/B/X? But you are listening to the noise. You're asking to throw the baby out with the bath water (?).
You are definitely not listening to the noise. You are listening to the effect of the noise on the music! If you were able to listen to the noise, you would be able to sonically characterize the noise separate from the music. This may be easy to do in the case of a 60 Hz hum. It is difficult or impossible for most people to do when the noise is randomly distributed and at a much lower amplitude than the music. Are you telling me that you can sonically separate and characterize randomly distributed low amplitude noise in a music sample?
In the AES paper I cited above, the subjects could not hear the noise. They could hear the effect of the noise on the music. This was perceived by the subjects as a change in the music's tonal character.It is amp A and amp B et al. The discrimination is in the unadulterated experience for the average user. I personally want equipment that will make a positive, clearly noticeable, impact for the average Joe. I just don't believe in treating the enjoyment of audio like training for a marathon.
Woooooooowwwwwwwwww. After all your years of pleading for scientific evidence of the things audiophiles have been claiming for years, you now want things "dumbed down"? First, you ridicule audiophiles when they say that specific performance differences can be perceived in stereophonic equipment. Now, you imply that I am being elitest when I advocate training to facilitate accurate evaluation.
Don't get it twisted. I am not advocating marathon level training. I am advocating becoming familiar with the relevant performance aspects of stereophonic audio. This is something that just about anyone with reasonably good hearing and a good, but not necessarily high-end, audio system can do. Notice that I used an untrained, disinterested subject in the case study in the JOSS paper.
If one wants to get the most out of any sensory experience, is it not reasonable that some training and education in the sensory stimuli produced by that experience and in the proper evaluative techniques for those stimuli would be essential? This is true whether the sensory stimuli are produced by wine, apple pie, armpits or a stereo system.
I realize that most people do not want to spend the required time that it takes to get the most from this hobby. We get scores of questions from new members asking us what do we think will sound best. They get frustrated...and sometimes angry...when we tell them that their question can only be answered by themselves after some personal listening evaluation.
With regard to average Joe, 99.99999% of the average Joes who have visited my home have been shocked to hear sounds coming from areas between and at the outsides of the speakers. They didn't know a stereo was supposed to do that!As far as the reading of your paper:
I still haven't made up my mind. In part because I personally want to have a speaker that is un-questionably up to the task of being revealing of source warts and all. That is the Mission Statements that I'm am going to start putting together this weekend. That part is all on my side of the court.
OK. I look forward to your well reasoned and scientifically rigorous review. I'm sure you will eventually be posting your thoughts here or at one of the other forums where you passionately promote the reading of selected Polk Forum threads.Since we are talking variability. Do you see any funding source for you to get a sizable N together for a longitudinal study.
I do. I do. More on that later.I am curious as to how reproducible results are given your postulated methodology.
The methodology I proposed is something that the "average Joe" can do in their own homes with equipment of moderate resolution and with appropriate training. The first thing you have to do is get average Joe to accept the stereo illusion wherein loudspeakers can project sound outside of their physical boundaries. If you sit most people in front of a stereo system and ask them to point to where the sound is coming from, most people will point to the speakers. Indeed, some people think that the concept of stereo imaging is a figment of audiophile imagination because they have never heard it. All they have heard is sound localized at the speakers.
I remember some trials I did with one of my sisters some years ago when I asked her to point to the locations of sounds generated by the speakers. For four playbacks of the same song, she pointed at both of the speakers. It wasn't until the fifth playback that she leaned forward, looked me and asked "is there a speaker in the middle?".
"Why do you think there is a speaker in the middle?"
"Because I hear sound coming from the middle. Is the other speaker in the cabinet?
"No, there is no speaker in the middle in the cabinet. See?"
On the sixth playback:
"Sometimes I don't hear the speaker on the left anymore."
Immediately prior to this, I had asked baby sister to just sit and listen and tell me where the sound was coming from, but without pointing to sound sources. In ten trials which represented close to an hour of listening, she could not perceive a stereophonic image. The image didn't come until I asked her to start pointing at the sound sources. By the way, she was a little irritable by the time we began the pointing trials, but this irritability gave way to wonderment when the stereophonic illusion formed in front of her.
"How does it do that?" she asked.
I have corresponded with some audio enthusiasts, some of whom are members of the AES society, who tested the methodology and found it performed as claimed. Common sense should inform that the JOSS review panel which approved the paper for publication had to have done some testing of the methodology...or do you think that they just took my word for it?
The Petri-Larmi JAES paper I cited earlier offers rigorous scientific analysis of results. One of the main points of the paper was that improved stereophonic equipment resulted in better noise discrimination of the subjects. The (circa 1980) audio equipment used was:
1. A Pioneer 530 turntable with Ortofon MC-20 moving coil cartridge and STM 72 transformer.
2. The phono preamplifier section of a Marantz 1200B integrated amplifier.
3. Electrocompaniet "The Two-Channel Power Amplifier". That was the actual name of the amplifier. It was a 25 watt class A design.
4. Yamaha NS 1000M loudspeakers.Proud and loyal citizen of the Digital Domain and Solid State Country! -
Reading that made me hungry, DK.
RT, +1
Let us know what that Soylent Green tastes like. I heard it tastes like PIE.:biggrin:Proud and loyal citizen of the Digital Domain and Solid State Country! -
ABX testing is a misleading crutch for those who do not know how to listen. Switching between samples gives you the the major difference between the two, not a summation of each. If you cannot be truthful enough to yourself due to some overwhelming bias or you cannot admit your $30K device sounds worse than something much cheaper or different then you are not up to the game anyway.Vinyl, the final frontier...
Avantgarde horns, 300b tubes, thats the kinda crap I want...
-
ABX testing is a misleading crutch for those who do not know how to listen. Switching between samples gives you the the major difference between the two, not a summation of each. If you cannot be truthful enough to yourself due to some overwhelming bias or you cannot admit your $30K device sounds worse than something much cheaper or different then you are not up to the game anyway.
Of course ABX has nothing to do with determining a preference, just determining a difference exists at all. Nice strawman though. -
Of course ABX has nothing to do with determining a preference, just determining a difference exists at all. Nice strawman though.
I wouldn't have gone that far to make that particular statement in general, but to the subject of this thread alone, sure.I don't read the newsssspaperssss because dey aaaallllllllll...... have ugly print.
Living Room: B&K Reference 5 S2 / Parasound HCA-1000A / Emotiva XDA-2 / Pioneer BDP-51FD / Paradigm 11se MKiii
Desk: Schiit Magni 2 Uber / Schiit Modi 2 Uber / ISK HD9999
Office: Schiit Magni 2 Uber / Schiit Modi 2 Uber / Dynaco SCA-80Q / Paradigm Legend V.3
HT: Denon AVR-X3400H / Sony UBP-X700 / RT16 / CS350LS / RT7 / SVS PB1000 -
DarqueKnight wrote: »You are definitely not listening to the noise. You are listening to the effect of the noise on the music! If you were able to listen to the noise, you would be able to sonically characterize the noise separate from the music.
In the AES paper I cited above, the subjects could not hear the noise. They could hear the effect of the noise on the music. This was perceived by the subjects as a change in the music's tonal character.
I should have said the amalgam... The listener is hearing everything et al.DarqueKnight wrote: »Woooooooowwwwwwwwww. After all your years of pleading for scientific evidence of the things audiophiles have been claiming for years, you now want things "dumbed down"? First, you ridicule audiophiles when they say that specific performance differences can be perceived in stereophonic equipment. Now, you imply that I am being elitest when I advocate training to facilitate accurate evaluation.
You are positing that when I speak of A/B/X that I was speaking of a highly trained group of N. I don't believe that you will find that in any of my statements made to date. The A/B/X need be carried out in a scientifically proper manner. That is all the control groups are documented and setup for. The listener is the variable. We know how equipment A and equipment B is going to perform once calibrated in a manner that allows for peer review.
All I have ever asked is show me a statistical meaningful result. Why do you think I offered to send out 4 terminated cables. Two with 400 plus hours and two with zero hours? I didn't stipulate the testing, I didn't stipulate the listener, nor the listener training. I only stipulated that you couldn't know which cable was which. Everything else was singularly up to YOUR preference. I can't possibly give anyone a better scenario.DarqueKnight wrote: »Don't get it twisted. I am not advocating marathon level training. I am advocating becoming familiar with the relevant performance aspects of stereophonic audio. This is something that just about anyone with reasonably good hearing and a good, but not necessarily high-end, audio system can do. Notice that I used an untrained, disinterested subject in the case study in the JOSS paper.
Fair enough and not twisted at all.DarqueKnight wrote: »If one wants to get the most out of any sensory experience, is it not reasonable that some training and education in the sensory stimuli produced by that experience and in the proper evaluative techniques for those stimuli would be essential? This is true whether the sensory stimuli are produced by wine, apple pie, armpits or a stereo system.
I could see a manufacturer training for BIAS that would favor their product. That is an extreme example though.DarqueKnight wrote: »I realize that most people do not want to spend the required time that it takes to get the most from this hobby. We get scores of questions from new members asking us what do we think will sound best. They get frustrated...and sometimes angry...when we tell them that their question can only be answered by themselves after some personal listening evaluation.
With regard to average Joe, 99.99999% of the average Joes who have visited my home have been shocked to hear sounds coming from areas between and at the outsides of the speakers. They didn't know a stereo was supposed to do that!
I have seen that too. I had some acoustic guitar music going on in the other room one time. A buddy stopped over and asked who was playing. Not who I had in the CD player, but who was in the room with an Acoustic guitar.DarqueKnight wrote: »OK. I look forward to your well reasoned and scientifically rigorous review. I'm sure you will eventually be posting your thoughts here or at one of the other forums where you passionately promote the reading of selected Polk Forum threads.
I'll be nice and make sure to post hereDarqueKnight wrote: »I do. I do. More on that later.
Cool, congratulations are in order then. -
Of course ABX has nothing to do with determining a preference, just determining a difference exists at all.
But almost everything and every combination sounds different. Not necessarily better or worse but different none the less.
Vinyl, the final frontier...
Avantgarde horns, 300b tubes, thats the kinda crap I want... -
But almost everything and every combination sounds different. Not necessarily better or worse but different none the less.
Right, but some on this forum constantly argue there is no difference between cables. But, if your ears work, and your gear is even semi-decent, different cables (and other tweaks) will produce different sounds. The trick is to add new cables that continually improve the sound, not degrade it.Lumin X1 file player, Westminster Labs interconnect cable
Sony XA-5400ES SACD; Pass XP-22 pre; X600.5 amps
Magico S5 MKII Mcast Rose speakers; SPOD spikes
Shunyata Triton v3/Typhon QR on source, Denali 2000 (2) on amps
Shunyata Sigma XLR analog ICs, Sigma speaker cables
Shunyata Sigma HC (2), Sigma Analog, Sigma Digital, Z Anaconda (3) power cables
Mapleshade Samson V.3 four shelf solid maple rack, Micropoint brass footers
Three 20 amp circuits. -
But almost everything and every combination sounds different. Not necessarily better or worse but different none the less.
Maybe, but for some items it's never been demonstrated. -
As usual, nothing but pointless pictures. Maybe the concept of ABX is a bit much for you to understand, like most subjects.
