Notice Of Sensory Science Journal Publication

DarqueKnight

Introduction

In March of this year, I conducted a scientific investigation to assess the validity of the ABX test for audio evaluation. During the course of that investigation, I was able to obtain a better understanding of a proper methodology for audio equipment evaluation. I was encouraged to submit my findings to a scientific journal for review and publication. There were several publication options available. I chose to submit my results to the Journal of Sensory Studies (JOSS), which is the premier peer-reviewed international sensory science journal. The Journal of Sensory Studies is also the official peer-reviewed journal of the Society of Sensory Professionals. An overview of JOSS can be read here: JOSS Overview.

Journal Paper Publication Acceptance

I have received notice that my 16-page manuscript, entitled "A Descriptive Evaluation Methodology For Consumer Audio Equipment", has been accepted for publication after rigorous peer review. The abstract, evaluation methodology summary and conclusion sections are as follows (all quoted text (in blue) and figures are © 2010 Wiley Periodicals, Inc.):

Abstract

"A pilot study was conducted in order to investigate the application of descriptive analysis techniques to the evaluation of consumer audio equipment. A global seven-step methodology for the descriptive evaluation of the sound characteristics of consumer audio equipment was developed. In contrast to the commonly used forced choice discrimination methods for audio equipment evaluation, the proposed methodology enhances, rather than compromises, a subject’s sensory input. A case study employing a minimally trained subject demonstrated the applicability of sensory evaluation techniques to sound. There has been some resistance in the audio community to the adoption of sensory science evaluation methods because such methods are erroneously thought to apply only to products that affect the senses of touch, taste, smell and sight.

Evaluation Methodology Summary

"The basic framework for the seven-step descriptive evaluation method for audio equipment comprises:

1. Evaluator sense organ assessment.

2. Evaluator training.

3. Evaluator experience.

4. Proper equipment setup, speaker placement and selection of
the optimum evaluator listening location (stereo sweet spot).

5. A quiet, comfortable and “acoustically friendly” listening
room, preferably with the capability of dimming the lights.

6. Careful listening, in the stereo sweet spot with familiar,
well-recorded music.

7. “Sound stage mapping” where the location of sounds (images)
within the sound stage and the attributes of those sounds are
carefully documented."

Conclusion

"This pilot study demonstrated that the descriptive analysis method for audio equipment evaluation is a sensitive, accurate, elegant, portable and easy to learn protocol. In stark contrast to the widely used forced choice discrimination evaluation methods, the protocol proposed in this paper is administered under realistic, non-stressful conditions and it maximizes and optimizes the amount of aural and tactile information received by the subject. The case study results emphasize the need for further study in audio evaluation with a fully trained descriptive analysis panel. Formal descriptive analysis would require evaluative panel members with training in descriptive techniques, adequate hearing ability, adequate full body tactile response, an extensive memory in real musical and vocal sounds, competence in setting up a moderate to high resolution audio system in an acoustically appropriate room and competence in documenting what is heard and felt."

Sample Figures

Figure 1. Audio test arrangement for descriptive method for stereophonic audio evaluation. © 2010 Wiley Periodicals, Inc.


Figure 3. Aerial and lateral sound stage charts for musical selection "Blue". © 2010 Wiley Periodicals, Inc.


Figure 4. Typical Duo-trio balanced reference (ABX) audio test arrangement. © 2010 Wiley Periodicals, Inc.

Motivation And Major Points

What I hope to achieve by publishing this paper is more consideration from the audiophile and audio community toward the adoption of scientifically validated and appropriate audio evaluation principles. The sensory science community is eager to see more research in the application of sensory science principles to sound.

Another important consideration is the deluge of misinformation, much coming from seemingly "authoritative" and "legitimate" sources via the Internet, that new entrants into our hobby and new entrants into professional audio practice are exposed to and influenced by.

The main points of the paper are:

1. Sensory science descriptive analysis (DA) techniques are applicable to and appropriate for the evaluation of audio equipment.

2. One does not need superhuman hearing to participate in accurate descriptive analysis of audio evaluation. A minimally trained subject quickly learned to discern subtle performance differences in audio equipment.

3. Forced choice discrimination methods, such as ABX, are not appropriate for the evaluation of audio equipment.


Publication Schedule

My copyright agreement with the journal prohibits me from posting a copy of the paper on the Internet. The paper will be available, in print and electronic formats, from the journal's website.

The Journal of Sensory Studies is published six times a year (Feb., Apr., Jun., Aug., Oct., and Dec.). "A Descriptive Evaluation Methodology For Consumer Audio Equipment" is scheduled to be published in the Journal's hard copy edition in December of 2010. This could possibly be moved up or back one issue (to either October 2010 or February 2011). Well in advance of the hard copy publication, the paper will be published on the Journal's website.

Further Study

I submitted a short paper on an audio-related topic to an international journal of the Institute of Electrical and Electronics Engineers (IEEE) in April of this year. After rigorous peer review, the journal editor informed me that they were interested in the topic, but they considered the paper to be "incomplete". The editor provided very specific instructions for revising the paper. Basically, I was advised that I needed to provide more theoretical and quantitative discussion pertaining to the results. I will provide a full discussion of this paper at a later date.

A common criticism, by the unknowledgeable, of high performance audio products is the absence of the science behind such products appearing in peer-reviewed scientific journals. The absence of such published information is more reflective of the realities and constraints of manufacturing in a competitive marketplace rather than high performance audio products being based on snake oil, voodoo and wishful thinking. Scientists who work for audio product companies typically do not have time to go 'round and 'round with the months-long peer review process. The "validation" of their work comes from the hearing of, and purchase by, the consumer rather than from acceptance by a scientific journal. Another consideration is the fact that the science behind many commercial audio products is proprietary information. Publishing such information in a journal, (or divulging such information in a patent application) might put a company at a severe competitive disadvantage.

I have submitted two audio related papers to scientific journals. The first paper, regarding audio equipment evaluation techniques, has been approved for publication in the leading international sensory science journal. The second paper has completed the first round of review by a leading international IEEE journal and is a work in progress.

From my limited experience, I can certainly extrapolate that scientific journals could be filled to the brim with audio related papers...if audio company scientists and engineers wanted to do that.

Such Good Science.:)

Wrote well you have my old apprentice. Much controversy awaits you.
Remember your training. Save you it can.

F1nut

Well done Raife, well done!

heiney9

I must say............OUTSTANDING!

quadzilla

I've long felt that if we could figure out what to measure, we could explain how to predict, given the specific auditory response curve and personal biases of a given person, what would sound best to them. Such would at least largely explain why what one person refers to as full treble extension is called overly bright by another, etc. Anyway, it's a great start, and congrats on getting published.

tonyb

Extremely well done....thanks for taking the time to do the study and posting.

george daniel

Bravo sir,,indeed

mhardy6647

Acceptance in a peer-reviewed journal. most impressive. Congratulations.

punk-roc

Publishing papers is a grueling process, but well worth it in the end. Good work DK!

Jason

markmarc

Awesome Raife!!!!!!

cnh

Evaluation Methodology Summary

"The basic framework for the seven-step descriptive evaluation method for audio equipment comprises:

1. Evaluator sense organ assessment.

2. Evaluator training.

3. Evaluator experience.

4. Proper equipment setup, speaker placement and selection of
the optimum evaluator listening location (stereo sweet spot).

5. A quiet, comfortable and “acoustically friendly” listening
room, preferably with the capability of dimming the lights.

6. Careful listening, in the stereo sweet spot with familiar,
well-recorded music.

7. “Sound stage mapping” where the location of sounds (images)
within the sound stage and the attributes of those sounds are
carefully documented."

Very nice work. If I am understanding the journal jargon correctly (and no judgment there since I have published any number of jargon-ridden articles in my own field).

Your main contribution is the development of a LISTENING ASSESSMENT, EDUCATION, AND PROTOCOL for the 'average' listener. To elaborate. It seems that what you do is take stock of a subject's listening abilities before you begin, then EDUCATE them on WHAT to LISTEN FOR and HOW TO LISTEN? This is then followed by actual audition/listening tests that are conducted in an as realistic environment as possible, i.e., one that simulates the normal at home listening experience? Subjects are then given a research protocol questionnaire.

And also providing an environment that is non-stressful--in other words one NOT full of measuring devices and gentlemen and women in white lab coats A/Bing equipment using a randomized sampling of listeners and a 'control' group? ...Yes??

cnh

headrott

A well deserved place for your research Raife. Congratulations.


Greg

Tony M

Congrats to you.

I love the chart of where sounds seem to come from. I've never seen one of those. I think it should be used in stereo publications asap to show what the reviewer heard while testing speakers. I love it. We all know that illusion we hear once you get up into higher products revelations.

What would a chart look like for someone auditioning bose with only 2 front cubes on w/sub? Probably not to spacious or deep of a soundfield huh.

That chart could revive a want for SDA speakers when reviewers chart sounds coming from the sides and vocals always (99%) coming from dead center too. I can almost tell if a singer is on a stool, standing and where the mike was in relation to their voices now that I have a pretty decent speaker/equipment combo. I have a long way to go I know but I love what I've heard since the purchase of my first Polks about 11 years ago.

It's nice to know the author of the article has a beautiful set of SRS's in his home to prove he knows what he's talking about.;):D

DarqueKnight

cnh wrote: »

Your main contribution is the development of a LISTENING ASSESSMENT, EDUCATION, AND PROTOCOL for the 'average' listener.

The protocol is for ANY listener, from totally inexperienced to professional audio reviewer.

cnh wrote: »

To elaborate. It seems that what you do is take stock of a subject's listening abilities before you begin, then EDUCATE them on WHAT to LISTEN FOR and HOW TO LISTEN? This is then followed by actual audition/listening tests that are conducted in an as realistic environment as possible, i.e., one that simulates the normal at home listening experience? Subjects are then given a research protocol questionnaire.

An important part of the "questionnaire" would be the sound stage mapping process.

The paper concentrates on the home stereo listening experience, but the protocol is extensible (with modification) to any listening environment (eg. automobiles). The important thing is to simulate (and ideally replicate) the actual "normal" listening environment during the evaluation.

cnh wrote: »

And also providing an environment that is non-stressful--in other words one NOT full of measuring devices and gentlemen and women in white lab coats A/Bing equipment using a randomized sampling of listeners and a 'control' group? ...Yes??

Yes. Of course...unless that is how you actually plan to listen to your audio system.:) Measurements have certainly have their place, but we still cannot determine how something will sound from the way it measures.

dorokusai

Sounds like a society of crackpots with pretty good ears and very good writing skills. Nice read DK.

reeltrouble1

Alas, I have been told by those who seek their truth's I am but a defective repository of self-senses, never to be believed, let alone trusted, still empiracally at the very least I am, and still have faith I can understand. Yet all what is, takes place within a living dimensionally complex system, ordered in constant flux events occuring with connected singularity to form a collected past which is assimilated into the collective whole of the person, the soul per say. One might endeavor to understand the being of self with equation with data, indeed, devote a lifetime of study to just the one let alone this duo of trio whose receptors are processing musical sound in a given moment, yet complete replication is like so much unobtanium, molecules set into motion cannot be recalled, cannot be reset, even the sheep dolly's become more indepedent as time and space move. Recollection of something musical possible yet born from the collective whole of an individual within a group likely influenced by each.

What looks to be may not be. Theory is fundamental to the order, invention to new order, so we seek because we are human, we listen because we must, afterall, we always have, listened that is.

RT1

Erik Tracy

Nice job - sounds like there could be a paradigm shift in how audio evaluations are to be conducted.

You may be subjected to vehement attack by the AVS wankers who only put stock in true blind A/B testing.

Imagine the furor if tests can substantiate subjects being able to hear differences in components!

DarqueKnight

Erik Tracy wrote: »

You may be subjected to vehement attack by the AVS wankers who only put stock in true blind A/B testing.

Imagine the furor if tests can substantiate subjects being able to hear differences in components!

I don't mind vehement attack and furor as long as it is mathematically and scientifically sound...and it promotes better understanding.;)

I also don't mind if people want to believe in witchcraft, voodoo and wishful thinking. I would just prefer it if the people standing on their heads and screaming the loudest for "scientific proof" would not use an evaluation method that is statistically inefficient and error prone and which mathematically converges to results very similar to guessing.

heiney9

I found this quote extremely timely for this thread, especially the part in red. Nelson has always believed that listening and enjoyment of music comes from your ears and brain not a set of measurements. (see his quote in my sig) I'm looking forward to reading the entire article this is quoted from when it's available.

Nelson's new paper on a Jean Hiraga-based, Jan Didden-inspired Arch Nemesis DIY amp project kicks off with these words: "A poster of Einstein once said, things should be made a simple as possible but no simpler. This can apply to audio amplifiers but if they are evaluated subjectively, the simplicity thing can get a little out of hand. Of itself, minimalism exerts a strong aesthetic attraction and there is a reasonable belief that fewer components in the signal path allow more information to get through with less coloration. If like me you are interested in understanding how we hear distortions with our brains—instead of our meters—you might appreciate that simple circuits help isolate these phenomena. I listen to all sorts of flawed circuits because I enjoy hearing the differences and it helps to train my ears. In this regard, reducing the number and types of flaws makes it easier to tweak a single parameter and hear the difference. I think it's also true that simple distortions are often more forgivable in a listening situation and create less fatigue."

H9

P.s. much of his evaluation is done by listening.

madmax

Excellent! We all need help in listening strategies.

"7. “Sound stage mapping” where the location of sounds (images)
within the sound stage and the attributes of those sounds are
carefully documented."

I especially like this part. I always pay attention to where sounds come from, whether or not they sound real and if the image size is realistic.

Well done!
madmax

Rev. Hayes

I would love to read the full article when it comes out...

As madmax said the sound stage mapping is a fantastic idea. It would really be something if the industry, or even a single manufacturer of speakers (;))developed some sort of base line, or boilerplate signal and listening room, thus allowing them to show a sound stage map of each of their products.

I guess I always imagined that there was a great deal of scientific method on the design end of audio but now that I think about it none of the product evaluation charts I have seen are based on large scale listener evaluation. It's always electronic specs and one individual's opinion. To see a statistical analysis of products based on many people's listening experience who have a simple way to evaluate and describe what they are actually hearing.... well, right on man.

unc2701

DarqueKnight wrote: »

I also don't mind if people want to believe in witchcraft, voodoo and wishful thinking. I would just prefer it if the people standing on their heads and screaming the loudest for "scientific proof" would not use an evaluation method that is statistically inefficient and error prone and which mathematically converges to results very similar to guessing.

Do you compare the statistical efficiency of this new method to ABX testing in the manuscript?

dorokusai

reeltrouble1 wrote: »

Alas, I have been told by those who seek their truth's I am but a defective repository of self-senses, never to be believed, let alone trusted, still empiracally at the very least I am, and still have faith I can understand. Yet all what is, takes place within a living dimensionally complex system, ordered in constant flux events occuring with connected singularity to form a collected past which is assimilated into the collective whole of the person, the soul per say. One might endeavor to understand the being of self with equation with data, indeed, devote a lifetime of study to just the one let alone this duo of trio whose receptors are processing musical sound in a given moment, yet complete replication is like so much unobtanium, molecules set into motion cannot be recalled, cannot be reset, even the sheep dolly's become more indepedent as time and space move. Recollection of something musical possible yet born from the collective whole of an individual within a group likely influenced by each.

What looks to be may not be. Theory is fundamental to the order, invention to new order, so we seek because we are human, we listen because we must, afterall, we always have, listened that is.

RT1

I've ceased explaining what sounds good and why to my social circle....it's a waste of time for the most part. I do end up installing more Such Good Sound for their rooms....so it must just work itself out I guess, who knows.

WilliamM2

Why is the listening position different for the descriptive method and the ABX method?

DarqueKnight

unc2701 wrote: »

Do you compare the statistical efficiency of this new method to ABX testing in the manuscript?

This method is not really "new". This is what I have been doing for years. My equipment review posts on this forum attest to this.:)

I evaluated the statistical efficiency of my proposed protocol. I did not need to do a direct comparison between its statistical efficiency and that of the ABX protocol because the statistical inefficiency of the ABX (duo-trio balanced reference) protocol is well documented in textbooks and in the audio literature.

I discussed the methodical and statistical failures of a well known ABX trial here: Power Cable ABX Test.

An often cited ABX test for power amplifiers appeared in the January 1987 issue of Stereo Review (p. 78-84)

Robert Harley's book devotes 12 pages to ABX discussion. The January 1987 issue of Stereo Review published a "classic" ABX
test for amplifiers.

In a 1987 Stereo Review article by David L. Clark entitled "Do All Amplifiers Sound The Same?", an ABX comparator device was used to switch between amplifiers. The amplifiers tested were:

1. Pioneer SX-1500 receiver (transistor), $200.
2. Hafler DH-1 (transistor), $320.
3. NAD 2200 (transistor), $548.
4. Counterpoint SA-12 (tube-transistor hybrid), $995.
5. Mark Levinson ML-11 (transistor), $2,000.
6. Julius Futterman OTL-1 (tube), $12,000.

The gains of all amplifiers were equalized by attenuators in the tape monitor loop of an Audio Research SP-11 tube preamplifier. The listening panel consisted of both "skeptics" and "believers" and consisted of 25 persons. Skeptics and believers were not mixed during the same trial. The 11 ABX test sessions involved from 1 to 8 persons at a time as follows:

Two 1 person trials.
Two 2 person trials.
One 3 person trial.
Two 4 person trials.
One 5 person trial.
Three 8 person trials.

The size of the listening room was 25 x 18 with a 10 foot ceiling. The author did not specifically state the seating pattern of the panelists for the 4, 5 and 8 person trials. However, considering the size of the room, I assume that some type of multi-row seating pattern was used when 4 or more panelists were used (similar to the seating arrangement in the power cable and other ABX tests found in the audio literature).

A total of 772 trials were performed and the panelists identified the correct amplifier in 388 trials, which was a success rate of 50.3% and which was statistically similar to guessing.

This test indicated that there was no audible difference between any of the amplifiers. To reiterate, there was no perceived audible difference between a $200 transistor receiver and a $12,000 tube amplifier!

The author concluded:

"But for now, the evidence would seem to suggest that distinctive amplifier sounds, if they exist at all, are so minute that they form a poor basis for choosing one amplifier over another. Certainly there are still differences between amps, but we are unlikely to hear them."

Such Good Entertainment.:)

In Robert Harley's "Complete Guide to High-End Audio" (3rd Ed.), he provides a thorough discussion of the fallacies of the application of ABX tests to audio. Mr. Harley discusses ABX tests from a common sense and reasoning viewpoint rather than a scientific and mathematical one. He also provides several examples.

As I have said many times before, I do not believe that a test which mathematically converges to results similar to guesswork and which limits the amount of sensory information received by the listener is appropriate for sound stimuli.

punk-roc

Out of curiosity, when you say, "statistically similar to guessing", do the investigators calculate the statistical power (β) to make sure they have a large enough sample to detect a difference?

We run into β issues frequently in medical literature, wasn't sure if those calculations were done in these studies that showed there was no audible difference.

Jason

DarqueKnight

punk-roc wrote: »

Out of curiosity, when you say, "statistically similar to guessing", do the investigators calculate the statistical power (β) to make sure they have a large enough sample to detect a difference?

We run into β issues frequently in medical literature, wasn't sure if those calculations were done in these studies that showed there was no audible difference.

I have not seen beta error calculated in any of the ABX audio trials I have read. I am going to go way, way, way out on a limb and speculate that the beta error was not calculated or even considered.

As I mentioned here, high beta error (false negative result) due to small population size is another disadvantage of the ABX protocol.

However, I must concede that the ABX protocol works very, very well when you want to "prove" that there is no appreciable audible difference between a rinky-dink consumer grade Pioneer SX-1500 transistor receiver ($200/45 watts per channel ) and a pair of Julius Futterman OTL-1 monoblock tube power amplifiers ($12,000 per pair/90 watts per channel) made of premium parts.:)

Such Good "Science".

punk-roc

That was my thought, sorry i missed that you had brought that up before. Even the well-to-do, hard Sciences have trouble using actual scientific methods all lot of the time, so its not too surprising

Jason

Jetmaker737

Your soundstage map is simple yet brilliant... a good data-driven tool which any individual could use in evaluating/comparing their gear at home.

I'm curious about how your test subjects evaluate and record the attributes of the sounds. For me that is always the most tricky part, and very hard to make comparisons because it's hard to "fix" them in memory.

DarqueKnight

punk-roc wrote: »

That was my thought, sorry i missed that you had brought that up before.

It was a good point. It certainly didn't hurt to revisit it.:)

Jetmaker737 wrote: »

I'm curious about how your test subjects evaluate and record the attributes of the sounds. For me that is always the most tricky part, and very hard to make comparisons because it's hard to "fix" them in memory.

In the paper, I specifically discuss how to "fix" things in memory and how to evaluate sonic attributes.

cnh

DarqueKnight wrote: »

The protocol is for ANY listener, from totally inexperienced to professional audio reviewer.



An important part of the "questionnaire" would be the sound stage mapping process.

The paper concentrates on the home stereo listening experience, but the protocol is extensible (with modification) to any listening environment (eg. automobiles). The important thing is to simulate (and ideally replicate) the actual "normal" listening environment during the evaluation.



Yes. Of course...unless that is how you actually plan to listen to your audio system.:) Measurements have certainly have their place, but we still cannot determine how something will sound from the way it measures.

Thanks for the confirmation...look forward to reading the whole piece. The listening diagrams are a very nice IDEA!

cnh

cnh

reeltrouble1 wrote: »

Alas, I have been told by those who seek their truth's I am but a defective repository of self-senses, never to be believed, let alone trusted, still empiracally at the very least I am, and still have faith I can understand. Yet all what is, takes place within a living dimensionally complex system, ordered in constant flux events occuring with connected singularity to form a collected past which is assimilated into the collective whole of the person, the soul per say. One might endeavor to understand the being of self with equation with data, indeed, devote a lifetime of study to just the one let alone this duo of trio whose receptors are processing musical sound in a given moment, yet complete replication is like so much unobtanium, molecules set into motion cannot be recalled, cannot be reset, even the sheep dolly's become more indepedent as time and space move. Recollection of something musical possible yet born from the collective whole of an individual within a group likely influenced by each.

What looks to be may not be. Theory is fundamental to the order, invention to new order, so we seek because we are human, we listen because we must, afterall, we always have, listened that is.

RT1

Ted, I actually 'understand' a lot of the above! But WHO taught you to WRITE like that? Unfortunately, Andre Breton you are not. But you do have your 'own' style. Interesting points! And I agree.

"Anyone who cannot imagine a horse galloping on a tomato is an idiot."
Andre Breton (a surrealist and a great author).

cnh