Developing a technical ear training program: Part two - Establishing clear goals

• posted by Mark Bassett - 23 February 2019

Developers of technical ear training programs (TETPs) – we assume – want their training programs to be effective. Users of TETPs most definitely want their hours of practice on the training program to be effective, and an efficient use of their time – worthwhile. To establish the effectiveness of a technical ear training program (TETP), one needs to establish whether the goals of the training program have been met; if the goals of the program have been met, the training is successful. However, it’s not possible, of course, to establish whether the goals of a program have been met if the goals are not stated. As such, the goals of a TETP must be clearly stated at the outset. Training program goals must be published – provided to the user prior to undertaking the training program.

Not all TETPs state their goals, clearly, in advance, or at all. How then, do the developers of these programs know that their training programs are effective? The short answer is that they don't. Developers may rely on anecdotal evidence from users, which isn't invalid, but it's far from reliable, and most often based on users’ performance on the training program itself.

Although some developers of TETPs do clearly state the goals of their training programs, such as improving: sensitivity to timbral changes (Quesnel & Woszczyk, 1994; Rościszewska, 2011); ‘discrimination of sound attributes’ (Liu et al., 2007a, p. 1); ‘the ability to systematically discriminate and identify sonic differences’ (Kaniwa et al., 2011, pp. Introduction, Para 1); ‘memory for timbre and develop[ing] the ability of associating the perceived characteristics of timbre with the spectral properties of sounds’ (Rościszewska & Miśkiewicz, 2015, p. 1); and the ability to ‘discriminate between different sounds’ (Kawahara et al., 2013, p. 1), performance relating to the training goals is not directly measured; rather, the goals are assumed to have been met if students simply show an improvement in performance on the TETP (or a closely related task). This brings us to the second point; how developers establish if the goals of their training program have been met.

Not only should the goals of a TETP be clearly stated at the outset, but valid and reliable tests that can measure whether the goals have been met are required. It's not good enough to simply state, or more commonly, assume that the goals of a training program have been met, or will be met, if users complete the training program. In the majority of published studies presenting student-targeted TETP performance data, the training task itself, or a closely-related task utilising the same practised skills as the training task, is used as the measurement tool to ascertain whether the goals of the training program have been met. In these studies, an improvement in students’ performance on the training program over time, or an improvement on a closely-related task (often delivered as a pre/post-training test), is used as evidence of the training program’s effectiveness.

As I argued in my doctoral thesis, a legitimate test of whether a TETP’s goal has been met will use in that test neither the TETP employed to train students or a closely-related task. This is because a TETP is only useful as a test if seeking to show whether students have improved their ability to undertake the TETP itself, and the purpose of a TETP is not to get better at the TETP itself. Contrary to the prevailing paradigm in TETP research, the purpose of a TETP is not to simply improve one’s performance on the training program itself, as improvement in performance on a TETP does not necessarily provide the best evidence that the training program’s goals have been met. But if measuring performance on the TETP itself is not a valid measure of the effectiveness of the TETP, then how can developers establish the effectiveness of their TETPs? By using tests that measure the transfer of developed critical listening skills outside of the TETP itself.

Several researchers in the field acknowledge this argument and advocate for an external, independent test of skill transfer outside of a TETP. In the earliest published paper detailing the Timbre Solfeggio TETP, Rakowski and Trybuła (1975) suggest a positive correlation exists between performance on a critical listening entrance exam and students’ subsequent ‘efficiency as recording specialists’, and assume a potential student’s ability to detect distortions in a 2AFC test is positively correlated to their professional skill (p. 3). However, the authors caution that to prove such an assumption it would be necessary to estimate the validity of the test using a well-established criterion of Tonmeisters [sic] professional efficiency’ (p. 3). Quesnel (1990) also suggested that ‘a true measure of the validity of such a [technical ear training] system is achieved only if improvements in the exercises can also be measured in the recording studio’ (p. 105). In reference to training participants for professional listening evaluations in the automobile industry, Shively and House (1998) also suggested that based on their experience, listeners achieving consistently high scores on a resonance training task ‘does not mean that they are well trained for listening evaluation purposes’ (p. 2). Furthermore, the assertion that performance on the TETP is insufficient to indicate whether the goals of a TETP have been met is supported by several authors (Indelicato, Hochgraf, & Kim, 2014; Kim, 2015; Liu et al., 2007a; Quesnel & Woszczyk, 1994; Rościszewska, 2011; Rościszewska & Miśkiewicz, 2014, 2015).

Regardless of the support for this approach, there are no published studies that seek to establish the effectiveness of student-targeted TETPs using evidence other than an improvement on the training program itself, or a similar related task.

Researchers in the field may at this stage point to Quesnel who concluded that his TETP was effective because students that trained using his program outperformed professional engineers on a critical listening test. Quesnel’s study, which compared the performance of students on a critical listening test to that of professional engineers, is one of the most frequently cited in the literature. Quesnel (2001) argued that the TETP was effective as it had trained students on a task that ‘did involve listening skills that professional audio engineers, in the absence of specific training, develop throughout their career’ (pp. 69- 70). Therefore, Quesnel hypothesised that the professional engineers would outperform students on the test. However, upon reviewing Quesnel’s test, it becomes apparent that summarising the results as ‘student subjects performed significantly better’ is inadequate (Quesnel, 2001, p. 96; Rościszewska & Miśkiewicz, 2015, p. 6). Indeed, there are several reasons unrelated to the development of a ‘memory for a set of octave resonance references’, and aside from the small sample size, which may explain why the students outperformed professional audio engineers on the test (Quesnel, 2001, p. 45).

The primary reason for this is that the students were familiar with many aspects of the test, largely because they had been trained to perform an identical task using identical software in an identical room during the Timbral Ear Trainer TETP. This provided students with a significant advantage over the professional audio engineers that was unrelated to any skill the students may have developed in identifying the properties of spectral irregularities introduced by a parametric equaliser. In addition, the test interface featured vowel buttons that Quesnel (2001) instructed participants ‘to use… only if they were familiar with the relationship between vowels and resonance frequencies’ (p. 91). Vowel buttons do not exist on professional audio equipment and are not used in commercial audio production. It is unsurprising, therefore, that every professional audio engineer stated that ‘they were not familiar with the concept’ of vowels as they pertain to equalisation (pp. 91–92). The inclusion of these vowel labels not only makes the simulated task less like one ‘common in sound recording practice’ (2001, p. 73), but probably further biased the comparison in favour of the students.

I developed three criteria to establish the minimum requirements for TETP goals if a TETP is to be assessed for effectiveness; they must be professionally relevant; researchers must measure performance relating to the goals; and this performance must be measured separately from the TETP, on a task that students have not previously trained with. The next post in this series will present the goals of the TETP that I am developing and state how I am planning to measure users’ performance relating to these goals separately from the TETP, on a task that students have not previously trained with.