From 61514c5bbd16a888250300616ad9311a262d24f0 Mon Sep 17 00:00:00 2001 From: WillyPillow Date: Fri, 15 Jun 2018 17:26:19 +0800 Subject: [PATCH] Add "Repeat" and "GroupRepeated" options --- README.html | 374 ++++++++++++++++++++++++++++----------------------- README.md | 2 + js/beaqle.js | 58 ++++++-- 3 files changed, 253 insertions(+), 181 deletions(-) diff --git a/README.html b/README.html index eb53674..d0d28b5 100644 --- a/README.html +++ b/README.html @@ -1,55 +1,91 @@ - - + + - - + - - + + README + + - +

Table of contents

-
    +
    1. Description
    2. Basic Setup
    3. Test Configuration
      4. @@ -69,14 +105,14 @@

      1. Description

      BeaqleJS has been presented at the Linux Audio Conference 2014 at the ZKM in Karlsruhe, Germany. The original paper and presentation slides are available from the conference archive. However, meanwhile most of the paper content has been updated and merged into this documentation.

      If you want to cite BeaqleJS in a publication please use

      -

      S. Kraft, U. Zölzer: "BeaqleJS: HTML5 and JavaScript based Framework for the Subjective Evaluation of Audio Quality", Linux Audio Conference, 2014, Karlsruhe, Germany

      +

      S. Kraft, U. Zölzer: “BeaqleJS: HTML5 and JavaScript based Framework for the Subjective Evaluation of Audio Quality”, Linux Audio Conference, 2014, Karlsruhe, Germany

      as a reference or link to our GitHub repository

      https://github.com/HSU-ANT/beaqlejs

      2. Basic Setup

      -
        +
        1. Download the test scripts

        2. Uncomment the line where the desired test class is initialized and loaded in the header of the index.html file.

          -
          <script type="text/javascript">
-  var testHandle;
-  window.onload=function() {
-    // Uncomment one of the following lines to choose the desired test class
-    //testHandle = new MushraTest(TestConfig);  // <= MUSHRA test class
-    //testHandle = new AbxTest(TestConfig);     // <= ABX test class
-  };
-</script>
          3. +

        3. Prepare a config file and set its path in the prepared <script></script> tag in the header of the index.html file.

          -
          <!-- load the test config file -->
-  <script src="config/YOUR_CONFIG.js" type="text/javascript"></script>
-<!---->
          +

          Two example config files for the MUSHRA and ABX test class are already supplied in the config/ folder to serve as a starting point. Detailed information about the different test classes and configuration can be found below.

        2.1 Docker Webserver

        @@ -130,66 +166,71 @@

        2.1.2 Docker Compose

        3. Test Configuration

        3.1 General Options

        The available options can be divided into a set of general options which apply to all test classes and other options, including file declarations, that are specific for a single test class.

        -
        var TestConfig = {
-  "TestName": "My Listening Test",   // <=  Name of the test
-  "LoopByDefault": true,             // <=  Enable looped playback by default
-  "AutoReturnByDefault": true,       // <=  Always start playback from loop/track begin
-  "ShowFileIDs": false,              // <=  Show file IDs for debugging (never
-                                     //     enable this during real test!)
-  "ShowResults": false,              // <=  Show table with test results at the end
-  "EnableABLoop": true,              // <=  Show controls to loop playback with an
-                                     //     AB range slider
-  "EnableOnlineSubmission": false,   // <=  Enable transmission of JSON encoded
-                                     //     results to a web service
-  "BeaqleServiceURL": "",            // <=  URL of the web service
-  "SupervisorContact": "",           // <=  Email address of supervisor to contact for
-                                     //     help or for submission of results by email
-  "RandomizeTestOrder": true,        // <=  Present test sets in a random order
-  "MaxTestsPerRun": -1,              // <=  Only run a random subset of all available
-                                     //     tests, set to -1 to disable
-  "Testsets": [ {...}, {...}, ... ], // <=  Definition of test sets and files, more
-                                     //     details below
-}
        +

        3.2 ABX

        In an ABX test three items named A, B and X are presented to the listener, whereas X is randomly selected to be either the same as A or B. The listener has to identify which item is hidden behind X, or which one (A or B) is closest to X. If the listener is able to find the correct item, it reveals that there are perceptual differences between A and B.

        A typical application of ABX tests would be the evaluation of the transparency of audio codecs. For example item A could be an unencoded audio snippet and B is the same snippet but encoded with a lossy codec. When the listener is not able to identify if A or B was hidden in X (results are randomly distributed), one can assume that the audio coding was transparent

        -
        ...                                  // <=  General options
-"Testsets": [
-  { "Name": "Schubert",              // <=  Name of the test set
-    "TestID": "id1_1",               // <=  Unique test set ID, necessary for internal
-    "Files": {                       // <=  Array with test files
-      "A": "audio/schubert_ref.wav", // <=  File A
-      "B": "audio/schubert_2.wav",   // <=  File B
-      }
-  },
-  { ... },                           // <=  Next test set starts here
-  ....
-]
        -

        3.3 MUSHRA

        + +

        3.3 Preference Testing

        +

        A preference test (or A/B test) is a simplified version of ABX in which the listener is to decide which of two stimuli sounds better. A typical application of preference tests would be the comparison of different speech synthesis methods. Over a range of stimuli and listeners, one method (A or B) might be shown to be better than the other (one might use a sign test for comparison of results for A and B). Configuration is identical to the ABX test; however, no X is presented to the user.

        +

        3.4 MUSHRA

        In a MUSHRA test (ITU-R BS.1116-1) the listener gets presented an item marked as reference together with several anonymous test items. By using a slider for each test item he has to rate how close the items are to the reference on top. Among the test items there is usually also one hidden reference and one, or several, anchor signals to prove the validity of the ratings and the qualification of the participants.

        Contrary to ABX tests the MUSHRA procedure allows more detailed evaluations as it is possible to compare more than one algorithm to a reference. Furthermore, the results are on a continuous scale allowing a direct numerical comparison of all algorithms under test.

        -
        ...                                   // <=  General options
-"RateMinValue": 0,                    // <=  Minimum rating
-"RateMaxValue": 100,                  // <=  Maximum rating
-"RateDefaultValue":0,                 // <=  Default rating
-"RequireMaxRating": false,            // <=  At least one of the ratings in a testset
-                                      //     has to be at the maximum value
-"Testsets": [
-  { "Name": "Schubert 1",             // <=  Name of the test set
-    "TestID": "id1_1",                // <=  Unique test set ID, necessary for
-                                      //     internal referencing
-    "Files": {                        // <=  Array with test files
-      "Reference": "audio/ref.wav",   // <=  Every MUSHRA test set needs exactly
-                                      //     one(!) file with a "Reference" label
-      "label_1": "audio/algo_1.wav",  // <=  Various files to be tested, the labels
-      "label_2": "audio/algo_2.wav",  //     can be freely chosen as desired but
-      "label_3": "audio/algo_3.wav",  //     have to be unique inside a test set
-      "anchor": "audio/algo_anc.wav", //      ...
-      }
-  },
-  { ... },                            // <=  Next test set starts here
-  ....
-]
        +

        4. Browser Support

        BeaqleJS in general will run well in any recent web browser out in the wild. The only noteworthy exceptions are the Internet Explorer versions below 9 which still have a market share of a few percent and unfortunately miss the required FileAPI. Participants will get a warning if they open the listening test with one of these old versions.

        4.1 Required HTML5 Features

        @@ -269,7 +310,7 @@

        4.2 Codecs

        5. Online Submission

        BeaqleJS can send the test results in JSON format to a web service to collect them in a central place. An exemplary server side PHP script which can be used to receive and store the results is included in the web_service/ subfolder. It only requires a webspace with PHP >= 5.6.

        5.1 Setup

        -
          +

          1. Upload the file web_service/beaqleJS_Service.php to a webserver. Create a folder named results/ next to the PHP script and make sure that the webserver has write permissions on it.

          2. Try to execute the script in your browser. For example, point your browser to http://yourdomain.com/mysubfolder/beaqleJS_Service.php. The script performs a self-test and checks PHP version and write permission of the results/ folder.

          3. Enable online submission in the BeaqleJS config ("EnableOnlineSubmission": true) and set the BeaqleServiceURL to http://yourdomain.com/mysubfolder/beaqleJS_Service.php.

            4. @@ -282,67 +323,66 @@

            5.2 Security

          4. File names in the results/ folder contain a random string, so it is not possible to access the submitted data without listing the whole directory

            5. 6. Internals

            -
            -BeaqleJS functional blocks -

            BeaqleJS functional blocks

            -
            +
            +BeaqleJS functional blocks
            BeaqleJS functional blocks
            +

            The general structure of BeaqleJS can be divided in three blocks as visualised in the diagram above. There is a common HTML5 index.html file to hold the main HTML structure with some basic place holder blocks whose content will be dynamically created by the JavaScript backend. The styling is completely independent and done with the help of cascading style sheets (CSS). Style sheets, config files and all necessary JavaScript libraries are loaded in the header of the index.html. Most of the descriptive text, like introduction and instructions, are placed in hidden blocks inside this file and their visibility is controlled by the scripts. For the user interface and to simplifiy Document Object Model (DOM) manipulations, the well known jQuery and jQueryUI libraries are used.

            The JavaScript backend consists of two main classes. The first one is the AudioPool which takes care of audio playback and buffering. It pools a set of HTML5 <audio>-tags in a certain AudioPool <div>-tag. There are simple functions to add and load a new file, connect and address it with an ID, manage playback and looping as well as synchronized pause and stop operations.

            -

            The ListeningTest class provides the main functions of an abstract listening test. This includes the setup and management of basic playback controls (play, pause, looping, time line display, ...), reading of the test configuration as well as storage of the results and also main control over the test sequence.

            +

            The ListeningTest class provides the main functions of an abstract listening test. This includes the setup and management of basic playback controls (play, pause, looping, time line display, …), reading of the test configuration as well as storage of the results and also main control over the test sequence.

            To create a certain test type the abstract ListeningTest class is inherited and specific functions for the actual arrangement of test items or storage and evaluation of the results need to be implemented. Based on this modular approach it is very easy to extend the framework with additional test types or to create variants of existing ones without the need to reimplement all the necessary basics.

            If the test is performed distributed over the internet or on several local computers, the ListeningTest main class is also able to send the final ratings to a web service for centralised collection and evaluation.

            6.1 Creating a new test class

            A new test class has to inherit the base functionality from the main ListeningTest class. Inheritance in JavaScript is achieved by prototypes and this means to define a new class MyTest and then set its prototype to the base class. As this overwrites the constructor it has to be reset to the child constructor afterwards:

            -
            // inherit from ListeningTest
-function MyTest(TestData) {
-  ListeningTest.apply(this, arguments);
-}
-MyTest.prototype = new ListeningTest();
-MyTest.prototype.constructor = MyTest;
-
-// implement the necessary functions
-MyTest.prototype.createTestDOM = ...
-MyTest.prototype.saveRatings   = ...
-MyTest.prototype.readRatings   = ...
-MyTest.prototype.formatResults = ...
            +

            The child class can access the TestState and the TestConfig structures from the parent:

            -
            ListeningTest.TestState = {
-    // main public members
-    'CurrentTest': -1,
-    'TestIsRunning': false,
-    'FileMappings': {},
-    'Ratings': {},
-    'EvalResults': {},
-    // ...
-    // optionally add own fields
-    // ...
-}
-
-ListeningTest.TestConfig = {
-    "TestName": "Test",
-    "LoopByDefault": true,
-    "EnableABLoop": true,
-    "EnableOnlineSubmission": false,
-    "BeaqleServiceURL": "http://...",
-    "SupervisorContact": "super@visor.com",
-    "Testsets": [
-        {
-            "Name": "Testset 1",
-            "Files": {
-                // ...
-            }
-        },
-        {
-            "Name": "Testset 2",
-            "Files": {
-                // ...
-            }
-        }
-        ],
-    // ...
-    // further test specific settings
-    // ...
-}
            +

            The TestState should be used to store random file mappings, ratings and other status variables, but can also be dynamically expanded with specific fields required by the child class. The TestConfig structure is just a mapping of the BeaqleJS config file into the class namespace. It has to contain at least the fields and structure as given above but it is possible to add additional sections which are required by the child class.

            Every new test class has to implement at least four new functions:

              diff --git a/README.md b/README.md index 58afd7d..653901d 100644 --- a/README.md +++ b/README.md @@ -146,6 +146,8 @@ var TestConfig = { "MaxTestsPerRun": -1, // <= Only run a random subset of all available // tests, set to -1 to disable "AudioRoot": "", // <= path to prepend to all audio URLs in the testsets + "Repeat": 1, // <= Each testset will be repeated x times + "GroupRepeated": true, // <= If set to false, automatically replicated testsets gets split apart when randomization is enabled "Testsets": [ {...}, {...}, ... ], // <= Definition of test sets and files, more // details below } diff --git a/js/beaqle.js b/js/beaqle.js index 2a19949..5058c63 100644 --- a/js/beaqle.js +++ b/js/beaqle.js @@ -418,6 +418,14 @@ $.extend({ alert: function (message, title) { this.TestConfig = TestData; this.setDefaults(this.TestConfig); + var tmp = Array(); + for (var i = 0; i < this.TestConfig.Testsets.length; i++) { + for (var j = 0; j < this.TestConfig.Repeat; j++) { + tmp.push(this.TestConfig.Testsets[i]); + } + } + this.TestConfig.Testsets = tmp; + // some state variables this.TestState = { "CurrentTest": -1, // the current test index @@ -513,17 +521,19 @@ $.extend({ alert: function (message, title) { // ################################################################### ListeningTest.prototype.setDefaults = function(config) { var defaults = { - "ShowFileIDs": false, - "ShowResults": false, - "LoopByDefault": true, - "AutoReturnByDefault": true, - "EnableABLoop": true, - "EnableOnlineSubmission": false, - "BeaqleServiceURL": "", - "SupervisorContact": "", - "RandomizeTestOrder": false, - "MaxTestsPerRun": -1, - "AudioRoot": "" + "ShowFileIDs": false, + "ShowResults": false, + "LoopByDefault": true, + "AutoReturnByDefault": true, + "EnableABLoop": true, + "EnableOnlineSubmission": false, + "BeaqleServiceURL": "", + "SupervisorContact": "", + "RandomizeTestOrder": false, + "MaxTestsPerRun": -1, + "AudioRoot": "", + "Repeat": 1, + "GroupRepeated": true } for (var property in defaults) { @@ -609,21 +619,41 @@ $.extend({ alert: function (message, title) { // ################################################################### ListeningTest.prototype.startTests = function() { - + + // if `GroupRepeated` is enabled, only the list of identical testsets are shuffled + // each testset is then duplicated `Repeat` times + const len = this.TestConfig.GroupRepeated ? + this.TestConfig.Testsets.length / this.TestConfig.Repeat : + this.TestConfig.Testsets.length; + const maxTake = this.TestConfig.GroupRepeated ? + Math.floor(this.TestConfig.MaxTestsPerRun / this.TestConfig.Repeat) : + this.TestConfig.MaxTestsPerRun; + // init linear test sequence this.TestState.TestSequence = Array(); - for (var i = 0; i < this.TestConfig.Testsets.length; i++) + for (var i = 0; i < len; i++) this.TestState.TestSequence[i] = i; // shorten and/or shuffle the sequence if ((this.TestConfig.MaxTestsPerRun > 0) && (this.TestConfig.MaxTestsPerRun < this.TestConfig.Testsets.length)) { this.TestConfig.RandomizeTestOrder = true; this.TestState.TestSequence = shuffleArray(this.TestState.TestSequence); - this.TestState.TestSequence = this.TestState.TestSequence.slice(0, this.TestConfig.MaxTestsPerRun); + this.TestState.TestSequence = this.TestState.TestSequence.slice(0, maxTake); } else if (this.TestConfig.RandomizeTestOrder == true) { this.TestState.TestSequence = shuffleArray(this.TestState.TestSequence); } + if (this.TestConfig.GroupRepeated) { + // repeat each element + var tmp = Array(); + for (var i = 0; i < this.TestState.TestSequence.length; i++) { + for (var j = 0; j < this.TestConfig.Repeat; j++) { + tmp.push(this.TestState.TestSequence[i] * this.TestConfig.Repeat + j); + } + } + this.TestState.TestSequence = tmp; + } + this.TestState.Ratings = Array(this.TestConfig.Testsets.length); this.TestState.Runtime = new Uint32Array(this.TestConfig.Testsets.length); // this.TestState.Runtime.forEach(function(element, index, array){array[index] = 0});