generalize to a portion of recordings

Author: bjarthur

README.md

@@ -1092,24 +1092,31 @@ file.
 ## Measuring Generalization ##
 
 Up to this point we have validated on a small portion of each recording.  Once
-you have annotated many recordings though, it is good to set aside entire WAV
-files to validate on.  In this way we measure the classifier's ability to
+you have annotated many recordings though, it is good to set aside entire
+recordings to validate on.  In this way we measure the classifier's ability to
 extrapolate to different microphones, individuals, or whatever other
 characteristics that are unique to the withheld recordings.
 
 To train one classifier with a single recording or set of recordings withheld
 for validation, first click on `Generalize` and then `Omit All`.  Use the `File
-Browser` to either select (1) specific WAV file(s), (2) a text file
-containing a list of WAV file(s) (either comma separated or one per line), or
-(3) the `Ground Truth` folder or a subdirectory therein.  Finally press the
-`Validation Files` button and `DoIt!`.
+Browser` to either select (1) specific WAV file(s), (2) a text file containing
+a list of WAV file(s) (either comma separated or one per line), or (3) a
+subdirectory within the `Ground Truth` folder.  Finally press the `Validation
+Files` button and `DoIt!`.
 
 To train multiple classifiers, each of which withholds a single recording in a
 set you specify, click on `Omit One`.  Select the set as described above for
 `Omit All`.  The `DoIt!` button will then iteratively launch a job for each WAV
 file that has been selected, storing the result in the same `Logs Folder` but in
-separate files and subdirectories that are suffixed with the letter "w".  Of
-course, training multiple classifiers is quickest when done simultaneously
+separate files and subdirectories that are suffixed with the letter "w".
+
+To train multiple classifiers, each of which withholds a portion of the
+recordings in the set you specify, click on `Omit Some`.  Select the set as
+before, and specify the number of partitions using `k-fold`.  For example, if
+`k-fold` is "4", then four models will be trained, each with a different fourth
+of the recordings in the chosen set withheld.
+
+Of course, training multiple classifiers is quickest when done simultaneously
 instead of sequentially.  If your model is small, you might be able to fit
 multiple on a single GPU (see the `models_per_job` variable in
 "configuration.py").  Otherwise, you'll need a machine with multiple GPUs,
@@ -1268,20 +1275,22 @@ To perform a simple grid search for the optimal value of a particular
 hyperparameter, first choose how many folds you want to partition your
 ground-truth data into using `k-fold`.  More folds permit characterizing the
 variance better, but take longer to train and also result in fewer annotations
-to measure the accuracy.  Ensure that you have at least 10 annotations for each
-label in the validation set if using many folds.  Then set the hyperparameter
-of interest to the first value you want to optimize and use the name of the
-hyperparameter and it's value as the `Logs Folder` (e.g. "mb64" for a
-mini-batch size of 64).  Suffix any additional hyperparameters of interest
-using underscores (e.g. "mb64_ks129_fm64" for a kernel size of 129 and 64
-feature maps).  If your model is small, use `models_per_job` in
-"configuration.py" to train multiple folds on a GPU.  Click the `X-Validate`
-button and then `DoIt!`.  One classifier will be trained for each fold, using
-it as the validation set and the remaining folds for training.  Separate files
-and subdirectories are created in the `Logs Folder` that are suffixed by the
-fold number and the letter "k".  Plot training curves with the `Accuracy`
-button, as before.  Repeat the above procedure for each of remaining
-hyperparameter values you want to try (e.g. "mb128_ks129_fm64",
+to measure the accuracy.  Unlike `Omit Some` in [Measuring
+Generalization](#measuring-generalization) above, the partitions here group
+individual annotations, as opposed to entire recordings.  Ensure that you have
+at least 10 annotations for each label in the validation set if using many
+folds.  Then set the hyperparameter of interest to the first value you want to
+optimize and use the name of the hyperparameter and it's value as the `Logs
+Folder` (e.g. "mb64" for a mini-batch size of 64).  Suffix any additional
+hyperparameters of interest using underscores (e.g.  "mb64_ks129_fm64" for a
+kernel size of 129 and 64 feature maps).  If your model is small, use
+`models_per_job` in "configuration.py" to train multiple folds on a GPU.  Click
+the `X-Validate` button and then `DoIt!`.  One classifier will be trained for
+each fold, using it as the validation set and the remaining folds for training.
+Separate files and subdirectories are created in the `Logs Folder` that are
+suffixed by the fold number and the letter "k".  Plot training curves with the
+`Accuracy` button, as before.  Repeat the above procedure for each of remaining
+hyperparameter values you want to try (e.g.  "mb128_ks129_fm64",
 "mb256_ks129_fm64", etc.).  Then use the `Compare` button to create a figure of
 the cross-validation data over the hyperparameter values, specifying for the
 `Logs Folder` the independent variable (e.g. "mb") suffixed with the fixed

src/gui/controller.py

@@ -844,7 +844,7 @@ def remaining_callback():
     V.context_update()
 
 def action_callback(thisaction, thisactuate):
-    M.action=None if M.action is thisaction else thisaction
+    M.action=None if M.action is thisaction and M.action is not V.leaveout else thisaction
     M.function=thisactuate
     V.buttons_update()
 
@@ -1272,11 +1272,16 @@ def generalize_xvalidate_succeeded(kind, logdir, currtime):
             return False
     return True
 
-async def leaveout_actuate(comma):
+async def leaveout_actuate(kind):
     test_files = _validation_test_files(V.test_files.value)[0]
     validation_files = list(filter(
             lambda x: not any([y!='' and y in x for y in test_files.split(',')]),
-            _validation_test_files(V.validation_files.value, comma)))
+            _validation_test_files(V.validation_files.value, False)))
+    if kind=="omit all":
+        validation_files = [','.join(validation_files)]
+    elif kind=="omit some":
+        k = int(V.kfold.value)
+        validation_files = [','.join(validation_files[x::k]) for x in range(k)]
     currtime = time.time()
     jobids = []
     os.makedirs(V.logs_folder.value, exist_ok=True)

src/gui/main.py

@@ -91,7 +91,7 @@
                 V.fixfalsenegatives, V.generalize, V.tunehyperparameters,
                 V.examineerrors, V.testdensely, V.findnovellabels,
                 V.doit, width=M.gui_width_pix),
-            row(V.detect, V.misses, V.train, V.leaveoneout, V.leaveallout, V.xvalidate,
+            row(V.detect, V.misses, V.train, V.leaveout, V.xvalidate,
                 V.mistakes, V.activations, V.cluster, V.visualize, V.accuracy, V.freeze,
                 V.ensemble, V.classify, V.ethogram, V.compare, V.congruence,
                 width=M.gui_width_pix),

src/gui/model.py

@@ -233,7 +233,7 @@ def next_pow2_sec(x_sec):
     return False, x_sec
 
 def init(_bokeh_document, _configuration_file, _use_aitch):
-    global changed_style
+    global changed_style, default_style, primary_style
     global bokeh_document, configuration_file, bindirs, repodir, srcdir
     global use_aitch, local_current_job, server_current_job
     global audio_tic_rate, audio_nchannels, video_channels
@@ -260,6 +260,8 @@ def init(_bokeh_document, _configuration_file, _use_aitch):
     exec(open(_configuration_file).read(), globals())
 
     changed_style = [".bk-input { background-color: #FFA500; }"]
+    default_style = [".bk-input { background-color: #FFFFFF; color: #000000; font-size: 12px; }"]
+    primary_style = [".bk-input { background-color: #428BCA; color: #FFFFFF; font-size: 12px; }"]
 
     time_units = gui_time_units
     time_scale = gui_time_scale

src/gui/view.py

@@ -1302,7 +1302,10 @@ def buttons_update():
     for button in wizard_buttons:
         button.button_type="success" if button==M.wizard else "default"
     for button in action_buttons:
-        button.button_type="primary" if button==M.action else "default"
+        if button == leaveout:
+            button.stylesheets = M.primary_style if button==M.action else M.default_style
+        else:
+            button.button_type="primary" if button==M.action else "default"
         button.disabled = button not in wizard2actions[M.wizard]
     if M.action in [detect,classify,ethogram]:
         wavcsv_files_button.label='wav files:'
@@ -1349,6 +1352,8 @@ def buttons_update():
                     textinput.disabled = cluster_parameters[thislogic[0]].value not in thislogic[1]
                 else:
                     textinput.disabled = False
+            elif M.action is leaveout and textinput is kfold:
+                textinput.disabled = leaveout.value != "omit some"
             else:
                 textinput.disabled=False
             if textinput.disabled==False and textinput.value=='':
@@ -1533,7 +1538,7 @@ def init(_bokeh_document):
     global load_multimedia, play, video_slider, load_multimedia_callback, play_callback, video_slider_callback, video_toggle, video_div
     global undo, redo, remaining
     global recordings
-    global detect, misses, train, leaveoneout, leaveallout, xvalidate, mistakes, activations, cluster, visualize, accuracy, freeze, ensemble, classify, ethogram, compare, congruence
+    global detect, misses, train, leaveout, xvalidate, mistakes, activations, cluster, visualize, accuracy, freeze, ensemble, classify, ethogram, compare, congruence
     global status_ticker, waitfor, deletefailures
     global file_dialog_source, configuration_contents
     global logs_folder_button, logs_folder, model_file_button, model_file, wavcsv_files_button, wavcsv_files, groundtruth_folder_button, groundtruth_folder, validation_files_button, test_files_button, validation_files, test_files, labels_touse_button, labels_touse, kinds_touse_button, kinds_touse, prevalences_button, prevalences, delete_ckpts, copy, labelsounds, makepredictions, fixfalsepositives, fixfalsenegatives, generalize, tunehyperparameters, findnovellabels, examineerrors, testdensely, doit, nsteps, restore_from, save_and_validate_period, validate_percentage, mini_batch, kfold, activations_equalize_ratio, activations_max_sounds, cluster_these_layers, precision_recall_ratios, congruence_portion, congruence_convolve, congruence_measure, context, shiftby, optimizer, loss, learning_rate, nreplicates, batch_seed, weights_seed, augment_volume, augment_noise, augment_dc, augment_reverse, augment_invert, file_dialog_string, file_dialog_table, readme_contents, model_summary, labelcounts, wizard_buttons, action_buttons, parameter_buttons, parameter_textinputs, wizard2actions, action2parameterbuttons, action2parametertextinputs, status_ticker_update, status_ticker_pre, status_ticker_post
@@ -1952,7 +1957,7 @@ def init(_bokeh_document):
     remaining = Button(label='add remaining', disabled=True)
     remaining.on_click(C.remaining_callback)
 
-    recordings = Select(title="recording:", height=50)
+    recordings = Select(title="recording:", height=48)
     recordings.on_change('value', C.recordings_callback)
 
     detect = Button(label='detect', width_policy='fit')
@@ -1964,13 +1969,11 @@ def init(_bokeh_document):
     train = Button(label='train', width_policy='fit')
     train.on_click(lambda: C.action_callback(train, C.train_actuate))
 
-    leaveoneout = Button(label='omit one', width_policy='fit')
-    leaveoneout.on_click(lambda: C.action_callback(leaveoneout,
-                                                   lambda: C.leaveout_actuate(False)))
-
-    leaveallout = Button(label='omit all', width_policy='fit')
-    leaveallout.on_click(lambda: C.action_callback(leaveallout,
-                                                   lambda: C.leaveout_actuate(True)))
+    leaveout = Select(title="", value="omit one", margin = 4,
+                      options=["omit one", "omit some", "omit all"],
+                      width_policy='fit', stylesheets=M.default_style)
+    leaveout.on_change('value',
+                       lambda a,o,n: C.action_callback(leaveout, lambda: C.leaveout_actuate(n)))
 
     xvalidate = Button(label='x-validate', width_policy='fit')
     xvalidate.on_click(lambda: C.action_callback(xvalidate, C.xvalidate_actuate))
@@ -2146,7 +2149,7 @@ def init(_bokeh_document):
                                                disabled=False, sizing_mode='stretch_width')
     precision_recall_ratios.on_change('value', lambda a,o,n: C.generic_parameters_callback(n))
 
-    congruence_portion = Select(title="portion", height=50,
+    congruence_portion = Select(title="portion", height=48,
                                 value=M.state['congruence_portion'],
                                 options=["union", "intersection"],
                                 sizing_mode='stretch_width')
@@ -2157,7 +2160,7 @@ def init(_bokeh_document):
                                     disabled=False, sizing_mode='stretch_width')
     congruence_convolve.on_change('value', lambda a,o,n: C.generic_parameters_callback(n))
 
-    congruence_measure = Select(title="measure", height=50,
+    congruence_measure = Select(title="measure", height=48,
                                 value=M.state['congruence_measure'],
                                 options=["label", "tic", "both"],
                                 sizing_mode='stretch_width')
@@ -2171,12 +2174,12 @@ def init(_bokeh_document):
                         disabled=False, sizing_mode='stretch_width')
     shiftby.on_change('value', lambda a,o,n: C.generic_parameters_callback(n))
 
-    optimizer = Select(title="optimizer", height=50, value=M.state['optimizer'],
+    optimizer = Select(title="optimizer", height=48, value=M.state['optimizer'],
                        options=["Adadelta", "Adagrad", "Adam", "Adamax", "Ftrl", "Nadam", "RMSProp", "SGD"],
                        sizing_mode="stretch_width")
     optimizer.on_change('value', lambda a,o,n: C.generic_parameters_callback(''))
 
-    loss = Select(title="loss", height=50, value=M.state['loss'],
+    loss = Select(title="loss", height=48, value=M.state['loss'],
                   options=["exclusive", "overlapped", "autoencoder"],
                   sizing_mode="stretch_width")
     loss.on_change('value', lambda a,o,n: C.generic_parameters_callback(''))
@@ -2207,7 +2210,7 @@ def parse_plugin_parameters(Mparameters, width, msu=False):
                 thisparameter = Select(value=M.state[parameter[0]],
                                        title=parameter[1],
                                        options=parameter[2],
-                                       height=50,
+                                       height=48,
                                        sizing_mode='stretch_width')
             thisparameter.on_change('value', get_callback(parameter[6], msu))
             parameters[parameter[0]] = thisparameter
@@ -2332,8 +2335,7 @@ def parse_plugin_parameters(Mparameters, width, msu=False):
     action_buttons = set([
         detect,
         train,
-        leaveoneout,
-        leaveallout,
+        leaveout,
         xvalidate,
         mistakes,
         activations,
@@ -2409,7 +2411,7 @@ def parse_plugin_parameters(Mparameters, width, msu=False):
             makepredictions: [train, accuracy, freeze, classify, ethogram, delete_ckpts],
             fixfalsepositives: [activations, cluster, visualize, delete_ckpts],
             fixfalsenegatives: [detect, misses, activations, cluster, visualize, delete_ckpts],
-            generalize: [leaveoneout, leaveallout, accuracy, delete_ckpts],
+            generalize: [leaveout, accuracy, delete_ckpts],
             tunehyperparameters: [xvalidate, accuracy, compare, delete_ckpts],
             findnovellabels: [detect, train, activations, cluster, visualize, delete_ckpts],
             examineerrors: [detect, mistakes, activations, cluster, visualize, delete_ckpts],
@@ -2419,8 +2421,7 @@ def parse_plugin_parameters(Mparameters, width, msu=False):
     action2parameterbuttons = {
             detect: [wavcsv_files_button],
             train: [logs_folder_button, groundtruth_folder_button, labels_touse_button, test_files_button, kinds_touse_button],
-            leaveoneout: [logs_folder_button, groundtruth_folder_button, validation_files_button, test_files_button, labels_touse_button, kinds_touse_button],
-            leaveallout: [logs_folder_button, groundtruth_folder_button, validation_files_button, test_files_button, labels_touse_button, kinds_touse_button],
+            leaveout: [logs_folder_button, groundtruth_folder_button, validation_files_button, test_files_button, labels_touse_button, kinds_touse_button],
             xvalidate: [logs_folder_button, groundtruth_folder_button, test_files_button, labels_touse_button, kinds_touse_button],
             mistakes: [groundtruth_folder_button],
             activations: [logs_folder_button, model_file_button, groundtruth_folder_button, labels_touse_button, kinds_touse_button],
@@ -2440,8 +2441,7 @@ def parse_plugin_parameters(Mparameters, width, msu=False):
     action2parametertextinputs = {
             detect: [wavcsv_files] + list(detect_parameters.values()),
             train: [context, shiftby, optimizer, loss, learning_rate, nreplicates, batch_seed, weights_seed, augment_volume, augment_noise, augment_dc, augment_reverse, augment_invert, logs_folder, groundtruth_folder, test_files, labels_touse, kinds_touse, nsteps, restore_from, save_and_validate_period, validate_percentage, mini_batch] + list(model_parameters.values()),
-            leaveoneout: [context, shiftby, optimizer, loss, learning_rate, batch_seed, weights_seed, augment_volume, augment_noise, augment_dc, augment_reverse, augment_invert, logs_folder, groundtruth_folder, validation_files, test_files, labels_touse, kinds_touse, nsteps, restore_from, save_and_validate_period, mini_batch] + list(model_parameters.values()),
-            leaveallout: [context, shiftby, optimizer, loss, learning_rate, batch_seed, weights_seed, augment_volume, augment_noise, augment_dc, augment_reverse, augment_invert, logs_folder, groundtruth_folder, validation_files, test_files, labels_touse, kinds_touse, nsteps, restore_from, save_and_validate_period, mini_batch] + list(model_parameters.values()),
+            leaveout: [context, shiftby, optimizer, loss, learning_rate, batch_seed, weights_seed, augment_volume, augment_noise, augment_dc, augment_reverse, augment_invert, logs_folder, groundtruth_folder, validation_files, test_files, labels_touse, kinds_touse, nsteps, restore_from, save_and_validate_period, mini_batch, kfold] + list(model_parameters.values()),
             xvalidate: [context, shiftby, optimizer, loss, learning_rate, batch_seed, weights_seed, augment_volume, augment_noise, augment_dc, augment_reverse, augment_invert, logs_folder, groundtruth_folder, test_files, labels_touse, kinds_touse, nsteps, restore_from, save_and_validate_period, mini_batch, kfold] + list(model_parameters.values()),
             mistakes: [groundtruth_folder],
             activations: [context, shiftby, logs_folder, model_file, groundtruth_folder, labels_touse, kinds_touse, activations_equalize_ratio, activations_max_sounds, mini_batch, batch_seed] + list(model_parameters.values()),