Extended Loss: Incorporating Long Context into Training Models when using Short Audio Frames

Quang Minh Dinh^1*

Yuanhao Yu²

Simon Fraser University¹, Noah's Ark Lab, Huawei Canada²

Overview of the training pipeline. (a) First \(k\) audio files are randomly selected from the training set, and STFT features for \(l\) consecutive short frames are extracted \(S^1 \dots S^k\). (b) Each batch is formed in a way that consecutive frames appear sequentially, and \(b=k \times l\). (c) The output of the AEC model is the enhanced audio per short frame. (d) Inverse STFT is applied on each short frame separately to obtain their reconstructed waveforms. (e) \(l\) consecutive waveforms that belong to the same audio are concatenated together to form a long waveform. (f) STFT is applied on the long sample which is used for loss calculation. In the regular non-extened setup, loss calculation is performed at stage (c) on predicted short audio frames.

Abstract

Recently deep learning solutions have been successfully applied to many signal processing tasks including acoustic echo cancellation (AEC). Most of the existing work focus on architecture design, and ignore practical issues such as the effect of frame length on the performance of end-to-end AEC models. In real-time applications, frame length can be as small as 10ms. Since observed context is very limited during training, it results in boundary discontinuities (glitches) in the final output. While using long frames or post-processing can help, it adds extra delay which may not be desirable depending on the application. In this paper, we investigate the practical issue of handling short frames for AEC, and propose an efficient remedy for it. By keeping the long context information in each batch and using it during loss calculation, we compensate for the short frames. Our solution is architecture independent and has no effect on the inference time.

^* Work done during internship at Huawei Canada

Double-talk (DT)

Microphone.

Far end.

Train: 368ms. Test: 368ms.
	AECMOS_e: 4.51, AECMOS_d: 3.6

Train: 10s. Test: 10s.
	AECMOS_e: 4.78, AECMOS_d: 4.55

Train: 10s. Test: 368ms.
	AECMOS_e: 4.71, AECMOS_d: 3.55

Train: 368ms (x3 frames). Test: 368ms.
	AECMOS_e: 4.79, AECMOS_d: 4.24

Train: 368ms (x5 frames). Test: 368ms.
	AECMOS_e: 4.79, AECMOS_d: 4.26

Far-end single-talk (FEST)

Microphone.
Far end.
Train: 368ms. Test: 368ms.	AECMOS_e: 3.89, ERLE: 32.53

Train: 10s. Test: 10s.	AECMOS_e: 4.45, ERLE: 48.02

Train: 10s. Test: 368ms.	AECMOS_e: 4.52, ERLE: 25.69

Train: 368ms (x3 frames). Test: 368ms.	AECMOS_e: 3.75, ERLE: 39.21

Train: 368ms (x5 frames). Test: 368ms.	AECMOS_e: 3.88, ERLE: 44.2

Additional information

We included samples only for the Conformer experiments in Table 1 of the paper.
The metrics for double-talk (DT) scenario are calculated only on the regions with actual double-talk.