Speech and Audio AI

Machine learning that listens — splitting recordings by who is speaking, in Hebrew, English and French, and predicting the emotions that music evokes.

Headline result: Language-independent speaker change detection.

Papers: 4
Years: 2021 to 2026
Co-authors: 7

Each square is one of these 4 papers. Hover to read it, click to open it.

Overview

A meeting recording, a call-centre conversation or a podcast is only useful once you know who said what. We build models that find the moments a new speaker starts talking, group the segments by speaker, and work across languages.

The Challenge

Speaker change detection (SCD) splits a recording at every speaker turn. Speaker diarization and automatic transcription both depend on it: when the turns are wrong, everything built on top of them is wrong too. Systems that listen to the voice signal alone struggle as the number of speakers grows and when recordings mix languages.

Our Approach

Reading the words, not only the voice — A word-embedding approach to speaker change detection that outperformed voice-based solutions for diarization, reaching 82.12% recall and an 89.02% F1 score on speaker turns (Interspeech 2021).
Hybrid speech and text models — A framework that learns from the speech signal, the transcript and metadata together, evaluated on a commercial Hebrew dataset and the ICSI Meeting Corpus, and robust across the two languages (IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021).
Multilingual diarization — A language-independent framework using transformer-based text models and speech embeddings, tested on English, French and Hebrew. Textual information improved on speech-only approaches without hurting diarization (Expert Systems with Applications, 2023).

Beyond Speech

Emotions in music — A two-layer AI model that predicts the main emotion a listener connects with a song they chose, built on songs selected by about 4,447 people, with and without demographic factors (Multimedia Systems, 2026).

Papers

4 publications, newest first.

All publications

2026

The Sound of Emotions: An Artificial Intelligence Approach to Predicting Emotions from Musical Selections

Ron Hirshprung, Ori Lesman, and Chen Hajaj

JournalMultimedia Systems

DOI

Music in the current digital era is consumed primarily on-demand, for example by streaming from Spotify or YouTube, rather than by broadcasting, for example, by listening to a radio station. A strong correlation has been found between music and emotions. Hence, with our ability to select our own music and the availability of artificial intelligence (AI) tools, the importance of music emotion recognition (MER) is significant. This research proposes a novel methodology for predicting emotions from a musical piece that is selected by the listener based on a two-layer AI. The dataset and the process are exceptional in two ways: first, by their qualitative character, which minimized the inherent subjectivity of self-reported emotions, and also the high reliability of the collected data in comparison to other methods such as a questionnaire; and second, by their quantitative character, given songs from about 4,447 people. The model succeeded in predicting the primary emotion with and one of the two leading emotions (the primary and the secondary) with, and without accommodating demographic factors in the model. We believe this knowledge may be a useful tool for therapists, coaches, teachers, etc. and may also raise public awareness of a potential privacy violation.
```
@article{hirshprung2026sound_of_emotions,
  author = {Hirshprung, Ron and Lesman, Ori and Hajaj, Chen},
  title = {The Sound of Emotions: An Artificial Intelligence Approach to Predicting Emotions from Musical Selections},
  journal = {Multimedia Systems},
  year = {2026},
  doi = {10.1007/s00530-026-02220-1}
}
```

2023

Speech and Multilingual Natural Language Framework for Speaker Change Detection and Diarization

Or Haim Anidjar, Yannick Estève, Chen Hajaj, Amit Dvir, and Itshak Lapidot

JournalExpert Systems with Applications

DOI

Speaker Change Detection (SCD) is the problem of splitting an audio-recording by its speaker-turns. Many real-world problems, such as the Speaker Diarization (SD) or automatic speech transcription, are influenced by the quality of the speaker-turns estimation. Previous works have already shown that auxiliary textual information (for mono-lingual systems) can be of great use for detection of speaker-turns and the diarization systems’ performance. In this paper, we suggest a framework for speaker-turn estimation, as well as the determination of clustered speaker identities to the SD system, and examine our approach over a multi-lingual dataset that consists of three mono-lingual datasets—in English, French, and Hebrew. As such, we propose a generic and language-independent framework for the SCD problem that is learned through textual information using state-of-the-art transformer-based techniques and speech-embedding modules. Comprehensive experimental evaluation shows that (i) our multi-lingual SCD framework is competitive enough when compared to a framework over mono-lingual datasets, and that (ii) textual information improves the solution’s quality compared to the speech signal-based approach. In addition, we show that our multi-lingual SCD approach does not harm the performance of SD systems.
```
@article{anidjar2023speech,
  title = {Speech and Multilingual Natural Language Framework for Speaker Change Detection and Diarization},
  author = {Anidjar, Or Haim and Est{\`e}ve, Yannick and Hajaj, Chen and Dvir, Amit and Lapidot, Itshak},
  journal = {Expert Systems with Applications},
  volume = {213},
  pages = {119238},
  year = {2023},
  publisher = {Pergamon},
  doi = {10.1016/j.eswa.2022.119238}
}
```

2021

A Thousand Words Are Worth More Than One Recording: Word-Embedding Based Speaker Change Detection

Or Haim Anidjar, Itshak Lapidot, Chen Hajaj, and Amit Dvir

ConferenceInterspeech

DOI

Speaker Change Detection (SCD) is the task of segmenting an input audio-recording according to speaker interchanges. This task is essential for many applications, such as automatic voice transcription or Speaker Diarization (SD). This paper focuses on the essential task of audio segmentation and suggests a word-embedding-based solution for the SCD problem. More-over, we show how to use our approach in order to outperform voice-based solutions for the SD problem. We empirically show that our method can accurately identify the speaker-turns in an audio-recording with 82.12% and 89.02% success in the Recall and F1-score measures.
```
@inproceedings{anidjar2021thousand,
  title = {A Thousand Words Are Worth More Than One Recording: Word-Embedding Based Speaker Change Detection},
  author = {Anidjar, Or Haim and Lapidot, Itshak and Hajaj, Chen and Dvir, Amit},
  booktitle = {Interspeech},
  pages = {3121--3125},
  year = {2021},
  doi = {10.21437/interspeech.2021-87}
}
```
Selected: Hybrid Speech and Text Analysis Methods for Speaker Change Detection

Or Haim Anidjar, Itshak Lapidot, Chen Hajaj, Amit Dvir, and Issachar Gilad

JournalIEEE/ACM Transactions on Audio, Speech, and Language Processing

DOI

Speaker Change Detection (SCD) is the task of segmenting an input audio-recording according to speaker interchanges. Nowadays, many applications, such as Speaker Diarization (SD) or automatic vocal transcription, depend on this segmentation task. In this paper, we focus on the essential task of the SD problem, the audio segmenting process, and suggest a solution for the SCD problem, as well as the assignment of clustered speaker labels for the extracted segments, and applying the solution over two datasets: a commercial dataset in Hebrew and the ICSI Meeting Corpus. As such, we propose a hybrid framework for the SCD problem that is learned by textual information and speech signals and the meta-data features that can be extracted from them. Moreover, we demonstrate the negative correlation between an increase in the number of speakers in the training dataset and the influence on the overall diarization system’s performance, which is improved using our efficient SCD component. Finally, we show how our proposed hybrid framework remains robust compared to the ICSI Meeting Corpus, as the experimental evaluation’s training and testing is based on two languages.
```
@article{anidjar2021hybrid,
  title = {Hybrid Speech and Text Analysis Methods for Speaker Change Detection},
  author = {Anidjar, Or Haim and Lapidot, Itshak and Hajaj, Chen and Dvir, Amit and Gilad, Issachar},
  journal = {IEEE/ACM Transactions on Audio, Speech, and Language Processing},
  volume = {29},
  pages = {2324--2338},
  year = {2021},
  publisher = {IEEE},
  doi = {10.1109/taslp.2021.3093817}
}
```

Work on this

The lab takes on PhD and MSc students and works with academic and industry partners on this line of research. Write with a short note on your background and what you would like to work on.

Email Chen about this project Code and datasets