Feasability of the implementation of a Speaker Enrollment pipeline.

[hadware]

Is your feature request related to a problem? Please describe.
The title is pretty self-explanatory. I'd just like to know how much work would be needed to implement a pipeline for a speaker unrolling task: are all the required building blocks already here in your opinion? If there isn't too much digging involved, i'd probably be willing to do it myself :)

[hbredin]

Can you please define "speaker unrolling"? I am not familiar with this wording.

Did you mean "speaker enrollment"?
If so, what do you have in mind exactly? Speaker identification?

[Rachine]

Hello,

Yes, it would be speaker enrollment.
Based on variable amount of target speakers, find all the segments in the audio from these speakers.
We were thinking to a pipeline that look like this https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/41939.pdf

[hbredin]

I didn't go through the paper but I think the SpeechTurnClosestAssignment pipeline might get you started.

Enrollment

Basic idea: gather all speaker embedding for each target and take the average.

pyannote-audio/pyannote/audio/pipeline/speech_turn_assignment.py

Lines 94 to 113 in 06f76a2

	# gather targets embedding
	labels = targets.labels()
	X_targets, targets_labels = [], []
	for l, label in enumerate(labels):
	timeline = targets.label_timeline(label, copy=False)
	# be more and more permissive until we have
	# at least one embedding for current speech turn
	for mode in ["strict", "center", "loose"]:
	x = embedding.crop(timeline, mode=mode)
	if len(x) > 0:
	break
	# skip labels so small we don't have any embedding for it
	if len(x) < 1:
	continue
	targets_labels.append(label)
	X_targets.append(np.mean(x, axis=0))

Recognition

Basic idea: for each test speech turn (or, here, speaker cluster), find closest target speaker (by comparing their average embedding. You might also want to consider the reject option if even the closest target speaker is too far.

pyannote-audio/pyannote/audio/pipeline/speech_turn_assignment.py

Lines 115 to 144 in 06f76a2

	# gather speech turns embedding
	labels = speech_turns.labels()
	X, assigned_labels, skipped_labels = [], [], []
	for l, label in enumerate(labels):
	timeline = speech_turns.label_timeline(label, copy=False)
	# be more and more permissive until we have
	# at least one embedding for current speech turn
	for mode in ["strict", "center", "loose"]:
	x = embedding.crop(timeline, mode=mode)
	if len(x) > 0:
	break
	# skip labels so small we don't have any embedding for it
	if len(x) < 1:
	skipped_labels.append(label)
	continue
	assigned_labels.append(label)
	X.append(np.mean(x, axis=0))
	# assign speech turns to closest class
	assignments = self.closest_assignment(np.vstack(X_targets), np.vstack(X))
	mapping = {
	label: targets_labels[k]
	for label, k in zip(assigned_labels, assignments)
	if not k < 0
	}
	return speech_turns.rename_labels(mapping=mapping)

[Rachine]

Amazing! Thank you! We will let you know how this goes and if it works on our 'special' data.

[hbredin]

Closing this issue as I believe the original question has been answered.
I'd still be interested in knowing how it went 👀

[Rachine]

Hey! This is still ongoing. We have the pipeline up and running but we are having a hard time to finetune correctly a spk emb model on our smallish dataset 😿