Adding a new AudioStream variant to play audio from an actual stream (continously adding data)

[Leroymilo]

Describe the project you are working on

An internet radio player receiving audio data through a StreamPeerTCP.

Describe the problem or limitation you are having in your project

I tried to play continous audio in multiple ways with all the options Godot's AudioStream types offer but I always end up with an issue: there is always a delay between the instant the stream in the AudioStreamPlayer ends and the instant when the finished signal is emitted, creating very grating stutters between chunks of streamed audio data, especially when the game's window is not focused. I suspect this is due to how the engine's event loop works, so I wouldn't be surprised if this did not happen for people with very good PC specs.
For completeness, I tried to use an AudioStreamPlaylist but the same issue emerges because its set_list_stream stops the audio for some reason, so calling play is still required and causes stuttering. Also I'm pretty sure it isn't a good idea to infinitely add streams to a playlist.

Some of the methods I tried with limited success are listed in this reddit post, I can reproduce minimal working examples for each case if necessary.

Here is a minimal working example I put together of the simplest way to see this happening:

extends AudioStreamPlayer

var stream_peer := StreamPeerTCP.new()

# Called when the node enters the scene tree for the first time.
func _ready() -> void:
	stream_peer.connect_to_host("das-edge16-live365-dal02.cdnstream.com", 80)
	stream_peer.poll()
	
	while stream_peer.get_status() == stream_peer.STATUS_CONNECTING:
		stream_peer.poll()
		await get_tree().process_frame
	assert(stream_peer.get_status() == stream_peer.STATUS_CONNECTED)
	
	var http_req = "\r\n".join([
		"GET /a25222 HTTP/1.1",
		"Host: das-edge16-live365-dal02.cdnstream.com",
		"", ""
	]).to_utf8_buffer()
	stream_peer.put_data(http_req)
	
	while true:
		stream_peer.poll()
		var available := stream_peer.get_available_bytes()
		if available < 16000:
			await get_tree().process_frame
			continue
			# waits for a minimum of data to be ready
		
		var data: PackedByteArray = stream_peer.get_data(available)[1]
		# the actual MP3 raw data is in an array in the 2nd element of the received data
		if playing: await finished
		stream.set_data(data)
		play()

(the radio link used in this example is the one I'm trying to connect to, please excuse the choice)

I know that the stuttering does not come from the data because concatenating multiple chunks of data from stream_peer.get_data(available)[1] and then playing the resulting PackedByteArray does not create any stutter. Also, other radio players (firefox, Gnome's Rythmbox) are able to play the radio with no issue.

I'd also like to add that this feature would be required to play .m3u files (Audio Playlist) because they can contain links to internet audio streams.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

Since it is not possible to play a continous stream with the existing AudioStream types, I propose the introduction of a new type: proposed name is AudioStreamMP3Stream. This new type would have a queue_data method to add a chunk of MP3 data to be played, and it would internally chain the data together, eventually discarding chunks that have finished playing.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

I am making this proposal before starting the implementation because it was advised to do so in the Godot documentation, I have some ideas about how to build what I am proposing, but I did not look into the source code yet, so it is based on my C++ knowledge, not on the Godot codebase.

The AudioStreamMP3Stream class I am planning to create has these methods:

• queue_data(data: PackedByteArray) -> void adds a chunk of data to an internal queue to be played.
• get_elapsed_time() -> float returns the total length of audio played since play() was called in milliseconds.

As well as the following signal:

• data_discarded(data: PackedByteArray) emited when the internal data chunk just finished playing and is discarded (freed) to play the next one in the queue.

From what I understand, it will also require some other internal features to handle all the methods of an AudioStreamPlayer like play, stop and stream_paused.

As mentioned earlier, I did not look into how an AudioStreamMP3 works internally yet, but I am confident that it is possible to continously play raw MP3 data by using a queue structure.

If this enhancement will not be used often, can it be worked around with a few lines of script?

Not to my knowledge. If it can, I would be glad to make an addon with the workaround instead of a new class.

Is there a reason why this should be core and not an add-on in the asset library?

From my understanding, what's in the asset library is built with what exists in Godot, but what I am proposing is not possible with what is already available. Feel free to correct me if this is wrong.

[sockeye-d]

I think you might want to look into AudioStreamGenerator, you can add samples to the buffer continuously from the TCP connection

[Leroymilo]

I think you might want to look into AudioStreamGenerator, you can add samples to the buffer continuously from the TCP connection

Do you have any documentation or pointers on how I could convert raw MP3 data to a suitable frame for the AudioStreamGenerator's playback.pushframe method?

[sockeye-d]

No. I assume you'll have to decompress it yourself, or use an external library? I had just assumed you were transmitting the raw uncompressed samples or something easier to decode

[Leroymilo]

I think there is a miscommunication here, what I receive is raw MP3, I am not the one sending the data, it's from an internet radio (could be any radio).

[Meorge]

From what I've understood of this conversation, I think it'd make more sense to add or expose methods for loading raw MP3 data and converting it into a format that AudioStreamGenerator can use. This keeps the simplicity of a single AudioStreamGenerator class/resource type, instead of making multiple for different data types.

The delay you described with the finished signal makes sense to me. Once the current audio chunk finishes playing, you then go and download the next chunk of MP3 data, load it into your program, and finally start playing the new chunk. That does seem like it'd take a non-zero amount of time, that would introduce a stutter. If possible, I think it'd make more sense to delay your playing of the audio chunks, so that you can be filling the buffer ahead of time - that way, by the time the current chunk is finished playing, the next chunk is already sitting there in the buffer.

[Leroymilo]

First, I see what you mean about extending the functionalities of the AudioStreamGenerator, I did not realize that my proposition was this close to what it does. I'll start looking into that.

Secondly, I feel like you missread the minimal working example I provided: downloading the next chunk of MP3 data is done in advance of waiting for finished, so that all there is to do is to swap the data in the stream when it's done playing:

# waiting for enough data to be ready
stream_peer.poll()
var available := stream_peer.get_available_bytes()
if available < 16000:
	await get_tree().process_frame
	continue

# downloading the data
var data: PackedByteArray = stream_peer.get_data(available)[1]

# waiting for the current chunk to finish playing
if playing: await finished

# replace stream data and play new data
stream.set_data(data)
play()

I tried to place the stream.set_data(data) line before waiting, and the opposite issue appears: the audio jumps forward because it starts the new chunk before the last chunk is finished. This also cconfirms that waiting for data and downloading it takes less time than the previous chunk takes to play.

I can also confirm that the stream.set_data(data) line is not the (main) source of delay because I also have a setup with 2 alternating AudioStreamPlayers so that nxt_stream can have the next chunk loaded while cur_stream is playing. This way, nxt_stream.play() is the literal first thing to happen in the method linked to cur_stream.finished. Even in this case, the delay still happens.