Strange audio artifacts from encoded data

[Lukaz32]

I'm capturing microphone data from an inputNode tap, downsampling the buffer to 24khz with AVAudioConverter and finally encoding it with .opus24khz . I then add both the downsampled buffer and the encoded data to two separate arrays in memory.
When I play the buffers they sound fine, but when I play the decoded .opus24khz data the audio is full of strange noises.

This is how Im doing it:

private var encodingFormat = AVAudioFormat(opusPCMFormat: .float32, sampleRate: .opus24khz, channels: 1)!

func encode(_ buffer: AVAudioPCMBuffer) {
        downsampledBufferData.append(buffer)

        var data = Data(count: 1500)
        
        do {
            encoder = try! Opus.Encoder(format: encodingFormat)
            _ = try encoder.encode(buffer, to: &data)
            encodedAudioData.append(data)
        } catch {
            print(error)
        }
    }

func playEncodedData() {
        do {
            let decoder = try Opus.Decoder(format: encodingFormat)
            try encodedAudioData
                .map { try decoder.decode($0) }
                .forEach { audioPlayer.scheduleBuffer($0) }
        } catch {
            print(error)
        }
    }

func playDownsampledBufferData() {
        downsampledBufferData.forEach {
            audioPlayer.scheduleBuffer($0) {
            }
        }
    }

This how it sounds like:
Encoded/Decoded: http://sndup.net/rp66
From Buffers: https://sndup.net/n3j6/

What am I missing here? 🤔

Thanks in advance