Real-time audio signal processing in python

Hello,

I've been trying to make jackclient work (if possible) for a real-time audio processing application in python. You can see the code below:

import jack
import numpy as np

jclient = jack.Client("jclient")
    
jclient.register_port("in", jack.IsInput)
jclient.register_port("out", jack.IsOutput)
jclient.activate()

jclient.connect("system:capture_1","jclient:in")
for n_playback_port in (1, 2):
    jclient.connect("jclient:out", "system:playback_{}".format(n_playback_port))
#sr = jclient.get_sample_rate()
buffer_size = jclient.get_buffer_size()

jcapture = np.zeros((1,buffer_size), 'f')    
jinput = np.zeros((1,buffer_size), 'f')
joutput = np.zeros((1,buffer_size), 'f')

try:
    while True:
        print("Capturing audio")
        try:
            jclient.process(joutput, jcapture)
        except jack.InputSyncError:
            print("InputSyncError")
        except jack.OutputSyncError:
            print("OutputSyncError")

        #process jcapture here

        print("Playing back\n")
        try:
            jclient.process(jcapture, jinput)
        except jack.InputSyncError:
            print("InputSyncError")
        except jack.OutputSyncError:
            print("OutputSyncError")            
except KeyboardInterrupt:
    pass

The problem is that even if I don't process the signal at all (use the code above) and just playback the input I hear the input but with interrupts between the consecutive frames (buffers). Am I doing something wrong or is there a better way to do this?
I'm using 16000 kHz sample rate and 1024 samples buffer size for the jackd server.

[mgeier]

It looks like you are using a different module that happens to be called "jack" as well. Probably https://pypi.org/project/py-jack/?

If you use the jack module from this repository, your code will not work at all because most of the functions you are using don't exist.

Please have a look at the documentation: https://jackclient-python.readthedocs.io/

And the example programs: https://github.com/spatialaudio/jackclient-python/tree/master/examples

Thanks for the prompt reply!

I was testing both pyjack and your module so I guess I have confused a bit the functions, sorry for the inconvenience.

So, I see that thru_client.py does what I wanted to do above so I'll read the documentation to find out how I can extract each buffer, process it and then send it to the output (are there any examples for this?)

Thanks a lot.

[mgeier]

Yes, thru_client.py is an example for this, but it uses plain Python "buffer" objects instead of NumPy arrays.

If you want to get the input buffer as NumPy array, use get_array() instead of get_buffer().

If you want to send stuff to the output, you have to get the output array (also with get_array()) and simply write to it.

Note that if you don't have output data available at the moment, you have to fill the output array with zeros. If your data is incomplete, you have to fill the remaining part of the array with zeros.

And note that all this has to be done in a "process callback" function, otherwise it won't work as expected.

I get it now and it seems to be working, thank you so much :)

[fotisdr]

Hello again,

So, to be more specific, I am trying to implement a real-time noise denoiser. I combined the thru_client and the play_file script so you can see my final script below.
It is working but I get a lot of dropouts (runs where the queue is empty). If you have any time to spare could you please check my code and tell me if there is any way to further improve it?

Thank you in advance.

Mortis19

"""Real-time single-channel denoising

"""
from __future__ import division
from __future__ import print_function
import argparse
#import os
import jack
try:
    import queue  # Python 3.x
except ImportError:
    import Queue as queue  # Python 2.x
import sys
import threading
import numpy as np

# jack options
parser = argparse.ArgumentParser(description=__doc__)
#parser.add_argument('filename', help='audio file to be played back')
parser.add_argument(
    '-b', '--buffersize', type=int, default=1,
    help='number of blocks used for buffering (default: %(default)s)')
#parser.add_argument('-c', '--clientname', default='file player',
#                    help='JACK client name')
parser.add_argument('-m', '--manual', action='store_true',
                    help="don't connect to output ports automatically")
args = parser.parse_args()
if args.buffersize < 1:
    parser.error('buffersize must be at least 1')

#argv = iter(sys.argv)
#defaultclientname = os.path.splitext(os.path.basename(next(argv)))[0]
#clientname = next(argv, defaultclientname)
#servername = next(argv, None)

#buffersize = 1024

q = queue.Queue(maxsize=args.buffersize)
#print(args.buffersize)
event = threading.Event()


def print_error(*args):
    print(*args, file=sys.stderr)


def xrun(delay):
    print_error("An xrun occured, increase JACK's period size?")


def shutdown(status, reason):
    print_error('JACK shutdown!')
    print_error('status:', status)
    print_error('reason:', reason)
    event.set()


def stop_callback(msg=''):
    if msg:
        print_error(msg)
    for port in client.outports:
        port.get_array().fill(0)
    event.set()
    #raise jack.CallbackExit


def process(frames):
    if frames != blocksize:
        stop_callback('blocksize must not be changed, I quit!')
    try:
        data = q.get_nowait()
	if data is None:
		stop_callback()  # Playback is finished
	else:
		client.outports[0].get_array()[:] = data.T
    except queue.Empty:
        stop_callback('Buffer is empty: increase buffersize?')


try:
    #import jack
    #import soundfile as sf

    client = jack.Client("thru_client")
    blocksize = client.blocksize
    samplerate = client.samplerate
    client.set_xrun_callback(xrun)
    client.set_shutdown_callback(shutdown)
    client.set_process_callback(process)

    client.inports.register('in_{0}'.format(1))
    client.outports.register('out_{0}'.format(1))
    i=client.inports[0]
    data=i.get_array()
    q.put_nowait(data)  # Pre-fill queue
    with client:
	if not args.manual:
		capture = client.get_ports(is_physical=True, is_output=True)
		client.inports[0].connect(capture[0])
		#print(capture[0])
		target_ports = client.get_ports(is_physical=True, is_input=True, is_audio=True)
		if len(client.outports) == 1 and len(target_ports) > 1:
			# Connect mono file to stereo output
			client.outports[0].connect(target_ports[0])
			client.outports[0].connect(target_ports[1])
		else:
			for source, target in zip(client.outports, target_ports):
				source.connect(target)
	timeout = blocksize * args.buffersize / samplerate
	while True:
		noisy=client.inports[0].get_array()
		#t=time.time()
		# processing input
		noisy.shape=(1,blocksize)
		#clean=np.zeros(frames,)
		noisy = np.expand_dims(noisy, axis = 2)
		clean = G_loaded.predict(noisy)
		clean = np.reshape(clean, (1,blocksize))
		clean = np.reshape(clean, (-1,)) # make it to 1d by dropping the extra dimension
		#print(time.time()-t)
		#
		q.put(clean,timeout=timeout)
	#q.put(None, timeout=timeout)  # Signal end of file
	event.wait()  # Wait until playback is finished
except KeyboardInterrupt:
    parser.exit('\nInterrupted by user')
except (queue.Full):
    # A timeout occured, i.e. there was an error in the callback
    parser.exit(1)
except Exception as e:
    parser.exit(type(e).__name__ + ': ' + str(e))

[mgeier]

I think the main problem here is that you use get_array() outside the process callback function. That is not allowed, please don't do that.

There is already one queue to move the audio data to the process callback after processing, but I guess you'll need a second queue to move the input data from the process callback to the main thread for processing.
Or, if the denoising is quick enough (given the block size you want to use), you can do everything in the process callback itself.

Other than that, you should probably simplify your example to only use one input and one output port. If you need more ports later, you can add them once the first draft works as expected.

[fotisdr]

Thanks a lot for your assistance.

Regarding the input/output ports, I'm using one input port and one output port for the client created, I just connect the output of the client to both speakers in the end.

I've managed to do the processing inside the process callback (although I didn't notice anything different to be honest), but I had no success so far using two queues (I've used a second queue to pass the data as you suggested but I'm not sure about the put/get commands and the corresponding timeout that I have to use and I get full/empty queue or xrun errors).

[mgeier]

If you show your updated code, I can have a look.

In the process callback, you should definitely not use a timeout, you should only use the non-blocking variants of put() and get(). I'm not sure if a timeout on the other side of the queues makes sense ...

You should also read through spatialaudio/python-sounddevice#148. It is not about JACK, but the mentioned usage of queues should be more or less the same.

[fotisdr]

Thanks for the prompt reply!

I managed to make it work after all using queues in the process callback. Specifically, I used another queue for passing the data to the main thread as you suggested by using the get_nowait and put commands for the two queues both in the main thread and in the process callback (without using any timeout).

Another question that I have in mind is whether I should transpose the data arrays that I put to and get from the queues (see lines 96 and 148 in my code below). Is there a reason for doing this? (I just copied it from the play_file example).
https://github.com/fotisdr/pi-gan/353ea63912754a76d0bd25d54ff92dcd05b26701/denoise_overlap.py

Thanks again for your help.

[mgeier]

I'm glad to hear that you made good progress!

I'm a bit surprised that you are using a buffersize of 1 ... if you only need a queue with one element, you might as well get rid of the queue and do everything directly in the process callback. Or am I missing something?

About transposing the data: The array provided by the soundfile module contains the audio channels as columns. In other words, the shape of the array is (B, C), where B is the block size (in frames) and C is the number of channels.

If I would iterate over such an array, I would get a sequence of frames, i.e. arrays with shape (C,). However, in the play_file.py example I wanted to have a sequence of channels (i.e. arrays with shape (B,), because each JACK port only takes a single channel. In order to get this, I just transpose the array first, and then iterate over it.

for channel, port in zip(data.T, client.outports):
    port.get_array()[:] = channel

For people not used to Python's iterator concept and zip() and all those good things, it might be more straightforward to do something like that:

for c in range(data.shape[1]):
    client.outports[c].get_array()[:] = data[:, c]

I didn't test this, but I think this should do the same thing. I personally prefer the first, more Pythonic, way of doing it.

But anyway, if you are only dealing with a mono signal in the first place, you don't have to transpose it. You can simply assign it to your one and only output port.

BTW, the register() function returns the new port (see https://jackclient-python.readthedocs.io/en/0.4.5/#jack.Ports.register), so you could directly assign it:

i = client.inports.register('in_1')

[fotisdr]

Thanks a lot for your valuable information.

Regarding the queues, I used them in the first place because I wanted to implement overlap between frames (i.e. half of the frame size for the jack server than the processing procedure's frame size), so I though it would not be possible inside the process callback. However, now that I'm thinking it, I guess I could do the same in the process callback as well, although I haven't tried it yet.
The buffersize of the queues was set to 1 because for higher sizes I didn't see any difference other than latency added up to the signal. So, my question is, is there any reason to use queues in a real-time application?

Thanks again for answering all of my questions...

EDIT: So, I tried doing the same inside the callback process, without using queues, but I get a lot of xrun errors for some reason ('thru_client' was not finished). You can find the code here:
https://github.com/fotisdr/pi-gan/master/denoise_overlap_noq.py
It should be the same code as before so I don't know why there are so many xrun errors introduced now. Any ideas?

[mgeier]

That's surprising to me, I would assume that if it works with a queue size of 1, it should work the same with everything done in the process callback.

I don't know why you get xruns in one case and not in the other. But you shouldn't worry about it, because in general it is a good idea to use queues anyway.

is there any reason to use queues in a real-time application?

Yes, definitely. It's the default way to do it.

The important limiting factor is that there is a limited amount of time for each audio block to be delivered to the sound card. If that deadline is missed, the output will be either silent or filled with garbage values (probably, but not reliably, the values of the previous output block). Either way, in most cases this leads to audible artifacts.

The problem is that there are multiple variables:

• the time needed for processing might change if you use a different computer, or if other resource-hungry processes are running at the same time

• the amount of processing might change if you make changes to the algorithm or if you change some parameters

• the JACK block size can be selected by the user, so you should ideally not rely on it having a certain value (though I typically assume/assert that it doesn't change during the runtime of the program)

If you use queues, you can to some extent handle those variables. For example, if a user selects a small JACK block size (there might be other JACK applications running that need a certain block size), you can choose a larger queue size to compensate.
Of course there are limits, it's not a magic wand.

If you are doing file or network IO, you should definitely use queues, because those processes might cause arbitrary delays, which can be taken care of by big enough queues.

Please also note that Python is in general not really suitable for real-time audio.

As I've written in the docs (https://jackclient-python.readthedocs.io/en/0.4.5/#jack.Client.set_process_callback):

Most Python interpreters use a global interpreter lock (GIL), which violates the above real-time requirement. Furthermore, Python’s garbage collector might become active at an inconvenient time and block the process callback for some time.

Because of this, Python is not really suitable for real-time processing. If you want to implement a reliable real-time audio/MIDI application, you should use a different programming language, such as C or C++.

One way to avoid the GIL and GC in the audio callback, is implementing the process function in C. If you are interested in this, you can have a look at my other project: https://github.com/spatialaudio/python-rtmixer.

BTW, the try/except in your process callback will not work, because the audio callback is called (by JACK) from a different thread. If you press Ctrl+C, this will only affect the main Python thread.
If you press Ctrl+C, the main thread will exit the context manager, your JACK client will be deactivated and the callback will no longer be called (but the except clause in the callback will never be reached).
In your case I think this will be fine, because the worst thing that can happen is that there is still some data in the queue in the end, which shouldn't really be a problem.

[fotisdr]

Okay, once again thank you very much for the lengthy and really helpful response :)

First of all, the truth is that, even with the use of queues in my previous example code (https://github.com/fotisdr/pi-gan/master/denoise_overlap.py), I am getting a somewhat distorted output at the moment, although I am still trying to find out the exact cause of this (if it's introduced from the algorithm or from python). The processing time needed right now for my code is barely lower than the available time so I might be getting these artifacts that you mentioned in certain callbacks.

I also just tried to increase the buffersize of the queues in my previous code a bit but I get these queue.empty errors after a while and the code stops. However, I tried this out in a rush so I need to look this more thoroughly.

I am aware of the limitations of Python but my processing algorithm is implemented in Python so I started doing everything in Python in the first place. However, your project seems really interesting and I think I should try this out, if it's possible to still use the python denoising algorithm.

About the try/except commands, I have already assumed that in the process callback they are probably unnecessary but I just left them unchanged (thanks for pointing this out).

Thanks a lot!

EDIT: I just changed the data = qin.get_nowait() command (line 148) to data = qin.get() and now it works for higher buffersize as well, although I don't really see a difference.

EDIT 2: I'm currently trying out your python-rtmixer (it looks very promising) but I'm having trouble finding a way to convert a numpy array to a buffer in order to play it back with the ringbuffer (i.e. I'm trying to do something equivalent to the lines 36-37 of your play_file.py example but for a numpy array: https://github.com/spatialaudio/python-rtmixer/blob/d7095f228f58d08fa69916d3657e4996526144cd/examples/play_file.py#L36-L37). What I'm trying to do is to record the input, get it as a numpy array and then produce the numpy array to the playback. You can see my code so far if you want of course but I'm still trying to find out the way the rtmixer works so don't except much!
https://github.com/fotisdr/pi-gan/master/test.py

EDIT 3 (and final): I actually made it to implement what I mentioned above, it was simpler than I thought it was in the first place, you can find my code below:
https://github.com/fotisdr/pi-gan/master/thru.py
The only thing that I am not certain about at the moment is the reason why the numpy array that I get from the buffer has twice the size of the blocksize that I define in the beginning (blocksize=512 returns a 1024 size array). I just have to experiment now a bit to understand the time restrictions.
Great work with the python-rtmixer, thanks a lot!!!

[mgeier]

The processing time needed right now for my code is barely lower than the available time so I might be getting these artifacts that you mentioned in certain callbacks.

Are you getting xruns at the same time?
If not, the artifacts may have a different cause.

I get these queue.empty errors after a while

Maybe you are not pre-filling enough?

your project seems really interesting and I think I should try this out, if it's possible to still use the python denoising algorithm.

Yes, because you would also use queues to transport the audio data to a Python thread.
But of course you would need to add some latency for that.

If you want very low latency, you might have to re-implement the algorithm in some compiled language like C. If you manage to avoid the GIL (see https://cython.readthedocs.io/en/latest/src/userguide/external_C_code.html#declaring-a-function-as-callable-without-the-gil), it might even work with Cython, but I've never tried that myself.

In jackclient-python it is not (yet) possible to use a callback like that (unlike the sounddevice module), but if you need it, I can implement it.

I just changed the data = qin.get_nowait() command (line 148) to data = qin.get()

Yes, that totally makes sense, because in the denoising thread you actually want to wait for the input data to appear. There is nothing else you could do before that, right?

It's just important that you don't block the audio thread waiting for something.

I actually made it to implement what I mentioned above, it was simpler than I thought it was in the first place

Cool!

If you take some extra care with your NumPy arrays, you don't even need .tobytes(). If they are C-contiguous and use the 'float32' data type, you can use them just like that, see:

https://github.com/spatialaudio/python-rtmixer/blob/d7095f228f58d08fa69916d3657e4996526144cd/examples/bleeperoo.py#L67-L69

You would need to do something like this:

np.zeros((blocksize, channels), dtype='float32')

Please note that play_buffer() and record_buffer() are not made to be called in rapid succession for "streaming" behavior.
As you have implemented it, there might be gaps (empty blocks) in the output and if you are unlucky, you might even write two different audio blocks to the same output block.
You should definitely use the ring buffer functions for your use case.

the reason why the numpy array that I get from the buffer has twice the size of the blocksize that I define in the beginning

I guess that's because np.zeros() by default uses dtype='float64', therefore you get twice the number of bytes you actually want.

[fotisdr]

Thanks again for the long comments. Thanks for solving the blocksize issue as well, I don't know why I forgot the dtype='float32' command in the np.zeros!

First of all, concerning the jackclient-python, I'm not getting xrun errors when I run the code usually (only rarely) so I think that it's the algorithm that is introducing the artifacts. The code is working properly now I think (https://github.com/fotisdr/pi-gan/blob/master/denoise_overlap.py), but I'm still not sure what difference it makes to use a higher than 1 buffersize for the queues.

Regarding the python-rtmixer, you can take a look at my code so far here:
https://github.com/fotisdr/pi-gan/blob/master/rtmixer_cnn.py
Right now I'm using the data.data command to get the buffer to bytes (line 94). The code seems to be working properly with the use of play_buffer() and record_buffer() commands, but I haven't fully understood the ringbuffer functions yet so I haven't succeded in using them properly. How could I implement the same code with the use of the ringbuffer commands instead of the play_buffer() and record_buffer() commands?

Right now both ways seem to function more or less the same, although I think that the python-rtmixer is a bit slower than the jackclient-python, probably because of the sounddevice module.