Skip to content

PROCESS_MAP.md

Latest commit

 

History

History
214 lines (149 loc) · 13.7 KB

PROCESS_MAP.md

File metadata and controls

214 lines (149 loc) · 13.7 KB

multi-midi Process Map

Part of the multi-midi documentation

© 2025 Harm Lammers

For questions or contributions open an issue or start a discussion on the GitHub repository.

This document provides a technical description of the inner workings of multi-midi. It also explores a few considerations for further development and testing, which might lead to total revision of the multi-midi library. I will be further looking into this before I will start working on integrating multi-midi into Cybo-Drummer, because both the way the current version of multi-midi works and the alternative(s) I have in mind require a major, structural refactoring of the Cybo-Drummer code.

Table of Contents

Consideration: Is Using asyncio the Right Approach?

The Challenge

The biggest challenge, when developing multi-midi was juggling the complexity of time-sensitive tasks which (ideally) should take place in parallel:

  • Processing up to 48 IN and OUT ports in total.
  • Slightly different processes for UART and PIO based hardware MIDI ports and totally different processes for USB-based MIDI ports.
  • Byte-by-byte processing of hardware MIDI data streams (including the decoding and encoding of running status) and packet-by-packet (4 bytes each, including a USB MIDI specific virtual cable / Code Index Number byte) processing of USB MIDI data streams (which do not support running status).
  • Prioritize (single-byte) MIDI Real-Time messages, for hardware MIDI to be inserted anywhere into the byte stream and for USB MIDI in between MIDI packets.
  • Dealing with SysEx messages, which – in particular when merging data from different sources – should not be interrupted by normal MIDI data (except MIDI Real-Time messages).
  • Etc.

The Benefits of Using asyncio

  • The asyncio model makes it relatively easy to write asynchronous code to deal with (semi-)parallel processes (e.g. by calling await sleep(0) to allow other coroutines to run without having to leave a loop).
  • The structure of asyncio based code based on awaitables and tasks is relatively easy to follow, which leads to easy to maintain code.
  • asyncio provides a framework for efficient, non-blocking handling of inputs and outputs, which allows processing MIDI messages and perform other tasks concurrently without busy-waiting and manually managing asynchronous tasks.
  • asyncio allows to easily add other asynchronous tasks (e.g. user input, GUI updates, logging) without major changes to the MIDI processing logic.

The Potential Disadvantages and Risks of Using asyncio

  • The cooperative multitasking of asyncio can introduce unpredictable scheduling delays, especially if other coroutines are running or if some code sections are accidentally blocking. This can result in increased latency or timing jitter, which is undesirable for real-time MIDI processing.
  • asyncio can only run on primary core of the RP2040/RP2350, which it shares with interrupt handlers and background processes such as MicroPython’s garbage collector. That may cause brief pauses in execution, which can disrupt MIDI timing if they occur during critical message processing.
  • All asynchronous tasks share a single event loop (that includes non-MIDI tasks), so heavy processing elsewhere in the application can impact MIDI responsiveness.
  • asyncio is based on soft timers, so it does not provide hard real-time guarantees, and there is no control over timing and no way to prioritize time-critical MIDI processes.
  • The performance of asyncio based code is severely limited by Python performance: it requires a lot of function calls (which are slow) and awaitables and tasks cannot be written in viper code (which is approximately 5 to 40 times faster and can get close to native C performance).

Alternative Approach: Dedicate the Second Core to MIDI Processing

If all MIDI processing would be moved to the second core, which would be only dedicated to do that, it should be possible to achieve much lower and more consistent latency for MIDI I/O compared to what would ever be possible using asyncio, especially if it is combined with using direct register access (bypassing MicroPython’s UART/PIO drivers for direct hardware reads/writes).

Potential Benefits

  • The full process could be covered by viper code, which gives a massive optimization potential, running at near C speed.
  • No asyncio means less overhead of MicroPython function calls and scheduler delays, and reading and writing MIDI bytes directly to registers would reduce the number of function calls even further.
  • The second core runs independently, so garbage collection or other main-thread Python activity can’t block or delay MIDI processing. This also limits how much the performance of the broader application which uses multi-midi can impact MIDI processing.
  • Tight, deterministic polling or interrupt-driven loops can be implemented, this minimizing jitter (timing variance).
  • Possibly it’s possible to reduce the number of buffering steps.
  • All data can be handled as raw bytes/bytearray, reducing Python object overhead and thus improving memory and cache usage (which, in turn, limits the frequency of garbage collection).
  • It would give application developers who’d like to use multi-midi total freedom to choose whether to use asyncio.

Disadvantages and Tradeoffs

  • It is much mode complex and difficult to write the code for a viper and direct register handling based MIDI manager which needs to juggle time-sensitive tasks without the help of an asyncio scheduler.
  • Such code would be much harder to debug, while both viper code and direct register handling easily lead to tricky bugs.
  • It will most certainly lead to complex, difficult to understand code, which is hard to maintain.
  • There’s much less documentation on low-level register programming and to a less extent on the use of viper code, and not many examples exist to learn from.
  • It would make multi-midi even more tied to RP2040/RP2350 than it already is in its current form.

Will It Be Worth the Effort?

Is it really necessary and worth the considerable effort to refactor the current asyncio based approach into an approach based on dedicating the second core to a viper code and direct register access based low-level approach?

To answer that question I probably first need to get a better understanding of how well the asyncio based implementation performs: try it out with real MIDI hardware (putting my collection of drum computers to work) and find a way to actually measure its current performance.

On the one hand I’m feeling drawn to taking the challenge of building a near-low-level version of multi-midi, but on the other hand I’m afraid it’s going to consume too much time – this project already took a little bit more time and effort than I expected…

Perhaps it’s also possible to look for hybrids of the current asyncio based approach and a complete low-level refactoring:

  • Only replacing the current IRQ + event based approach to I/O with a polling approach based on direct register access.

    Disadvantage: This goes against the asyncio philosophy and it can be tricky to maintain the flow of the event loop, but it can be done…

  • Only move the lowest level MIDI processing (merging data streams and I/O) to the second core.

    Disadvantage: In such case the second core could not be used anymore for off-loading heavy tasks, introducing a risk of more jitter because some MIDI processes depend on the main core.

MIDI Input Process Map

Flow Diagram

flowchart TD
    UART_IN[/"From hardware<br/>via UART<br/>(<code>InPortUART</code>)"/] -->|"Incoming bytes"| UART_READ_BUF["Low-level buffer <code>read_buf</code>"]
    UART_READ_BUF --> PROCESS_BYTE["Process byte-by-byte<br/><code>_process_midi_byte()</code>"]

    PIO_IN[/"From hardware<br/>via PIO<br/>(<code>InPortPIO</code>)"/] -->|"Incoming bytes"| PIO_READ_BUF["Low-level buffer <code>read_buf</code>"]
    PIO_READ_BUF --> PROCESS_BYTE

    PROCESS_BYTE --> MSG_TYPE{"Message type?"}

    MSG_TYPE -->|"MIDI Real-Time"| HW_RT{{Prioritize}}
    HW_SRC -->|"Yes"| HW_DEST[/"To OUT destinations<br/><code>write_real_time()</code><br/>(<code>OutPort*</code>)"/]
    HW_RT -.-> HW_SRC{"Fast-track<br/>routing<br/>source?"}
    HW_RT ---> HW_CB_RT[/"To callback<br/><code>_g_cb_rt</code>"/]

    MSG_TYPE -->|"SysEx"| HW_SYSEX_BUF["Buffer <code>sysex_buf</code>"]
    HW_SYSEX_BUF --> HW_SYSEX_COMPLETE{"Complete?"}
    HW_SYSEX_COMPLETE -->|"Yes"| HW_CB_SYSEX[/"To callback<br/><code>_g_cb_sysex</code>"/]
    HW_SYSEX_COMPLETE -->|"No"| PROCESS_BYTE

    MSG_TYPE -->|"Standard data"| DATA_BUF["Buffer <code>data_buf</code>"]
    DATA_BUF --> DATA_COMPLETE{"Complete?"}
    DATA_COMPLETE -->|"Yes"| HW_CB_DATA[/"To callback<br/><code>_g_cb_data</code>"/]
    DATA_COMPLETE -->|"No"| PROCESS_BYTE

    USB_IN[/"From USB host<br/>(<code>MidiUSB</code>)"/] -->|"Incoming packets"| RX_BUF["<code>RingIO</code> buffer <code>_rx_buf</code>"]
    RX_BUF --> PROCESS_PACKET["Process packet-by-packet<br/><code>_process_midi_packet()</code><br/>(<code>InPortUSB</code>)"]

    PROCESS_PACKET --->|"MIDI Real-Time"| USB_RT{{Prioritize}}
    USB_RT -.-> USB_SRC{"Fast-track<br/>routing<br/>source?"}
    USB_SRC -->|"Yes"| USB_DEST[/"To OUT destinations<br/><code>write_real_time()</code><br/>(<code>OutPort*</code>)"/]
    USB_RT ---> USB_CB_RT[/"To callback<br/><code>_g_cb_rt</code>"/]

    PROCESS_PACKET --->|"SysEx"| USB_SYSEX_BUF["Buffer <code>sysex_buf</code>"]
    USB_SYSEX_BUF --> USB_SYSEX_COMPLETE{"Complete?"}
    USB_SYSEX_COMPLETE -->|"Yes"| USB_CB_SYSEX[/"To callback<br/><code>_g_cb_sysex</code>"/]
    USB_SYSEX_COMPLETE -->|"No"| PROCESS_PACKET

    PROCESS_PACKET ----->|"Standard data"| USB_CB_DATA[/"To callback<br/><code>_g_cb_data</code>"/]
Loading

Key Points

  • All incoming data is first buffered, then parsed into higher-level buffers.
  • Awaitables ensure the event loop remains responsive.
  • When a message is ready, the corresponding callback is invoked directly with the message data.
  • There’s an optional immediate distribution path For MIDI Real-Time Messages:
    • If configured, Real-Time messages are sent directly to MIDI OUT port handlers as soon as they are recognized (in addition to normal callback handling).
    • This allows for low latency MIDI clock, start/stop and other Real-Time signal forwarding.

MIDI Output Process Map

Flow Chart

flowchart TD
    HW_WRITE_RT[/"From<br/><code>write_real_time()</code> (<code>OutPortUART</code>/<br/><code>OutPortPIO</code>)"/]
    HW_WRITE_RT -->|"Real-Time"| HW_RT_BUF["<code>RingIO</code> buffer<br/><code>_rt_buf</code>"]
    HW_WRITE_SYSEX[/"From<br/><code>write_sysex()</code> (<code>OutPortUART</code>/<br/><code>OutPortPIO</code>)"/]
    HW_WRITE_SYSEX -->|"SysEx"| HW_SYSEX_BUF["<code>RingIO</code> buffer<br/><code>_sysex_buf</code><br/>(via <code>scratch_buf</code>)"]
    HW_WRITE_DATA[/"From<br/><code>write_data()</code> (<code>OutPortUART</code>/<br/><code>OutPortPIO</code>)"/]
    HW_WRITE_DATA -->|"Standard data"| HW_DATA_BUF["<code>RingIO</code><br/>buffer <code>_data_buf</code>"]
    HW_RT_BUF & HW_SYSEX_BUF & HW_DATA_BUF --> HW_DATA_FLAG["Set <code>_data_flag</code>"]
    HW_DATA_FLAG --> HW_RUN["Await <code>_data_flag</code>"]
    HW_RUN --> HW_HAS_DATA{"Which<br/>buffer has<br/>data?"}
    HW_HAS_DATA ---->|"Real-Time (prioritize)"| HW_SEND_RT[/"Send Real-Time bytes<br/>to hardware (UART/PIO)"/]
    HW_HAS_DATA ---->|"SysEx"| HW_SEND_SYSEX[/"Send SysEx data<br/>to hardware (UART/PIO)"/]
    HW_HAS_DATA ---->|"Standard data"| HW_SEND_DATA[/"Send standard MIDI data<br/>to hardware (UART/PIO)"/]

    USB_WRITE_RT[/"From<br/><code>write_real_time()</code> (<code>OutPortUSB</code>)"/]
    USB_WRITE_RT -->|"Real-Time"| USB_RT_BUF["<code>RingIO</code> buffer<br/><code>_rt_buf</code>"]
    USB_WRITE_SYSEX[/"From<br/><code>write_sysex()</code></br>(<code>OutPortUSB</code>)"/]
    USB_WRITE_SYSEX -->|"SysEx"| USB_SYSEX_BUF["<code>RingIO</code> buffer<br/><code>_sysex_buf</code><br/>(via <code>scratch_buf</code>)"]
    USB_WRITE_DATA[/"From<br/><code>write_data()</code><br/>(<code>OutPortUSB</code>)"/]
    USB_WRITE_DATA -->|"Standard data"| USB_DATA_BUF["<code>RingIO</code> buffer<br/><code>_data_buf</code>"]
    USB_RT_BUF & USB_SYSEX_BUF & USB_DATA_BUF --> USB_DATA_FLAG["Set <code>_data_flag</code>"]
    USB_DATA_FLAG --> USB_RUN["Await <code>_data_flag</code>"]
    USB_RUN --> USB_HAS_DATA{"Which<br/>buffer has<br/>data?"}
    USB_HAS_DATA -->|"Real-Time (prioritize)"| USB_MIDI_WRITE_DATA["<code>write_event</code><br/>(<code>USBMidi</code>)"]
    USB_HAS_DATA -->|"Standard data"| USB_MIDI_WRITE_DATA
    USB_MIDI_WRITE_DATA --> USB_TX_BUF["<code>RingIO</code> buffer <code>_tx_buf</code>"]
    USB_HAS_DATA -->|"SysEx"| USB_MIDI_WRITE_SYSEX["<code>write_sysex</code><br/>(<code>USBMidi</code>)"]
    USB_MIDI_WRITE_SYSEX --> USB_TX_BUF["<code>RingIO</code> buffer <code>_tx_buf</code>"]
    USB_TX_BUF --> USB_SEND[/"Send MIDI data<br/>to USB host"/]
Loading

Key Points

  • All outgoing messages are first queued in ring buffers.
  • An asyncio background task merges and sends messages in correct priority order.
  • Awaitables (_data_flag, asyncio.sleep) ensure non-blocking operation and responsiveness.