Qubesd takes a really long time to query and manage USB devices with multiple USB hubs

[Atrate]

How to file a helpful issue

Qubes OS release

r4.3

Brief summary

Qubesd takes a really long time to query and manage USB devices with multiple USB hubs. This is quite annoying as qubesd uses 100% of one CPU core while doing that (in a blocking way), making it impossible to interact with Qubes GUI widgets or qubes-qube-manager (and other apps that rely on querying qubesd).

Steps to reproduce

1. Connect multiple USB hubs to your computer. In my case it's 2 daisy-chained hubs (one plugged into the other) that are integrated into my displays.
2. Attach a USB device to a qube or query usb devices

Expected behavior

Fast.

Actual behavior

Slow and making almost all dom0 Qubes GUI components unusable.

Additional information

timed while not much else was happening on the system:

user@dom0 ~ % time qvm-usb attach personal sys-usb:2-1.1.1
qvm-usb attach personal sys-usb:2-1.1.1  0.11s user 0.04s system 0% cpu 30.660 total

user@dom0 ~ % time qvm-usb >/dev/null 
qvm-usb > /dev/null  0.15s user 0.06s system 2% cpu 7.730 total

[Atrate]

Another timing test:

Restarting sys-usb on a running system makes it so that qubesd takes a whole 5 minutes to query all the USB devices after sys-usb starts!

[Atrate]

user@dom0 ~ % time qvm-usb attach personal sys-usb:2-1.1.3 
qvm-usb attach personal sys-usb:2-1.1.3  0.11s user 0.04s system 1% cpu 11.661 total

The situation has gotten better, possibly with some recent update (or UEFI update), but 11 (and up under load) seconds still leaves a lot of room for improvement.

[ben-grande]

Won't be able to continue on this issue at this moment. Talking with Marek, he indicated that there might be an issue of looping through more domains than necessary to find devices, and it should focus only on important qubes, in this case, the ones that have assigned and attached devices.

[ben-grande]

user@dom0 ~ % time qvm-usb >/dev/null
qvm-usb > /dev/null 0.15s user 0.06s system 2% cpu 7.730 total

I never got close to that total time. I did get:

• 0.3s with sys-usb not running
• 0.5s with sys-usb running
• Plugged two usb hubs with 4 ports, one connected to the last port of the other, same time while they are empty
• I added a device one by one, I noticed a continuous increase of something closer to 0.1s for every device I add
• 1.0-1.5s after I filled 1x port using the following tree format:
  • 1x USB 3.0 HUB
    • 3x 8GB USB 2.0
    • 1x USB 2.0 HUB
      • 1x security key,
      • 1x 8GB USB 2.0
      • 1x 128GB USB 3.0
      • 1x 256GB USB 3.0

Notes:

• Size of storage device and moving them between hubs and ports didn't seem to affect the time to generate the listing.
• I only got listing delays in multiple seconds when I just had plugged the device to the system, but that's expected I presume.
• I did notice that when plugging multiple devices simultaneously or querying with qvm-usb, qvm-ls --raw-list would take more than a second, which when such device action is not happening, it takes 0.2s. It normally waited for the query to end, apparently.
• I did notice that my PR to make some parts of the client async Schedule synchronous function to separate thread qubes-desktop-linux-manager#312, still blocked reopening the widget when plugging multiple devices at the same time (plug a USB HUB that's filled), and I think we need to fix this in the server, as it's blocking other API calls, after that, we can check if there is blocking calls that take a long time on the client.
• I did notice that with my PR Schedule synchronous function to separate thread qubes-desktop-linux-manager#312, attaching devices to disposables that aren't running yet, didn't block reopening the widget, so this is my way of testing a long attaching routine on the widget

Can you share more details so we can attempt to reproduce this issue with as much delay as you got? I think I have some starting ground, but if I can increase the delay with devices instead of adding artificial delays, would be much easier to understand the behavior and debug.

[Atrate]

Can you share more details so we can attempt to reproduce this issue with as much delay as you got? I think I have some starting ground, but if I can increase the delay with devices instead of adding artificial delays, would be much easier to understand the behavior and debug.

Could I send more details, such as the list of devices or extensive logs in a more private way, such as via e-mail + PGP? What address should I use?

[marmarek]

Maybe it's related to the number of qubes, or number of running qubes?