Making multi-keystroke macros more reliable?

[RedBearAK]

I've noticed that when trying to use macros they have a tendency to somehow crash or just stop outputting in the middle. Specifically this is with a multi-keystroke type of macro, like:

[K("x"), K("y"), K("z"), K("Space"), K("x"), K("y"), K("z"), K("Enter")], 

Ordinarily I would expect this to output:

xyz xyz
xyz xyz
xyz xyz
xyz xyz

With one line for each time I trigger the shortcut.

But what often happens, even if I wait a second between each trigger of the shortcut, is things like this:

xyz xyz
xyz xyz
xyz xyz
xyz xyz
xyz xyz
xyz xyz
xxyz xyz
xyz xyz
xyz xyz
xyz xyz
xyz xyz
xyz xyz
xyz xyz
xyz xyz
xyz xyz
xyz xyz
xyz xyz
xyz xyz
xyz xyz
xx
xyz xyz
xyz xyz
xyz xyz
xyz xyz
xxyz xyz
xyz xyz
xyz xyz
xyz xyz
xyxyz xyz
xyz xyz
xyz xyz
xyzxyz xyz
xyz xyz
xy
xyz xyz
xyz xyz
xyz xyz

I thought maybe the more keys in the macro, the more likely one of these hiccups will happen. But it actually often seems to happen right near the beginning of the macro sequence, if it's going to happen at all. Notice how there are a number of lines where the macro stopped after the first, second or third character, but really no lines that stop near the end.

Is there any way to tighten up the code to keep this from happening? Seems like the function responsible just develops a poor working memory at times. Or a buffer is getting overwhelmed with keystrokes coming in too fast? Maybe the keystrokes need a few milliseconds of delay before the next one?

I don't know, I'm just throwing out ideas here.

@mooz @Lenbok @rbreaves

[joshgoebel]

Maybe the keystrokes need a few milliseconds of delay before the next one?

It would be something if you could actually confirm that hypothesis... the code you're looking for is in transform.py#handle_commands... you want the for command in commands loop which is sending the array of "commands" to the output... it'd be easy to add a delay there.

Otherwise that sounds very strange behavior indeed.

[RedBearAK]

@joshgoebel

I've tried putting many instances of sleep(1) throughout that loop in handle_commmands. Other than the fact that I have to do a second import time to keep from producing a crash when I start xkeysnail, the sleep lines seem to have no effect on how a macro gets sent to an application. The fact that there is a crash when I get the syntax wrong means I'm editing the right file, but nothing I do changes the speed of the macro.

/usr/local/lib/python3.10/dist-packages/xkeysnail/transform.py is the file I'm editing.

I use this somewhat lengthy macro in Firefox to test:

define_keymap(re.compile("Firefox", re.IGNORECASE),{
    K("C-comma"): [
        K("C-T"),K("a"),K("b"),K("o"),K("u"),K("t"),
        K("Shift-SEMICOLON"),K("p"),K("r"),K("e"),K("f"),
        K("e"),K("r"),K("e"),K("n"),K("c"),K("e"),K("s"),K("Enter")
    ],
})

When this works it seems to type at about the speed of someone typing at maybe 500 wpm. But it only works properly about 50% of the time.

[joshgoebel]

And what does the console say... does it log all the keys there despite them not seeming to get all the way to Firefox?

the sleep lines seem to have no effect on how a macro gets sent to an application.

Can you show me where exactly you're adding the delays?

[joshgoebel]

Other than the fact that I have to do a second import time to keep from producing a crash when I start xkeysnail

Do you have an isuse for that? That sounds strange - perhaps something is weird with your system... I'd try to figure out the simpel things first before delving into key delay issues. :)

[RedBearAK]

@joshgoebel

The crash was probably down to some misuse of python syntax, like using sleep instead of time.sleep.

I've seen the same results on multiple different computers and multiple unrelated Linux distros over the course of the last couple of years. I don't think it has anything to do with my system.

Do you not have any issues with macros at all? Or have you not run the kind of test I ran that started this thread?

If you can do the same sort of test with perfect results 100% of the time, then maybe this is actually a problem with the specific config from Kinto.sh or the patched branch of xkeysnail that Kinto.sh uses to support "held keys" combos like Alt+Tab-Tab-Tab...

The view in the terminal when I remove --quiet is just showing the key combo I'm using, such as RC-Comma. I don't know how to see what xkeysnail is actually trying to send to the application.

As for where I tried to put the sleep(1) lines, I put one right after every line in handle_commands with a colon at the end of the line. All the if/elif lines inside the for loop. None of them seemed to have any effect on anything.

[joshgoebel]

Do you not have any issues with macros at all? Or have you not run the kind of test I ran that started this thread?

I'm new here. Haven't used the software much trying to decide if it's right for me and if I want to step up as part of a new small team to fork and continue development, respond and fix issues like these, etc. :-)

the patched branch of xkeysnail that Kinto.sh uses

Possibly but his changes to output have more to do with modifier states than plain keys... actually the plain key stuff is pretty boring which is why these glitches are surprising to me.

I don't know how to see what xkeysnail is actually trying to send to the application.

Ah, you're right... i was thinking of poiting you d7901ec but that's only logging in the transform, what you want is some logging in the output.

using sleep instead of time.sleep.

Yeah it's time.sleep... and you'd need to import time of course...

I put one right after every line in handle_commands

That'd be hard to miss but you only really need one at the top of the main loop:

for command in command_list:
  # add a delay

But see my next message about adding this to output itself, I think that'd be better.

[joshgoebel]

I don't know how to see what xkeysnail is actually trying to send to the application.

You'd want to add logging to send_key_action and send_combo, just a print at the top to print the argument should be sufficient to give you some workable output... if you see the output on the console but the keystrokes aren't coming thru then something most be gummed up in uinput... perhaps adding a small delay there... since all output goes thru send_sync() perhaps try:

def send_sync():
    _uinput.syn()
    time.sleep(0.1)

You shouldn't need anything crazy like sleeping for a whole second.

[joshgoebel]

@rbreaves Have you heard of any similar bug reports?

[joshgoebel]

https://stackoverflow.com/questions/70324709/uinput-seems-to-break-while-sending-inputs-in-quick-succession

This seems super relevant... (though no answer), but it's the same problem you're describing... So perhaps even a 25 ms sleep in sync would help.

output.py

def send_sync():
    _uinput.syn()
    time.sleep(25/1000) # sleep 25ms

That would kind of suck for typing out long macros though if true... maybe play with a delay before or after sync and if it makes a difference...

[joshgoebel]

I use this somewhat lengthy macro in Firefox to test:

I just ran this 20-30 times in rapid succession doing:

• Ctrl-, (trigger the macro, open preferences tab)
• Ctrl-w close the tab
• rinse/repeat

Every once in a blue moon Firefox would lag on the input a hair, but it worked every time, 100% consistently. I definitely think you're experience some type of edge case that not everyone is running into.

Note: This using 0.4.0/master... if you think the problem is the kinto version you'd need to test on the mainline version to confirm whether the issue persists or not. Should be easy to rule kinto in or out as the source of the trouble with that one test.

[RedBearAK]

@joshgoebel

Of course I know just enough about how a for loop works that I put a sleep line right at the beginning of the loop. The use of one second was just because smaller values seemed to have no effect. I like to make things blindingly obvious when I'm troubleshooting.

def send_sync():
    _uinput.syn()
    time.sleep(25/1000) # sleep 25ms

Alright, so I found this in output.py, and wow, that's something to be really careful with. Finally had an effect, at least. But 25ms is quite a long delay, and it puts that delay between every key press or combo you type. EVERY key press. So 25ms delay before Alt+Tab switches to another app, etc.

BTW, had to do another import time in that file above the def send_sync to avoid an error when trying this.

I shortened it down to 5ms (5/1000) and that's short enough to not interfere much with regular typing and shortcuts, but it definitely still has a strong effect on how quickly the macro gets entered.

And, even that very short delay seems to make the macro a lot more reliable for me. I just did the Firefox prefs shortcut like 30 times in a row without a failure. There are the usual variations in how fast it seems to happen, even appearing to enter all the text almost instantaneously at times, while other times it's slow enough to look like someone "typing" really fast. But it worked 100% of the time for as long as I had the patience to test it.

If I do it so fast that another tab opens before the first is finished, the full text will now always be there in the URL bar but sometimes it apparently loses keyboard focus at the point where it would have sent the Enter key to finish the macro, so the tab is just sitting there with about:preferences in the URL bar, but a blank window and the generic "New Tab" Firefox icon instead of the Preferences gear icon. Obviously this is not a normal thing the user would do.

Now to tune the delay. I'm kind of surprised that such a short delay is already effective. The 25ms that the person was talking about in the link about uinput is almost an order of magnitude longer delay. That would indicate a rather severe problem with uinput receiving mouse clicks.

I still would like to understand why none of my attempts to insert a delay in the for loop amounted to anything. Don't understand that at all.

...

Still working with as little as 1ms as a value. Still seeing an extraordinary amount of variability in terms of how fast the macro actually seems to get sent to the app. Sometimes too fast to see anything, as if the full text was just pasted all at once. Other times, almost as slow as my fastest typing speed on a good day. Very visibly "typing" each individual letter slowly enough for me to watch it.

But, it's still succeeding at actually completing the entire macro without errors or missing characters, even after doing about 50 preferences tabs. Even when it acts much more slowly on occasion.

I think maybe the variability is simply down to the reliance on the system sleep command, which isn't real-time accurate in any way, from what I understand. And it looks like 1ms is actually the minimum possible value that would be meaningful.

It looks like my supposition that some kind of input buffer was failing must be correct, in a way. The question is what, and where.

I've even had a lowlatency kernel installed on Ubuntu for a while now, just for kicks. It appeared to have no effect on this issue.

[joshgoebel]

The use of one second was just because smaller values seemed to have no effect.

Yeah I understand, just with 1 second you're starting to get into the area where the delay could cause it's own kind of problems is all. :-)

Still working with as little as 1ms as a value.

Unfortunately I don't know what to tell you - or where we'd go from here... I don't think adding a 1ms sleep here when no one else is reporting any issues is the correct solution, but I'm also not sure exactly how we'd go about gathering additional data.

Still seeing an extraordinary amount of variability in terms of how fast the macro actually seems to get sent to the app.

That seems strange to me since all that is happening on the Python side is a tight loop and a tiny amount of I/O... I saw some variability but I'd assume it was the Firefox UI, not xkeysnail.

It looks like my supposition that some kind of input buffer was failing must be correct,

You'd think... but we're talking a buffer with almost 0 capacity? Since it short circuits right away in so many of your tests it would seem the buffer is almost non-existant, which is weird.

[joshgoebel]

It looks like time.sleep() only has millisecond resolution. Though it could range between a few milliseconds to over 20 milliseconds (best case scenarios), based on what OS you are using, it does not look like you will get accurate microsecond sleep execution from it.

https://stackoverflow.com/questions/58553072/how-to-make-a-delay-at-100-200us

Could by Python itself that's the issue as evidentally this isn't what sleep was designed to do...

[joshgoebel]

https://github.com/torvalds/linux/blob/master/drivers/input/misc/uinput.c

Looks like uinput doesn't change that much and the buffer size is 16... 16 events I'm assuming...

[RedBearAK]

@joshgoebel

Yeah I understand, just with 1 second you're starting to get into the area where the delay could cause it's own kind of problems is all. :-)

Hey, when you're not catching any fish the usual way, a round stick of material capable of rapidly oxidizing after ignition can come in handy. 🤘

Unfortunately I don't know what to tell you - or where we'd go from here... I don't think adding a 1ms sleep here when no one else is reporting any issues is the correct solution, but I'm also not sure exactly how we'd go about gathering additional data.

I find it very strange that nobody else has seemingly ever had this issue. I would attribute that to most users simply not using macros much, and an even smaller percentage attempting to use long macros with a higher probability of failing. Unless it really is somehow the Kinto branch that has this issue, but in that case I would expect other Kinto users to complain, especially about the Firefox preferences shortcut. But it's not even limited to Firefox, I had a similar issue with Chrome at times when I tried to use a similar macro to get to settings. And of course the testing in the text editor from the first post.

I don't see adding a 1ms delay to be harmful, but that's just my perspective. The question is why does such a ridiculously short delay, apparently the absolute minimum possible with time.sleep, seem to completely fix this. I literally haven't been able to get macros to fail since this change. I just did 100 lines of this text with a macro in the new GNOME Text Editor app, with no failures:

now is the time for all good men to come to the aid of their country.
now is the time for all good men to come to the aid of their country.
now is the time for all good men to come to the aid of their country.
...

There is a significant delay when I try to hit the shortcut multiple times, but when the entire buffered output (could be many lines, nothing shows up as long as I keep tapping the shortcut too quickly) finally shows up in the app, there are never any errors. No missing Enter at the end of line, no incomplete lines. Nothing. Works perfectly. (Well, see below.)

I just held Ctrl+Shift+Alt and "buzzed" the "t" as fast as I could, to do this macro. Nothing shows up for at least 5 seconds, then I get 25 new perfect lines of the same text. No problems other than the delayed output.

I can't even pretend to understand exactly what is failing without the delay. The buffered macro lines I'm inserting into the text editor are obviously WAY more than 16 characters, yet they never have errors. So the uinput buffer would not appear to be the problem... unless it really doesn't like receiving characters too fast, maybe?

I need the patched 0.3.0 xkeysnail to make things like Alt+Grave and Alt+Tab work correctly, so I'm not sure how to safely test the 0.4.0 branch, but I might do it sometime. I really would like to get this permanently solved.

Uh oh, looks like I was too optimistic. I finally got a couple of lines to show up in the text editor with missing characters. Haven't changed anything since previous testing.

now is the time for all good men to come to the aid of their country.
now is the time for all good men to come to the aid of their country.
now is the time for all good men to come to the aid of their country.
now is the time for all good men to come to the aid of their country.
now is the time for all good men to come to the aid of their country.
now is the time for all good men to come to the aid of tntry.
now is the time for all good men to come to the aid of their country.
now is the time for all good men to come to the aid of their country.
now is the time for all good men to come to the aid of their country.
now is the time for all good men to come to the aid of their country.
now is the time for all good men to come to the aid of their country.
now is the time for all good men to come to the aid of their country.
now is the time for all good men to come to the aid of their country.
now is the time for all good men to come to the aid of their country.
now is the time for all good men to come to the aid of their country.
now is the time for all good men to come to the aid of their country.
now is the time for all good men to come to the aid of their country.
now is the time for all good men to come to the aid of theountry.
now is the time for all good men to come to the aid of their country.
now is the time for all good men to come to the aid of their country.
now is the time for all good men to come to the aid of their country.
now is the time for all good men to come to the aid of their country.

This is doing it slowly, letting each line appear before trying to do it again.

Interesting that these failures are near the end, where testing in the first post almost exclusively saw failures near the beginning of the macro.

Doing dozens more lines in the same way results in no other errors.

I don't know what to tell me either.