feat: performance improvements inspired by SAQ

[oliverlambson]

TODO:

• handle db migrations
• review

[hmax]

Hey, are there any plans/rough timeline for this one? I am in need of some of the features that you implement in this PR and have actually implemented something similar in my fork (but would definitely prefer an upstream version).

Maybe there's something I could help with to make it move forward?

[oliverlambson]

Hey, are there any plans/rough timeline for this one? I am in need of some of the features that you implement in this PR and have actually implemented something similar in my fork (but would definitely prefer an upstream version).

Maybe there's something I could help with to make it move forward?

We're actually using a different worker/messaging system now internally, so maintaining this library is lower priority at the moment.

This PR is in a half-done state and I hadn't finished thinking through the implications of the changes. If you'd like finish it (or open a similar new one), I would be happy to review it and merge if all looks good.