I was getting some issues with my thinkpad middle button and drawterm. Turns out it can be fixed in sway config https://makerdon.org/@nocko/104492157322589780
Should be applicable to X11 too with xinput magic.
Thanks to the magnificent grid people for the troubleshoot!

cc. @solderpunk