If you wish to assign other functions to the X1 & X2 mouse buttons besides Previous & Next, it seems like the only way would be to use some kind of 3rd party software, or the native driver configuration software that came with the mouse (if any).
Basically you would have to configure those buttons to produce a keystroke combo (in addition to or instead of the native X1 X2 click output).
One potential candidate (and a very powerful one) that comes to mind is AutoHotKey (which is also free and very well supported). I would suggest having those mouse buttons trigger a very difficult or impossible to type key combination, and then you can assign it to any QTTabBar function you like from the list.
For instance, you could setup your X1 & X2 mouse buttons to also trigger [Control]+F23 & F24 (respectively) by adding the following lines to the default AutoHotKey script (which is loaded by default at login). You would need to uncheck the option to capture/use the X1 X2 buttons in QTTabBar (so they don't trigger that as well), and then assign the buttons to whatever function you like.
~XButton1::SendInput ^{F23}
~XButton2::SendInput ^{F24}
The tilde (~) at the beginning of each line prevents the suppression of the normal output of each mouse button (so you can still use it in your web browser or whatever), and the caret (^) represents the [Control] key. And the function keys are inside curly brackets {} otherwise the output would actually be like literally typing F23- or rather [Control]+F followed by 23. You really only need the caret if you haven't checked the box at the top of the Keyboard Shortcuts page (in QTTabBar options) to allow the use of single keys (without modifiers)…
Personally, I think that the function keys (F1-F24) should be automatically excluded from that requirement because they are designed exclusively to be used as hotkeys- to trigger functions… unlike ALL the other regular keys on a keyboard. But that is not how Quizo made QTTabBar.
Hope that helps someone.
There are certainly lots of other, much more complex & specialized, ways you could configure AutoHotKey (such as having the output of those buttons vary based on the active app/window, etc), but the example I gave is the simplest and best possibility I could think of that would be least likely to affect or interfere with any other use/functions you may already have for the 4th & 5th mouse buttons.