Here is what I did if I remember: For everything that what "grab" I used the standard interactor and interactable prefabs. For things like buttons or drives based on physics, I copied the interactor game object, turned off "Is Trigger," and set the constraints in the rigid bodies.
I also made a simple script where that physics interactor would enable if I was in a proximity to the thing I was to interact with, like a button. I normally use a game object with a sphere collider to detect first, enable, and then disable when I leave the collider.
This way I had two interactors on each hand but only one would be active at a time... Again, from what I remember