twisting handlebar with aero bar

I have clip on aero bars on my handlebar and I can't get the stem clamp
on the handlebar tight enough to maintain the position of the
handlebar. Normally, the handlebar stays in position but when I'm
riding on the aero bars (which is the norm) and I hit a bump, I hear a
little click noise and the handlebar rotates downward which puts my
aero bar out of position (tilted down). I've tried re-positioning the
handlebar and re-tightening as tight as I can but the handlebar
continues to rotate when I hit a bump.

So I'm forced to conclude that there is not enough friction between the
stem clamp and the handlebar. I've considered that maybe I need to
remove the stem clamp (it's threadless and easy to do) and sandpaper
the clamp to roughen it. Or perhaps I could insert double-sided
sandpaper to give some grip.

Anybody else have this problem and any suggested solutions?

Denis