Actually I'm still shopping for the bike rack, but that's a very good
point to remember. I would probably remove the bike rack anyway, just to
make trunk access easier (I don't want the extra weight and expense of
the fancier racks that swing out of the way).

What effect, if any, does just the receiver hitch have on the car in a
rear impact?

SMS wrote:
On a unibody vehicle (nearly every car and car like SUV) even a small
impact on the rear rack will cause many thousands of dollars worth of
damage. The bumper is out of the equation, and the impact goes to the
floor pan where the hitch will be mounted.

I know this from experience unfortunately.

If you use a hitch rack remove it completely unless you have a bicycle
on it.