Would putting a taller stem and different bars on the Avanti Sport be a
viable idea (I liked the bike) or would it screw up the geometry?

Sure you should try putting a taller stem. And then widest tires Avanti
can take. Suspension is for cross-country.

I had a Trek MTB with suspension, and it always hurt after 3-4 hours
riding. Now I have an older than me drop bars roady with a tall(ish)
stem and it doesn't hurt after 5-6 hours.