355 Shift Linkage Mod Question

While driving home last night I shifted into 3rd gear and the shift linkage came apart at the 'bar buffer' shown in the photos below. As the photos show, the molded rubber isolators broke apart and completely decoupled the shift lever from the trans.

I know that the purpose of the rubber is to dampen the system vibration and prevent buzzing at the shift lever. However, I've never been happy with the vague feel of the shift lever in my 355 so the question is whether there's any downside to replacing the rubber isolators with aluminum spacers? I did this years ago on a modded Mustang and it was vastly improved. Other than buzziness in the lever, any thoughts regarding the loads going into the trans that would argue against this? I'm wondering if the Challenge cars removed the buffers?
Image Unavailable, Please Login
Image Unavailable, Please Login