I suppose you could, but you would possibly be running the risk of damaging the insert or wheel trying to get them out or putting them back in. You would also still need to mask the hole that the insert sits in so you don't really again anything by removing the insert.
