360 - Removal of Passenger Grab Handle

I am in the process of doing a DIY de-sticky on my 360. Everything was going smoothly until I got to the passenger door. The passenger "oh sh*t" handle is really sticky, so I removed the 2 allen bolts on the top bracket and the single allen bolt for the bracket by the door handle. The upper bracket will not separate from the door. It will wiggle so I know it is loose, but it will not pull out without a hell of a lot force - which I don't want to apply since I am afraid to damage something.

Is there some trick to getting the upper bracket to separate from the door?

 

Thanks Speedster. I was assuming that the fastener set-up was the same as on the 430, which apparently only uses 2 bolts into the door form inside the car.

How difficult is it to remove the entire door panel?