Welcome aboard CoachK!!!
Going from 20 - 24 won't be that bad. You'll be used to it in no time. Remove the door and the mounting brackets and you should be fine to get the fridge out. I suppose if hand tools, grunge, rust and dust bunnies makes you squeamish, then sure it'd be a 'nightmare' I guess.