I think he doesn't touch Olive as much (well, he touches everyone very little) because it's a physical manifestation of proximity to someone else - and he's been distancing himself from everyone his entire life. Because Olive wants to touch Ned, I think that frightens him as well.