I agree with Arianna that Montauk is very chill and has a nice community feel. Though it seems to be attracting more and more 18-24 crowd hitting the bars and overrunning certain areas on the weekends.
My wife and I go out for a few weeks every year. We are not big beach goers but do enjoy walking through towns and sitting at an outdoor cafe and people watching. Our favorites for that are Sag Harbor (nice town, less sleepy than it used to be but still my fave), East Hampton (largest town, more upscale, mobbed on the weekends) and Montauk.