Duke provides this website to help graduate students find local housing. The housing listed is off-campus, which is typical for physics graduate students. Many people looking for roommates advertise, but the site is also useful for learning about specific realtors, complexes, and neighborhoods -- not all are created equal in Durham.
Examine the comments in the annual Housing Survey to learn what people who have lived in a location you are considering think.
Though housing near Duke's west campus (where the physics building is located) is usually either expensive or dangerous, other areas of Durham along the Duke bus routes provide great places to live without the need of a car.