Although "having" is not wrong in that sentence, it does sound a bit odd. It could be replaced by "that has" or "which has":
The computer is a machine that has different parts.
The computer is a machine which has different parts.
I feel that "having" is reserved for more unusual/interesting information:
The piano is the instrument having the greatest number of octaves.
So by saying "having" you're implying that the information is more interesting than usual. Most machines have different parts, so it's not unusual that a computer, being a machine, would also have different parts.
Using "having" places emphasis on the information, whereas "with" does not.