The past perfect and simple past are correct but mean different things. Please consider the following situations:
1) When I came to your house yesterday, you had gone to your uncle's house. [At the time I went to your house, you had gone to your uncle's house, before I arrived.]
2) Question and Answer: When I came to your house yesterday, you were not there. Where were you?
Answer: I had gone [I'd gone] to my uncle's house. [same explanation as 1)]
compare that to:
3) When I came to your house yesterday, you went to your uncle's house.
Here, the meaning is very different: It means that when "I came to your house", "you went to your uncle's house" at the time I arrived at "your house".
[please note: we would say your house here rather than home. Home is used in the expression to go or come home or at home, but otherwise we say house unless we want to differentiate it from, say, the office, temple or church.]