Sure, there can modify you.
As McCawley says, pronouns don't "stand for" nouns but for entire noun phrases. Consequently you is a noun phrase, and (by default) its own Head.
You could pick up McCawley's recategorization of adverbs as Ad-Vs, Ad-Ns, Ad-Ss, &c and say that there here acts as an Ad-N´.
And if you want to sound really impressive and be really confusing, you could parallel that recategorization by saying that just as you is a Pro-N´, what there really is is a Pro-P´ or Pro-S, or whatever you think it stands for:
Oh, sorry, Weasley, didn't see you there.
Oh, sorry, Weasley, didn't see you in that place.
Oh, sorry, Weasley, didn't see you standing right in front of me.
Oh, sorry, Weasley, didn't see you blinking stupidly like the proletarian cretin you are.
Oh, sorry, Weasley, didn't see you offering an irresistible target.