Firstly, you want to say "take [X] off you", or "off of you", not just "of you".
Secondly, preferred usage will differ from place to place. In my answer, I am speaking for Australian English.
I have to take two marks off you.
This is correct. If your score would have been 18 out of 20, but your punishment is to receive 16 out of 20 instead, this is a good way to say it.
I have to take two grades off you.
This is probably not correct. A "grade" is a final or overall result. It typically goes on a short scale like "A, B, C, D, F", or "High Distinction, Distinction, Credit, Pass, Fail", or even just "Pass, Fail". So to "take two grades off" would mean either:
- To reduce this result by two steps on the scale (e.g. from B to D).
- To entirely remove two results from the student's record. This is a very strange thing to do!
In both cases, this is an awkward way of saying it. If you actually do mean this, say instead "I have to reduce your grade by two steps", or "I have to remove these two results from your record".
I have to take two scores off you.
This is almost certainly not correct. It is like "grade", option #2 (remove two results from the record).
I have to take two points of you.
This is correct. "Points" and "marks" are interchangeable words for "the basic unit of judging an assessment". "Marks" is somewhat more common among people I work with or teach.