What can you be thinking of?
What could you be thinking of?
I don't see any semantic difference between these two sentences.
[EDIT:] It's possible that can is a little more direct and suggests a rebuke because the speaker thinks the other person is thinking something odd, and that could suggests puzzlement rather than a rebuke.
In another pair that uses can and could, however, there is a difference in register:
Can you please close the door?
Could you please close the door?
The second one is slightly more polite than the first, but they both mean the same thing.