Is it telling someone that he/she does not realize that what he/she is saying is wrong,bad,strange,etc and that they're asking them that to check on what they saying is wrong,bad,strange,etc?
Yes, it can be used that way, but it goes a bit deeper than that. It’s more if someone begins saying something ridiculous or with strange implications, or that shouldn’t be said in the current social climate, you might ask ‘do you know what you’re saying?’ — or alternatively ‘do you realise what you’re saying?’ — and then spell out to them why it’s ridiculous or strange or shouldn’t be said.
So if they said something ridiculous, like ‘Antarctica should be handed over to the government of Malaysia,’you might respond with: ‘Do you know what you’re saying?’.
The second kind is where someone says something that has, perhaps, political or philosophical implications. If, for example, someone said ‘Negative eugenics is excellent!’, I might ask ‘Do you know what you’re saying? If that’s true, Hitler was right!’
The third kind, where what they’re saying probably shouldn’t be said in the current social climate or political climate, is more common in spy or dystopian fiction. For example, if one character in a book about Nazi Germany said out loud ‘Down with Hitler!’, their friend would grab them and hiss ‘Do you know what you’re saying?’, because they would get arrested.
Hope that helps!