They all have slightly different meanings:
Realize (or realise, UK) - means you have come to a realisation of something. It is the end of a thought process, but sometimes perhaps an "unintentional" or subconscious thinking. One metaphor sometimes used in place of "a sudden realisation" is "a light-bulb moment".
Notice - literally means that you have seen something. That can also lead to a realisation of something, but it is not technically the same.
Find out - this describes the very deliberate process of discovery. Perhaps researching something.
Figure out - would normally be used idiomatically in place of "find out", but I would say it better describes the thinking stage of that process whereas "find out" suggests investigation, perhaps going out searching for information. I am a British English speaker, but I am aware that "figure" is used more often and a little more glibly to mean "thought".
As regards which word to use in your sentence - I actually think none of them are completely ideal:
I (knew/noticed/found/found out/figured/figured out/realized) that it could be a famous writer signing books.
This is because you are not identifying the person signing books as a famous writer. In fact the sentence implies that you are not sure, as you say it only "could be" a famous writer.
Personally I would say:
I wondered if it could be a famous writer signing books.
But perhaps, considering that "glib" use of "figured" that I mentioned seems to exist in American English, maybe you could say:
I figured that it could be a famous writer signing books.
I figured that it might be a famous writer signing books.