Why do we say the sun is out? We don’t say the candle is out or the light bulb is out. Why don’t we say the sun is visible?
That's a good question, and for many questions like this, there may not really be a single, known answer about how it came about, or why people say something one way but something similar a different way (known as "idiomatic speech"). Nevertheless, here's my opinion or educated guess:
Unlike other light sources, the sun is, first of all, something that's entirely beyond human control. There's no turning it on or off, no controlling when it's visible, etc. Because it is something that really exists and behaves on its own, it is common for people to use language phrases to describe such a thing which give the object volition, that is, that make it sound like the object is the one acting on its own behalf, like a person would. That is why you will often encounter things like:
The sun was hiding behind the clouds all day today.
(whereas you wouldn't usually hear somebody talk about a light bulb "hiding" behind a lamp shade)
Obviously, the sun isn't actually deciding whether or not to be behind the clouds, but it's a fun way to add some color to talking about things that happen that we have no control over. (Of course, in many cultures, historically, many people did actually believe that the sun and other celestial bodies had wills of their own, which may also have contributed to this sort of way of looking at and describing things as well.)
So what happens when the sun isn't hiding anymore? Well, given its volition, it must have decided not to be behind the clouds anymore and:
The sun has come out from behind the clouds.
(which is also a fairly common expression)
Obviously, at this point, the sun is "out" from behind the clouds. Over time, I suspect this phrase came to be used even when there weren't any clouds around as just a general meaning of "it's sunny":
The sun is out, and it's a beautiful day today.
And there we go.
(As a side-note, there's really nothing wrong with saying "The sun is visible", either, though it does sound a bit more technical/formal, rather than something you'd say in casual speech. An alternative to "The sun is out" which does have the same casual feel might be "The sun is shining", which people do also often say a lot.)
"The Sun is out" is understood by analogy with out in sentences like these:
Is Johnny coming out to play today?
Hot jazz guitarist Ronnie Jordan's new album is out. (Source)
Never did I love anything so much as that spring, when the trees burst into leaf and the primroses came out. (That is, when the primroses bloomed. Source.)
The one place I was allowed to go by myself was the movies. … My parents wouldn't let me go out anywhere else, even when I was twenty-four. I didn't care. I wasn't used to going out, going anywhere outside of going to the movies. (Source: Going Out: The Rise and Fall of Public Amusements by David Nasaw (1999).)
When the boat is out on the water, you can get a feel for its ride, responsiveness and performance characteristics. (Source)
Who let the dogs out? (Wikipedia)
We do light the Shabbat candles very often and I am happy when my children think of it themselves and bring the candles out on Friday evenings. (Source)
These all have in common the idea that when someone or something is out, it is in the public sphere, active, participating in public life or at least publicly visible. "In" is not (usually) the opposite of "out" in this sense. The opposite of "out" in this sense is to be in a private, secluded place, inactive, protected, or perhaps in preparation for public participation in the future, as in "My new film is not out yet. We're still editing."
Hopefully now it's easy to see how the Sun makes a natural analogy here. Saying that the Sun is "out" likens its emergence from the clouds to "coming out" in public. You could say, fancifully, that the Sun's public activity is shining light on public life. A similar analogy occurs here:
In the country, you can see the stars come out at night.
You said this in your question:
We don’t say the candle is out or the light bulb is out.
Actually, we do say both of those things in English. However, they mean something different, because the word out has several meanings.
In the case of a candle:
out (adverb) extinguished
We use this sense of out with flames or fire: The candle is out. The forest fire is finally out.
In the case of a light bulb:
out (adjective) not operating or operational
For example: The light bulb is out. Do we have any new ones around here?
We might also use this sense of out when describing a blown or faulty fuse. This is also the same sense of out when you see a sign on a vending machine that says: OUT OF ORDER.
As for the sun, there are a couple meanings that align with out on a sunny day:
out (adverb) in the open air; outside; into view.
So, on a sunny day, we can say, "The sun is out," or, when a bright moon is prominent in the night sky, we might say, "The moon is out."
The moon was out and visibility was good.
Source: B. Parker, The Power of the Sea: Tsunamis, Storm Surges, Rogue Waves, and Our Quest to Predict Disasters, 2012.
It's a metaphor that treats clouds like a hiding place. When you're in a hiding place, you can't be seen. You become visible when you come out of the hiding place.
The word "out" is used similarly in a phrase like "Come out wherever you are", which would be said when playing Hide and Seek and admitting defeat.
A candle or a light bulb is either out or lit, whereas the sun is either out or in.
Per @James K (comment on the question), the British say "the sun has gone in" to mean that a cloud has come in front of it. I've never heard that in American English, but I think that is a strong indicator that the use of the word "out" in this context, even in American English, is meant as opposed to in.
The sun, unlike candles and lightbulbs, is always lit (insert pedantic caveat about billions of years in the past or future here). It is simply visible or not, obscured by the atmosphere, the Earth, the moon, or something else. So the "off" interpretation of "out" is not relevant here.
Candles and light bulbs, on the other hand, don't generally move around nor become obscured from a given viewpoint, so the "in" interpretation doesn't apply there.
English is hard.