The adverb casually simply means "in a casual manner", so it'll map to one of those definitions that are so easily found in the dictionary.
NOAD lists four main meanings:
1 relaxed and unconcerned : she regarded his affairs with a casual indulgence | he tried to make his voice sound casual.
• made or done without much thought or premeditation : a casual remark.
• done or acting in a desultory way : to the casual observer, rugby looks something like soccer.
• done or acting without sufficient care or thoroughness : the casual way in which victims were treated.
2 not regular or permanent : the tent is ideal for casual outdoor use | casual jobs.
• (of a worker) employed on a temporary or irregular basis : casual staff.
3 [ attrib. ] happening by chance; accidental : he pretended it was a casual meeting.
4 without formality of style, manner, or procedure, in particular
• (of a place or environment) relaxed and friendly : the inn's casual atmosphere.
I'm thinking that your question is really getting at: Which of these meanings does casually map to?
Most of time time, it'll be that first meaning: "relaxed and unconcerned," although there might be times when Meaning #4 would be a more precise match. That said, in regards to conversation, the difference between Meanings #1 and #4 would be trivial.