Ordinals are used to describe something's order in a sequence. These are the "first, second, third" words you have mentioned. However, numbers by themselves can be used to denote points on a list.
Therefore, we should use ordinals when describing things to do in sequence:
First, we need to till the soil. Second, we'll plant the flower seeds. After that, it's just a matter of waiting.
On the other hand, if someone has multiple, separate points to address, they might verbally number them for emphasis:
Speaker A: What's wrong with you?
Speaker B: Well, for one, my feet are killing me after standing around all day in these heels. Two, I'm starting to get a migrane. Three, I just found out the train is out of service today so I'll have to pay for a ride home.
Ordinals can be used in the second example, too, but it implies that the points are listed in order of most important to least important. Adding -ly to the end just turns the ordinal into its adverb form.