U. S. Usage
Officially, it’s called an ice cream bar or people will use the brand name “Popsicle” or, less commonly, “Fudgesicle.”
Except for the brand names, we don’t have a term (a generic word) for “one with a stick”. An ice cream bar without a stick could also be an ice cream sandwich.
The grocery store sometimes labels these items “frozen novelties” but that’s not a term we normally (ever) use.
U. K. Usage
Based on the comments: in the UK it may be called an ice lolly (with a stick) or a choc ice (with nonstick, and a chocolate coating.)
“An ice cream” (with the word an) is used to refer to one with a stick, and “ice cream” (with the zero article, and considered a mass noun) means something that might be served from a large tub.
I’ll have an ice cream
I’ll have some ice cream, or I want ice cream.
The ice cream bar is distinct from the popsicle, which does not contain any ice cream.
A popsicle is
a brand name (trademark) for a sweet piece of ice with a fruit flavour on a small stick
Common Usage (How U. S. Kids Talk)
So according to the dictionary, a chocolate or vanilla one is an ice cream bar, but only a fruit one is a popsicle.
I’m here to tell you that at my house, regardless of flavor, my kids would say “I want a popsicle.”
When I correct them, they would say “Can I please have a popsicle?
They should say, “May I (please) have a popsicle?”