Strictly speaking, adverbs modify verbs. (Although they can also modify clauses, adjectives, or other adverbs.)
There is only one verb in your sentences: speak.
So, well modifies either to speak or speaking (the different verb forms).
In the first sentence, the verb to speak could be considered to be composed of two words. But ice cream is considered to be a single word—even though you can see both ice and cream.
You could claim that well modifies two words in the first sentence, to and speak, but to speak is considered to be a single unit. You can refer to to speak as a word just as you can refer to ice cream as a word.
In practice, people switch between the one use of word and the other but are understood in context. If forced to answer the question "How many words are in ice cream?" most people would reply, "In what sense?" just to clarify the terminology being used by the person asking the question.
However, it's not quite as simple as that, depending on how you look at it.
Speak five languages describes something specific. The use of five languages limits and clarifies what is being spoken.
There is a difference between these two sentences:
Speaking well is an advantage.
Speaking five languages well is an advantage.
In the second sentence, it's something specific that's being spoken well, not just speech in general.
It could even be contrasted with this:
However, speaking six languages well is not an advantage.
So, even though well modifies the verb speak (expressed as either to speak or speaking), it also modifies the qualification of the verb and its object.