Is it a prepositional phrase or an infinitive phrase? In this phrase, we have to preceding a word that could be either a verb in base form or a noun in singular form.
Some other examples of this pattern, along with some approximate frequency data as to how the last word is used generally, include:
(1) He went to work. (48% V, 52% N)
(2) He went to school. (1% V, 99% N)
(3) He went to run. (75% V, 25% N)
I think (1) is entirely ambiguous as to whether work is a noun or verb. As a noun, work usually refers to the act of working, but it also connotes a place at which one is employed. In the latter sense, it can serve as a complement to the preposition to. But, depending upon context, the speaker might intend the verb sense of the word. In the case of (1), context makes a difference.
As for (2), the data suggests that school could be regarded as one of those singular nouns that can occur without a determiner and so is acting as a noun complement to the preposition to. Yet, why not interpret school as an intransitive verb having the sense: to educate in an institution of learning? An example of that usage is, "The child was schooled at great cost to her family." In that light, it seems natural to read to school as an infinitive phrase. In the case of (2), context seems to make no difference.
In (3), I think run must be a verb. I don't perceive any noun sense for the word that could be used without a determiner.
Like school, bed is used as a noun about 99% of the time, it has an intransitive verb sense, and context seems to be of no help in establishing the function of the word.
Perhaps the answer is that the absence of determiners conclusively indicates that to bed is an infinitive phrase. I'm really not sure.