I know that "to" can be a preposition and an adverb but what is the term for it in front of an infinitive?
It is referred to in many different ways. Because a lot of people aren't sure what part of speech it is, it is often just called infinitival to. It is also sometimes referred to as the infinitival particle.
For many modern grammarians, to is a subordinator with the syntactic function of marking infinitival verb phrases as subordinate (and therefore also termed a Marker - but note that Marker is a syntactic function and not a part of speech).
Because historically infinitival to was originally a preposition, some dictionaries list it as a preposition. However, very few modern grammarians or linguists think that infinitival to is a preposition in modern English.
Other grammarians argue that to is a unique word that doesn't belong to any other part of speech. They say that it is syncategorematic.
However, the most convincing argument is that infinitival to is a non-finite (in other words tenseless) auxiliary verb. This is what has been argued by linguists such as Geoffrey K Pullum.
Note for linguistics students:
If you are a linguist, or a linguistics students, you can read a linguistics paper about why infinitival to is a non-finite auxiliary verb here. It's quite tough, but interesting.
It's interesting to observe that Dictionaries present different definitions, but the majority has explained it under title (label) of preposition.
Cambridge: Preposition: used before a verb to show that it is in the infinitive.
Oxford: Infinitive Marker: Used with the base form of a verb to indicate that the verb is in the infinitive.
M.Webster: Preposition: used as a function word to indicate that the following verb is an infinitive.
The A.H.: Preposition: Used before a verb to indicate the infinitive.