About 2 and 4 ...
What must follow relative pronouns in general, including which, must be a clause with subject, verb, and anything the verb needs. A mere phrase without a subject or verb can't follow which. (Which is also an interrogative word but you are not asking a question in this example.)
It's a perfect time in which I can let my hair down.
Because a clause must follow which, to let my hair down won't work because let here is an infinitive.
4 might be technically correct but sounds excessively wordy and awkward. Avoid 4.
About 3 ...
Time is not a place, so where is not the appropriate pronoun to use to relate "I can let my hair down" it to "It's a perfect time."