You understand this correctly.
This fullest form of this comparative construction is
too ADJECTIVE [for SUBJECT to VERB COMPLEMENTS]
The piece in square brackets [] is what we call an infinitival clause: it has its own SUBJECT, VERB and COMPLEMENTS, with VERB cast in the infinitive (to VERB).
SUBJ VERBinf COMPL
[for me to take out ]
The entire comparative construction (too pretty for me to take out) "acts like" a single adjective modifying a noun or noun phrase. In your sentence it is a predicate adjective modifying the subject of the main clause, you, in exactly the same way as the bare adjective pretty would. In the next graphic the pieces which act like adjectives are enclosed in curly brackets {}.
You are { pretty }
.
You are { too pretty [for me to take out] }
.
There are two different ways of using the construction.
If the subject of the infinitival clause is different from the word which the construction modifies, then you use the full infinitival, with for SUBJECT. In the graphic below I've marked the word modified with '1' and the subject of the infinitival with '2'
You1 are {too pretty [for me2 to ask out]}.
As you say, you is understood to be the missing term in the infinitival: the direct object of take.
But if the subject of the infinitival is the same as the word which the construction modifies, for SUBJECT is deleted. In the graphic below I've marked both the word modified and the subject of the infinitival with 1
You1 are {too pretty [for you1 to be my girlfriend]}.