I believe you're thinking of the passive voice construction, or of the perfect construction (which is variously called an "aspect" or a "tense"). Both of these constructions use the past participle form of a verb, and the past participle of the verb make is "made".
The perfect construction doesn't generally use forms of "to be" in modern English; it's made using a form of the verb "to have" + a past participle.
The passive voice does use a form of the verb "to be" + a past participle. However, it is not a type of past tense. In any case, "Windows are made of glass" is not a passive-voice sentence: most people would do a double-take if you responded to this sentence with a question like "By whom?" or "Who does this?"
The word "made" in this sentence is not a participle, but an adjective. See the definition in Collins' Advanced English Dictionary:
adjective If something is made of or made out of a particular substance, that substance was used to build it.
The structure of the predicate is just "are," the simple present plural form of "be", followed by the predicative adjective phrase "made of glass".
There are a number of adjectives like "made" that look exactly like past participles. So some sentences are ambiguous between "be + predicative adjective" (active voice) and "be + past participle" (passive voice). Usually the context resolves any confusion. In this case, a general statement about the composition of windows (active voice) seems more likely than a passive-voice statement.
Another relevant question about this topic: How to identify whether the sentence is in passive voice or Simple past (state)