I built a giant house of playing cards an hour ago.
says exactly that. That is, it refers to an action completed in the past. By completed I don't mean finished, I mean accomplished.
It means 'An hour ago I accomplished the action of building a giant house of playing cards--and I am not still doing/accomplishing this action at the present time.' (The simple past strictly excludes present time.)
The simple past tense says nothing about the duration of the activity. The activitiy could have taken you three hours or three minutes. The simple past tense also says nothing about the beginning and ending point of the activitiy. It views the past action (the entirety of it) as a point in time, and can be represented as an X.
If the activity took you three minutes, you most likely wouldn't be asking the question that you are asking. You are asking this question because the activity's duration in real time is currently 3 times longer than the real time that has passed since the accomplishment.
The notion of duration (and the notions of the beginning and ending of the activity) is included in the verb activity itself (build a giant house of playing cards). It does not come from the use of the simple past.
Let's look at it from another perspective. If you say
I built a giant house of playing cards a year ago.
You are also not talking about the duration of the activity, nor about its beginning and ending points. Just about the accomplishment of an activity in its entirety in the past. And since in real time, the duration of the activity is insignificant compared to when the accomplishment occurred, the notion of what does this statement mean? does not even come up.
It is the fact that you are reporting on a three-hour activity only an hour after accomplishing it that makes you ask your question. So, here accomplishing it and finishing it are construed in your mind as being the same.
Here's another example:
I eat a leisurely breakfast over the newspaper from 7am to 10am. At 11am, a friend calls and invites me to lunch. I decline the invitation and say
I (just) ate breakfast an hour ago.
The simple past says nothing about how long (the duration) it took me to eat breakfast. It also, in and of itself, says nothing of the ending time. However, since in real time the activity was accomplished only an hour ago, the accomplishment of the entirety of the activity and the ending time of that accomplishment are synonymous.
So, in this case, I (just) ate breakfast an hour ago and I finished eating breakfast an hour ago are synonymous. But this is only fortuitous. It has to do with the passage of real time. It has nothing to do, per se, with the past simple tense.
The way to actually refer to the duration of the activity accomplished in the past is to directly tell us, like you did in your question or to use the progressive aspect.
The simple past views the past activity as a point in time, thus as an X.
The progressive past views the activity as having duration, thus it can be represented as a wavy line, ~~~~~~~~.
I built a giant house of playing cards an hour ago. It took me three hours.
still looks at the past event as a point in time X. It just so happens that, psychologically speaking, the punctiliar X is at present longer than "an hour ago". Wait a year and the X "becomes" (psychologically speaking) much more of a punctiliar activity. Although this is what it always was, as far as the simple past tense is concerned.