Numbers in your sentence are not a part of speech in the English language. Parts of speech (nouns, pronouns, adjective, verb, adverb, preposition, conjunction, interjection, and articles) are used for words.
The numbers in a sentence may be ordinal (represent rank or order of things), temporal (represent time), or nominal (identify something as in your sentence "1,2,3,4, that's it.").
Some idioms and phrases may also contain numbers that have special meanings. For instance,
- Easy as 1-2-3 (simple and straightforward).
- At sixes and sevens (total confusion or disarray).
- Behind the eight ball (mentally struggling or unable to keep up with a situation).
- A picture is worth a thousand words (pictures convey more information than words).
- A bunch of fives (punch someone).
- Back to square one (back to where one started).