A: What are your memories of the U. S. - Soviet Union game in 1980, Seth? ;-)
B: I was all of six months old that day. ;)
Does he mean he's exactly six months old that day?
When used with age, the all of means that someone is or was too young for something or doing something.
all of - As much as (often used ironically of an amount or quantity considered small by the speaker)
This means, he was too young to have memories.