Normally we use both in the singular, "as a matter of fact". Even if you then cite two facts. If you're citing them together, presumably they are related, so it is one "matter". Like if I said, "No, George deserves the credit. As a matter of fact, it was George's idea, Fred contributed very little", yes, there are two facts -- "it was George's idea" and "Fred contributed very little" -- but both relate to the same subject, who deserves the credit, so it's one matter.
We're using "fact" here as a collective noun, not referring to individual facts. Like suppose I said, "As a question of law, kidnapping is a more serious crime than assault". I'm referring to at least two laws, those against kidnapping and those against assault, but the phrase "of law" is referring to "the law" as a collective thing, not an individual law or laws.
I suppose the idiom could be strained if you brought up two completely unrelated subjects. Like suppose you said, "As a matter[s?] of fact[s?], Rome is the capital of Italy and the square root of 16 is 4". Is this one matter or two matters? But if you're not making any connection whatsoever between the two statements, why are you combining them in one sentence? Even if the only relation is that they were both questions you heard on a game show or something of the sort, well then, they have a connection.