I am wondering when I want to refer to for example, third instance of something, should the rd be superscript or subscript? What about when the exact digit is replaced with a variable like M?
It's never a subscript. Either write it "on level" with the number as in "3rd", or write it as a superscript as in "3rd".
This is the sort of thing people need style guides for. Almost any style guide is going to tell you to spell out numbers lower than 10 (Chicago likes spelling out numbers lower than 100), so you would actually write "third."
For higher numbers you can write it inline or superscript. Just choose one and always do it the same way. And whichever way you choose, treat ordinal variables the same, too.