The problem here is that the possessive gets confused with the plural. Let's take your second example: "student union" is a noun adjunct and is the most commonly used variant (in the UK at least). Student is singular, as is usual with noun adjuncts. As Wikipedia notes, plurals are becoming more common but personally I find them grating. Most native English speakers instinctively use the singular. On the other hand, "students' union" (note the apostrophe) is quite correct, but it's a possessive of a plural noun, not a plural noun adjunct.
An easy way to see this distinction is to look for words whose plural is not formed by adding an s. For example, female police officers and doctors used to be called "woman police constables" and "lady doctors" (note the singulars); but for possessives we have "women's magazines" and "ladies' toilets" using plurals.
Returning to your examples, the first is similar to the second: "student welfare officer" (singular) or "students' welfare officer" (plural, but possessive, not adjunct). Here though, the former, though more natural-sounding, is potentially confusing because it might imply that the officer is a student, rather than being for students.
The same confusion can't arise for your third example. "Student loans" sounds far more natural than "students' loans" but there is still a nuance to think about. The former described almost the institution of given loans to students, whereas the latter might describe some loans given to some students.
Your fourth example raises a completely different issue. The possessive would never be used here, since sport/s is/are inanimate and the centre doesn't belong to it/them. The issue is whether you consider sport in general to be singular or plural, regardless of whether it's used in a noun adjunct. American newspapers have sections called "sports" whereas British ones have sections called "sport."
In summary:
- When using a possessive, use the plural if there is more than one possessor.
- When using a noun adjunct, use the singular.
- But don't use the singular if you would never use it outside a noun adjunct, e.g. an "arms dealer" almost never sells an "arm."
- There is an (alas, growing) list of idiomatic examples where the plural is common, e.g. "people smugglers/"