Something is said to be ultimate if it is original or final for something else and therefore very important. A thing is primary as long as it is happening or coming first, hence important too.
I suggest these words may have a space of interchangeability, but probably they sometimes have distinguishable meanings, for example when we speak about responsibility.
Parents must have ultimate responsibility for their children’s safety.
From my point of view, the speaker here trys to convey the idea neither a children' school nor a local government must have power to issue obligatory rules in respect with children’s safety. Rather, their parents would have the final (of utmost importance) word on this topic.
For that matter, if we mention someone's ultimate liability for damages, it means although there are may be some intermediary (an insurance company) who must pay to a victim, but it is the very person who will finally be charged (by the insurer, in fact).
The CEO has primary responsibility for making day-to-day investment decisions for each fund.
In contrast, this sentence has meaning that the main CEO obligation is to make those decisions; all other ones pale in comparison to it.
Is my understanding correct? If so in some degree, are there cases where these differences not matter and responsibility can at once be ultimate and primary? For instance, I think it may be true for the following:
Governments of resource-rich countries bear the ultimate responsibility for translating those resources into a blessing for their people.
Nuclear-weapon States bore primary responsibility for implementing measures to reduce and eliminate nuclear arsenals.
albeit it is hard to me to say for sure what's the meaning conveyed here (both? ambiguity with no further context?).