From the Oxford English Dictionary:
First used to designate the chief minister of certain non-British rulers. In British use, originally merely a descriptive title for the minister considered to be chief among the advisers to the monarch (as Lord Burghley or the Earl of Leicester under Elizabeth I). Sir Robert Walpole was the first political leader to whom the term was (frequently disparagingly) applied with any consistency, coinciding with George I's withdrawal from attendance at cabinet meetings (1717). The term became more widely used as, under William Pitt the Younger and his successors, the position took on more political importance and gained a higher profile.
The term prime minister in its French form, premier ministre, is attested in 17th Century sources referring to Cardinal Richelieu after he was named to head the royal council in 1624. The title was however informal ... The term prime minister in the sense that we know it originated in the 18th century in the United Kingdom when members of parliament disparagingly used the title in reference to Sir Robert Walpole. During the whole of the 18th Century, Britain was involved in a prolonged conflict with France, periodically bursting into all-out war, and Britons took outspoken pride in their "Liberty" as contrasted to the "Tyranny" of French Absolute Monarchy; therefore, being implicitly compared with Richelieu was no compliment to Walpole. Over time, however, the title became honorific and remains so in the 21st century.
So, it isn't accurate to say that the term began as a joke, but this is a reference to the way it was used the first time when it was first applied in its modern sense to British politics.