Without a determiner ("a" or "the"), "hardware" very broadly refers to any amount of equipment. I suspect that as we are talking about "virtualisation of hardware" this is speaking specifically about computer hardware, but even so it is very broad - there is a lot of computer hardware in the world.
"Virtualisation of hardware" would be used when talking generally about the process of virtualising any or all hardware. It doesn't surprise me that the majority of hits on Google contain this version of your phrase.
"Virtualisation of the hardware" specifically refers to a limited set of hardware which the document must have already mentioned or alluded to. This is probably used quite often, but more likely in private documents where people are writing about their own localised environment, such as an organisation documenting the virtualisation of some specific hardware of their own. Such private documentation is far less likely to be publically available for Google to search.
"Virtualisation of a hardware" doesn't sound correct to me, as "hardware" is mainly used as a collective noun. Yes, you can refer to a computer as "hardware", but technically it is "a piece of hardware". There may be some contexts where the setup allows for correct usage of this, but I can't think of any.