You'll find that English evolves over time. Sometimes two frequently-paired words become oft-hyphenated when they are used as a common, descriptive phrase. Those two words may eventually evolve further and morph into a compound word.
When English evolves like this, the good news is that there is no "wrong" answer.
I did a brute-force search through the literature, and found examples both with and without a hyphen.
Susan Snedaker wrote in 2003:
This type of attack is also called a brute force attack because the hacker keeps hammering away at the account until it cracks.
Source: How to Cheat at Managing Windows Small Business Server 2003: In the Land of the Blind, the One-Eyed Man is King
But the hyphenated version seemed much more common, like this one by James M. Stewart:
A brute-force attack is designed to try every possible valid combination.
Source: CompTIA Security+ Review Guide: Exam SY0-301
When I searched Google books for the compound word version, the results were rather interest-
ing. Time and time again, Google was returning a hit for bruteforce attack, yet the book would spell
the term like this: brute-force attack. Why the discrepancy? Because when the search-term word happened to be located near the end of a line, Google seemed to interpret the hyphen as a symbol to break the word at the right-hand margin (much like I did with the word "interesting" at the top of this paragraph). Here are a couple screenshots to show what I mean:
I did manage to find two instances where the term was used as a single, unhyphenated word. Take this with a grain of salt, but those two were in:
Handbook of Research on Wireless Security, edited by Yan Zhang, Jun Zheng, Miao Ma
A to Z Email Hacking and Security, by Rajendra Maurya
So maybe Asian writers are on the leading edge of making bruteforce a single word?
I would go with the hyphenated term for now, but only because it seems to be most prevalent, not because the others are wrong.
I'll close with this excerpt from a blogger who also happens to be a dictionary editor:
With terms that are still changing, it is not easy to say whether it is definitively one word or two. The same person might write humblebrag on one occasion and humble brag on another, but in both cases the exact same meaning is intended and understood. The only difference is a matter of spelling. Humble brag is no less a word than humblebrag...
NOTE: The scope of this answer applies to the one framed by the O.P. (namely, computer security, as in: brute force algorithm, brute-force cracking, etc.) However, there are times when this two-word phrase isn't used adjectively. In those cases, a hyphen would be unwarranted: The thief used brute force to open the door.