As you note, awk is software - more specifically, a program run from the command line. By convention, many command-line programs will print a detailed message to the console if they are unable to understand the arguments they were given. This message may include various information:
- Expected arguments
- Documentation
- Which argument(s) were not understood
- What went wrong in trying to process the arguments
In this sentence, the author refers to this message as a "usage diagnostic". In this context, "usage" further describes or clarifies "diagnostic" - the diagnostic corresponds to an error in how the program was used, not environmental factors or other problems.
"Diagnostic" can take either an adjective or noun form. In its noun form, as seen here, it identifies a "practice or technique of diagnosis."
Had the author included another noun here, such as "message", then "diagnostic" would have taken its adjective form because it was describing the message.
Flipping these, to "diagnostic usage", does not make sense - a "usage" is not something you would expect to receive from a program.
"Sort" means "kind" or "variety". To be honest, it really doesn't add anything useful to the sentence, other than to suggest that the message might vary based on the arguments you passed.
So this sentence could be rewritten for clarity as follows:
Doing this leads to confusing behavior -- most likely a diagnostic message from awk, including details of the failure or expected usage.