My deduction is that the target is a file, so the "that" you have indicated refers to that target.
In that case, the sentence could be improved by setting off the clause this way: "A target, created in the same directory, that specifies any output...". It is still a poor explanation, but at least would clear up the antecedent of 'that'.
I'm going to make a guess or two as to what this is referring to: it sounds like an instruction from an "integrated development environment" (IDE) for a programmer, and the target is a file generated by the "custom command" referred to. Some IDEs allow a user to create a custom command that will generate one or more files, and those files are then used as input to other commands (whether custom or not).
For example, you could have a file that describes a data structure (in JSON, say), and you could create a program that would convert that JSON into a Java class that was a 'bean' for that JSON structure. You could then create a 'custom command' that ran your program, which would use the JSON file to create a Java source file. The IDE's normal Java compilation step could then use the Java source file to generate one or more Java class files, which could then be part of your java program.
If all this guessing is correct, the 'target' referred to would, in this case, be the 'target' referred to in your sentence, and the 'rule' would be one that caused the IDE to compile Java files when indicated.
I think it could be better explained, again assuming all my guessing is correct:
A target that specifies any output of the custom command as a source file, where the target is in the same directory, is given a rule to generate the output using the custom command at build time.
It's also possible that 'target' is an artifact of creating the custom command, so that instead of being the file itself, it only specifies the file. It's hard to tell without more information about the situation.
I hope 'same directory' is clear from context, because there is no clue here as to what directory they mean.