Can I write
I checked it, but nothing
I checked it but couldn't find anything
"But nothing" is too incomplete to be an independent clause, so "I checked it, but nothing" would not be a well formed sentence.
Very informally, one might possibly say:
I checked. Nothin'.
… but I would recommend against it for an English learner.
What could be an acceptable sentence would be:
I checked it, but found nothing.
It depends on the context. "I checked it, but nothing" is colloquial or informal. It is not formal.
For example, you would not say that as a doctor talking to a patient. I would not use it in any professional context.
I checked it, but nothing.
This not a sentence, even in the most radical of grammars. However, it would certainly be seen in reported speech, in journalism or fiction. There, it might be taken to mean:
I checked it, but nothing (was there/was happening/had happened/was new/&c.)
Your proposed replacement sentence is grammatical, but would be improved by the addition of a comma:
I checked it, but couldn't find anything.