It can be either.
The clause in question is copular. It’s also subordinate and relative. It might be easier to examine as an independent clause without adjuncts:
Phonograph parlors had proven successful.
Phonograph parlors had proven a success.
Both of these clauses have a subject, a verb, and a complement. The first has a predicate adjective subject complement. The second has a predicate nominative subject complement. Most linking verbs license an argument that can be either adjectival or nominative. The most typical linking verb is the copula to be:
Phonograph parlors were successful.
Phonograph parlors were a success.
The noun “success” has both a countable and an uncountable sense. The countable sense is marked by a determinder such as the indefinite article. It’s this sense that is a more natural choice in this clause.
The adjective “successful”, like most adjectives in English, has no grammatical number.
These Kinetoscope arcades were modeled on phonograph parlors, which had proven a success for Edison several years earlier.
This version of the sentence is correct. The original version of the sentence is also correct.
The verb to prove is not always copular. It can license a direct object and an object complement:
This evidence proves that statement true.
The object complement is, of course, optional:
This example proves that rule.
You may have been parsing “successful” as if it were a direct object, and you may have assumed that the direct object should be a noun rather than an adjective. With an ordinary transitive verb, you would have been right. You would even have been right to prefer the uncountable sense of “success” as that object.
The verb to prove isn’t an ordinary transitive verb. It’s ergative. The verb of the original clause behaves more like to become than to have:
. . . which had become successful . . .
. . . which had had success . . .