The two phrasings have different meanings and are used in different contexts. Most native english-speakers would consider only the first appropriate here. Let's break it down:
In case :
To keep your car in a good condition, you would do well to maintain it regularly.
Here, 'to do well to' is a semi-colloquial expression meaning the same as, "it is generally a good idea to". As such, the intent of the sentence becomes:
"It is a good idea to maintain your car regularly," [as already stated] "to keep it in a good condition."
There are no problems with this sentence.
In case :
To keep your car in a good condition, you would do well by maintaining it regularly.
This is not correct, because 'to do well by' refers to something different than 'to do well to'.
The phrase 'to do well by [something/someone]' is an idiom meaning that entering into a working relationship with the object of the phrase will result in a beneficial outcome, usually material gain or generous treatment, for that person or organization.
As an example:
The mayor did well by her constituents.
Meaning: 'the mayor treated her constituents well'.
Note that 'do well by' is not normally used with a gerund, and as such cannot be applied to the sentence in the exam.
So  is correct.
Incidentally, 'do well by' as an idiom is somewhat rare in modern English. The exam seems quite rigorous.