"You had better keep doing this and figuring out other complicated problems."
"You had better keep doing this, figuring out other complicated problems."
The first sentence is correct, although it can be a made a bit clearer. Instead of writing:
you had better keep doing this and figuring out other complicated problems
I would write:
you had better keep doing this and keep figuring out other complicated problems
It is a bit difficult to understand the meaning of the sentence if not seen in context: it is not clear where "this" refers to.
The second sentence seems incorrect to me. Instead of a comma, I would expect a colon:
you had better keep doing this: keep figuring out other complicated problems
The first sentence lists two things you should do separated by "and".
The second sentence separates them by a comma, which, in this case, is incorrect. The comma breaks up a sentence for a pause or as part of listing more than two options. For example:
do this, this and that
In your case there are only two options: "keep doing" and "figuring out". So they don't need a comma, just "and".
If you replace the comma with a colon, then the meaning changes. Instead of two things you have to do, there is only one ("keep doing this") and the colon introduces related information. The related information is what you should keep doing.
The phrase "you had better" means "I very strongly advise you to". What then follows is what you should be doing. This is used in written text. When speaking, "had" can be dropped or contracted:
you better study hard if you want to pass your exams.
you'd better study hard if you want to pass your exams
The two sentences are not equivalent.
In the first sentence, "this" and "figuring out other complicated problems" are two separate things that the listener had better keep doing. Previous sentences must have established the meaning of "this".
In the second sentence, the phrase "figuring out other complicated problems" is an appositive, which provides the meaning of "this". An appositive is a phrase, placed immediately adjacent to another phrase, indicating that the two phrases describe the same thing. Here are a examples of the two kinds of appositives in English:
My teacher, a man from Georgia, used to work as a cotton farmer.
That means the same as "My teacher used to work as a cotton farmer. He is a man from Georgia." The commas mean that the appositive makes an additional statement about "teacher", beyond what the verb ("used to work…") says. This is called a non-restrictive appositive.
My cat Josephine likes to chase ping-pong balls.
That means "One of my cats likes to chase ping-pong balls. The cat that I am referring to is the one named Josephine." The lack of commas around "Josephine" means that the appositive indicates which cat I'm talking about. This is called a restrictive appositive.
So, your second sentence means that "figuring out other complicated problems" is what "this" refers to. However, depending on the preceding sentences, many people will think that the sentence is an error, because normally you would say something to establish the meaning of "this" before adding information to it with an appositive. Or, if there is only one thing that the listener had better keep doing, normally you would word the sentence like this:
You had better keep figuring out other complicated problems.