The past perfect discusses whether something occurred as of some past time. It can also be used to discuss counterfactuals. So "if the weather had been fine" either refers to a counterfactual, hypothetical being fine that never happened, or refers to some even more past possibility. That is, while the simple past would indicate that when we decided whether to go swimming, we would look at how the weather was at that moment (both the decision and the weather are happening in the past, but at the same past moment), the perfect tense would indicate that when we decided whether to go swimming, we would look at the weather as it had been in the past relative to that decision. That is, the time period for the weather being fine would be even more in the past than the moment of deciding whether to go swimming.
The counterfactual meaning dominates; with the past perfect, this would probably be interpreted as saying the weather was not in fact fine. But a "more past" interpretation is also possible. Either way, it wouldn't have the same meaning as the simple past.