A joint optimization algorithm which combines M-QAM with HARQ protocol is proposed for the multi-branch two-top cooperation diversity system to minimize the symbol error rate (SER) and maximize the throughput. Using moment generating function (MGF), the probability density function of SNR among relay nodes and the destination node were calculated, and the formulas of the SER for M-QAM were derived in non-cooperative and cooperative HARQ protocols over Nakagami-mfading channels. Then, by utilizing the state transition probability of relay, the throughput in closed-form in cooperative HARQ protocols was derived. The simulation results showed that the decrease of modulation orderMcould lead to the decrease of the SER, and cooperative HARQ protocols by MGF could obtain lower SER and better throughput compared with non-cooperative HARQ protocols. In view of M-QAM with HARQ protocol, the increment of relay could lead to the increment of system throughput and the decrement of the SER, and then the performance of System is improved.
