The present algorithms of iterative detection for serially concatenated continuous phase modulation,which was defined in physical layer of downlink in DVB-RCS2,had a high computational complexity and poor convergence at low signal-to-noise ratio (SNR).The joint iterative demodulation and decoding algorithm based on chaos theory was proposed. A modified Gaussian wavelet of fast convergence was used to control the dynamic region.Combined with delayed feedback control method,transient chaos was suppressed.It improved the convergence of system in low SNR region effectively and reduced the number of iterations.A large number of matched filters were required to compute a priori information of codeword in iterative decoding.Because of large condition number, effective matched filters had a strong correlation.So truncated small eigenvalues method was used to reduce the number of matched filters,which was more suitable for practical implementation with only four matched filters.Simulation results showed that the proposed methods exhibited a better convergence and lower complexity compared with the traditional algorithm,and suitable for all kinds of CPM schemes.