For reducing the pilot overhead used in time-varying channel estimation of orthogonal frequency division multiplexing system with cyclic delay diversity (CDD-OFDM), an improved expectation-maximization (EM) based channel estimation scheme with recursive fashion was proposed. In this scheme, the mathematical relationship between the basis expansion model (BEM) of time-varying channels and the symbol time-averaged channel impulse response was firstly established and then used to predict the initial value of the EM based channel estimator, therefore the parsimonious parameterization properties of the BEM and the slow variations of the BEM parameters could be exploited to improve the channel prediction accuracy and make the algorithm real-time, respectively. Simulation results showed that when the signal to noise ratio is above 15 dB, the proposed scheme effectively reduces the error propagation of decision directed channel tracking scheme, and has nearly the same bit error rate performance as that with perfect channel state information.