Based on image energy correlation, a novel second-order prediction algorithm was proposed to solve the low coding efficiency caused by inaccurate prediction and no detailed analysis of the correlation between the pixels. By analyzing the statistical properties between intra residual and inter residual, the energy correlation between pixels was defined. Then, the statistics of the energy correlation and first-order inter residual curve were applied to filter out second intra prediction reference pixels dynamically and the second order prediction became more accurate which makes a bitrate reduction and the coded image quality improvement. Experimental results showed that the bitrate and the quality of image are significantly improved. Especially for high-resolution video with complex movement, this algorithm achieves significant PSNR increment and bit rate reduce on average. The rate-distortion optimization has a significantly improvement.