Transmission over wireless fast time varying channels can lead to Inter-cell interference (ICeI) and Inter-carrier Interference (ICaI) in multiple-input multiple output (MIMO) Orthogonal Frequency Division Multiplexing (OFDM) systems. To decrease the impact of the interference, it is considered that each cell can select different order Fractional Fourier transform (FrFT) suitable for the different interference environments to modulate and demodulate the symbols,and a model for MIMO-OFDM based on FrFT is proposed.The Interference Restricted Combination (IRC) equalizer is designed to suppress the interference. Because of the use of Hybrid Automatic Repeat Request (HARQ) technique in practical communication, it is unnecessary to minimize the bit error rate of first transmission. Therefore, more reasonable rule for optimizing FrFT order is proposed, which maximizes the channel capacity under the constraint condition that received symbol SINR reach threshold value,and the optimal IRC equalizer based on optimal FrFT are derived. The system simulation results proved that the rule is available and the ICeI and ICaI can be suppressed effectively. Furthermore, the system throughput also increases obviously by updating the optimal order.
