If the effects of the local oscillator (LO) phase noise and the power amplifier nonlinearity on the digital transmitter error vector magnitude (EVM) can be predicted easily and effectively, and the bit error rate (BER) can be predicted using the EVM, a new way will be provided to estimate the performance of the modern communication systems. In order to solve this problem, the relationships between EVM and two effects, LO phase noise and power amplifier nonlinearity, were derived and expressed separately. And the third order intermodulation distortion (IMD3) was used to simplify the expression of the EVM. Then, an expression for the EVM was derived based on the digital transmitter model that considers local oscillator phase noise and power amplifier nonlinearity. Finally, the math formula of bit error rate (BER) versus EVM,which can be easier and more useful to predict BER,was given. Through the simulation, it was found that a simple way was provided by using IMD3 to calculate EVM, without considering the input power, output power, 3rd interception point and 3rd intermodulation product power. And predicting BER with the value of EVM can save the necessary closed loop equipment in the conventional method and shorten measure time.