The sealing characteristics of the nitrile rubber O-ring, restrained unilaterally in the seal groove of a mechanical seal, were numerically simulated using the Mooney-Rivlin model with high-order terms. Effects of the precompression ratio and the medium pressure on contact stress, contact width, and peak stress were mainly studied.Simulated results showed well agreement with the Lindley’s semi-empirical relationship and the Wendt’s experimental data. The contact stresses have a parabolic distribution along the main contact surface. The precompression ratio of the O-ring has a great effect on the main contact surface while an insignificant effect on the restrained contact surface. The greater the precompression ratio is, the larger the peak stress on the main contact surface is, whereas the peak stress on the restrained contact surface is almost unchanged. The so-called secondary compression of the O-ring results from the effect of medium pressure.The results also showed that the O-ring has characteristics of self-sealing when it is precompressed and contacts with the fluid medium.The greater the medium pressure is, the larger the peak stresses on both the main and the restrained contact surfaces are. The sufficient contact stresses could be achieved under the condition of a small precompression ratio for an O-ring.It was suggested that the small precompression ratio of the O-ring for a mechanical seal would be maintained to meet requirements of both the unrestricted axial motion of the seal ring and the seal face closure.