Analyzed the traditional Mean Shift tracking algorithm in appearance model is sensitive to illumination changes and the disadvantages such as easily accumulated error on appearance model updating, combines the traditional Mean Shift tracking algorithm calculation speed is fast and easy to combination, the advantages of the design of the two different appearance modeling Mean Shift tracking algorithm. The first kind of Mean Shift tracking algorithm using traditional RGB color model to extract the appearance model, the second is not sensitive to illumination change of color and gradient information extraction model appearance. Combining these two tracking algorithm, through the two track of target tracking algorithm weighted target location, and the principle of based on the update of the cooperation of the two kinds of the appearance of the tracker template updates. This not only makes the tracking accuracy has been improved, and the ability to adapt to change the appearance is greatly improved.