To solve the drifting problem of a varying target object during target tracking, an adaptive prior appearance model was presented. The hierarchical Dirichlet process evolutionary clustering model and online boosting learning were combined into a coherent framework. By taking the hierarchical Dirichlet process as prior distribution, the prior appearance knowledge could adapt to change over time. Further, the appearance model was smoothly constrained by the mixture proportion of a type of appearance cluster at each moment. To balance the classification error of appearance model and the cost for splitting the clusters, the multi-modal appearance model was automatically learned by the use of Bayesian posterior inference. Finally, based on the weighting factor of appearance clusters, the target object was discriminated by combining the outputs of appearance classifiers. The simulation results showed that the learned appearance model could robustly adapt to the appearance variations, as well as the tracking results with high accuracy.