Upstream damming can greatly alter the flow and sediment conditions entering downstream reaches in the middle reaches of the Yangtze River, with the bed material continuously coarsened, which has a significant influence on the sediment transport and bed evolution in the downstream of the Three Gorges Project (TGP). In order to study the riverbed armoring and sediment exchange processes among bed material, bed load, and suspended load layer in the Yizhi Reach and Zhijiang Reach, the model with three-state transition probability (Markov Chain) and hiding-exposure effects of non-uniform sediment are introduced. The equilibrium equation of sediment quality in active layer is proposed for calculating grain size distribution. In this model, the influences of flow and sediment regimes, riverbed erosion and deposition on SEP are discussed. The results show that, the composition of surface bed material at Zhicheng station was obviously coarsening, and the grain size distribution curves shifted to the left. The grain size range of surface bed material in 2003 was 0.002~41.000 mm, and changed to 0.031~86.000 mm in 2017. In addition, the median grain size of surface bed material at Zhicheng station showed an upward trend and fluctuated after 2010, but the fluctuation range was relatively small. The monthly change process of the median grain size can be divided into three categories according to the time node of flood periods. The probabilistic model developed in this study is verified against the field measures with both bed load and suspended load particles, and the modeling results show a reasonable agreement with the measured data at Zhicheng station from 2003~2009. Accordingly, the probabilistic model can be used to predict the riverbed armoring and annual process of sediment exchange in sand-gravel bed in the Yizhi Reach and Zhijiang Reach. The model can be used to reveal the mechanism of sediment motion and non-equilibrium sediment transport in sand-gravel bed rivers.