Similar to the spread of infectious diseases, rumor spread has a significant impact on human lives. As the saying goes, “rumors end with the wise” and the proportion of the wisemen in the crowd has a certain influence on the spread of rumors. Based on this fact, a rumor transmission model considering the proportion of wisemen in the crows is established. It provides a new angle of view on the issue of rumor spreading. Different from previous studies, we consider the speed of rumor propagation as the variable over time in our model. Then, locally asymptotic stability of each of the feasible equilibriums is presented by using Routh–Hurwitz criteria. Numerical simulations are carried out to illustrate the feasibility of our results and the influence of different parameters on the rumor spreading. The results in this paper provide theoretical support to rumor control.