To investigate the effects of replacement rate and sample size on the direct shear performance and damage of recycled aggregate concrete (RAC), 45 RAC specimens were manufactured. The failure modes of RAC with different sizes were characterized, and the effects of replacement rate and specimen size on the direct shear performance and damage constitutive model of RAC with different sizes were investigated via direct shear tests. The results show that the peak deformation and ductility coefficient have a size effect, the damage development rate of natural concrete becomes slow, and the damage development rate of RAC becomes faster. The peak deformation firstly decreases and then increases, while the ductility coefficient firstly increases and then decreases with the increase of the replacement rate. The damage development law does not have a linear relationship with the replacement rate. The constitutive model can well reflect the shear load-deformation relationship and the damage evolution process of the RAC with different sizes.