In order to prevent the data from being tampered in power communication in smart grid, security chip is essential, and RSA algorithm is one of the most widely used public key algorithms. However, due to the high complexity and power consumption of hardware implementation, it is unable to take the performance, power consumption, security into account. In this paper, a high secure and high performance RSA coprocessor was designed. The security strategy proposed enhanced the coprocessor's ability to resist side channel attack, differential power attack and EMA electromagnetic attack. Two levels of algorithm optimization were used to improve the coprocessor performance, and the improved Montgomery modular multiplication algorithm combined with CIOS square algorithm and Karatsuba algorithm made the 1 024 bit RSA algorithm with protection have an area of 48 000 gates @30 MHz and a power consumption of 4.62 mW @30 MHz under UMC 55 nm process. The performance of API test on FPGA board was 709.3 kbit/s.