In order to solve the problem of high resource consumption and low precision in the hardware implementation of the activation function Tanh, a Tanh function approximation algorithm combining second-order approximation and error compensation was proposed. This method first performed log2 transformation on the traditional second-order approximation function’s coefficients, which was convenient for hardware implementation and reduced the resource consumption. Then the compensation interval was divided according to the approximate function’s error curve to improve the function’s accuracy. On this basis, the circuit structure of the hardware implementation of the Tanh function was designed. The ModelSim platform was used for functional simulation, and the logic synthesis was performed under the SMIC 018 μm process. Experimental results showed that the implemented circuit’s maximum absolute error was 0007 8, which was the highest accuracy that the 8 bit output width could represent.