Lithography hotspot detection plays a critical role in realizing the manufacturability design of integrated circuits (IC) and ensuring the final yield of IC chips. Considering that conventional lithography hotspot detection methods based on deep learning are challenging to meet the inspection precision requirement of advanced IC manufacturing, we propose a detection algorithm based on improved Yolov5s for the precise detection of hotspot defects in the lithography layout. In the algorithm, a coordinate attention mechanism is introduced into the backbone network, which can improve the attention of the Yolov5s model to the patterned area in the layout. Thereby, the performance of the lithography hotspots based on the Yolov5s detection algorithm can be greatly promoted. Meanwhile, the Sigmoid linear unit activation function is used to improve the nonlinear expression of the entire neural network, and the Scylla intersection over union loss function is adopted to realize the quantitative evaluation of the bounding box regression loss more quickly, which can enhance the convergence speed and accuracy of the algorithm. Using the ICCAD (The International Conference on Computer-Aided Design) 2012 contest benchmark and the optical proximity correction optimized lithography patterns as the dataset, performance test experiments are carried out to verify the excellent detection accuracy of the proposed algorithm. The experimental results indicate that the mean precision, mean recall, mean F1-score, and mean average precision of the algorithm reach 97.7%, 98.0%, 97.8%, and 98.4%, respectively, which are significantly better than those of other hotspot detection algorithms and show its good application prospects.