Hyperspectral Image (HSI) has rich information, it has been widely used in various fields. Due to the limitations of various factors, such as lighting conditions, transmission conditions and imaging instruments, HSI is polluted by various noises, which not only reduces the visual quality but also brings difficulties to subsequent processing. Many existing traditional denoising models still use nuclear norm minimization to iteratively solve the matrix rank minimization, and each iteration involves singular value decomposition, so these algorithms have a high computational complexity; in addition, total variation item fails to explore shared group sparsity patterns of difference images. In summary, how to express low rank more quickly and express sparsity more accurately is still a difficult problem. Under the framework of combining local low-rank and global group sparsity, this paper proposes the Fast Tri-factorization and Group Sparsity (FTFGS) model. In local modules, FTFGS model partitions the HSI into overlapping 3-D patches and converts patches into a matrix by lexicographical sorting. This operation conforms to the physical characteristics of HSI, avoids the formation of ill-conditioned matrices, and can better protect the details in the local blocks. This patchwise approach can reduce the dependence on the hypothesis that noise in HSIs is independent and identically distributed. When dealing with small-scale matrices, the Fast Tri-factorization (FTF) is used to decompose these matrices into two orthogonal factor matrices and a core matrix, the size of the core matrix and its L_2,1 norm minimization are used to more accurately and quickly represent the local low rank. FTF explores the low rank, which has the advantages of lower computational complexity and faster speed than the nuclear norm, furthermore, FTF digs deeper into the low rank because the low rank constraints are transferred to a smaller core matrix. When exploring the sparsity, the existing total variation regularizations do not consider the group sparsity property of HSI and so on, the local area structure is the same for all bands, as is the smoothed structure. This paper proposes a new weighted spatial-spectral group sparse regularization to explore the shared group sparse pattern in each gradient direction of HSI. With this strategy, the local and global modules are executed alternately to express the local low-rank and global group sparsity properties of HSI and remove complex mixed noises. In the comparative experiments, intuitive visual effects, quantitative numerical evaluation and qualitative comparisons are used for evaluation. From the visual effects, the FTFGS model better preserves image details and texture information, and the visual effect is significantly improved. Compared with the five classical denoising methods, the average peak signal-to-noise ratio index is increased by 1.75 dB, the average structural similarity index is increased by 0.003, the average feature similarity index is increased by 0.002, and the denoising accuracy is significantly improved. Moreover, in the qualitative comparison experiment, the spectral curve of our model is closest to the original image. The validation effect on the real dataset further proves its effectiveness. The reason for the good results is that compared to other models, FTFGS not only improves the local low-rank term, but also better explores the sparse prior of the image with the group sparse term. The time complexity analysis of the model verifies the effectiveness of the FTF framework. The model makes full use of the prior information of the HSI, which not only develops more accurate approximate representations for the low-rank and sparse, but also improves the speed while ensuring the optimal solution. The model is robust, fast and effective, and has certain research value for remote sensing and other application fields.