Layered Space-time code (LST) in free space optical communication system can greatly improve system transmission speed rate, but the corresponding error rate of the method impacts the reliability of system badly. Firstly, based on the analysis of optical intensity fluctuation, a Multiple Input Multiple Output (MIMO) channel model is proposed. Secondly, the decoding of LST is completed by using Serial Interference Cancellation (SIC) in free space optical communication, and then its performance is compared with that of linear decode algorithm. At last, the feasibility of the technique is proved by simulation analysis, which shows that in free space optical communication system, the error-rate performance by using SIC algorithm in LST is better than that of linear decoding algorithm.