Aiming at the problems of feature sparseness and noise sensitivity when the temporal-spatial local direction angle mode is applied to the video emotion recognition, we propose a new feature extraction algorithm, the spatiotemporal local ternary orientation pattern (SLTOP). Considering the complementarity of facial expression and posture characteristics in recognition, a classification method based on the cloud weighted decision fusion is proposed. The video image is preprocessed to obtain the sequence of the two modes of facial expression and gesture. For reducing the sparseness of the feature histogram, we extract the SLTOP feature of the sequences of expression and posture, learning from the idea of gray level co-occurrence matrix. In the stage of decision fusion, the cloud model is introduced to implement the cloud weighted decision fusion for the two modes of expression and posture making to realize the final recognition of dual-modal emotion. The average recognition rate of the single modal of facial expression and body posture in the FABO database is 92.21% and 96.76%, respectively. And they are approximately 18.42%, 22.01% and 9.15% higher in expression, respectively, when compared with the volume local binary mode, local binary mode three orthogonal planes (LBP-TOP) and temporal-spatial local ternary pattern moment (TSLTPM). In the single-posture modal, they are 26.59%, 29.53%, 1.98% higher, respectively. The average recognition rate obtained by cloud-weighted fusion is 97.54%, which is higher than that of other experiments. The proposed SLTOP has good robustness to the noise and illumination. The weighted decision fusion method of cloud model is used to greatly express the performance of two classifiers with expression and posture. The superiority of the recognition results in this paper is shown comparing with other classification methods.