A multitask recognition model based on convolutional neural network is proposed to avoid single task recognition ignoring supervision information of related tasks. The proposed model introduces an attention mechanism to perform feature recalibration of the task shared layer and combines the multiscale structure for feature fusion. Finally, multi-task recognition is performed on the task-specific layers. Center loss and mean square error loss functions are employed together with the traditional cross entropy loss function to solve the generalization degradation problem caused by uncompact class distribution in the shared feature space. Experimental results on 6 human activities and 15 identities show that the model can achieve the maximum recognition accuracies of 100% and 99.93% on each task, respectively, and the multitask accuracy is up to 99.93%. The results are better than those obtained by the single task models. This shows that the model can simultaneously perform human activity and identity recognition more effectively.