In this paper, a visual focus of attention (VFOA) detection method based on the improved hybrid incremental dynamic Bayesian network (IHIDBN) constructed with the fusion of head, gaze and prediction sub-models is proposed aiming at solving the problem of the complexity and uncertainty in dynamic scenes. Firstly, gaze detection sub-model is improved based on the traditional human eye model to enhance the recognition rate and robustness for different subjects which are detected. Secondly, the related sub-models are described, and conditional probability is used to establish regression models respectively. Also an incremental learning method is used to dynamically update the parameters to improve adaptability of this model. The method has been evaluated on two public datasets and daily experiments. The results show that the method proposed in this paper can effectively estimate VFOA from user, and it is robust to the free deflection of the head and distance change.