Aiming at solving the problems such as low efficiency, high labor intensity and unsatisfying detection accuracy in traditional automatic brick stacking, a machine vision based automatic brick anomaly detection and recognition method is proposed. Brick images are captured from the brick delivering machine and the pit car are de-noised by applying an improved cross-like median filtering. Edges of bricks are extracted using the Canny edge detector. Vertical edges are detected by constraining polar angles in the Hough transform for analyzing the shape of the bricks. Anomaly detection is performed by measuring the length and width of the bricks in each column. Experimental results indicate that the average detection accuracy is 98.2% for brick-missing, brick-shifting and brick-tilting in one-scale brick stacks and multi-scale brick stacks. This meets the requirement of auto detection and recognition of brick anomaly in the automatic brick stack system of firing common bricks.