Single frame infrared (IR) small dim target detection with high detection rate, low false alarm rate and high detection speed is a difficult task, since the targets are usually very small and dim, and different types of interferences exist, such as high brightness backgrounds, complex background edges and Pixel-sized Noises with High Brightness (PNHB). The single frame detecting algorithms based on HVS can usually achieve a better performance than traditional algorithms. However, for an algorithm based on HVS, how to define the formula for local contrast is one of the key issues, which directly determines the performance of the algorithm.By now, researchers have not reached a consensus on how to define the local contrast, and many local contrast definitions have been proposed. Existing algorithms, such as the ratio form local contrast methods and the difference form local contrast methods, cannot effectively enhance real targets and suppress all the interferences simultaneously, they just simply take the local surrounding areas as background without taking into account the diversity of the local surrounding background itself and the local diversity information which can be used to further suppress the complex backgrounds is wasted. A Multi-scale Ratio-Difference joint Local Contrast Measure (MRDLCM) was proposed. It could combine the advantages of the ratio form methods and the difference form methods, so it could suppress all the types of interferences while enhancing different sizes of real targets, and did not need any preprocessing algorithms. Besides, a weighted function utilizing the Reversed Local Diversity (RLD) was proposed, it utilized the local diversity of the local surrounding areas to suppress the complex backgrounds further. Experimental results show the effectiveness and the robustness of the proposed MRDLCM_RLD algorithm against existing algorithms in detection rate and false alarm rate. Besides, the proposed algorithm has the potential of parallel processing, which is very useful for improving the detection speed.