The detection of high-dimensional entanglement is an important challenge in both theoretical and experimental aspects of quantum information science and technology. The traditional entanglement witness based on fidelity is not suitable for detecting all kinds of entanglement states, and the number of measurement combinations is not optimal. A new algorithm based on convex optimization theory has been proposed to remedy the situation. The performance of both approaches on a bipartite time-bin-entangled qutrit is experimentally test. When the traditional entanglement witness cannot determine its entanglement property unambiguously, it’s switched to the optimization-based method. The results show that a simpler and more effective measurement scheme can be obtained by using the new method, which can efficiently and accurately verify the entanglement dimension of the target qutrit. In addition, compared with the traditional entanglement witness, the new method only requires counting 6 different pairs of coincidences instead of 15.