One of the most important issues in biomedical research is in vivo fluorescence-based multispectral imaging, but quantitative image analysis has been generally perturbed by the cross talk problem of fluorescence signals and significant autofluorescence problem of tissues. Spectral unmixing is a useful technique in multispectral fluorescence imaging for reducing the effects of native tissue autofluorescence and separating multiple fluorescence probes in vivio. While spectral unmixing methods are well established in fluorescence microscopy, which typically rely on precharacterized spectra for each fluorophore. A spectral unmixing algorithm tailored for in vivo optical imaging that is able to find the signal distribution and the pure spectrum of each component is introduced. It is derived from multivariate curve resolution-alternative least square (MCR-ALS) method using multiple constraints such as nonnegative, equality, closure, unimodal, spectral range and normalization. The signal distribution maps help to separate autofluorescence from other probes in the raw images and hence provide better quantification and localization for each probe. The test of mixed spectral samples with abundance fractions generated by Dirichlet distribution demonstrates that the algorithm is robust with different noises and pure spectral. And a quantum dots mouse multispectral fluorescence image has convinced this method.