The fusion of a high-spatial-resolution (HSR) panchromatic band and several multispectral bands with a relative low spatial resolution has become a research focus with the development of HSR remote sensing technology. Previous studies have demonstrated that fused spectra of mixed pixels (MPs) remain mixed, which considerably contributes to spectral distortions observed in fused images produced by most of the current pansharpening methods. Several works have attempted to reduce spectral distortions of fused spectra of MPs to improve the quality of fused products generated by some fusion methods based on component substitution (CS). An image fusion framework for reducing spectral distortions caused by the incorrect fusion of MPs is proposed for both CS and fusion methods based on multiresolution analysis (MRA). Using the proposed framework based on image segmentation, the fused products of two classic MRA-based pansharpening methods were improved by improving the fusion spectra of MPs. The improved fused images were compared with the original fusion products through a fusion experiment using three datasets recorded by WorldView-2, GeoEye-1, and WorldView-3. Experimental results showed that the improved fused products yielded higher Q2ⁿ and quality with no reference values and lower relative average spectral error, dimensionless global relative error of synthesis, and spectral angle mapper values than the corresponding original fusion products. This indicates that the proposed image fusion framework is effective for reducing spectral distortions of fused images generated by the two MRA-based fusion methods.