It is very challenging to reconstruct the 3D structure from a single image and perceive the semantic information of 3D objects. Aiming at the problem that it is difficult to directly generate a 3D reconstruction model from a single image input, a joint optimization network model combining PointNet and 3D-LMNet is proposed for single image 3D reconstruction and semantic segmentation. First, a 3D point cloud is generated by training based on the 3D-LMNet network, and then local segmentation is performed. Meanwhile, the network loss function is jointly optimized to predict the segmented 3D point cloud. Then, the reconstruction effect is improved through the semantics information of segmented point cloud, and a 3D point cloud reconstruction model is generated with semantic segmentation information. Finally, in view of the problem that there is no point-to-point correspondence between the true value point cloud and the predicted point cloud category label during the joint training, the joint optimization loss function is introduced into the joint optimization network to improve the reconstruction and segmentation effect, and the 3D reconstructed model is made. Through verification on the ShapeNet dataset, and comparation with PointNet and 3D-LMNet training, the model in this paper improves mean intersection over union (mIoU) by 4.23%, and reduces chamfer distance (CD) and earth mover’s distance (EMD) by 7.97% and 6.04%, respectively. The joint optimization network significantly improves the reconstruction and segmented point cloud model.