Point clouds are a common method for representing three-dimensional (3D) data. With the rapid advancement of technologies such as LiDAR and other 3D scanning techniques, capturing point clouds from real-world scenes has become feasible^[1,2]. As a result, 3D point clouds have found extensive applications in various domains including military technology, autonomous driving^[3,4], and cartography^[5]. However, due to limitations imposed by capture equipment and acquisition conditions, raw point clouds produced by 3D scanning are often locally sparse and non-uniform, posing difficulties for tasks such as point cloud classification^[6–8], segmentation^[9,10], and detection^[4,11–13]. Therefore, the task of point cloud upsampling, which converts sparse, incomplete, and noisy point clouds into denser, complete, and cleaner forms, has attracted much attention.

In addition to traditional 3D scanners like LiDAR, tactile sensing represents another approach to perceiving 3D shapes. Most tactile sensors can measure force distribution or geometric shapes on small contact areas, offering an alternative means of obtaining 3D models. By utilizing the sensor’s position and orientation during touch, the sensor can assist in reconstructing the shape of an object. However, tactile perception is constrained by the size and scale of the sensor. Since each touch only provides information from a localized area, multiple touches and a considerable amount of time may be required to fully reconstruct the complete shape of the object. With the development of tactile sensors, prevalent devices like Gelsight^[14] and DIGIT^[15] can capture localized geometric shapes of contact surfaces through touch. High-resolution local geometric data, valued for its accuracy, is often applied in tasks involving 3D reconstruction^[16–18]. 3D reconstruction often leverages RGB images and tactile data sequentially, by first using RGB images to learn 3D object shape priors on simulated data, and subsequently using touch to refine the vision reconstructions^[19,20]. However, the fusion of tactile information with other modalities remains largely unexplored.

With the successful application of deep learning in image super-resolution^[21–23], many researchers have designed more advanced deep learning networks to achieve better image super-resolution^[24–26]. 3D point cloud super-resolution shares similarities with 2D image super-resolution, and as such, various algorithms and concepts from 2D image super-resolution can be adapted and applied to the field of 3D point cloud super-resolution. However, different from 2D image data, the data structure of 3D point clouds is more complex, and the data are sparse and irregular. So, super-resolution for 3D point clouds is still a very challenging problem. With the pioneering introduction of PointNet^[6], which employed deep learning networks for processing point cloud data, researchers have shifted their focus toward constructing deep learning networks to accomplish point cloud upsampling tasks. The general approach employed by prevailing learning-based methods involves the initial design of an upsampling module aimed at augmenting the number of points within the feature space. Subsequently, losses are formulated to impose constraints on the output points, ensuring both distribution uniformity and proximity to the surface^[27–30]. In other words, these methods share a key to point cloud upsampling: learning the representative features of given points to estimate the distribution of new points. However, in high magnification point cloud super-resolution tasks, the dense points generated by existing works still tend to exhibit non-uniform noise or retain excessive noise. This issue may arise from the insufficient point density in low-resolution point clouds, resulting in an ineffective extraction of point cloud features.

In recent times, there has been a swift progression in the utilization of industrial robots. In many cases, achieving specific tasks requires the coordinated operation of visual sensors and robotic arms equipped with tactile sensors. Our integration of tactile sensing primarily aims to supplement visual data in challenging conditions. For example, in high-intensity light situations such as robot arm welding, visual sensors often struggle. Similarly, in low-light conditions, like underwater archaeological explorations in murky seas, relying solely on visual data can be inadequate. Moreover, in scenarios with adverse weather conditions, such as heavy fog or smog, visual sensors frequently face significant challenges. In these cases, the use of tactile information from robot arms or other equipped machinery becomes invaluable for effective point cloud upsampling. Additionally, in fast-paced industrial assembly lines where visual sensors may not capture 3D data quickly enough, tactile data from existing sensors can significantly enhance the upsampling process. By integrating tactile information with visual data and leveraging tactile assistance for point cloud upsampling, it becomes possible to accomplish this task even with a limited number of point clouds. Consequently, this approach can provide high-quality point clouds for high-level vision tasks such as point cloud classification and object detection, setting the stage for further advancements.

In this paper, we leverage both visual information and local tactile data to enhance point cloud upsampling. To do so, we introduce a feature fusion module that integrates tactile features with visual features. By progressively refining visual features using tactile information obtained during each touch, this module exploits the complementarity of both modalities, leading to a substantial performance boost. Inspired by the PU-Transformer^[29], we input well-fused features along with the 3D coordinates into the cascaded transformer encoders to generate a more comprehensive feature map. Finally, we use the shuffle operation^[31]to form a dense feature map and reconstruct the 3D coordinates with a multilayer perceptron (MLP). Our main contributions can be summarized as follows: (1) we introduce both visual and tactile information into the point cloud upsampling task and achieve improved qualitative and quantitative results compared to using a single modality; (2) we introduce a feature fusion module that effectively leverages information from both tactile and visual modalities; (3) we build a dataset containing tactile information to benchmark point cloud upsampling algorithms in this setting.

Compared to optimization-based approaches^[32–34], deep learning methods exhibit a promising advancement due to their data-driven nature and the learning capacity of neural networks. Through deep neural networks, it has become possible to directly learn features from point clouds, such as PointNet^[6], PointNet++^[7], and PointCNN^[35]. The earliest use of deep learning networks to process low-resolution point clouds was the PU-Net^[36], which drew inspiration from PointNet. It initially divides the low-resolution point cloud into small blocks of different scales and then employs multiple MLPs for feature extraction at different scales. These multi-scale features are aggregated and fed into an upsampling module to expand the features. Finally, a coordinate reconstruction module remaps the features back into the 3D coordinate space to obtain high-resolution point clouds. As the pioneer in the field of deep-learning-based point cloud super-resolution tasks, there have been many outstanding works that have drawn inspiration from its foundations. Deep-learning-based point cloud super-resolution algorithms can generally be divided into two modules: the feature extraction module and the upsampling module. In the feature extraction module, the 3D coordinates are mapped to a feature space, resulting in high-dimensional features that are then input to the upsampling module. Through feature expansion, a denser set of features is generated. Finally, these features are remapped back into the 3D coordinate space to obtain high-resolution point clouds. Wang et al. proposed MPU^[37], a patch-based upsampling pipeline that can flexibly upsample point cloud patches with rich local details. Furthermore, MPU employs skip connections between each upsampling unit to facilitate information sharing, enabling the utilization of fewer parameters while achieving enhanced generalization performance. However, MPU is computationally expensive due to its progressive nature. Li et al.^[30] introduced DisPU, which disentangles the upsampling task into two sub-goals. First, it generates a coarse but dense point set and then refines these points over the underlying surface to improve distribution uniformity. More recently, Qiu et al.^[29] proposed PU-Transformer, the first introduction of a transformer-based model for point cloud upsampling. The PU-Transformer gradually encodes a more comprehensive feature map through cascaded encoders using the preliminary feature map. Subsequently, a coordinate reconstruction module is employed to map the refined features back into the 3D coordinate space. Existing point cloud super-resolution networks exclusively rely on low-resolution point cloud information, thus leading to suboptimal performance when dealing with high magnification point cloud upsampling tasks where the quality of the low-resolution point clouds is poor. We propose a new approach to incorporate denser and more precise tactile point cloud information as a supplementary input, resulting in high-resolution point clouds that exhibit improved uniformity and superior local representation.

Traditionally, researchers have predominantly relied on visual information for shape reconstruction. However, with the advancement of tactile sensors, Li et al.^[14] introduced a novel tactile sensor known as Gelsight. Using the Gelsight sensor, it becomes possible to utilize photometric stereo algorithms to reconstruct the 3D geometric shape of object surfaces. Wang et al.^[38] introduced a convolutional neural network (CNN)-based approach that combines visual and tactile information for 3D reconstruction. They reconstructed the 3D model of the object from RGB images and then proceeded to touch the areas of the 3D model with the highest uncertainty. Tactile data provides accurate shape information of the object’s surface, which is then employed as a constraint to refine the shape of the 3D model. Smith et al.^[28] proposed a visual-tactile fusion 3D reconstruction network based on graph convolutional network (GCN). They employed a U-Net network to transform tactile information into a mesh representation. Then, a predefined 3D spherical object was used to construct an adjacency matrix for the graph structure. RGB visual information and the tactile mesh were fed into the GCN, continuously updating the shape of the 3D object and ultimately obtaining an accurate point cloud model. Rustler et al.^[39] introduced ActVH, a method that constructs a point cloud model from depth images acquired by a depth camera. They then actively interact with regions of higher uncertainty and iteratively refine the 3D shape through tactile feedback, further enhancing the precision of the reconstruction. Currently, tactile information is primarily employed in 3D reconstruction tasks, demonstrating its capacity to provide accurate local information. We introduce tactile information into the point cloud upsampling task and reconstruct a point cloud upsampling dataset named TSR-PD with tactile information based on the 3D reconstruction dataset proposed by Smith et al.^[19]

Essentially, 3D scanning is a sampling problem in 3D physical space, while upsampling is a predictive task aimed at inferring additional samples on the original surface, particularly when dealing with sparse samples obtained during scanning. Given a point set $\mathcal{P}\in {\mathbb{R}}^{N\times 3}$, the point cloud upsampling process with an upsampling ratio of $r$ produces a dense point cloud $\mathcal{Q}\in {\mathbb{R}}^{rN\times 3}$ based on the geometric information of the sparse point cloud $\mathcal{P}$. The resulting high-resolution point cloud $Q$ should have uniformly distributed points and have an edge surface similar to ground truth. Considering the inherent complexity and ill-posed nature of this upsampling process, which is compounded by the incomplete representation of the original geometry, achieving a high upsampling rate of $r$ introduces even greater challenges. Different from the existing methods that primarily focus on the low-resolution point cloud itself, our approach integrates locally dense tactile information to generate high-resolution point clouds that accurately represent the underlying object surface. It holds importance for industrial robots that rely on both 3D scanners and tactile sensors.

The overall framework of our method is shown in Fig. 1. We formulate the point cloud upsampling model as follows: $$\mathcal{Q}=f(\mathcal{P},\mathcal{T}),$$(1)where $f(·)$ represents our tactile-assisted point cloud super-resolution network (TAPSR-Net). Given a sparse point cloud $\mathcal{P}\in {\mathbb{R}}^{N\times 3}$ and dense tactile point cloud $\mathcal{T}\in {\mathbb{R}}^{N\times 3}$, our proposed TAPSR-Net can generate a dense point cloud $\mathcal{Q}\in {\mathbb{R}}^{rN\times 3}$. First, the feature extraction block extracts preliminary feature maps $F_p$ and $F_t$ from the input. Then we feed them into the feature fusion block to produce the fused feature map $F_f$. Next, we use the transformer encoder^[29] to process $\mathcal{P}$ and the fused feature $F_f$, thereby refining the feature map. Finally, we proceed to reconstruct points from the latent space back to the coordinate space, resulting in a denser point cloud of size $rN\times 3$.

As shown in Fig. 2, given $\mathcal{P}\in {\mathbb{R}}^{N\times 3}$ and $\mathcal{T}\in {\mathbb{R}}^{N\times 3}$, our feature extraction block produces preliminary feature maps $F_p\in {\mathbb{R}}^{N\times mC}$ and $F_t\in {\mathbb{R}}^{N\times C}$, where $C$ is the number of feature channels and $m$ is the ratio of tactile point cloud features to low-resolution point cloud features. Inspired by the application of U-Net^[40] in the fields of point clouds and images, we employ cross-layer connections to retain visual and tactile information at different scales, utilizing this approach for initial feature extraction. Specifically, the feature extraction block is downsampled two times by the max-pooling layer and upsampled two times by the deconvolution, and the skip connection is used at the same stage, which ensures that the final recovered feature map incorporates more low-level features and also enables the fusion of features of different scale point cloud features. The initial processing of the tactile point cloud and visual point cloud follows a similar approach to produce feature maps, and the difference between $F_p$ and $F_t$ lies in the number of feature channels. In this work, we empirically determined that the optimal feature dimension for visual point clouds is 4 times that of tactile features. This allows the tactile point cloud to better assist during feature fusion, without excessively diverting attention to tactile information.

Many approaches exploiting vision and touch for 3D tasks, especially in 3D reconstruction, tend to independently process visual and tactile information^[20,39]. Specifically, researchers used touch and partial depth maps separately to predict independent voxel models, which were then combined to produce a final prediction. Indeed, this strategy effectively leveraged tactile information and has proven beneficial. However, a principal limitation of the experimental approach is that its utilization of tactile information is restricted to a small area that has been touched. Different from previous works that generate preliminary results of the final target and gradually refine them using tactile feedback, we prefer generating fused features to integrate both tactile and visual information. This approach enables us to comprehensively learn both the local and global features of tactile information, allowing touch information not only to enhance the upsampling at the touch site but also to extrapolate its local neighborhood. Therefore, we introduce a feature fusion block as shown in Fig. 3. We feed tactile features $F_t$ and visual features $F_p$ into the network sequentially and iteratively optimize visual features through tactile features, finally obtaining fused features $F_f$.

In Algorithm 1, we present the basic operations that are employed to build our feature fusion block. We first feed the low-resolution point cloud feature $f_p$ and $\alpha$ tactile features $f_{ti}$ into the proposed module, where $\alpha$ represents the number of tactile iterations. Each iteration can be formulated as follows: $$F_m=\mathrm{MLP}[\text{Concat}(f_{\text{ti}},f_i)],$$(2)$$f_f=\text{Deconv}[\text{Concat}(F_m,f_p)].$$(3)

As discussed in Sec. 3.2, in order to provide better assistance to visual information using tactile cues, the dimension of the visual features is 4 times that of the tactile features. Specifically, when adding the first tactile iteration, the feature $f_p$ is directly used as the input feature $f_1$. Next, we concatenate $f_p$ with $f_{ti}$ to obtain the refined point feature. A common approach is to apply MLPs followed by max-pooling. Then we concatenate this feature with the low-resolution point cloud feature and then perform a deconvolution to obtain the fused feature. As we merge the subsequent tactile features, this fused feature will be fed into the network as input $f_i$. By repeating this step, we obtain the final fused feature map.

The design of the fusion module combines low-resolution point cloud features with tactile features for more accurate feature correction. In each fusion process, the module utilizes low-resolution point cloud features as the basis and gradually introduces tactile features as auxiliary information. This gradual fusion not only helps improve the accuracy of the overall features, but also effectively corrects possible errors. At the same time, the number of input and output channels of the module is set to a fixed value, and this design makes the module highly flexible and scalable. As the number of tactile touches increases, we can easily connect multiple fusion modules by cascading to further enhance the fusion effect of the features.

The PU-Transformer^[29] encodes the initial feature mapping into a comprehensive feature map via a cascaded transformer encoder containing a positional fusion block and a shifted-channel multi-head self-attention (SC-MSA) block. It fully utilizes the transformer’s power in feature representation. So, we chose to employ the same cascaded transformer encoder as the PU-Transformer to encode a more comprehensive feature map. After extracting the fused features, we used the fused feature map along with the inherent 3D coordinates as the input to the transformer encoder.

The transformer encoder utilizes a positional fusion block to encode and combine both the provided 3D coordinates $\mathcal{P}\in {\mathbb{R}}^{N\times 3}$ and the fused feature map $F_f\in {\mathbb{R}}^{N\times C}$ for a point cloud. This process follows the local geometric relationships between scattered points. The positional fusion block not only encodes positional information for a set of unordered points, facilitating the transformer’s processing, but also aggregates comprehensive local details for precise point cloud upsampling. The main purpose of the SC-MSA block in the transformer encoder is to enhance inter-point relations in a multi-head fashion and strengthen channel-wise connections by introducing overlapping channels between consecutive heads.

We reconstruct points from the latent space to the coordinate space, resulting in the desired denser point cloud of size $rN\times 3$, where $r$ represents the upsampling scale. The output of the last transformer encoder serves as the input and undergoes a periodic shuffling operation^[31] to reorganize the channels and create a dense feature map. Finally, another MLP is applied to estimate the 3D coordinates of the upsampled point cloud $\mathcal{P}\in {\mathbb{R}}^{rN\times 3}$.

In order to validate the effectiveness of tactile information in point cloud super-resolution tasks, we reconstructed a tactile-enhanced point cloud super-resolution dataset, named the tactile super-res point cloud dataset (TSR-PD), based on the 3D reconstruction dataset introduced by Smith et al. (2020). We also conduct ablation studies to show the benefits of the proposed modules. To be specific, there are two key components: the feature extraction block and the feature fusion block.

Smith et al.^[19] built a new dataset that aims to capture the interactions between a robotic hand and an object it is touching for 3D reconstruction. They loaded example objects into the 3D robotics simulator Pybullet^[41], placed the hand randomly on its surface, and then closed its fingers to attempt to produce contact between the sensors and some point on the object using inverse kinematics. Touch signals were ultimately obtained through the simulation of the DIGIT principle^[15]. In this paper, we chose 8192 points from the object model in ABC Dataset^[42] as the ground truth, which were then downsampled to 512 points to produce the low-resolution point cloud. Figure 4 depicts an object from our dataset. Tactile point clouds are generated by tactile sensors, and the number of points in a tactile point cloud depends on the area of contact with the tactile sensor. A tactile point cloud containing at least 1000 points will be regarded as a successful touch. Initially, these tactile point clouds were aligned with the object point cloud using Euler angles and then downsampled to 512 points to serve as the tactile input for our network. The dataset comprises a total of 12,732 samples, covers a large semantic range of 3D objects, and includes simple, as well as complex, shapes to evaluate the generalization capability of the model.

We use the Chamfer distance (CD) loss to minimize the distance of the predicted point cloud and the referenced ground truth in our experiments: $$\mathrm{C}(\mathrm{P},\mathrm{Q})=\frac{1}{|P|}\sum _{p\in P}\underset{q\in Q}{\min }{\Vert p-q\Vert }_2^2+\frac{1}{|Q|}\sum _{q\in Q}\underset{p\in P}{\min }{\Vert p-q\Vert }_2^2,$$(4)where P is the predicted point cloud, Q is the ground truth, $p$ is a 3D point from P, and $q$ is a 3D point from Q. The first term represents the sum of the minimum distances from any point $p$ in P to Q. The second term represents the sum of the minimum distances from any point $q$ in Q to P. A larger CD indicates greater differences between the two sets of point clouds, while a smaller inverse CD indicates that the predicted point cloud is closer to the ground truth point cloud, resulting in better prediction accuracy.

We train our network with a single NVIDIA RTXA6000 running on the Linux OS in all the experiments. In terms of the hyperparameters for training, we have a batch size of 32 for 300 training epochs, optimized using Adam with an initial learning rate of 0.001 with a 0.7 decay rate. Similar to previous work, we perform point cloud normalization and augmentation^[29] (rotation, scaling, and random perturbations). We report results using a $\times 16$ upsampling rate, i.e., $r=16$.

Table 1 shows the quantitative comparisons on the dataset under upsampling rates $r=16$. The input low-resolution point cloud consists of 512 points, each tactile point cloud contains 512 points, and the output high-resolution point cloud comprises 8192 points. The evaluation metrics used include CD, Hausdorff distance (HD), and Earth-moving distance (EMD). We can see that, for this task, incorporating tactile information yields better results compared to not using tactile information.

However, we found that it is not the case that upsampling performs better as the number of tactile points increases; as the number of tactile iterations increases, the rate of decrease in CD and HD slows down. However, when adding a fifth tactile iteration, CD and HD increase compared to four tactile iterations. When adding a second tactile iteration, the EMD reaches its minimum value. This observation emphasizes the existence of an upper limit to the beneficial effects of tactile feedback on vision. We consider that the reason for this result may be that the tactile data is not perfect, and the tactile point cloud we obtained by simulating the DIGIT principle may contain errors. Within a certain range, the gain brought by adding tactile information is greater than the mistakes brought, so the performance of upsampling is improved as the number of touches increases. Beyond this range, adding tactile information multiple times increases the mistakes, which affects the overall point cloud features and makes the upsampling performance degrade. In addition, multiple tactile inputs cause the network to focus excessively on local tactile information and reduce the sensitivity to visual information, which should dominate the features of the visual point cloud in the point cloud upsampling process.

The qualitative results of different point cloud upsampling models are presented in Fig. 5. Three object models were selected for visualization, and local regions were magnified for closer examination. From left to right, the three columns represent the joint, arch, and lamp post, respectively. The first row shows the GT with 8192 points. The second row shows the input low-resolution point cloud to the network (512 points), with the red regions indicating four densely arranged tactile point clouds. Each tactile point cloud also consists of 512 points. The third row shows the upsampling points (8192 points) without incorporating tactile information. The fourth row shows the upsampling points (8192 points) produced by the proposed module and incorporates four instances of tactile information.

Comparing the dense points produced in the cases of incorporating tactile information and not incorporating tactile information, we can see that the method without adding tactile information tends to introduce excessive noise [e.g., Fig. 5(a)], cluster points together with a non-uniform distribution [e.g., Fig. 5(b)], or destroy some tiny structures [e.g., Fig. 5(c)] in the results. In contrast, our method incorporating tactile information produces the most similar visual results to the target points, and our dense points can well preserve tiny local structures with a uniform point distribution. Consequently, it can be inferred that the addition of tactile information not only impacts the local effects during point cloud upsampling but also integrates tactile cues as part of the global information, influencing the overall results. This can be particularly observed in the magnified views depicted in Fig. 5.

To further validate the effectiveness of tactile fusion and the superiority of the proposed algorithm, we employed several point cloud super-resolution algorithms for training and testing on the TSR-PD dataset, including PU-GAN^[27], PU-GCN^[28], Grad-PU^[43], and PU-Transformer^[29]. To provide a more intuitive representation of the differences in high-resolution point clouds obtained by different algorithms, we conducted 16× point cloud super-resolution experiments using models trained in this section on the test set and performed qualitative analysis.

As shown in Fig. 6, we selected a buckle for visualization. Figure 6(a) represents the original high-resolution point cloud, containing 8192 points. Figure 6(b) depicts the low-resolution point cloud, containing 512 points, with shaded regions representing dense tactile point clouds incorporated using our algorithm, where each tactile point cloud comprises 512 points. Figure 6(c) shows the high-resolution point cloud reconstructed using the PU-GCN. Figure 6(d) shows the high-resolution point cloud reconstructed using the Grad-PU. Figure 6(e) shows the high-resolution point cloud reconstructed by the PU-Transformer^[29]. Figure 6(f) shows the high-resolution point cloud reconstructed by TAPSR and incorporates four instances of tactile information.

We can observe that several comparative algorithms demonstrate a certain degree of super-resolution effect compared to the input low-resolution point cloud, but the super-resolution effect is unsatisfactory. For example, as shown in Fig. 6(c), the PU-GCN algorithm can reconstruct the buckle’s outline, but it generates numerous outlier points on the left side of the buckle. Conversely, Fig. 6(d) generated fewer outlier points, but there are some missing parts in the outline of the buckle. In comparison, our algorithm can better reconstruct the buckle’s outline with fewer outlier points. This indicates that the additional local information provided by the tactile point cloud effectively assists in the point cloud super-resolution task.

As shown in Table 2, we compared the quantitative performance of different algorithms in point cloud super-resolution on the TSR-PD test set. Compared to other point cloud super-resolution algorithms that do not incorporate tactile information, our model achieves optimal super-resolution performance, exhibiting the lowest CD, HD, and EMD. This indicates the effectiveness of utilizing tactile information to assist point cloud super-resolution through iterative fusion.

Table 3 shows the quantitative comparisons on the dataset under different upsampling rates. Different upsampling rates use the same model, while the network we designed is mainly used for point cloud upsampling at $16\times$ upsampling rates. Thus, the best CD values were obtained at $16\times$. More importantly, we can see that at different upsampling rates, for this task, incorporating tactile information yields better results compared to not using tactile information.

To evaluate the effectiveness of the major components in our framework, we conduct ablation studies on the feature extraction block and feature fusion block. All the models are trained and evaluated on TSR-PD. The results in Table 4 show the effectiveness of our fusion module when incorporating tactile information. Specifically, we remove the feature extraction module and the feature fusion module under different tactile iterations. In the first row, we removed the feature extraction and feature fusion modules and directly concatenated the tactile point clouds. This led to a decrease in the quality of upsampling. Due to the disruptive effect of dense local information on the overall structure of the point cloud, the performance without incorporating tactile information was better than those with tactile information. In the second row, we removed the feature fusion module and directly concatenated the sparse features with the tactile features. Clearly, our complete pipeline consistently achieves the best performance with the lowest CD value across different numbers of tactile iterations. Furthermore, removing any component leads to a reduction in overall performance, meaning that each component in our framework contributes.

Table 5 shows the time required for the model to train each epoch and the inference time required to reconstruct from a low-resolution point cloud to a high-resolution point cloud with different numbers of touches. It can be seen that with the increase in the number of touches, the training time of the model has a certain increase, and the increase is also within a reasonable range.

In this paper, we propose a method of point cloud upsampling assisted with tactile information. Specifically, we design a feature fusion module that effectively leverages information from both tactile and visual modalities. With the assistance of tactile information, our approach can significantly improve the quality of point cloud upsampling both quantitatively and qualitatively compared with using visual information only, since the tactile point cloud data contains both local and global features of objects.

The network in this paper is built on the PU-Transformer^[29] network architecture, but this idea of utilizing multimodal information to assist in the task of point cloud upsampling can be applied to other networks for point cloud upsampling as well. We note that a number of feature extraction methods have been proposed; in the future, we plan to explore new feature extraction methods^[44]. Additionally, we expect to further explore the application of visual and tactile fusion, expanding its adaptability in high-level 3D visual tasks.