The volume-to-point (VP) heat conduction problem is one of the fundamental problems of cooling for electronic devices. The existed reports about the VP problem are mainly based on the Fourier’s law which works well at the macroscopic scale. However, the length scale of modern electronic devices has reduced to micro- and nano-scale, at which optimization methods that are capable of dealing with the non-Fourier heat conduction are desired now. In this paper, phonon Boltzmann transport equation (BTE) and solid isotropic material with penalization (SIMP) method are coupled to develop a topology optimization method for ballistic-diffusive heat conduction. Phonon BTE is transformed into equation of phonon radiative transport, which is solved by the discrete ordinate method. To realize the topology optimization, SIMP method is adopted to penalize the phonon extinction coefficient, which equals to the reciprocal of phonon mean-free-path, and an explicit constraint on the global gradient of the nominal material density is used to ensure the solutions being well-posed and mesh-independent. By using the developed topology optimization method, it is found that the optimal material distributions for the VP problem in ballistic-diffusive heat conduction significantly deviate from the traditional tree-like structure obtained in diffusive heat conduction, and the results vary with the Knudsen number (Kn). This is related to the different coefficient interpolation ways in the SIMP method and phonon ballistic transport. When Kn → 0, instead of converging to the conventional tree-like structure which fully stretches into the interior zone, the new method gradually produces the result obtained by the topology optimization which interpolates the reciprocal of the thermal conductivity in diffusive heat conduction. As Kn increases, the high thermal-conductive filling materials show a trend to gather around the low-temperature boundary, and there are more thick and strong trunk structures, less tiny and thin branch structures in the optimized material distributions. In addition, the ratio of the optimized average temperature to the value of the uniform material distribution $\left( {T_{{\rm{ave}},{\rm{opt}}}^{\rm{*}}/T_{{\rm{ave}},{\rm{uni}}}^{\rm{*}}} \right)$ also increases. The dependence of the topology optimization results on Kn can be attributed to the size effect of the thermal conductivity caused by phonon ballistic transport. In the diffusive heat conduction, filling materials with different length scales have the same efficiency to build high thermal-conductive channels. However, with ballistic effect enhancing, size effect makes the effective thermal conductivities of the branch structure lower than those of the trunk structure, as the former is smaller than the latter. As a result, the branch structures are less efficient compared with the trunk structures in terms of building high thermal-conductive channels, and the optimal material distributions have more trunk structures and fewer branch structures. When the ballistic effect becomes significant enough, say at Kn = 0.1, the topology optimization gets a dough-like material distribution in which branches merge into trunks. The proposed topology optimization method have the potential to provide guidance in designing nanoscale electronic devices for improving the heat dissipation capability.