Controlling and manipulating the internal states by optical pulses are crucial in some physical systems. However, it is a challenge to achieve the high-precision quantum computing in imperfect physical systems, because it is limited by various dephasing factors such as frequency detuning or fluctuation in a tightly packed frequency interval, the unwanted off-resonant excitation outside this interval, decoherence, and Rabi frequency fluctuations. At present, different methods, such as Lewis-Riesenfeld Invariant (LRI) and Transitionless Quantum Driving (TQD), have been put forward and been implemented experimentally to inversely engineer the time-dependent Hamiltonian of a quantum system and to accelerate slow adiabatic processes via nonadiabatic shortcuts. Here, we use the TQD method to speed up adiabatic passage technique in the Rare-Earth Ions (REI) system and to eliminate the microwave field that is hard to implement experimentally. According to the characteristics of the REI system, to achieve the high-precision quantum control demands that the light-matter interaction treat different frequencies as a single one, but shut off abruptly. That means, the quantum control should have the same manipulation over the ensemble qubit ions with a frequency distribution in the range of ±170 kHz. At the same time, it can’t affect the background qubit ions which are about 3.5 MHz away from the addressing frequency of the target qubit as these excited background qubit ions have a probability of interfering with the target qubit ions. Therefore, it is necessary and important to control the light-matter interaction by designing appropriate optical pulses specifically, so that the quantum control can overcome the influence of the restrictive factors present in the system, eventually achieving high-fidelity quantum manipulation in such systems. In this article, we made an analysis on the three limitation factors: 1) the frequency detuning between the ensemble qubit ions, 2) the off-resonant excitation to the background qubits around the target qubit, 3) the infeasibility of using a microwave field to directly couple the two qubit levels. We proposed a theoretical scheme for non-adiabatic high-fidelity quantum operations in a three-level system to overcome the three limitation factors. The main work includes: 1) The theoretical scheme to construct the optical pulses by utilizing the time evolution operator is proposed. It is constructed based on a set of orthogonal auxiliary states containing time-dependent parameters, and the Hamiltonian of the system is inversely constructed from the time evolution operator. From the one-to-one correspondence between the matrix elements in the Hamiltonian and the Rabi frequency of optical pulses, the representation of optical pulses is obtained. 2) A method of eliminating a microwave field to directly couple the two qubit levels is proposed. By confining the relationship between the time-dependent parameters, which are introduced in the orthogonal auxiliary states and time evolution operators, the microwave coupling term in the Hamiltonian is eliminated. The elimination of the microwave field can simplify the experimental operation. 3) The non-adiabatic optical pulses to manipulate the quantum state of REI ensemble qubits with high fidelity are designed. The performance of the optical pulses is improved by optimizing the multiple degrees of freedom in the pulses. In summary, the optical pulses developed in this scheme can not only eliminate the direct-coupling microwave field between the two qubit levels, but also achieve high-fidelity (99.86%) quantum control over the ensemble qubits which are distributed in a frequency range of about ±170 kHz, and in the mean while suppress the unwanted excitation of other qubits with a distance to the qubit-ion addressing frequency ≥3.5 MHz. This scheme is not only applicable to REI ensemble qubit system, but also to other quantum systems where qubits are addressed in frequency.