The head-mounted display system is widely applied in the fields of modern education, medical treatment, and entertainment. The off-axis reflective head-mounted display optical system is a design form of head-mounted display system, and it can not only meet the requirements of miniaturization and compactness, but also realize broadband spectrum imaging without color aberration correction. However, the existing designs of off-axis reflective head-mounted display optical systems cannot compatibly meet the demands of large diameter of exit pupil and small F number together. To solve this problem, we design an off-axis head-mounted display optical system based on two reflective free-form surfaces to realize the requirements of large diameter of exit pupil and small F number. The double-curvature free-form surface and XY polynomial freeform surface are used in our designed head-mounted display optical system, its pupil diameter is 10 mm, F number is 3.0, field of view is 28°, and eye relief is larger than 15 mm. The final imaging performance of the system meets the requirements, and it is better than the current design results.