To be useful for most scientific and medical applications, compact particle accelerators will require much higher average current than enabled by current architectures. For this purpose, we propose a photonic crystal architecture for a dielectric laser accelerator, referred to as a multi-input multi-output silicon accelerator (MIMOSA), that enables simultaneous acceleration of multiple electron beams, increasing the total electron throughput by at least 1 order of magnitude. To achieve this, we show that the photonic crystal must support a mode at the $\mathit{\Gamma }$ point in reciprocal space, with a normalized frequency equal to the normalized speed of the phase-matched electron. We show that the figure of merit of the MIMOSA can be inferred from the eigenmodes of the corresponding infinitely periodic structure, which provides a powerful approach to design such devices. Additionally, we extend the MIMOSA architecture to electron deflectors and other electron manipulation functionalities. These additional functionalities, combined with the increased electron throughput of these devices, permit all-optical on-chip manipulation of electron beams in a fully integrated architecture compatible with current fabrication technologies, which opens the way to unconventional electron beam shaping, imaging, and radiation generation.