The optical properties of gratings written in nonperiodical multilayer fibers are investigated both theoretically and experimentally. The coupled-mode theory for gratings is extended to be applicable to the fibers with more than one photosensitive layers. The experimental results of fiber Bragg gratings written in laboratory-fabricated ordinary and Er3+ doped multilayer single mode fibers confirm that such gratings possess the common characteristics of the gratings written in ordinary one-layer-core fibers, which indicates that in a device using multilayer fiber as its critical part, gratings written in the multilayer fibers are able to fulfill the requests for grating parts. The experimental results also suggest that the inter-layer stress in core area may be a major cause of photosensitive compensation for hydrogen-loaded Er3+ doped multilayer fiber.