We propose a novel convolutional neural network architecture AIFNet for removing spatially-varying defocus blur from a single defocused image. To remedy the lack of real defocused image datasets, we leverage light field synthetic aperture and refocusing techniques to generate a large set of realistic defocused and all-in-focus image pairs depicting a variety of natural scenes for network training.