In this talk, I will first discuss relationship between some deep learning models and traditional algorithms such as finite element and multigrid methods. Such relationships can be used to study, explain and improve the model structures, mathematical properties and relevant training algorithms for deep neural networks. I will report a class of new training algorithms that can be used to improve the efficiency of convolutional neural networks (CNN) by significantly reducing the redundancy of the model without losing accuracy. By combining multigrid and deep learning methodologies, I will present a unified model, known as MgNet, that simultaneously recovers some CNNs for image classification and multigrid methods for solving discretized PDEs. MgNet can also be used to derive a new class of CNNs that mathematically unify many existing CNN models and practically prove to be computationally competitive.