We present structure preserving integrators for solving linear quadratic optimal control problems. The goal is to build methods which can also be used for the integration of nonlinear problems if they are previously linearized. The equations are solved using an iterative method on a fixed mesh with the constraint that at each iteration one can only use results obtained in previous iterations on that fixed mesh. On the other hand, this problem requires the numerical integration of matrix Riccati differential equations whose exact solution is a symmetric positive definite time-dependent matrix which controls the stability of the equation for the state. This property is not preserved, in general, by the numerical methods. We analyze how to build methods for the linear problem taking into account the previous constraints, and we propose second order exponential methods based on the Magnus series expansion which unconditionally preserve positivity for this problem and analyze higher order Magnus integrators. The performance of the algorithms is illustrated with the stabilization of a quadrotor which is an unmanned aerial vehicle.