Differential dynamic microscopy (DDM) typically relies on movies containing hundreds or thousands of frames to accurately quantify motion in soft matter systems. Using movies much shorter in duration produces noisier and less accurate results. This limits the applicability of DDM to situations where the dynamics are stationary over extended times. Here, we investigate a method to denoise the DDM process, particularly suited to when a limited number of imaging frames are available or when dynamics are quickly evolving in time. We use a convolutional neural network encoder-decoder (CNN-ED) model to reduce the noise in the intermediate scattering function that is computed via DDM. We demonstrate this approach of combining machine learning and DDM on samples containing diffusing micron-sized colloidal particles. We quantify how the particles' diffusivities change over time as the fluid they are suspended in gels. We also quantify how the diffusivity of particles varies with position in a sample containing a viscosity gradient. These test cases demonstrate how studies of non-equilibrium dynamics and high-throughput screens could benefit from a method to denoise the outputs of DDM.