The aberration and Doppler coupling effects of the Cosmic Microwave Background (CMB) were recently measured by the Planck satellite. The most straightforward interpretation leads to a direct detection of our peculiar velocity β, consistent with the measurement of the well-known dipole. In this paper we discuss the assumptions behind such interpretation. We show that Doppler-like couplings appear from two effects: our peculiar velocity and a second order large-scale effect due to the dipolar part of the gravitational potential. We find that the two effects are exactly degenerate but only if we assume second-order initial conditions from single-field Inflation. Thus, detecting a discrepancy in the value of β from the dipole and the Doppler couplings implies the presence of a primordial non-Gaussianity. We also show that aberration-like signals likewise arise from two independent effects: our peculiar velocity and lensing due to a first order large-scale dipolar gravitational potential, independently on Gaussianity of the initial conditions. In general such effects are not degenerate and so a discrepancy between the measured β from the dipole and aberration could be accounted for by a dipolar gravitational potential. Only through a fine-tuning of the radial profile of the potential it is possible to have a complete degeneracy with a boost effect. Finally we discuss that we also expect other signatures due to integrated second order terms, which may be further used to disentangle this scenario from a simple boost.