Quantum error mitigation (QEM) is critical for harnessing the potential of near-term quantum devices. Particularly, QEM protocols can be designed based on machine learning, where the mapping between noisy computational outputs and ideal ones can be learned on a training set consisting of Clifford circuits or near-Clifford circuits that contain only a limited number of non-Clifford gates. This learned mapping is then applied to noisy target circuits to estimate the ideal computational output. In this work, we propose a learning-based error mitigation framework called Clifford Perturbation Data Regression (CPDR), which constructs training sets by Clifford circuits with small perturbations. Specifically, these circuits are parameterized quantum circuits, where the rotation angles of the gates are restricted to a narrow range, ensuring that the gates remain close to Clifford gates. This design enables the efficient simulation of the training circuits using the Sparse Pauli Dynamics method. As a result, CPDR is able to utilize training sets with a better diversity to train the model, compared with previous learning-based QEM models that construct training sets with only Clifford or near-Clifford circuits. Numerical simulations on small-scale Ising model circuits demonstrate that the performance of CPDR dramatically outperforms that of existing methods such as Zero-Noise Extrapolation and learning-based Probabilistic Error Cancellation. Furthermore, using the experimental data from IBM's 127-qubit Eagle processor, our findings suggest that CPDR demonstrates improved accuracy compared to the original mitigation results reported in [Nature 618, 500 (2023)].