This study addresses how the competitive impact between the effective stress and the CH4-CO2 counter diffusion-induced coal matrix swelling/shrinking affects the CO2 enhanced coalbed methane recovery. This issue is important both for the extraction of methane as an energy resource and for the safe sequestration of carbon dioxide in a coal seam. Through this study, a novel fully coupled coal deformation, gas flow, and CH4-CO2 counter-diffusion finite element (FE) model was developed and applied to the quantification of the combined net effects on coal permeability among the coal matrix swelling/shrinking due to gas displacement, pore pressure and in-situ stress. These combined effects are quantified through solving a set of coupled field equations which govern the coal deformation, prescribe the transport and interaction of gas flow in a similar way to poroelastic theory, and define CH4-CO2 counter diffusion and flow in a coal seam. Numerical results demonstrate that the evolution of porosity and permeability is controlled by the competing influences between the effective stress and the CH4-CO2 counter-diffusion induced volume change. This achievement extends our ability to quantify the opposite impacts of effective stress-induced and CH4-CO2 counter diffusion-induced coal matrix deformation on the CO2 enhanced coalbed methane recovery under field conditions.