The mechanical properties of the extracellular matrix (ECM)–a complex, 3D, fibrillar scaffold of cells in physiological environments–modulate cell behavior and can drive tissue morphogenesis, regeneration, and disease progression. For simplicity, it is often convenient to assume these properties to be time-invariant. In living systems, however, cells dynamically remodel the ECM and create time-dependent local microenvironments. Here, we show how cell-generated contractile forces produce substantial irreversible changes to the density and architecture of physiologically relevant ECMs–collagen I and fibrin–in a matter of minutes. We measure the 3D deformation profiles of the ECM surrounding cancer and endothelial cells during stages when force generation is active or inactive. We further correlate these ECM measurements to both discrete fiber simulations that incorporate fiber crosslink unbinding kinetics and continuum-scale simulations that account for viscoplastic and damage features. Our findings further confirm that plasticity, as a mechanical law to capture remodeling in these networks, is fundamentally tied to material damage via force-driven unbinding of fiber crosslinks. These results characterize in a multiscale manner the dynamic nature of the mechanical environment of physiologically mimicking cell-in-gel systems.