End-to-End Data Movement: Paradigm Reexamination and Principles for Efficiency
High-performance data transfer is often viewed through raw bandwidth, with 100+ Gbps international links seen as the primary enabler. Yet this network-centric view confuses provisioned speed with sustainable throughput. Suboptimal rates occur even on 10 Gbps links, and faster networks only magnify the issue. We examine six paradigms - network latency, TCP congestion control, CPU performance, virtualization, and others - that critically impact data movement workflows. These reflect common engineering assumptions shaping system design, procurement, and operations. To bridge the gap between raw bandwidth and application-level throughput, we introduce the "Drainage Basin Pattern" - a conceptual model for reasoning about end-to-end constraints across heterogeneous hardware and software at varying target rates. Our findings are validated via production-scale deployments, from 10 Gbps links to U.S. DOE ESnet technical evaluations and transcontinental trials over 100 Gbps operational links. Results show that bottlenecks typically lie outside the network core, and that holistic hardware-software co-design delivers consistent, predictable performance for demanding bulk and streaming transfers. A burst buffer subsystem, together with data staging, is introduced at every tier to decouple data movement from erratic production storage and sustain wide-area transfer, with a quantitative bound for sizing the buffer capacity it requires. The primary goal is to transform such transfers from unpredictable struggles into routine, line-rate operations accessible to any regular user. Finally, we correct two industry misconceptions: using aggregated traffic rate as a measure of application efficiency, and conflating operational complexity with technical expertise.