Neutralizing antibodies (nAbs) elicited against the receptor binding site (RBS) of the spike protein of wild-type severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) are generally less effective against recent variants of concern. RBS residues Glu⁴⁸⁴, Lys⁴¹⁷, and Asn⁵⁰¹ are mutated in variants first described in South Africa (B.1.351) and Brazil (P.1). We analyzed their effects on angiotensin-converting enzyme 2 binding, as well as the effects of two of these mutations (K417N and E484K) on nAbs isolated from COVID-19 patients. Binding and neutralization of the two most frequently elicited antibody families (IGHV3-53/3-66 and IGHV1-2), which can both bind the RBS in alternative binding modes, are abrogated by K417N, E484K, or both. These effects can be structurally explained by their extensive interactions with RBS nAbs. However, nAbs to the more conserved, cross-neutralizing CR3022 and S309 sites were largely unaffected. The results have implications for next-generation vaccines and antibody therapies.