You can remove buyers at any time by first opening the list of all buyers and then clicking the remove icon next to the respective user: