I believe the function you are looking for is wp_set_object_terms(). You will need to ensure that you are not giving duplicate IDs when append is set to false. I'm not sure what the behavior is when append is set to true.
EDIT: Having read through the source code for the function, it looks like if you're appending, then it does check for duplicates.