This will work:
wp_set_object_terms( '2', wp_get_object_terms( '1', 'my_taxonomy' ), 'my_taxonomy',array("fields"=>"ids"));
Explanation: added this: ,array("fields"=>"ids") to the attempt i wrote in the question to make the wp_get_object_terms return array of ids alone (what the wp_set_object_terms want to get.