I'm using jQuery UI for a few custom controls on the page edit screen and I've noticed that the styling of the jQuery UI dialogs are conflicting with the TinyMCE dialog. Specifically the buttons and title area look especially bad. I've looked through the jQuery UI css to see what could be removed to avoid any conflicts and they're spread pretty much throughout the whole stylesheet. Is there a way to run both together without conflicting?
Turns out what I had to do was use the CSS Scope option in the jQuery UI download application. I was using the normal wp_register_style/wp_enqueue_style to put it in place and it would always conflict with the jQuery UI dialog styling that the Wordpress install of TinyMCE uses. After switching to using a scoped version of the jQuery UI styles everything worked fine.
It should work just fine if you enqueue your script & stylesheet within functions.php file.
This is how I do that (feel free to copy & paste, it should work out of the box):
Don't forget to delete your previous links to jQuery UI.
If something is still wrong, then your custom JS code is corrupted and affects other elements. Can't say anything more without investigating the code itself.