I think the best way is to use the [before_wp_tiny_mce()][1] hook.  Then, you can define the url in PHP; and pass it to the page so it is available to TinyMCE.

    function sgp_before_wp_tiny_mce() {

        ?>
        <script type="text/javscript">
            var sgp_plugin_url = "<?php echo plugins_url('shortcode_generator_popup.php', __FILE__); ?>";
        </script>
        <?php
    }
    add_action('before_wp_tiny_mce', 'sgp_before_wp_tiny_mce');

You may need to change the [plugins_url()][2] to match the appropriate location of your php file.

Then, you can use the variable in your plugin tinymce script to get the location.

    ed.windowManager.open({
        file : sgp_plugin_url,

I haven't tested this... it's from memory. Let me know if you encounter any issues and I'll adjust accordingly.


  [1]: http://wpseek.com/hook/before_wp_tiny_mce/
  [2]: http://codex.wordpress.org/Function_Reference/plugins_url