How can I remove the tag base that WordPress creates by default in permalinks? Any reason why it would be there in the first place?
It's probably there to help visually and programmatically distinguish tag URLs from pages; I'm not sure, I'm not a core developer of WordPress.
But http://wordpress.org/extend/plugins/wp-no-tag-base/ and http://wordpress.org/extend/plugins/wp-no-category-base/ work well and do not make permanent changes to files or GUIDs, so they can be deactivated with no ill effects.
Use the plugins as "regular" plugins, or as mu-plugins http://codex.wordpress.org/Must_Use_Plugins
The permalink bases for taxonomies exist to make it easier to distinguish those requests from page or post requests. Let’s say there is a request to
/world/ and you have a tag and a page with that slug.
What will WordPress send to the visitor? That is not predictable for the end user without a clear, visible distinction. It would be the term page probably, depending on the inner workings of the plugins @markratledge mentioned.