Use JS To Remove A Script Import?

I've got a website that uses AJAX to obtain all of the page. The site contains hundreds of pages and only about 3 or 4 of them show a map. So, in order to not have to import the google maps api for every single user when only a minority of users are ever going to make use of it, I defer the import and then include with javascript, like so:

Code:

var script = document.createElement("script");
script.type = "text/javascript";
script.src = "http://maps.google.com/maps/api/js?sensor=false";
document.body.appendChild(script);

This seems to work nicely, however I am wondering if is possible to use a similar method to remove the import once the user navigates away from a map page. I figure it's a waste of time having all of that stuff in memory wasting space when we are unlikely to use it again. When profiling, the Google Maps API code causes both the longest load time and highest memory usage of any page on the entire website.

I've done a similar thing with styles, using an empty style tag with a name property set. I can then insert page-specific styles and remove them when the page changes. The trouble is, the script tag doesn't have a name property, or seemingly any properties that can be used to identify it other than the src. I could check against this and remove the tag, but is that actually going to remove the JS and free up memory, or will it really just remove the entry from the DOM but not release the mem?