I was reminded today about the difference of removing vs. deleting a web part from a site page today.  There’s already a great blog on the subject and it can be found here. 

The quick and dirty is that when you either choose to “close” on a web part or click the “X” in edit view, this does NOT remove it from the page, it only hides it.  There is a Closed Web Parts gallery where this closed web part gets placed, and if there are enough closed web parts, it can significantly decrease your pages load time, as it still loads the closed web parts. 

To close the web part, the above blog shows you how to navigate to the Close Web Parts gallery and remove it so it will no longer be a drag on the page.  It has also been pointed out that there is a quicker way to delete web parts. 

Before you have closed or clicked the X on the web part, add ?contents=1 at the end of your URL which loads the site maintenance page where you can easily click the web part via checkboxes and delete it.

SharePoint Web Part Page Maintenance