Every file in the modules folder is read for inclusion in the Drupal stack. If you store large third party libraries, such as teh YUI library or third party WYSIWYG editors, this will slow down Drupal. The modules folder is usually in /sites/all/modules or /sites/all/modules/contrib. To avoid such unnecessary file reading, move all your third party libraries to a special folder at/sites/all/libraries.
For example, think of the following libraries: Wysiwyg, YUI, jQuery UI, Facebook SDK, Geshi filters, CKeditor, CKFinder, Getid3, Jwplayer5, Phpmailer, Swfobject, TCPDF and more... Remove them all from the /modules folder.
Optionally work with the libraries API module.