Often overlooked by Drupal beginners, but still worth mentioning: do not forget to enable core caching at Administer > Site Configuration > Performance (/admin/settings/performance). To avoid issues with Drupal modules, simply select Cache Mode: Normal. And enable Page Compression.
Here you can also enable bandwidth optimizations and block cache. We will come back to that later.