PhantomJS's load time is heavily dependent on the number of fonts that it must load into its cache. Disabling fonts that aren't required can dramatically decrease load time.
How to disable fonts (eHow)
A word of caution - some programs may not render properly if fonts that they rely on are removed.