In his Top 10 Data Mining Mistakes John Elder shares lessons learned from 20 years of data science consulting experience. Avoiding these mistakes are cornerstones to any successful analytics project.
In this paper about Mistake #6 you will learn that outliers and leverage points can greatly affect summary results and cloud general trends. Yet, one must not routinely dismiss them; they could be the result.
To the degree possible, visualize your data to help decide whether outliers are mistakes to be purged or findings to be explored.