Dealing with Non-Normal Data

March 16, 2011

How often do you have to deal with non-normal data? Do you know what to do with it? In his article “Dealing with Non-normal Data: Strategies and ToolsArne Buthmann explains the common reasons for non-normal data and how to handle it.

Addressing Reasons for Non-normality

Reason 1: Extreme Values

Reason 2: Overlap of Two or More Processes

Reason 3: Insufficient Data Discrimination

Reason 4: Sorted Data

Reason 5: Values Close to Zero or a Natural Limit

Reason 6: Data Follows a Different Distribution

No Normality Required

He states that “Some statistical tools do not require normally distributed data. To help practitioners understand when and how these tools can be used, the table below shows a comparison of tools that do not require normal distribution with their normal-distribution equivalents.”

Comparison of Statistical Analysis Tools for Normally and Non-Normally Distributed Data
Tools for Normally Distributed Data Equivalent Tools for Non-Normally Distributed Data Distribution Required
T-test Mann-Whitney test; Mood’s median test; Kruskal-Wallis test Any
ANOVA Mood’s median test; Kruskal-Wallis test Any
Paired t-test One-sample sign test Any
F-test; Bartlett’s test Levene’s test Any
Individuals control chart Run Chart Any
Cp/Cpk analysis Cp/Cpk analysis Weibull; log-normal; largest extreme value; Poisson; exponential; binomial

Application: Missing Data

March 16, 2011

In the real world we will almost always come across missing values in data due to many reasons. This problem must be adressed to produce reliable statistical results.  First of all we need to identify what is missing. Then ask yourself why the data is missing and what it means. After you have answered those questions you need to deal with the missing values. Those are the steps to take:

1. When missing values are few and lay far apart then do nothing.

2. When a column has a significant number of missing values then create a variable for missing, present values (0/1).

3. When a column has a significant number of missing values then replace the missing value with a constant value e.g. mean, median or mode.

4. When a column and its values are essential to producing accurate predictions then estimate the missing value based on other, non-missing data elements.

Any question- Quora gives the answer

March 16, 2011

Need some hints about useful statistical books, free public data sets or have any question about statistic got to:

Einstein once said

May 23, 2010

“Not everything that can be counted counts, and not everything that counts can be counted.”

To read more about this click on the article Metric Mania by John Allen Paulos for The New York Times.