Data, DataViz, and Stats with the Stars

Orange is a visual drag-and-drop tool for

• Data visualization
  
• Statistical Tests
• Machine Learning
  
• Data mining

and much more. You can download and install Orange from here:

https://orangedatamining.com/download/

• Let’s use the Datasets widget
• Click on it to select one of the built-in CSV files : Auto MPG
• Let’s look at the Data using the Data Table widget
• And create a Scatter Plot with the Scatter Plot widget (Horsepower vs Displacement)
• Try the menu options on the left side to see how they alter the plot

Steven Stigler (2016) in “The Seven Pillars of Statistical Wisdom”:

• One of the Big Ideas in Statistics is: Aggregation
• How is it revolutionary?
• By stipulating that, given a number of observations, you can actually gain information by throwing information away
• In taking a simple arithmetic mean, we discard the individuality of the measures, subsuming them to one summary.

• So what do we throw away now? And how?

• Let’s look at the existing dataset, Auto MPG
• Pull in the Feature Statistics widget
• Connect the Datasets widget to the Feature Statistics widget
• Appreciate the Summary Statistics that are generated

• To summarize is to understand.
• Add to that the fact that our Working Memories can hold maybe 7 items, so it means information retention too.
• Borges wrote, “To think is to forget details, generalize, make abstractions. In the teeming world of “Funes the Memorious,”, there were only details.”
• Brad Pitt aka Billy Beane was throwing away the details, and looking at the aggregated picture to pick his future Oakland A’s team.

• Using Conjectures:
• Symbols -> Letters
• Based on well-known Counts of letters (Zipf’s Law)
• Holmes deduces that the most common letter in the message is “E”
• He then deduces that the second most common letter is “T”

• In “The Accountant,” Christian Wolff is heard reciting “Solomon Grundy,”
• The nursery rhyme tells the life and death of a man named Solomon Grundy, all within a single week.
• It was innocently used to help children learn their days of the week.
• However, when we look into the fact that Thursday through Sunday detail the tragic end of Mr. Grundy due to an unspecified illness…
• it’s hard to ignore the dark undertones.

• The data is the days of the week.
• The data is the number of events that happen on each day.
• The y-variable is a Quant variable, a number
• The x-variable is also Quant variable, a time variable

• The data is the number of births in the USA, by day, month, and year
• Let us use the Group By widget to group by day_of_week
• AND compute the mean(births) in the same widget
• We plot the mean(births) vs month, and colour by day_of_week

• Time series data is a special kind of Quantitative data, where the x-variable is a time variable.
• The y-variable is a Quant variable, a number.
• The x-variable is a Quant variable, a time variable.
• By colouring by day_of_week, we can see how the number of births varies by day of the week.

What is the Story Here?

• Could there be a staffing shortage at hospitals on weekends?
• Is this a “revealed preference” thing?
• Or should we watch Grey’s Anatomy, or even House?

• Katie Ledecky is a swimmer, and a mermaid.
• She has won 7 Olympic gold medals, and 15 World Championship gold medals.
• She is the world record holder in the 400, 800, and 1500 meter freestyle events, and in the 4x100 meter freestyle relay, and the 4x200 meter freestyle relay.
• What does that make her? An Outlier…

• Outliers are data points that are significantly different from the rest of the data.
• They can be identified using box plots, which show the distribution of the data.
• Box plots show the median, quartiles, and outliers of the data.
• Of course, Ledecky was in the water! Well in.

• Let’s get some data to plot:

Rows: 397 Columns: 6
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (3): rank, discipline, sex
dbl (3): yrs.since.phd, yrs.service, salary

ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.

• And let’s use a pre-set Workflow in Orange Download the Orange Box Plot Workflow

• Box Plots tell us distributions of Quant variables, and show us outliers.
• They show us the median, quartiles, and outliers of the data.
• They are useful for comparing distributions of Quant variables across Qual variables. ( E.g Sex or Rank)

What is the Story Here?

• Could there be a systemic bias in salaries?
• A statistical t-test / ANOVA would tell us if that is true.
• Look in the figures for a t-test and ANOVA report at the bottom.

• What was the “data” Gabbar was looking at?

• The number of people in the village, and the number of people who were armed. (Quant)
• The number of bullets in the gun (Quant)
• The ID of the pistol chamber which contains a bullet (Qual)

• So it appears the call percentage is different for the two ethnicities, afam and cauc
• But is it statistically significant? Would Gabbar be surprised?
• Let us pretend ethnicity does not matter and spin the revolver!!
• We mess with the ethnicity variable, some 5000 times

• We are not able to mimic Mother Nature aka Reality
• The red line is the observed difference in proportions, and it is way out of the null distribution.
• So we can reject the NULL Hypothesis that ethnicity does not matter.
• Hence we infer that there was bias in the hiring process, and that afam candidates were discriminated against.