M6 – Organising and Exploring Data

M6: Organising and Exploring Data

💡 Learning Outcomes

By the end of this module, learners will be able to:

• Recognise why raw data needs cleaning and organisation before analysis.
• Apply basic techniques to tidy, structure, and validate datasets.
• Explore data to identify common features, patterns, and relationships.
• Move from descriptive observations to forming hypotheses and simple predictions.

❓ Key Questions

• Why is real-world data often messy or inconsistent?
• How can cleaning and organising data improve accuracy?
• What can we learn through basic data exploration and summary?
• How do insights and hypotheses support later prediction and analysis?

Structure & Agenda

1. Organising Messy Data (~20 min)
   
2. Exploring Data (~20 min)
   
3. Finding Patterns (~20 min)
   
4. Predicting Outcomes (~20 min)

Submodule 1: Organising Messy Data

What is “Messy Data”?

Data rarely arrives clean. Messy data is any dataset that violates the rules required for analysis.

💬 It may include missing values, inconsistent labels, extra spaces, or mixed types.

Example Messy Dataset

Name    Age   Snack     Height   Home Country
Sarah   21    Chips     1.63m    UK
sarah   21    crisps    1.63     United Kingdom
NA             Cookies  1.7      UK

🕵️ Think of this stage as being a data detective — scanning the scene, not solving the case yet.

Problems Noticed

Problem	Example	Impact on Analysis
Inconsistent Entries	Sarah vs. sarah	Impossible to count unique names
Mixed Types/Formats	1.63m vs 163 vs 5’4”	Blocks averages
Missing Values	NA or blanks	Misleading averages and errors
Synonyms/Variations	Chips vs. crisps	Hides true popularity

⚠️ If data is inconsistent, your results can be misleading or invalid.

Why Organising Matters

Organising data helps you:

• Ensure accuracy and consistency
  
• Remove duplicates
  
• Handle missing values
  
• Format data for analysis

🧩 Clean data = reliable analysis

The Four Pillars of Data Tidy-Up

1. Standardisation
   Convert inconsistent formats → e.g., ‘22 yrs’, ‘Twenty-Four’ → 22, 24

2. Validation
   Check values make sense → e.g., ‘Age: 200’

3. Deduplication
   Remove duplicates → e.g., repeated “Alex C.”

4. Transformation
   Convert data types → e.g., height text → numeric

❌ Clean data isn’t perfect — just ready for reliable calculation.

Categorical vs Numerical Data

Type	Description	Examples	Operations Allowed
Categorical	Qualities, groups, names	Gender, Country, Snack	Counting, grouping
Numerical	Quantities that can be measured	Age, Steps, Commute	Mean, median, sum

⚙️ Mixed numerical values must be converted before calculating averages.

Activity: The Data Detective Game

Dataset:

Setup

• Present dataset
  
• Identify errors (missing values, inconsistencies, duplicates)

#Questions#

1. Unique Students
   • Remove duplicates
     
   • Answer: 9
2. Most Frequent Home Country
   • Standardise country names
     
   • Answer: United States (3 occurrences)
3. Average Age of Chips Lovers
   • Standardise chips/crisps
     
   • Convert ages
     
   • Answer: 21.5

Submodule 2: Exploring Data

What Does It Mean to Explore Data?

Exploration is the first look at a dataset:

• What variables exist?
  
• What types are they?
  
• What common values appear?

💡 Exploration helps catch cleaning mistakes and shapes further questions.

Basic Exploration Techniques

Technique	Tells You	Example Question
Mean	Typical value	Average height?
Median	Middle value	Median commute time?
Mode	Most frequent	Most common snack?
Min/Max	Range	Youngest and oldest?

📈 These require clean data.

Visualising Distributions

• Symmetry
• Skew
• Outliers

A simple bar chart quickly shows common categories (e.g., transport type).

The Importance of the Median

Statistic	Meaning	Best Use Case
Mean	Average	Symmetric data (e.g., height)
Median	Middle value	Skewed data (e.g., commute time)

💬 Median often better represents “typical” experience.

Summary Game

Dataset: {insert link}

Statistic	Question	Answer
Mode	Most common transport method	Bus
Mean	Average hours slept	6.44
Min/Max	Min/max age	19 / 25

Submodule 3: Finding Patterns

From Observation to Patterns

Example:

• Observation: average steps = 7,000
  
• Pattern: walkers average 10,000 vs bus users 4,500

💡 Patterns show correlations, not causation.

Common Pattern Techniques

Technique	Example	Shows
Frequency counts	Most common snack	Popularity
Group comparison	Average age by snack type	Demographic patterns
Cross-tabulation	Snack by home country	Relationships
Sorting/filtering	Top 3 snacks	Quick insights

Why Patterns Matter

Help you:

• Form hypotheses
  
• Understand relationships
  
• Decide what to analyse next

Building Hypotheses

Example table:

Snack	Avg Age	Avg Steps
Crisps	20.1	4,000
Cookies	21.3	5,500
Fruit	23.2	7,000

Examples:

• Fruit eaters may walk more
  
• Snack choice not strongly age-related

Quick Insights Race

Dataset: {insert link}

Find 3 facts in 5 minutes.

Example insights:

1. Chips/Crisps most popular
   
2. 22.22% have ≥ 1 hr commute
   
3. 22.22% don’t use library
   

Submodule 4: Predicting Outcomes

What Does Prediction Mean?

Prediction = using patterns to estimate unknown values.

🧠 Predictions rely on similarity: past behaviour → future expectation.

Hypothesis → Prediction

Example:

• Hypothesis: age affects snack preference
  
• Prediction: 25+ choose savoury snacks

Simple Predictive Thinking

1. Identify question
   
2. Choose variables
   
3. Examine patterns
   
4. Apply pattern to new situation

Prediction is informed guessing.

Forms of Prediction

Approach	Example	Concept
Rule-based	Age > 20 → savoury	Heuristics
Trend extrapolation	Snack preference over time	Trend extension
Probabilistic	60% chance chips	Likelihood

📌 Weather forecasts are predictions.

Uncertainty Matters

• Predictions ≠ guarantees
  
• High probability ≠ certainty
  
• Aim to be less wrong

Prediction Game

Groups predict Estimated Daily Energy Level (High/Low).

Steps

1. Form hypothesis
   
2. Select variables
   
3. Define rule
   
4. Calculate % Low Energy

Benchmark model:

Low Energy if (Hours Slept ≤6.0) OR (Steps < 5000)

Result: 6 / 10 students = 60% Low Energy

📚 Keypoints

Key takeaways:

• Organising data ensures accuracy
  
• Exploration builds understanding
  
• Patterns inspire hypotheses
  
• Predictions extend insights
  
• Uncertainty is inherent
  

🔦 Hints

Additional learning:

Submodule	Python Connection	Why It Matters
Organising Messy Data	pandas cleaning tools	Foundation of real analysis
Exploring Data	.describe(), .info(), .value_counts()	快速 understanding
Finding Patterns	groupby, pivot_table, clustering	Reveals relationships
Predicting Outcomes	simple models, sklearn basics	Intro to ML thinking