Messy datasets are everywhere. If you want to analyze data, it’s inevitable that you will need to clean data. In this tutorial, we're going to take a look at how to do that using R and some nifty tidyverse tools.
The tidyverse tools provide powerful methods to diagnose and clean messy datasets in R. While there's far more we can do with the tidyverse, in this tutorial we'll focus on learning how to:
- Import comma-separated values (CSV) and Microsoft Excel flat files into R
- Combine data frames
- Clean up column names
- And more!
The tidyverse is a collection of R packages designed for working with data. The tidyverse packages share a common design philosophy, grammar, and data structures. Tidyverse packages “play well together”. The tidyverse enables you to spend less time cleaning data so that you can focus more on analyzing, visualizing, and modeling data.
1. Characteristics of Clean Data and Messy Data
What exactly is clean data? Clean data is accurate, complete, and in a format that is ready to analyze. Characteristics of clean data include data that are:
- Free of duplicate rows/values
- Error-free (e.g. free of misspellings)
- Relevant (e.g. free of special characters)
- The appropriate data type for analysis
- Free of outliers (or only contain outliers have been identified/understood), and
- Follows a “tidy data” structure
Common symptoms of messy data include data that contain:
- Special characters (e.g. commas in numeric values)
- Numeric values stored as text/character data types
- Duplicate rows
- White space
- Missing data
- Zeros instead of null values
In this blog post, we will work with five property-sales datasets that are publicly available on the New York City Department of Finance Rolling Sales Data website. We encourage you to download the datasets and follow along! Each file contains one year of real estate sales data for one of New York City’s five boroughs. We will work with the following Microsoft Excel files:
As we work through this blog post, imagine that you are helping a friend launch their home-inspection business in New York City. You offer to help them by analyzing the data to better understand the real-estate market. But you realize that before you can analyze the data in R, you will need to diagnose and clean it first. And before you can diagnose the data, you will need to load it into R!
3. Load Data into R with readxl
Benefits of using tidyverse tools are often evident in the data-loading process. In many cases, the tidyverse package
readxl will clean some data for you as Microsoft Excel data is loaded into R. If you are working with CSV data, the tidyverse
readr package function
read_csv() is the function to use (we’ll cover that later).
Let’s look at an example. Here’s how the Excel file for the Brooklyn borough looks:
The Brooklyn Excel file
Now let’s load the Brooklyn dataset into R from an Excel file. We’ll use the
readxl package. We specify the function argument
skip = 4 because the row that we want to use as the header (i.e. column names) is actually row 5. We can ignore the first four rows entirely and load the data into R beginning at row 5. Here’s the code:
library(readxl) # Load Excel files brooklyn <- read_excel("rollingsales_brooklyn.xls", skip = 4)
Note we saved this dataset with the variable name
brooklyn for future use.
4. View the Data with tidyr::glimpse()
The tidyverse offers a user-friendly way to view this data with the
glimpse() function that is part of the
tibble package. To use this package, we will need to load it for use in our current session. But rather than loading this package alone, we can load many of the tidyverse packages at one time. If you do not have the tidyverse collection of packages, install it on your machine using the following command in your R or R Studio session:
Once the package is installed, load it to memory:
tidyverse is loaded into memory, take a “glimpse” of the Brooklyn dataset:
## Observations: 20,185
## Variables: 21
## $ BOROUGH <chr> "3", "3", "3", "3", "3", "3", "…
## $ NEIGHBORHOOD <chr> "BATH BEACH", "BATH BEACH", "BA…
## $ `BUILDING CLASS CATEGORY` <chr> "01 ONE FAMILY DWELLINGS", "01 …
## $ `TAX CLASS AT PRESENT` <chr> "1", "1", "1", "1", "1", "1", "…
## $ BLOCK <dbl> 6359, 6360, 6364, 6367, 6371, 6…
## $ LOT <dbl> 70, 48, 74, 24, 19, 32, 65, 20,…
## $ `EASE-MENT` <lgl> NA, NA, NA, NA, NA, NA, NA, NA,…
## $ `BUILDING CLASS AT PRESENT` <chr> "S1", "A5", "A5", "A9", "A9", "…
## $ ADDRESS <chr> "8684 15TH AVENUE", "14 BAY 10T…
## $ `APARTMENT NUMBER` <chr> NA, NA, NA, NA, NA, NA, NA, NA,…
## $ `ZIP CODE` <dbl> 11228, 11228, 11214, 11214, 112…
## $ `RESIDENTIAL UNITS` <dbl> 1, 1, 1, 1, 1, 1, 1, 1, 2, 1, 1…
## $ `COMMERCIAL UNITS` <dbl> 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0…
## $ `TOTAL UNITS` <dbl> 2, 1, 1, 1, 1, 1, 1, 1, 2, 1, 1…
## $ `LAND SQUARE FEET` <dbl> 1933, 2513, 2492, 1571, 2320, 3…
## $ `GROSS SQUARE FEET` <dbl> 4080, 1428, 972, 1456, 1566, 22…
## $ `YEAR BUILT` <dbl> 1930, 1930, 1950, 1935, 1930, 1…
## $ `TAX CLASS AT TIME OF SALE` <chr> "1", "1", "1", "1", "1", "1", "…
## $ `BUILDING CLASS AT TIME OF SALE` <chr> "S1", "A5", "A5", "A9", "A9", "…
## $ `SALE PRICE` <dbl> 1300000, 849000, 0, 830000, 0, …
## $ `SALE DATE` <dttm> 2020-04-28, 2020-03-18, 2019-0…
glimpse() function provides a user-friendly way to view the column names and data types for all columns, or variables, in the data frame. With this function, we are also able to view the first few observations in the data frame. This data frame has 20,185 observations, or property sales records. And there are 21 variables, or columns.
5. Data Types
Looking at the data types for each column, we see that, in general, the data is stored in a format that is ready to use! For example:
NEIGHBORHOODis “character” type, also known as a string.
GROSS SQUARE FEET(i.e. the size of the property) is of type “double”, which part of the “numeric” class in R.
SALE PRICEis also numeric.
SALE DATEis stored in a format that represents calendar dates and times.
So why does this matter? Because
GROSS SQUARE FEET and
SALE PRICE are numeric, we can perform arithmetic operations on the data right away. For example, we can calculate the average sale price for all properties:
##  1098644
6. Ready to Plot!
It’s useful that
SALE DATE is stored in a format that represents calendar dates and times because this enables us to use a single line of code to make a histogram of property sales by date:
qplot(`SALE DATE`, data = brooklyn)