Tutorial 4: Intermediate Python for Data Science

Have you ever felt like your Python skills were holding you back from tackling more complex data science challenges? As I've grown in my role as director of course development at Dataquest, I've seen firsthand how aspiring data scientists who improve their Python skills can work with any dataset―no matter how messy―and extract actionable insights.

A recent experience really drove this point home for me. I was working on automating our course prerequisites system, a task that initially seemed overwhelming. But as I applied more advanced Python techniques, I watched the project transform from a complex puzzle into a manageable challenge. Cleaning our course metadata was essential―I wrote a function to standardize it, which greatly improved the accuracy of our AI model for generating prerequisites. By automating much of the process, I was able to reduce the time spent on manual data cleaning and focus on more strategic tasks.

In this tutorial, we'll explore intermediate Python skills and how you can apply them to your own data science projects. We'll look at techniques for efficient data analysis and automation, and discover how to leverage Python libraries for various applications, like AI. As we go through each topic, I'll share real-world examples from my work at Dataquest as well as examples from our Intermdeiate Python for Data Science course that uses a modified version of the Museum of Modern Art (MoMA) dataset, showing you how these skills can be applied in practice.

Each row of the MoMA dataset represents a unique piece of art and contains the following columns:

• Title: the title of the artwork
• Artist: the name of the artist who created the artwork
• Nationality: the nationality of the artist
• BeginDate: the year in which the artist was born
• EndDate: the year in which the artist died
• Gender: the gender of the artist
• Date: the date that the artwork was created
• Department: the department inside MoMA to which the artwork belongs

We'll start by exploring one of the most fundamental skills in data science: cleaning and preparing data in Python. You'll learn how to standardize and structure your data, making it ready for processing and analysis. With these skills, you'll be able to tackle more complex projects with confidence.

Lesson 1 – Cleaning and Preparing Data in Python

When I started working with real-world datasets, I quickly realized that data rarely comes in a neat, ready-to-analyze format. It's often messy, with inconsistencies, missing values, and formatting issues. The replace string method is a good first step in cleaning data:

Let's take a closer look at a common task you'll run into a lot with many datasets: cleaning date data. Dates can be tricky because they come in various formats. Here's a function we can use to clean and convert date strings into an integer using the replace method to delete unwanted characters:

def clean_and_convert(date):
    # check that we don't have an empty string
    if date != "":
        # move the rest of the function inside
        # the if statement
        date = date.replace("(", "")
        date = date.replace(")", "")
        date = int(date)
    return date

This function performs three important tasks:

• It checks if the date string is empty to avoid errors.
• It removes parentheses from the date string, which can interfere with conversion.
• It converts the cleaned string to an integer for easier manipulation.

I used a similar function when working on a project analyzing student enrollment data at Dataquest. I noticed that the dates were inconsistently formatted, with some years enclosed in parentheses and others not. This simple function saved us hours of manual data cleaning and allowed us to standardize our process.

Removing Multiple Characters

Another useful data cleaning technique I use all the time is removing multiple unwanted characters from strings. Here's a function that does just that using a list of bad characters and some test data:

bad_chars = ["(",")","c","C",".","s","'", " "]
test_data = ["1912", "1929", "1913-1923",
             "(1951)", "1994", "1934",
             "c. 1915", "1995", "c. 1912",
             "(1988)", "2002", "1957-1959",
             "c. 1955.", "c. 1970's", 
             "C. 1990-1999"]

def strip_characters(string):
    for char in bad_chars:
        string = string.replace(char,"")
    return string

stripped_test_data = []
for d in test_data:
    date = strip_characters(d)
    stripped_test_data.append(date)

This function iterates through the bad_chars list of unwanted characters and removes them from the test_data list. It's a particularly useful technique for dealing with data that contains multiple types of unwanted characters.

We've found these techniques useful in various scenarios at Dataquest. When analyzing feedback from our course surveys, we often need to clean text responses to remove special characters and standardize formatting. These simple yet powerful functions make that process much more efficient.

While we've focused on date and string cleaning here, these techniques can be applied to many other data cleaning tasks. You might use similar approaches to clean numerical data, standardize categorical variables, or even preprocess text for tasks like sentiment analysis.

Pro tip: always document your data cleaning steps. This not only helps you remember what you've done but also allows others to reproduce your work. At Dataquest, we often include comments in our cleaning functions explaining why certain steps are necessary, which has been incredibly helpful when revisiting projects months later.

By learning these data cleaning techniques, you'll be able to handle real-world datasets with confidence. Remember, clean data is the foundation of good analysis. As you practice these skills, you'll find that you can spend less time wrestling with data inconsistencies and more time uncovering valuable insights.

In the next lesson, we'll explore how to take your cleaned data and perform some basic analysis tasks in Python. But for now, I encourage you to try out these cleaning techniques on some messy datasets of your own. You might be surprised at how much cleaner and more manageable your data becomes!

Lesson 2 – Python Data Analysis Basics

Now that we've covered some data cleaning techniques, let's explore some essential Python data analysis skills that will help you work more efficiently with data and extract valuable insights.

When I first started working with data, I quickly realized the importance of having a solid grasp of Python's data exploration capabilities. One common challenge I faced was getting to know my data before jumping into any kind of analysis. Over time, I developed a set of techniques and strategies that streamlined that process.

As an example, let's take a look at a function we could use to get a summary of any artist in the MoMA dataset:

artist_freq = {}

for row in moma:
    artist = row[1]
    if artist not in artist_freq:
        artist_freq[artist] = 1
    else:
        artist_freq[artist] += 1

def artist_summary(artist):
    num_artworks = artist_freq[artist]
    template = "There are {num} artworks by {name} in the dataset"
    output = template.format(name=artist, num=num_artworks)
    print(output)

I've used similar functions many times when working with our course enrollment data at Dataquest. We often need to gain a little context on our data before proceeding with any analysis tasks. A version of this simple function saves us hours of manually exploring the data before attempting to extract insights from it.

Pro tip: Before forming an analysis on a dataset, consider creating a function like this to get to know your data. Not only will it help you gain context, but it'll often make you think of other kinds of analyses you should be performing.

Functions like this are a vital part of efficient data analysis in Python. They allow you to simplify common operations, making your code more readable and reusable. I'm always grateful for well-written functions when revisiting old analysis projects or sharing code with colleagues.

String Formatting

Another important aspect of data analysis is presenting your results clearly. Python's string formatting capabilities are incredibly useful for this. Here's an example of how I often format output:

num = 32.554865
print("I own {pct:.2f}% of the company".format(pct=num))

Here's a quick breakdown of how this code works:

When run, this code produces output like:

 I own 32.55% of the company

Using this kind of formatting makes your analysis results much easier to read and understand, especially when you're sharing them with non-technical team members. I remember when I first started using these techniques at Dataquest, our reports became much clearer and more accessible to the whole team.

Pro tip: When presenting your analysis results, think about your audience. If you're sharing with non-technical stakeholders, consider using string formatting to create clear, readable output. It can make a big difference in how your insights are received and understood.

These basic Python data analysis techniques can significantly improve your work. The artist_summary() function shows how you can better understand your data, while the string formatting example demonstrates how to present your results effectively.

As you continue to develop your Python skills, you'll find even more ways to enhance your data analysis capabilities. Remember, the key is to keep experimenting and finding ways to make your analysis more efficient and your results more clear. Whether you're cleaning data, creating reusable functions, or formatting your output, these intermediate Python skills will serve you well in your data science journey.

In the next lesson, we'll explore how object-oriented programming can help us create more scalable and maintainable data science solutions. This approach will allow us to handle more complex data structures and analyses.

Lesson 3 – Object-Oriented Python

I was initially skeptical about the relevance of Object-Oriented Programming (OOP) to data analysis. However, after applying OOP principles in my work at Dataquest, I realized its value in creating efficient, scalable, and maintainable data analysis solutions.

Organizing Code with Classes

OOP is all about organizing code into reusable structures called classes. These classes serve as blueprints for creating objects, which are instances of the class with their own unique data and behaviors. In simple terms, OOP is a way of writing code where we create objects that hold both data and the tools to work with that data. This approach is particularly useful when dealing with complex datasets or analysis pipelines.

Let's look at a simple example to illustrate these concepts:

class MyClass:

    def __init__(self, initial_data):
        self.data = initial_data

    def append(self, new_item):
        self.data = self.data + [new_item]

In this code, we define a class called MyClass. The __init__ method initializes new objects with some initial data. The append method allows us to add new items to our data.

Using the Class

Now, let's see how you can use this class:

my_list = MyClass([1, 2, 3, 4, 5])
print(my_list.data)

my_list.append(6)
print(my_list.data)

When you run this code, you'll get:

[1, 2, 3, 4, 5]
[1, 2, 3, 4, 5, 6]

This example demonstrates four key OOP concepts:

1. Classes: Think of these as blueprints. They define what our objects will look like and how they'll behave. (MyClass)
2. Instances: These are the actual objects we create from our class blueprints. (my_list)
3. Attributes: This is the data stored inside our class objects. (data)
4. Methods: These are functions that belong to our objects and can work with the object's data. (append)

Real-World Applications

In my work at Dataquest, I've found OOP particularly useful for managing complex data structures. For instance, when developing our course on AI conversation history, I created a Conversation class to handle storing, updating, and retrieving chat data. This made the code much more organized and easier to maintain compared to using separate functions and data structures.

Let me share another example of how we could use classes. If we wanted to analyze student engagement across different courses, we could take an OOP approach. Imagine if we had data on course completions, time spent on lessons, and quiz scores. Instead of handling these as separate datasets, we could create a Student class that encapsulates all this information. Here's a simplified version of what that might look like:

class Student:
    def __init__(self, student_id):
        self.student_id = student_id
        self.courses = {}

    def add_course(self, course_name, completion_status, time_spent, quiz_score):
        self.courses[course_name] = {
            'completion': completion_status,
            'time_spent': time_spent,
            'quiz_score': quiz_score
        }

    def get_average_quiz_score(self):
        scores = [course['quiz_score'] for course in self.courses.values()]
        return sum(scores) / len(scores) if scores else 0

This approach would allow us to easily manage and analyze data for each student across multiple courses. We could quickly calculate metrics like average quiz scores or total time spent across all courses.

The Power of OOP in Data Science

By using OOP, you can store related data and functionality, making your code more organized and efficient. You can also create custom data types tailored to your specific analysis needs, saving you time and reducing errors. This approach can significantly enhance your data science capabilities, allowing you to tackle more complex problems and create more robust solutions.

Getting Started with OOP

If you're new to OOP, here are a few tips to get started:

1. Start small: Begin by identifying a simple aspect of your data analysis that could benefit from being encapsulated in a class.
2. Focus on real-world entities: Classes often represent real-world concepts. In data science, this could be things like datasets, models, or analysis pipelines.
3. Use meaningful names: Choose class and method names that clearly describe their purpose.
4. Keep it simple: Don't try to put everything into one class. It's often better to have multiple smaller, focused classes than one large, complex one.

As you become more comfortable with OOP, you'll find it opens up new possibilities for structuring your data science projects. It's not just about writing code differently―it's about thinking about your data and analysis in a more structured, modular way. This approach can significantly enhance your data science capabilities, allowing you to tackle more complex problems and create more robust solutions.

In the next lesson, we'll explore how to work with dates and times in Python, another essential skill for many data analysis tasks. But keep OOP in mind as we move forward―you might be surprised at how often you can apply these concepts to make your code more efficient and easier to understand.

Lesson 4 – Working with Dates and Times in Python

When working with datasets, you'll frequently encounter dates and times that need to be manipulated and analyzed. I've found that these skills are essential for many data science tasks, from parsing timestamps to calculating time differences and formatting dates for output.

To work with dates and times in Python, we'll use the datetime module, which provides classes for manipulating dates and times. Although the diagram above shows a few different ways to import it, here's how you'd typically do it using the standard alias dt:

import datetime as dt

We often use an alias like dt to keep our code concise. With this module, you can create datetime objects that represent specific points in time:

event_date = dt.datetime(2023, 6, 15, 14, 30)

This creates a datetime object for June 15, 2023, at 2:30 PM. You can easily access different parts of this datetime object using its attributes:

print(event_date.year)  # 2023
print(event_date.month)  # 6
print(event_date.day)  # 15

Parsing Dates from Strings

One of the most common tasks you'll face is parsing dates from strings. The strptime function is particularly useful for this. Here's an example I've used when working with a dataset of White House visitors:

date_format = "%m/%d/%y %H:%M"
for row in potus:
    start_date = row[2]
    start_date = dt.datetime.strptime(start_date, date_format)
    row[2] = start_date

In this code, we're converting a string representation of a date into a datetime object. The "%m/%d/%y %H:%M" format string helps Python interpret the date string. This is incredibly useful when you're dealing with data from various sources, each potentially using different date formats. Here's

Just to show you how much easier it is to parse dates using strptime, here's how we'd create the same datetime object using standard Python string methods:

Formatting Dates

Once you have your dates in a workable format, you often need to output them in a specific way. The strftime method is great for this. It's like the opposite of strptime―instead of parsing strings into dates, it formats dates into strings. Here's an example:

formatted_date = start_date.strftime("%B %d, %Y")

This would convert a datetime object into a string like "January 15, 2023". Pretty neat, right?

Frequently, we need to analyze our course completion data, which involves working with start and end dates, calculating the difference, and then formatting the result in a readable way. In one project, we were trying to understand seasonal trends in our enrollment data. We had to parse thousands of timestamp strings, group them by month and year, and then calculate average daily sign-ups for each period. It was challenging, especially because we had to account for different time zones in our global user base. By using the datetime module effectively and being mindful of time zone differences, we were able to uncover some interesting patterns in our data.

We discovered that enrollment spikes occurred consistently in January and September, aligning with New Year's resolutions and the start of the academic year. This insight helped us tailor our marketing efforts and course offerings to these peak periods, resulting in an increase in new student sign-ups during these months.

Working with dates and times might seem challenging at first, but with practice, it becomes second nature. Don't be afraid to experiment with these functions and methods. The more you use them, the more comfortable you'll become, and the more insights you'll be able to extract from your temporal data.

In the final section of this tutorial, we'll put all these skills together in a guided project, analyzing posts from Hacker News. This will give you a chance to see how data cleaning, object-oriented programming, and date/time manipulation come together in a real-world data analysis scenario.

Guided Project: Exploring Hacker News Posts

Let's put our Python skills to the test with a real-world project. We're going to explore a dataset from Hacker News, a popular tech news site where users post and discuss all things technology.

I've found that hands-on projects are where the real learning happens. It's one thing to understand concepts, but it's another to apply them to messy, real-world data. At Dataquest, we've seen time and again how projects help our learners solidify their understanding and boost their confidence in using Python for data science tasks.

Our dataset is a CSV file with information about Hacker News posts, including titles, URLs, points, comment counts, and creation times. We're going to analyze this data to uncover insights about user engagement and posting patterns.

First, let's categorize the posts based on their titles. Hacker News has two special types of posts: "Ask HN" and "Show HN". Here's how we can separate these from the rest:

ask_posts = []
show_posts = []
other_posts = []

for row in hn:
    title = row[1]
    if title.lower().startswith('ask hn'):
        ask_posts.append(row)
    elif title.lower().startswith('show hn'):
        show_posts.append(row)
    else:
        other_posts.append(row)

print(len(ask_posts))
print(len(show_posts))
print(len(other_posts))

This code is simple but effective. It loops through each post, checks its title, and sorts it into the appropriate list. By printing the length of each list, we get a quick overview of how many posts fall into each category.

Timing Is Everything

Next, let's look at how the time of posting affects engagement. We'll focus on "Ask HN" posts and analyze the number of comments they receive based on the hour they were posted:

result_list = []

for row in ask_posts:
    created_at = row[6]
    num_comments = int(row[4])
    result_list.append([created_at, num_comments])

counts_by_hour = {}
comments_by_hour = {}

for row in result_list:
    date = row[0]
    comment = row[1]
    hour = dt.datetime.strptime(date, '%m/%d/%Y %H:%M').strftime('%H')
    if hour not in counts_by_hour:
        counts_by_hour[hour] = 1
        comments_by_hour[hour] = comment
    else:
        counts_by_hour[hour] += 1
        comments_by_hour[hour] += comment

This code does a few things: it creates a list of "Ask HN" posts with their creation times and comment counts, loops through this list to extract the hour from each post's creation time, and finally counts the number of posts and comments for each hour of the day.

From this analysis, we might find that posts created at certain times tend to get more comments. For example, we could discover that posts made in the evening hours receive more engagement, possibly because more users are online after work.

I recall a similar analysis we did on our Dataquest course data. We found that our learners were most active during weekday evenings. This insight was incredibly valuable―it helped us optimize when to release new content and when to schedule our support hours. It's a great example of how data analysis can directly inform business decisions.

Project Conclusions

The techniques we're using here have wide-ranging applications. If you're a content creator, understanding the best times to post can significantly boost your engagement. If you're in marketing, this kind of analysis can help you time your campaigns for maximum impact.

Here are a few tips for when you're working on your own data projects:

1. Start by exploring your data. Print out a few rows, check for missing values, and understand what each column represents.
2. Break your analysis down into steps. Start with simple questions and build up to more complex ones.
3. Don't be afraid to iterate. Your first attempt at analysis might reveal new questions or approaches.
4. Visualize your results. Sometimes a simple graph can reveal patterns that aren't obvious in the raw numbers.

I encourage you to take this project further. Try analyzing the "Show HN" posts. Look for correlations between comments and points. The more you explore, the more comfortable you'll become with these techniques.

Remember, becoming proficient in data science is all about practice and curiosity. Keep asking questions of your data, and you'll be surprised at the insights you can uncover. Happy analyzing!

Advice from a Python Expert

As I look back on my journey with intermediate Python, I'm struck by the vast possibilities it offers in data science. I started by learning new concepts―advanced data cleaning, object-oriented programming, and more―and watched my analytical toolkit grow. Now, as I work with students at Dataquest, I've seen how these skills help them tackle complex datasets and streamline tedious tasks.

We've covered a range of topics in this piece, from data cleaning and analysis basics to object-oriented programming and working with dates and times. The Hacker News project put these skills to the test, showing how they can be applied in a real-world scenario. I've seen firsthand how mastering these areas can significantly boost efficiency and problem-solving abilities in data science roles.

If you're motivated to improve your Python skills, here's my advice: practice consistently and apply these concepts to your own projects. Don't be discouraged by errors―they're valuable learning opportunities. Keep experimenting with your data, and you'll be surprised by the insights you'll uncover. As you work through challenges, remember that persistence and curiosity are key to becoming proficient in data science.

As you continue your Python journey, you'll find that each new skill you acquire opens up new possibilities in data science. You might find yourself cleaning messy datasets with ease, building complex models using object-oriented principles, or uncovering hidden patterns in time-series data. Whatever path you choose, celebrate your progress, and enjoy the process of learning.

If you're looking to further develop these skills in a structured environment, you might find our Intermediate Python for Data Science course helpful. It's designed to build on the basics and guide you through applying these concepts to real-world scenarios.

Remember, even the most complex data analysis starts with a single line of Python code. You've got this! Keep learning, keep practicing, and most importantly, keep applying these skills to projects you're passionate about. You'll be amazed at the data science challenges you can overcome.

Frequently Asked Questions

bad_chars = ["(",")","c","C",".","s","'", " "]
def strip_characters(string):
    for char in bad_chars:
        string = string.replace(char,"")
    return string

date_format = "%m/%d/%y %H:%M"
start_date = dt.datetime.strptime(start_date, date_format)

formatted_date = start_date.strftime("%B %d, %Y")

bad_chars = ["(",")","c","C",".","s","'", " "]
def strip_characters(string):
    for char in bad_chars:
        string = string.replace(char,"")
    return string

num = 32.554865
print("I own {pct:.2f}% of the company".format(pct=num))

I own 32.55% of the company

class Student:
    def __init__(self, student_id):
        self.student_id = student_id
        self.courses = {}

    def add_course(self, course_name, completion_status, time_spent, quiz_score):
        self.courses[course_name] = {
            'completion': completion_status,
            'time_spent': time_spent,
            'quiz_score': quiz_score
        }

    def get_average_quiz_score(self):
        scores = [course['quiz_score'] for course in self.courses.values()]
        return sum(scores) / len(scores) if scores else 0

date_format = "%m/%d/%y %H:%M"
start_date = dt.datetime.strptime(start_date, date_format)

formatted_date = start_date.strftime("%B %d, %Y")