Analyzing Wikipedia Pages
- Guided Project
- 0.4 hours
- Advanced
- Python
Practice using MapReduce in Python to efficiently analyze a large Wikipedia dataset and build in-demand data skills.
Overview
In this project, you'll take on the role of a data analyst and process over 54 MB of Wikipedia articles to find specific text matches. Using Python and MapReduce, you'll build a parallel solution to efficiently search the dataset and return match details.
You'll develop a simplified version of the grep command-line utility to find strings across multiple files. Through hands-on practice with text processing, parallel computing, and data engineering, you'll gain valuable skills in analyzing large unstructured datasets.
Objective: Efficiently search a large text dataset using MapReduce and Python to find specific strings and build valuable big data skills.
What You'll Learn
- ✓ Analyze Wikipedia pages using MapReduce computing
Before You Start
- ✓ Starting multiple processes in Python to parallelize analysis of Wikipedia pages
- ✓ Running functions on several processes simultaneously to efficiently process Wikipedia articles
- ✓ Sharing data between multiple processes for coordinated analysis of Wikipedia content
- ✓ Implementing the MapReduce framework to distribute processing of Wikipedia pages
Project Steps
8 steps
- 1 Introducing Wikipedia Data
- 2 Adding the MapReduce Framework
- 3 Grep Exact Match
- 4 Grep Case Insensitive
- 5 Checking the Implementation
- 6 Finding Match Positions on Lines
- 7 Displaying the Results
- 8 Next Steps
Join 1M+ data learners on Dataquest.
- 1
Create a free account
- 2
Choose a learning path
- 3
Complete exercises and projects
- 4
Advance your career