Web scrapping in Python is s the practice of gathering data from the internet from other means apart from the traditional web browser.
In today’s data-driven world, web scrapping has become an essential skill for researchers, analysts, and developers.
It is widely used in machine learning, business forecasting, medical reporting and diagnostics and so on.
With the emergence of data-driven decision-making, web scraping has emerged as a powerful technique for extracting valuable information from websites.
Basically, web scrapping involves fetching HTML content of web pages, parsing the data, and extracting the desired information.
For instance, you can gather structured data, such as product details, pricing information, news articles, and more, for analysis, research, or business intelligence purposes.
Interestingly, Python provides a wide range of libraries, such as Beautiful Soup and Scrapy, which make web scrapping easier and more efficient.
In this article, we will learn web scraping in Python using the BeautifulSoup.
Beautiful Soup Library
Beautiful Soup is a popular Python library for web scrapping.
It provides a convenient way to extract data from HTML and XML documents, making it easier for developers to parse and navigate through the structure of web pages.
It provides an intuitive interface to navigate and search the parsed data, allowing you to extract specific elements easily.
BeautifulSoup is simple and intuitive, making it a suitable option for beginners.
Installing Required Libraries
To get started with BeautifulSoup, you need to install the BeautifulSoup.
For the sake of this tutorial, you will also need to install the Request library.
Open your command prompt or terminal and run the following commands:
pip install beautifulsoup4 pip install requests
Having installed these libraries, you are set for web scrapping.
Take a look at this example:
import requests
from bs4 import BeautifulSoup
import csv
url = 'https://wikipedia.org'
response = requests.get(url)
soup = BeautifulSoup(response.content, "html.parser")
links = soup.find_all("a")
for link in links:
print(link["href"])
with open("data.csv", "w", newline="") as csvfile:
writer = csv.writer(csvfile)
for link in links:
writer.writerow([link["href"]])
The example above fetches all the links on the Wikipedia home page.
The requests library is for making HTTP requests to the target website, while the bs4 module from BeautifulSoup is for parsing and navigating the HTML or XML content.
A BeautifulSoup object was created by passing the HTML content and a parser (in this case “html.parser”) to the BeautifulSoup constructor.
The find_all() method and attributes of the BeautifulSoup are used to locate the specific elements or data to be extracted from the HTML.
Finally, the csv module was used to store the extracted data into a csv file.
However, before delving deeper into scrapping, it is important that you understand the workflow for web scrapping.
Basic Web Scraping Workflow with BeautifulSoup
The basic workflow for web scraping using BeautifulSoup involves a few key steps:
1. Inspect the website’s HTML structure
Use the browser’s developer tools to analyze the HTML structure of the web page you want to scrape.
This step helps identify the relevant data and its location within the HTML document.
2. Fetch the web page
Utilize the Requests library to send an HTTP request to the website and retrieve the HTML content.
3. Parse the HTML
Use Beautiful Soup to parse the HTML content into a structured representation.
This makes the traversal and extraction of data easy.
4. Extract the desired data
Employ Beautiful Soup’s powerful methods to navigate and locate the specific data elements you want to scrape.
5. Store or process the data
Once you’ve extracted the data, you can save it to a file, store it in a database, or process it further for analysis or visualization.
Commonly used BeautifulSoup Methods and attributes
Beautiful Soup provides several methods and attributes that you can use to navigate, search, and extract data from HTML or XML documents.
Here are some commonly used methods:
find() method
This method searches for the first occurrence of a specified tag or a combination of tags, attributes, or text.
It returns a single BeautifulSoup object.
soup.find("h1") # Find the first occurrence of a h1 tag
soup.find("h2", id="main") # Find a h2 tag with an id="main"
soup.find(text="text to locate") # Find a specific text within the document
find_all() method
This method searches for all occurrences of a specified tag or a combination of tags, attributes, or text.
It returns a list of BeautifulSoup objects.
soup.find_all("h2") # Find all occurrences of a h2
soup.find_all("h3", id="main") # Find all h3 tags with id of main
soup.find_all(text="text to locate") # Find all occurrences of a specific text
select() method
This method allows you to use CSS selectors to search for elements in the document.
It returns a list of BeautifulSoup objects.
# Select all elements with class "my-class"
elements = soup.select(".my-class")
# Select the element with ID "my-id"
element = soup.select("#my-id")
# Select all <a> tags within a <div> element with class "container"
links = soup.select("div.container a")
get() method
This method retrieves the value of a specific attribute from a tag.
link = soup.find("a")
href_value = link.get("href")
text attribute
The text attribute is used to retrieve the text content within a tag, excluding any HTML tags or markup.
tags = soup.find_all("a")
for tag in tags:
tag_text = tag.text
print(tag_text)
Navitaging Trees
Tree navigation is employed if you want to find a tag based on the location in the document.
Here, you have to deal with children, parents, siblings and descendants.
Children
Children are exactly one tag below the parents, while descendants can be at any level below the parents.
for child in bs.find_all(id="main").children:
print(child)
Legal and Ethical Considerations
When engaging in web scraping, it’s essential to consider the legal and ethical implications.
Here are some key considerations:
Legality
Make sure that your web scraping activities comply with relevant laws, regulations, and terms of service of the target website.
Some websites explicitly prohibit web scraping in their terms of service.
Therefore, review the website’s robots.txt file, which provides guidelines on what content is allowed to be scraped.
Additionally, be aware of any applicable data protection or privacy laws, especially when handling personal or sensitive data.
Rate Limiting and Respectful Crawling
Implement procedures to ensure that the target website’s server load and bandwidth are not exceeded.
This involves introducing delays between consecutive requests to prevent server overload.
Avoid aggressive or disruptive crawling techniques that may cause harm or inconvenience to the website or its users.
Attribution and Copyright
Respect intellectual property rights by giving appropriate attribution when using or republishing scraped content.
Ensure that you comply with copyright laws and do not infringe on the rights of content creators.
Obtain permission when necessary, respect copyright restrictions, and avoid scraping private or sensitive data without consent.
Personal Data and Privacy
Be cautious when handling personal data obtained through web scraping.
Take necessary measures to protect user privacy and follow applicable data protection regulations.
Anonymize or aggregate data whenever possible to avoid disclosing personally identifiable information.
Data quality and validation
Scrutinize the scraped data for inconsistencies or errors.
Validate and clean the data to ensure its accuracy before further analysis or integration.
Conclusion
Web scraping using BeautifulSoup is a valuable skill for extracting data from websites efficiently.
Python, combined with BeautifulSoup’s simplicity and powerful parsing capabilities, provides an excellent environment for web scraping projects.
Remember to always scrape responsibly, respecting website policies and ensuring your actions align with legal and ethical standards.
With continued practice and exploration, you’ll become adept at harnessing the power of web scraping to uncover valuable insights and automate repetitive tasks.