Let’s finish up the function: if not is_valid(href): # not a valid URL continue if href in internal_urls: # already in the set continue if domain_name not in href: # external link if href not in external_urls: print(f"") links = get_all_website_links(url) for link in links: if total_urls_visited > max_urls: break crawl(link, max_urls=max_urls) href = parsed_href.scheme + "://" + parsed_loc + parsed_href.path
#Http url extractor code
Now we need to remove HTTP GET parameters from the URLs, since this will cause redundancy in the set, the below code handles that: parsed_href = urlparse(href) # remove URL GET parameters, URL fragments, etc. Since not all links are absolute, we gonna need to join relative URLs with their domain name (e.g when href is “/search” and URL is “”, the result will be “/search”): # join the URL if it's relative (not absolute link) href = urljoin(url, href) Otherwise, we just continue to the next link. So we get the href attribute and check if there is something there. Let’s get all HTML tags (anchor tags that contains all the links to the web page): for a_tag in soup.findAll("a"): href = a_("href") if href = "" or href is None: # href empty tag continue Third, I’ve downloaded the HTML content of the web page and wrapped it with a soup object to ease HTML parsing. Second, I’ve extracted the domain name from the URL, we gonna need it to check whether the link we grabbed is external or internal. Now let’s build a function to return all the valid URLs of a web page: def get_all_website_links(url): """ Returns all URLs that is found on `url` in which it belongs to the same website """ # all URLs of `url` urls = set() # domain name of the URL without the protocol domain_name = urlparse(url).netloc soup = BeautifulSoup(requests.get(url).content, "html.parser")įirst, I initialized the URLs set variable, I’ve used Python sets here because we don’t want redundant links. This will make sure that a proper scheme (protocol, e.g HTTP or HTTPS) and domain name exist in the URL. """ parsed = urlparse(url) return bool(loc) and bool(parsed.scheme) Since not all links in anchor tags (tags) are valid (I’ve experimented with this), some are links to parts of the website, and some are javascript, so let’s write a function to validate URLs: def is_valid(url): """ Checks whether `url` is a valid URL. External links are URLs that link to other websites.Internal links are URLs that link to other pages of the same website.We gonna need two global variables, one for all internal links of the website and the other for all the external links: # initialize the set of links (unique links) internal_urls = set() external_urls = set() We are going to use colorama just for using different colors when printing, to distinguish between internal and external links: # init the colorama module colorama.init() GREEN = GRAY = _EX RESET = YELLOW = Open up a new Python file and follow along, let’s import the modules we need: import requests from urllib.parse import urlparse, urljoin from bs4 import BeautifulSoup import colorama We’ll be using requests to make HTTP requests conveniently, BeautifulSoup for parsing HTML, and colorama for changing text color.
#Http url extractor install
Let’s install the dependencies: pip3 install requests bs4 colorama In this tutorial, you will learn how you can build a link extractor tool in Python from Scratch using only requests and BeautifulSoup libraries. Extracting all links of a web page is a common task among web scrapers, it is useful to build advanced scrapers that crawl every page of a certain website to extract data, it can also be used for SEO diagnostics process or even information gathering phase for penetration testers.
0 kommentar(er)
