URL categorization generally refers to using machine learning techniques to automatically classify the content of a website into one or more categories.

Use Cases - Cybersecurity

URL categorization has many use cases from a wide range of fields. One important application of website categorization is cybersecurity, where we classify websites into potential spam, phishing and other kinds of "warning" websites that we do not want to be visited by users.

Consumer facing products such as web browsers already use website categorization as they have built-in filters that block phishing, malware and adult content websites. In this case we are interested in identifying the type of website rather than just blocking them based on their URL.

URL categorization can also be used to improve how you organize your bookmarks, and even to decide which websites you want to visit in the first place!

Use cases - Marketing

A lot of business partners and other companies get to know each other because they have a mutual interest in the same product or service. However, some companies are so large that they divide their interests into categories or “verticals,” which are essentially specialized sections of the company that deal with specific products or services.

Website categorization is one of the most important tools for companies to know what verticals their business partners fall into. Not only does it help them to identify what area of the company holds a partnership, but it also helps them to understand how to market their product or service to be more appealing to a wider audience. Website categorization can also provide information about demographics which helps advertisers determine where ads should be placed on websites that have certain audiences.

Taxonomies for URL categorization

Been wondering how to classify your website for advertising purposes?

The solution is simple: the IAB .

The Internet Advertising Bureau is a nonprofit organization that's at the forefront of the digital advertising industry. They're the ones who created and continue to develop the taxonomy used by companies to categorize websites for their ad platforms, which means you can use IAB's classifications to tell those companies what kind of site you have.

IAB has updated their categories over the years, so make sure you're using their most recent version---that way, you'll know you're getting an accurate description of your site.

Benefits of URL categorization for Online Stores

As an online store owner, you are probably aware of the importance of accurate, well-organized product categorization. But did you know that proper categorization can bring your store many benefits?

An important one is improving user experience. Well-organized and categorized products not only help your customers find what they're looking for with ease, they also make them more likely to buy!

But did you know that ecommerce site categorization can also improve your SEO? When used to group products by their categories, it allows you to generate more subpages for indexing in search engines. This means you'll get more visits from them.

The added specificity and content relevant keyword brought on by proper categorization also leads to better signaling for search engine ranking algorithms. So not only will you have a happier customer base and more traffic overall, but you'll also start appearing higher in search engine results pages.

Machine learning models for URL Categorization

Automated URL categorization is the process of labeling websites using a predetermined taxonomy. It’s usually done using a supervised machine learning model (ML) developed specifically for this purpose.

When starting to work on your solution, however, take a step back and think about what you're doing first.

First, how are you going to train your model? What data set will you use? How big will it be? The bigger the data set, the more accurate your model will be. However, with larger data sets also comes greater complexity in preparing them for training.

Second, you should think about which taxonomy you would like to use when classifying websites. There are various options available such as IAB (Interactive Advertising Bureau), Google or Facebook taxonomies but if none of those match your use case exactly you can also go and prepare your own, e.g. by buying the data sets or scraping the data on your own.

Text pre-processing and content extraction

Before you can build a URL categorization model, you have to extract the text from a web page—but that's easier said than done.

Most websites are made of two parts: the content (the main article or blog post), and the supplementary parts like menus, footers, and sidebars. In most cases, we don't want the latter to be part of our text used in website categorization.

There are specially built content extractors that can help you in this regard. We have used the following python libraries in the past:

• goose3 (https://github.com/goose3/goose3)
• newspaper (https://github.com/codelucas/newspaper)

Conclusion

URL categorization of domains and full-path URLs is an important field of AI and relevant for many applications in cybersecurity, marketing and other areas.

URL categorization is also usually done in automated way, using a machine learning model.

Special care should be given to proper content extraction from websites when deploying a website categorization solution in production.