WhoisXML API Blog

Python Script for Transforming Domain Names from First Watch Malicious Domains Data Feed into STIX 2.1

WhoisXML API recently created a Python script to help users of First Watch Malicious Domains Data Feed, also known as “First Watch,” transform predictive domain intelligence into a machine-readable format—STIX 2.1—for automated data processing.

The script reads domain names from First Watch files, converts them to STIX 2.1 Indicator objects with domain-name observable types, and generates a TAXII 2.1-compatible STIX bundle. The output is downloadable as a JSON file in TAXII format.

Managing a Passive DNS Database Using PostgreSQL

Abstract

This document outlines the setup of a PostgreSQL database on Ubuntu Linux to efficiently manage and query WHOISXMLAPI’s Premium DNS database. Designed to store and analyze billions of DNS records, this database will handle large-scale data ingestion, facilitate rapid data retrieval, and support extensive analytical operations. PostgreSQL's robust performance, scalability, and support for advanced indexing make it ideal for managing DNS data, while its compatibility with open-source tools provides a flexible environment for future scaling and data processing.

Managing a Passive DNS Database Using Apache Cassandra

Abstract

Apache Cassandra is a highly scalable, distributed NoSQL database designed for handling massive volumes of data across many commodity servers without a single point of failure. Its decentralized nature and robust architecture make it particularly well-suited for applications that require high availability, fault tolerance, and horizontal scalability. Cassandra is engineered to handle very large datasets, supporting billions of records with ease, making it an ideal choice for organizations dealing with large-scale, real-time applications such as time-series data, IoT data, and customer logs. Through its use of a partitioned architecture and the ability to add nodes seamlessly as data grows, Cassandra offers an efficient means of managing big data with low latency and high throughput.

Importing and Indexing First Watch Malicious Domains Data Feed into MySQL

Abstract

This white paper provides a comprehensive guide for importing First Watch Malicious Domain data from a CSV file into a MySQL database and indexing it on the `domainName` column. The steps outlined here cover the creation of a database, the table structure design, and data import using MySQL. By following this approach, users can efficiently handle domain-related datasets for querying and analysis purposes.

Introducing Server-to-Server OAuth to Secure API Integrations

We’re thrilled to announce the availability of Server-to-Server OAuth for all our API users. Using this authentication method helps developers build robust integrations without compromising security and efficiency.

Server-to-Server OAuth or two-legged OAuth eliminates user interaction for authentication, making it ideal for automated workflows. It simplifies API integration by allowing a server to directly request and manage access tokens. The authorization method is also designed to handle high-volume API requests, supporting application scalability.

WhoisXML API Tech Announcement Service Is Now Available for Data Feed Users

We’re excited to announce that WhoisXML API now offers a tech announcement service page to help customers keep track of events, such as the release of new or revised data feed files. The service aims to provide users access to relevant product updates covering events related to Newly Registered Domains, the Typosquatting Feed, the Early DGA Detection Feed, the Early Warning Phishing Feed, and the Quarterly WHOIS Database.

The tech announcement service is available on two channels. The RSS feed can be accessed at https://www.whoisxmlapi.com/tech_announce/announce.rss, while the JSON list is available at https://www.whoisxmlapi.com/tech_announce/announce.json.

Scripts for Building and Querying Interval Trees with IP Geolocation Database

WhoisXML API now offers scripts for IP Geolocation Database users to easily create and use interval trees for efficient IPv4 and IPv6 geolocation data searches, including a function to check if an IP address falls within any GeoIP range.

WHOIS API Now Supports POST Requests

We're thrilled to announce that WHOIS queries can now be processed through POST requests in addition to the current HTTP GET request method. This new capability offers a more secure way to access domain registration information.

Using POST requests significantly reduces the risk of unauthorized access to API keys, as they encrypt the HTTP message body, including the API key. To illustrate, here is an example of a POST request.