It goes without saying that we live in a world where data is more valuable than oil, we like to think that most of this data is locked in the secure servers of the many private companies that collect it. Anyone who hasn’t been sleeping for the past decade also…

Recently, I completed a small project that required me to make suggestions for optimizing an elevator configuration within a theoretical high-rise in New York City. The building is set up as follows. Entrants to the building must first swipe badges through a security system, then they can push the elevator…

How to save Python objects persistently using the built-in shelve module — I’ve had my share of experience writing classes and utilizing OOP in Python, but I have only developed objects that are used in the current running script and saved in the cache. Those cached objects are not available when you run your program in the future. My first thought when…

Open-source Intelligence, or OSINT, consists of using publicly available information to collect data on subject. Used primarily by investigators, law-enforcement, and penetration testers, this information could include anything from public records to social media data. …

Sometimes, it’s necessary to add some visual output to Python scripts in order to provide users with feedback as to the progress of a program. In this article, we are going to touch on two ways you can incorporate this feedback: progress bars and ANSI color codes. Using these two…

I recently was involved in a project that involved the creation of an automated tool that collects a large amount of documents. In order to save these documents, it was necessary to make create files and folders on the local filesystem where the rest of the script could access the…

I recently was involved with a project that required parsing of a PDF in order to identify the regions of page and return the text from those regions. The text regions would then be fed to a Q/A model (farm-haystack), and return extracted data from the PDF. Essentially, we wanted…

I’m going to walk through how to develop a basic spider using the Scrapy package. The purpose of this tutorial will be to introduce the basic concepts needed to understand and use the Scrapy package, and give you the tools to develop custom spiders for your data collection needs. …

Similar to regular expressions, Xpath can be thought of as a language for finding information in and XML/HTML document. It has many uses, but personally I use it most for developing web crawlers and grabbing information from websites. We’re going to go over the basics of the language, and how…