Day 98 of #100daysofnetworks
How Curiosity Led Me from Web Development to OSINT
Hello everyone. I want to do another free article today. I currently spend a lot of time doing OSINT work, but my origins come from software engineering, data, and cybersecurity. I have felt a bit unusual ever since learning about and then obsessively learning network science and OSINT, so tonight I did a bit of remembering, thinking back to when exactly I transformed from a Data Engineer into a Platform Engineer and then into an OSINT Platform Architect.
I thought that by writing this I might encourage some of you to follow your own curiosity as I did. I am very happy with this career progression. I think it’s really cool and also practical.
My Career Progression
I started coding at age six, so when did my career really begin… Wrote my first script at age 6, ran a server at 12, learned about databases at 16. At age 20, I decided that I wanted to be a web developer, not a web designer. This is important nuance. In the early 2000s, web development was cutting edge and exciting, like AI engineering is today. So, I’ll start this progression at Web Developer even though I was a coder for 14 years before that. I’ll call it Software Engineer, to be fair, because I was.
Software Engineering and Web Development
SQL Development and Database Administration
Data Operations and Site Reliability Engineering
Data Engineering
AI and Machine Learning Platform Engineering + OSINT Analysis (Simultaneously)
OSINT Platform Engineering
OSINT AI Architecture
My career has been rich and exciting. I feel like I have lived so many different lives and enjoyed each one. I am now in the OSINT AI Architecture phase of my life and career. What does that mean? It means that I build World AI. I build intelligence that is able to give insight into things happening in the world in real-time.
I seem a bit unusual because I forked off at that third-to-last stage. I have done the learning and the work. As I have spent a lot of time learning and building things, I am excited to provide training!
Enrollment is Now Open!
I am offering the following courses, and enrollment is NOW OPEN!
OSINT for Everyone (Expected beginning 6/30)
GraphRAG from Scratch (Expected beginning 6/23)
You can enroll here! I’m excited to offer these!
My previous article also has more information.
Ok, let’s continue. I hope to see some of you in class.
Curiosity was my guide
When people have asked me how to do things or how to learn things, I often tell people to follow their own natural curiosity. Learn to listen to it. Curiosity is my guide. From age six, all I wanted to know was what I could do with a computer. I didn’t want to disappear into a computer world. I wanted to use a computer to interact with my physical world.
Now, as an adult, nothing has changed. The computer isn’t the star attraction to me, nor is AI. They are my tools for understanding reality.
ChatGPT just drew this picture to kinda stitch together my career in one image… and it really doesn’t look too different from my desk, other than that I have a whole bunch of GrooveSeeker stickers, fancy rocks and minerals, and a bag of cotton candy instead of books… Anyway, let me describe this progression of how I fell into OSINT. I think a lot of you would enjoy learning OSINT.
Software Engineering and Web Development
I learned to write code at age six. I learned servers, databases, and a bit of cybersecurity before turning twenty. When I was in high school, the world wide web went public, and I was immediately obsessed. I mean OBSESSED. My whole life became web development, running online forums, creating e-commerce sites, all of that. I was a web developer for twelve years, which informs my AI and OSINT skills. Think about hyperlinks. :)
In this phase of my life, I built a lot of websites and communities, and this made me very popular. Those were fun days. The original GrooveSeeker was created during these times.
SQL Development and Database Administration
Web developers are software engineers. In the process of creating dozens of web applications, I also had to create dozens of different kinds of databases to support those applications. I have worked on databases for 32 years. At first, I spent my time learning as much about SQL as I could. I wanted to find answers. I jumped into databases because I wanted to UNDERSTAND things. I don’t care about databases as databases. I care about the reason they exist. I followed my curiosity from software engineering to web development to becoming obsessed with SQL.
After enough time working on SQL, you will inevitably have to troubleshoot slow SQL code, and do other annoying things. We don’t get to select * our way to a lifetime of bliss. After learning anything for more than a couple weeks, reality sets in. But the headaches of reality are where you get good at operations work. So, my SQL Development work led me to getting good at working with databases themselves, and I worked as a Database Administrator (DBA) several times.
Data Operations and Site Reliability Engineering
Cool, so, now I can code, I can get data, and I can keep those databases running. What’s next? Data Operations and Site Reliability Engineering! I loved working in Data Operations so much. I used to say that my job was to dissect ancient undocumented orphaned production servers, but that was only half of the job. After reverse-engineering how this unknown stuff worked, I then had to uplift it to modern infrastructure.
In plain English, people used to ask me to look at 15 year old production servers that had no documentation and no owners, and when they broke problems happened.
During this time, I learned about Network Science and Social Network Analysis. In the previous stage of my career, I had learned about Entity-Relationship Diagrams, but they were less useful to me in Data Operations, because I needed to map out dataflows from server to server, or data center to data center. I dove head first into Network Science and Social Network Analysis, but my interest was in mapping dataflows, not mapping people. I was doing what is now known as Data Observability work, but it didn’t have the name.
I learned to convert code into graphs that I could explore, and I could use those graphs to troubleshoot outages in minutes. Prior to my graphs, it used to take the team an average of about two days to troubleshoot these ancient undocumented orphaned machines whenever they’d break.
So, you should take my dataflow mapping class when I offer it, because it will probably change your life if you learn it well. Same can probably be said for any course that I offer. Enroll today.
Data Engineering
I worked in Data Operations around Big Data technologies. In 2015, that was Hadoop, and at the time Spark, Storm, and Flink were contenders. I loved Spark, so I dove straight in and was the first person on my team to learn it. Since I was the only person on my team who knew how to do anything with Spark, people came to me for Data Engineering even though I worked in Data Operations. Finally, at one point, I was pulled off of my Data Operations work and onto a Data Engineering team.
So, I did that for a while, doing a bunch of Spark automation, Airflow, Redshift, AWS, blah blah blah blah blah. I did all that and I am happy that I learned to process massive amounts of data, because that is a very useful thing to know how to do if you want to do OSINT or build World AI.
AI and ML Platform Engineering + OSINT Analysis
2020 is when things started to get weird for me and my new career began to invent itself. I am not exactly an AI Engineer, and I am not exactly an OSINT Analyst. I am absolutely both and more. But in 2020, I was losing interest in learning ML and gaining interest in using it for positive impact in the world.
Losing interest in ML? Yes. I was obsessed with ML from about 2016 to 2019, and by 2020 it was muscle memory to me. Just workflows. Do I have data? Yes. Have I looked at it? Yes. Do I have a rough idea of what models will do ok? Yes. Blah blah blah. Data Science workflows become repetitive with time (and so do OSINT workflows).
In 2020, I was working on a cool project to predict future phishing attacks using social media data.
Alright, let’s break that down:
In 2020 (after I was comfortable working with ML)
I was working on predicting future phishing attacks (Machine Learning)
Using social media data (OSINT)
In 2020, I was already combining ML with real-world data to try to find useful insights that could help keep people safe. Over the course of that year, my Data Science interests had physical world reach. I didn’t just want to play with Kaggle datasets and whatever company data I had access to.
I started seeing what data was out there and accessible. I learned web crawling and how to use the Twitter API. I built my first Wikipedia crawlers. It was around this time that I created the first #100daysofnetworks blog. This was an important period of time.
Nobody taught me to do what I do. I learned a bit of how to use networkx from one book, but other than that I mainly read Social Science books. I already knew ML well enough. I wanted to understand people and the real world. I fell into OSINT analysis.
OSINT Platform Engineering and AI Architecture
After about a year of that, I decided to start a company, and then another, and then another. I have had different sets of cofounders, and I am still at this, still building, not going to give up until I build the thing that is in my mind. I am the CEO now and my company is Verdant Intelligence.
I have built what I call World AI. It is not a LLM. It is World AI. It is aware of things that have happened in the world moments ago. People argue with me because they do not understand that this kind of thing can exist. It is not an LLM. It is a different fish. I call it World AI, but it could also just be called OSINT AI.
In order to build World AI, you have to build World databases. You have to store information about things happening in the world. I figured out how to collect this kind of data during my OSINT phase, but not only there. I was using cURL in 2004.
I have OSINT data.
Yes, that is data.
See those red lines? The data is in my database at 9:51:17pm and the clock on my computer shows 9:52pm. Less than a minute has gone by. My AI can tell me things about the world.
With one of my endpoints, I can use it to investigate anything that happens. I have used this to:
Track Data Center information (Coming Soon)
In other words, I use this to do OSINT investigations because I want people to have important awareness of certain things.
Want to learn OSINT? Enroll now!
The other World AI endpoint is GrooveSeeker, which I use to create an Event Paper that supports Hillsboro, Beaverton, and Portland, Oregon. This is the more exciting of the two, and it is useful Event Intelligence, which is just what I am calling the combination of OSINT + Music/Art.
How did I build my World AI?
I built my World databases
I built two GraphRAG AI interfaces
Want to learn to build GraphRAG from Scratch? Enroll today!
So, you see, I am simultaneously OSINT and AI Architect and Engineer, and that is why my creations are a bit different. They continuously monitor the real world, not fire off web searches. My path has been different, and I think differently. I am far down the ML, AI, and OSINT path, and I have used my book and this blog to try to inspire others to think bigger and differently, to aim for outcomes, and to not just perpetually chase news and hype. Builders build.
Long strange trip
It has been a long strange trip. I can still remember making computers go beep boop when I was six years old. That moment is frozen in my memories. Everything since then has been me falling deeper and deeper down the rabbit hole. I feel like I am in Alice’s Wonderland, or in the data wilderness. I don’t chase news or hype or follow crowds, I build what needs to be built to solve problems to reach outcomes.
I build what I build to do two things:
Help us live
Help us enjoy life while we are alive
That is my mission in life and what I build for. I don’t build for money. If you showed up with a wad of cash and asked me to spend a year building something that I don’t see any value in, I won’t want the cash, because you cannot get your time back. I use my time to build things that support life and happiness.
I will probably always be a wanderer. I’m the one that teams throw at problems they need to understand. Sign up for my courses if you want to learn to think differently, or if you want to learn practical OSINT and AI Engineering skills.
Thanks everyone. That was a fun trip down memory lane.
And don’t forget to upgrade to a paid subscription if you want to read my more technical articles.
Have a great day and week!