Day 50 of #100daysofnetworks
Milestone Day! Knowledge Graphs, Neo4j, GraphRAG, Artificial Intelligence
Hi everybody! Today is a big day! Today is day 50, the supposed halfway point of this series. For those who do not know, this is actually the SECOND iteration of #100daysofnetworks, with the first taking place about five years ago. The first iteration went well past 100 days, and I expect this one to go beyond 100, as well, but, it feels good to be fifty days in! We are lightyears beyond the first iteration, now within the realms of Artificial Intelligence.
I move really fast, and so have some nice news:
The blog now has 749 subscribers. I’d like to see this hit 800 soon. I think it will, if I write more often, and if I promote previous articles better. Please let your friends and coworkers know about this blog.
The blog has six paying readers, and that’s enough to support this blog’s Neo4j instance that I will write about today. Thank you very much to them! Today’s article would not be possible without them!
Enthusiastic readers have been sharing their use-cases with me, and occasionally meeting to collaborate and share ideas! That was my hope when writing my book and creating this blog! I was hoping to meet more people who understood the value and were building with graph, ML, and AI.
I’ve even found a cool graph group to join and hang out with! Check out Graph Geeks! You should get involved with them!
And here are some very cool technical milestones:
I’ve found a very useful tool for creating Knowledge Graphs. Cognee FTW. I only have good things to say about them!
I learned how to push this Knowledge Graph to various databases, and decided to push it to Neo4j Aura. I will explore other Graph Databases, later.
I used a dataset of Artificial Life research to create a Cognee Knowledge graph that I can chat with, and I pushed the Knowledge Graph to Neo4j.
I have figured out how to give access to the Neo4j instance to paying readers. If you are a paying reader and would like access, please reach out to me on LinkedIn and I can set you up with access in minutes.
I am now using Cognee and Neo4j for Temporal Graph RAG and getting accurate and useful answers about anything I use this approach for.
A lot has been going on. I move very fast and learn fast. Things will slow down a bit after this article, as I essentially figured out a bunch of end-to-end stuff very quickly. From day 51, things will slow down and simplify.
Write Knowledge Graph to Neo4j
If you read my previous article (https://100daysofnetworks.substack.com/p/day-49-of-100daysofnetworks), you will learn how easy it is to create a Knowledge Graph (KG) with Cognee. However, that is only part of the equation. After creating a KG, you can chat with it, but if you do not persist the KG, then you will need to recreate it to chat with it again. I am still figuring things out, but pushing to a graph database is simple. You need a .env file, and you need settings inside it. I’ll show mine, without creds, so you can see the simplicity.
GRAPH_DATABASE_PROVIDER="neo4j"
GRAPH_DATABASE_URL="<your instance here>"
GRAPH_DATABASE_NAME="neo4j"
GRAPH_DATABASE_USERNAME="neo4j"
GRAPH_DATABASE_PASSWORD="<your password here>"
LLM_API_KEY="<your key here>"
LLM_PROVIDER="openai"
LLM_MODEL="gpt-4o-mini"
VECTOR_DB_PROVIDER="neo4j"That last line took some time to figure out. Include that last line if you want everything on Neo4j. With that, Cognee knows what LLM to use, and what databases to store data to. It takes care of the rest. I love this simplicity!
You can use your LLM of choice, Graph DB of choice, etc. Cognee is flexible.
If you run the code from the Day 49 article and you have a .env file with correct settings, you should get what you want. Cognee also support other databases (relational, graph, vector).
BREAK POINT: I have pushed 5% of a dataset on Artificial Life research into a Neo4j Knowledge Graph. I am planning to continue to scale this up. I am evaluating cost and ability to scale, gradually. I could use some help:
I would like to scale this database up, so if you are one of the other 742 readers, please consider upgrading to a paid account. This blog provides oversized value compared to its low cost. Please go here (https://100daysofnetworks.substack.com/subscribe) and you can easily upgrade, if you would like to support the blog. I will provide Neo4j access to paying subscribers who want access. The database is literally running on 1GB of memory, so it would be nice to go a little higher. If we go 4x higher, I can make use of Neo4j’s AI features, and the database will run better. 🙂
Show and Tell!
That’s a lot of words. Let’s look at some stuff! Yesterday I used a dataset of Artificial Life research that I pulled two years ago on Day 18. On that day, I created a way for you to pull these kinds of datasets, on any topic of interest. I am interested in Artificial Life.
The dataset I pulled two years ago contains Artificial Life research going back 30 years! It contains the context of 20,000 Artificial Life articles, a context rich and valuable dataset.
Yesterday, I first used 1% (random sampling) of that dataset to create the first knowledge graph. With only 1% of the data, these were the results:
Status: SUCCESS
Time to Status: 43 minutes
Cost to Status: $3
Then, before going to bed, I set to 5% and rebuilt the KG, and pushed it to Neo4j:
Status: SUCCESS
Time to Status: I fell asleep and didn’t see… lol. Hours.
Cost to Status: $10
Here is what 1% of the data looks like, in a Knowledge Graph!
Stunning. Incredible. I have since increased the scale 5x and pushed the data to Neo4j. I now have a much better KG than this one, but do not have a screenshot.
I can chat with it, and I get accurate answers. You heard that right, I get accurate answers from AI. I have a reliable AI interface for learning about Artificial Life.
I can ask it any question. I can do all of the things that we prompt for in AI Engineering. I find GraphRAG reliability really soothing, after so many headaches with ChatGPT and other LLMs. It is nice to get straight answers to questions I want to know about, and not be led in circles or have my time wasted by an overly verbose and unreliable chatbot.
And now the data is in Neo4j, so I can explore the Knowledge Graph there as well.
In total, the Knowledge Graph currently contains roughly 18k nodes and 41k edges, and this above screenshot is showing nine of the nodes and nine of the edges, 0.05% of the Knowledge Graph. There is so much more to explore.
Each node can be expanded, each author and research paper can be inspected.
The purpose of this is the ability to create accurate and reliable Artificial Intelligence interfaces, not things that hallucinate all over the place and make errors constantly. I wanted to reach the outcome of being able to quickly find out about Artificial Life research that interested me. I have seen other approaches, and I wanted to build this using my approach. So, I did.
Use Cases for Knowledge Engineering
This isn’t just AI Engineering work, this is also Knowledge Engineering. Since the beginning of this series, I have shown how to create datasets that are of interest to YOU, and now I am showing how to take it even further.
Literally, my interfaces talk to me. I listen to audio. I don’t just read the text outputs. I can use this to hear information that I want to know.
The use case is:
I can get information that I want to know.
That is a generalizable use-case. If you work in entertainment, you want answers about specific information. If you work in a hospital, you want answers about specific information. If you work in OSINT, you want answers about specific information.
So, please think about this, and think creatively about your own use-cases. I find this useful in general.
The use-case is generalizable. It comes down to curiosity and creativity. What do you want to learn about? Where will you get the data? Once you have the data, how will you get the insights out of it? How will those insights be communicated to you?
Big Day! What’s Next!
That was a lot to cover! We now have a cool graph database to explore, learn to use, and play with. Please reach out to me if you become a paying reader and want database access.
What’s next: I plan to simplify. I plan to do shorter articles for a bit. I brought us to Knowledge Graphs, Graph Databases, and Artificial Intelligence. Next, let’s slow down and learn how to use a Graph Database. We have a lot to work with, and this will be generally useful, beyond Knowledge Graphs.
That means future articles will be about writing queries in the Neo4j UI, writing queries against Neo4j programmatically, learning the UI, learning simple administration, etc. Let’s play with Graph Databases for a while and get comfortable.
We’ll slow down, simplify, and have a more peaceful learning experience. I have been going very fast, and it will be good to slow down!
Please Support this Work!
I have written 50 articles for this series. Each one takes about roughly four hours of research, and several pages of writing and editing. Here are some ways you can support the blog!
Please subscribe, if you have not.
BIGGEST HELP: Please consider upgrading if you are a subscriber. Thank you to all current paying subscribers for making this research and development possible!
Please buy my book to understand how I think about Natural Language Processing and Network Science combined.
Feel free to hang out in the comments and have a good time!
We have come so far since the very first day of the very first #100daysofnetworks. I love writing for this series. Thank you for being a part of it!
