In-Memory Search and the Basics I Skipped

I have been building by making the product work.

That is a real way to learn. It is also a narrow one. You grab the tool that unblocks the feature, you ship, and you move on. A lot of the broader ideas never get a rep. Week 1 of Alexey Grigorev’s AI Engineering Buildcamp: From RAG to Agents put me back in those reps.

The week ran September 21 to September 27. The foundation module is LLMs, RAG, and search. The lessons and the office hours have been helpful. I have not sat down with Alexey one on one. The content and the live session were enough to make the idea concrete.

I am glad to be in the course. So far it is a 10 out of 10.

What In-Memory Search Is

In-memory search is retrieval that lives with the program.

You hold the documents in process memory. You search them there. Keyword match, or a similarity score, on data you already loaded. No separate vector database. No service to stand up before you can ask a question.

To understand this properly, compare it with the habit I already had.

When I needed retrieval, I reached for a vector database. Embed the chunks. Store the vectors. Query the index. That works. It is also a lot of infrastructure for a question you could have answered on one machine.

The reality is I treated the database as the starting point because that is what I had seen work in products. I skipped the simpler object: a search index that exists only as long as the process does.

Why It Matters

Think about where the data actually lives.

A local assistant does not need a hosted index to answer questions about a folder of notes. A small deployment can load the corpus, search it, and throw the index away when the process ends. A distributed setup can do the same thing on each node: search the data that node already has, instead of shipping every query to one central vector store.

This means the tool should match the job.

A vector database is the right move when the corpus is large, shared, and needs to stay up. It is the wrong first move when you are still learning what retrieval even is, or when the data is already on the machine that needs to search it.

In my opinion, that is the opening. Local AI and distributed AI both get more interesting once search is allowed to be ordinary memory, not a product you have to adopt.

The Reps

A lot of my learning has been trying to get a product to work.

That teaches you the path from bug to ship. It does not teach you the set of ideas sitting next to that path. I had missed a broad set of those ideas. Formal practice is how you get them back.

The recorded lessons, the homework, and the office hours are that practice. Alexey walks the foundation in public: what the model is doing, what search is doing, and how a RAG pipeline is just those two things put in order. I did not need a private meeting for that to land. The course content has been the guide.

Now the world looks more open.

Revisiting the basics did that. Not a new framework. Not a new model drop. The simple version of search, done on purpose, instead of inherited from the last product I shipped.

The Hype

The issue with the public conversation is that it stays at the surface.

Twitter will tell you RAG is solved, or dead, or replaced by whatever shipped this week. Most of that is hype. Most people have not scratched the surface of usefulness. They have a diagram. They have a vendor. They have not sat with the search itself.

What if we began to treat the basics as the actual work?

In-memory search is not a toy step you graduate out of and forget. It is a real design. It is also a test. If you cannot explain retrieval without naming a database, you do not yet know what you are building.

What This Means Next

Week 1 gave me a usable frame.

Search can live in memory. A vector database is a choice, not a reflex. The office hours and the lessons made that distinction practical instead of theoretical.

The course moves from RAG toward agents. I am staying in the reps, and I am writing it down in public. I want the next weeks to keep opening the same door: less hype, more of the machinery, and a wider set of ideas than the ones I needed to ship the last product.

Overall, I am glad I stopped long enough to do the foundation again.