Forward Deployed Engineer and RAG for AI Jobs Essay
FDE and RAG systems are frequently mentioned together in job offers for AI positions. There is a good reason for that: the companies want AI that can understand the data belonging to the company. Retrieval augmented generation or RAG allows for that.
Creating a RAG system that delivers in the office environment is not easy. This is where the Forward Deployed Engineers step in. Below we describe what they do and how they interact with RAG systems.
What Is a RAG System?
Large language models (LLMs) have vast knowledge. However, they do not know what goes on in the companies that use them. Furthermore, they cannot assimilate new information that comes after training. That is where the RAG system steps in.
There are different types of RAG. In essence, the RAG system searches through documents and finds the information relevant to the query. It then passes it to the language model, which uses that information to craft a reply. As such, the RAG systems deliver high quality and recent information that can be traced back to its origin.
What Is a Forward Deployed Engineer?
A Forward Deployed Engineer or FDE is a software engineer that works directly with the customer. They understand the problem at hand and develop a solution. Some FDEs work onsite while others work remotely but in close contact with the client.
The FDE concept was developed by Palantir. Other AI companies and start-ups employ FDEs as well.
Why RAG Systems Need FDEs
The RAG example can be created in a matter of hours. Creating a functional RAG system requires much more effort.
The data of a company can be vast and varied. It includes many document types and formats. Some of the data may even be contradictory. The documents can be old, or only some of the employees can see them. None of that is a problem for the FDE, who can look at the big picture and see what the best RAG system would be. After all, the FDE works with the customer, meaning that they understand the specific demands of the system.
How FDEs Build RAG Systems
The creation of a RAG system for companies usually follows the same six steps.
First, the FDE collects and prepares the documents. They can be in various formats from shared drives, wikis, or databases. The FDE then applies some cleaning to the data. They may discard old documents or reduce the size of the data set.
Next, the FDE splits the documents into manageable pieces called chunks. It is important to make the chunks neither too big nor too small. Too small and the model will lose the context of the information. Too big and the needed information will not appear. FDEs use trial and error to find the right size.
The next step is to create an embedding. The FDE takes a chunk and encodes it into a list of numbers. Then they use a vector database to store those numbers.
At the next step, the FDE retrieves the information. When the user asks a question, the system looks for the relevant chunks of information. FDEs often use a combination of contextual and keyword search. That way, the best results can be found. The FDEs then apply filters, making sure that users only see the information that they can see.
The next step is to generate a response. The model gets the question and the relevant chunks of information. A good RAG system will use the prompt to make sure that the model only uses the given information. If the information is not there, the model should note that. The FDE makes sure that they can cite the source of the information.
Finally, the FDE tests the system. It is not enough to have a system that works on paper. The FDE develops a set of test questions with known answers. Then they see if the system finds the right information and delivers an appropriate response. After launching the system, they still work on it, looking for ways to improve it.
A Simple Example
Let us consider a manufacturing company with hundreds of pages of documentation. The staff spends hours trying to find the relevant information. An FDE can come to the company, learning what they are looking for. They can clean up the documents and divide them into chunks. Then they develop a search engine that can answer specific questions, citing the relevant page. Finally, they test the system, making sure that questions get appropriate answers.
In effect, the FDE made the model better without changing it. The same knowledge was applied using a better search system.
Common RAG Problems and How FDEs Fix Them
Wrong chunks refer to the system delivering irrelevant information. FDEs fix that by changing the size of the chunks and using keyword search.
Hallucinations are when the model makes up information. FDEs solve that by making the model cite its sources. They can also make it possible for the model to note that it does not have the information.
Outdated information can be the result of the documents being old. The FDE can update the documents or make a procedure for updating them.
Leaked private information is the result of users seeing documents that they should not see. FDEs can use permissions to stop that from happening.
Slow response time can be the result of the RAG system. FDEs make the system as efficient as possible and limit the prompt. That way, the model does not spend time on irrelevant information.
Skills for FDE and RAG Roles
Companies look for a mix of technical and social skills from FDEs.
The technical skills include:
Proficiency in Python, TypeScript, and SQL Knowledge of embeddings and vector databases Experience with search, including keyword and hybrid search Prompt engineering Testing and evaluation Cloud computing
The social skills needed are:
• Communication: The FDE needs to be able to listen to the customer and explain what they are doing
• Acanthophobia: Some customers can be harder to work with. A good FDE is not discouraged by that
• Agency: The FDE needs to take ownership of the project
• Speed of learning: The FDE needs to be able to pick up new skills in a new industry
Both sets of skills are important. Someone who is fast at coding but has no social skills will fail as an FDE. The same person might make a good RAG system but be unable to explain how it works.
Why Demand Is Growing
There are several factors that contribute to the rise in demand for FDEs.
More companies want AI that uses their data. Fewer companies know how to create a RAG system. The latter is especially true for enterprise software. At the same time, the enterprise software industry demands solutions from FDEs. Finally, privacy and security laws are a concern for all industries. The FDEs are the link between LLMs and the data of the companies.
The FDE Position in a Rising Field
The FDE field offers opportunities for advancement. Some FDEs move into product management. Others become leaders within the engineering team. The more experienced FDEs can work as consultants or start their companies. No matter what path they choose, the FDE has options.
How to Start Your FDE Journey
It is not necessary to have the perfect resume to start as an FDE. It helps to have some background in Python and SQL. One should also learn about LLMs and how embeddings work. They should then try to make a RAG application with their own data. Experimenting with the size of the chunks is important. They can also try to implement a test set.
The aspiring FDE should also learn to explain what they do. Finally, it is a good idea to join a training program.
Projects are more important than certifications. The portfolio may include a few projects, but it should demonstrate the breadth and depth of one’s skills. As the FDE field grows, there will be opportunities ahead for those that prepare for it.
Closing Thoughts
FDE and RAG systems are a perfect match. For all the promise of LLMs, they still need the right information. FDEs deliver the right information by using the right tools. Anyone considering a turn to the FDE profession should start today.