Introduction
A forward deployed engineer rarely has the luxury of choosing their technology stack in the comfort of their office. They are in the trenches of their customer’s environment, battling against data and time constraints, and seeking to solve their business problem with generative AI.
This guide covers the tools needed to succeed in this endeavor.
Why This Tech Stack Matters for an FDE
An FDE rarely builds a product for their employer, but rather, they build a solution to a unique problem that their client faces.
They might spend one week supporting a customer in creating a chatbot to handle customer support, while the next week, they extract information from hundreds of contracts. Because of this, we need something that can be rapidly prototyped and built upon, as well as easily explained to our customers.
FDEs should know how to build a prototype within days, not months.
Programming Languages and Core Tools
Python underlies most software development in generative AI. It has excellent libraries and first-class support from most AI companies. If you only know one programming language, it should be this one!
In addition to Python, one should know a few other minor tools to reduce time spent in the field:
- SQL – for querying and sanity-checking customer data
- JS or TS – for quickly making a frontend demo
- Git – for version control
- Docker – for packaging software
These are all incredibly useful skills to have as an FDE and could save you hours of frustration at a customer site.
Large Language Models (LLMs)
LLMs are the foundation of the generative AI tech stack. You will spend a lot of time working with models from various companies.
Some examples of LLMs are commercial offerings such as OpenAI, Anthropic, or Google, while others, such as Llama and Mistral, are open-source alternatives.
It is unlikely you will have a strong preference between open-source and closed models, since every customer has different requirements, and you have to pick the best model for their needs.
Prompt Engineering and Evaluation
Prompts are the instructions given to a model and are one of the primary ways of controlling its behavior. You should be able to write effective prompts that get the response you want from a model.
In addition to writing prompts, you should also know how to evaluate whether a model’s response is good or not. Most FDEs I know skip this critical step and waste a lot of time later because the model wasn’t doing what they wanted it to do.
I recommend having a baseline of some judgments on real customer data. These can be used to sanity-check any prompt or model you encounter.
Tools like LangSmith, Promptfoo, and Ragas can help you with this, as they provide metrics and ways to share results with stakeholders.
RAG and Vector Databases
Retrieval Augmented Generation, or RAG, allows a model to use information from documents provided by the customer.
This is without a doubt the most common use case of generative AI I’ve seen as an FDE. Almost every company has some information that is difficult for computers to process but easy for humans to understand.
The RAG process typically consists of 3 steps:
- Chunking a document into manageable-sized texts
- Embedding these texts into a vector database
- Using the retrieved information along with the initial query to answer a question asked by the user
Some examples of vector databases are Pinecone, Weaviate, Qdrant, Chroma, and pgvector.
Of these, pgvector in Postgres is a good option if the customer is already using Postgres, as it has decent performance for smaller-scale projects. For your first project, it’s best to use whatever is simple.
Orchestration Frameworks and Agents
LangChain and LlamaIndex provide generic tools for building a wide variety of applications. They provide memory, methods for retrieval-augmented generation, and tools for interacting with the outside world.
Meanwhile, agents make use of these tools in novel and interesting ways. An agent could be as simple as providing suggestions for a customer service representative or as complicated as performing multiple API calls in sequence.
Agents tend to be more difficult to debug and can have surprising behaviors. It is best to start with simple agents until you are more comfortable with the tools.
APIs and Integration Tools
FDEs need to be able to interface with their customers’ systems, and this usually involves APIs.
You should have a good understanding of how these systems work and how to build applications on top of them. You should also understand authentication and event-driven architectures with webhooks.
FastAPI is an excellent general-purpose API framework that can be used to expose some small endpoints to the internet.
Most of your customer’s systems will require some sort of external integration, and this is usually the single largest time sink for an FDE. You can’t really do much if your model can’t fulfill a certain request within the customer’s database or CRM.
Cloud Platforms and Deployment
All FDEs eventually have to deploy their work somewhere, and the cloud is the usual place for this.
AWS, Azure, and GCP provide managed AI services, security, and storage, while Docker and Kubernetes provide deployment and scaling.
It is best to be intimately familiar with at least one major cloud provider and have a more superficial knowledge of the others.
Most customers will have a cloud preference, and you should be able to work within their constraints. Additionally, consider the economics of your deployment, as it is all too easy to spin up a GPU instance that costs thousands of dollars a month.
Monitoring and Guardrails
A deployed model requires maintenance, and you should be able to monitor it for errors, performance, token usage, and costs.
Tools like Langfuse and Helicone can help you with some of these tasks.
Additionally, you should be aware of standard security practices, such as encrypting data at rest and in transit, rotating secrets, masking PII when possible, and so on.
Similarly, you should have some knowledge of model guardrails, which prevent your model from leaking information or behaving unexpectedly.
Almost all customers will look favorably upon you if you can demonstrate experience with security and privacy best practices. In fact, this is one of the places where many FDEs can differentiate themselves from the competition.
How to Build Your Own Stack
You don’t need to be an expert in all of these fields. Rather, you should deeply understand some while having a superficial knowledge of the others.
To start building your own FDE stack, begin with the following:
- Learn Python and APIs
- Make a rudimentary chatbot with an LLM
- Add RAG capabilities with your own documents
- Package your application with FastAPI
- Deploy your application on a cloud provider
- Set up logging and monitoring
These are all excellent steps to begin your FDE journey and serve as a nice portfolio piece.
Final Thoughts
The generative AI tech stack for FDEs is expansive, but hopefully, this article gives you some ideas about what to learn to become a successful FDE.
There are some obvious parallels to traditional software engineering, but with the addition of LLMs, RAG, agents, orchestration, and deployment to production.
With that said, testing, evaluation, security, and model guardrails are always important considerations for any FDE endeavor.
The tools will inevitably change, but the habits described in this article will take you far in your FDE career.