Typescript & LLMs: Lessons Learned from 9 Months in Production
In early 2024, I set out to build a research AI agent capable of autonomously browsing the web, gathering facts, and returning metadata about their sources. Like many developers in the AI space, I started with Python, FastAPI, and LangChain — popular choices due to their flexibility and speed for prototyping.
However, as the project grew, so did its complexity. I soon realized that Python, while excellent for rapid iteration, became a bottleneck when transitioning to production-grade systems. The dynamic nature of Python, especially its weak typing, introduced bugs that only surfaced at runtime. Managing concurrency became cumbersome, requiring complex workarounds. In contrast, Typescript provided the type safety and scalability I needed to handle production workloads efficiently.
This shift — from Python to Typescript — was a crucial turning point. In this article, I’ll dive deeper into why I made these choices, where frameworks like LangChain fell short, and how these technical decisions impacted business outcomes.
Python vs. Typescript: The Concurrency Struggle
Python’s strength lies in its flexibility. It’s fantastic for prototyping, which is why I began with it. But once I needed to scale, Python’s lack of strong concurrency support became a serious issue. Libraries like asyncio and frameworks like FastAPI offer concurrency but come with significant complexity. I had to manually manage event loops and concurrency controls, making tasks like scraping web pages or querying multiple APIs more challenging.
Additionally, Python’s Global Interpreter Lock (GIL) limits its ability to handle true multi-threaded parallelism — a problem that doesn’t exist in Node.js. Node.js is naturally built around an event-driven, non-blocking architecture. With Node.js and Typescript, using async/await felt much more intuitive and robust, particularly when managing large-scale web scraping or data extraction tasks where handling multiple I/O-bound operations concurrently is critical.
As I scaled my system to hundreds of concurrent tasks, Python presented issues with thread management and performance bottlenecks. Switching to Typescript allowed me to leverage tools like the Bottleneck library, providing fine-grained control over concurrency and ensuring that my system could scale without overwhelming resources.
In practical terms, this shift reduced my infrastructure costs. My system could process more tasks in parallel, reducing the time spent on expensive cloud compute resources.
Where LangChain Fell Short: Customization and Type Safety
LangChain is a fantastic tool for prototyping LLM workflows, offering a simple interface for chaining prompts, tools, and memories. However, as my system became more complex, LangChain’s abstractions began to limit my ability to customize workflows.
One specific challenge I encountered was handling custom data extraction tasks that required tight integration between LLM outputs and custom code, such as web scraping and data transformation. LangChain’s rigid chaining mechanism made it difficult to insert custom logic that didn’t fit into its predefined structures. The flow between LLM responses and my custom code was tightly coupled to LangChain’s framework, which limited my ability to apply custom transformations or validations on the data between workflow steps.
By moving to Typescript and building my own modular classes, I gained granular control over each workflow step. I could define custom handlers that seamlessly integrated with existing business logic, such as applying validation and transformation to LLM outputs. This flexibility also allowed me to incorporate advanced features like prompt versioning and schema validation without relying on an external framework.
Embracing Typescript’s Type Safety
Typescript’s ability to enforce types at compile-time became invaluable as my workflows grew more intricate. With LangChain, I often encountered runtime errors due to missing variables or mismatched data types — issues that only surfaced when the workflow was executed. These runtime errors increased debugging time and made the system less reliable in production.
In contrast, Typescript’s compile-time checks prevented a whole class of bugs before they ever reached production. For example, when defining a prompt template with placeholders, Typescript would alert me if any required variables were missing from the input object.
This level of type safety reduced runtime errors by approximately 30%, as issues were caught during development rather than in production. The system became far more reliable, and we could deploy updates with greater confidence.
LangChain’s Lack of Type Safety
LangChain, being Python-based, lacks strong type enforcement. While Python’s dynamic typing allows for rapid prototyping, it becomes a liability in larger systems. Errors related to data types often only surface at runtime, sometimes under specific conditions that are hard to reproduce.
For example, a prompt expecting a variable {{userInput}} might silently fail if the shape of userInput has changed somewhere else in the system, leading to unexpected behavior from the LLM. Debugging such issues can be time-consuming, especially in complex workflows.
While LangChain does offer a Typescript compatible version, it is not designed to be type safe by default.
By leveraging Typescript’s robust type system, I eliminated many of these runtime errors. The compiler acted as a safety net, catching mistakes early in the development process. This not only improved reliability but also increased developer productivity, as less time was spent tracking down elusive bugs.
Built-In Prompt Versioning: Enhancing Collaboration and Maintainability
As our workflows grew in complexity, managing changes to prompt templates became increasingly important. In dynamic AI systems, prompts evolve over time to improve performance, adapt to new requirements, or fix issues. Without a robust versioning system, keeping track of these changes becomes a daunting task.
In our Typescript framework, we built a prompt versioning system directly into the workflow. Each prompt template is assigned a unique version ID and timestamp whenever it’s created or modified. This metadata is stored alongside the prompt, allowing us to:
- Audit Changes: Track the history of prompt modifications, understanding who made changes and why. This is crucial for maintaining accountability and reviewing the evolution of prompts.
- Rollback Capabilities: Easily revert to a previous version if a new prompt introduces unintended behavior. This minimizes downtime and reduces the risk of errors in production.
- Collaborative Development: Enable multiple team members to work on prompt templates concurrently. Versioning ensures that changes don’t inadvertently overwrite each other, and merging updates becomes more manageable.
Contrast with LangChain
LangChain lacks built-in support for prompt versioning. Without this feature, managing prompt changes requires manual processes or external tools, leading to challenges such as:
- Difficulty Tracking Changes: Developers might overwrite each other’s modifications without realizing it, causing confusion and potential loss of work.
- Increased Risk of Errors: Without versioning, pinpointing when and where a problematic change was introduced becomes harder, complicating debugging efforts.
- Reduced Collaboration Efficiency: Coordinating prompt updates is challenging, slowing down development cycles and innovation.
By integrating prompt versioning into our Typescript framework, we streamlined the development process, improved collaboration, and enhanced system maintainability. For complex applications, built-in solutions like this simplify maintenance and foster a more productive development environment.
Managing LLM Outputs and System Scalability
LLMs inherently return unstructured text-based outputs, even when you ask them for structured data formats like JSON. This presents a challenge when building scalable systems that depend on predictable outputs from the model. LangChain provides output parsers to handle this, but in my experience, these parsers added unnecessary complexity without significant value. Often, they are simple regex functions that you could write yourself.
In my system, I took a different approach, integrating output parsing directly into my Typescript utilities. By writing custom parsers, I could handle edge cases more efficiently — such as when an LLM returned partially correct JSON with extra tokens. My parsers cleaned the data before passing it to the next workflow step, eliminating the need for additional parsing steps and reducing code clutter.
This focus on structured outputs was critical when scaling the system. In production, my system processes thousands of LLM responses per hour. Having reliable JSON output allowed for seamless integration with downstream services like databases and API endpoints. This not only made the system more efficient but also reduced post-processing needs, further speeding up response times.
Improved Developer Experience and Maintainability
Transitioning to Typescript significantly enhanced the developer experience. Our framework’s design made it easier to trace issues and maintain the codebase, leading to faster development cycles and more efficient onboarding of new team members.
Simplified Debugging
Typescript’s static typing and compile-time checks simplified the debugging process. Errors related to data types, missing variables, or incorrect function signatures were caught early, preventing them from becoming runtime issues. This proactive error detection reduced time spent hunting down bugs in production and allowed developers to focus on building new features.
Additionally, our framework incorporates clear error messages and stack traces that pointed directly to the source of problems. We leveraged tools like source maps and integrated logging to enhance observability, making it straightforward to identify and fix issues quickly.
Code Readability and Approachability
Typescript’s type annotations and interfaces improved code readability, making it easier for developers to understand the structure and flow of the application. The explicit definitions of data types and function contracts acted as self-documenting code, reducing the need for external documentation.
By reducing unnecessary abstraction layers, the codebase became more approachable. Developers could navigate the code without getting lost in complex inheritance hierarchies or deeply nested abstractions. This simplicity not only made the existing team more productive but also eased the onboarding process for new engineers joining the project.
Why This Matters
A better developer experience directly translates to faster development cycles, higher-quality code, and more resilient systems. For senior engineers, maintainability and ease of collaboration are critical factors in project success. Our framework’s emphasis on clear, readable code and simplified debugging processes addressed these concerns head-on.
The Market Impact: Why These Choices Matter
From a business perspective, making these technical decisions early on had a profound impact on scalability and operational efficiency. As LLM workflows become increasingly central to businesses — whether for data extraction, customer service, or content generation — the ability to scale workflows efficiently directly translates to cost savings and faster time-to-market.
By building a system that efficiently manages concurrency and maintains type safety throughout the workflow, I reduced operational costs, minimized downtime, and ensured the system could handle large-scale data extraction tasks reliably. This was particularly important as I integrated the system with AWS Lambda for cost-effective task execution.
Running LLM-based workflows in serverless environments like AWS Lambda is more efficient when your system is built with concurrency control in mind. Using the Bottleneck library in Typescript allowed me to control the flow of tasks into these serverless functions, ensuring that no function exceeded its resource limits.
The native concurrency control in Typescript also allowed me to optimize resource usage, ensuring the system performed efficiently under heavy loads without exceeding the cost thresholds typical of large-scale deployments.
These real-world benefits highlight why choosing the right technology stack early in the development process is crucial for long-term success. The switch to Typescript allowed me to build a more maintainable, scalable, and cost-effective system, directly contributing to business outcomes like lower infrastructure costs and faster development cycles.
Balancing Prompts with System Flexibility
Initially, I spent considerable time designing a system that abstracted prompts away from workflow logic. My goal was to keep workflows readable and maintainable by hiding the complexity of prompt management. While this worked for smaller workflows, it introduced unnecessary complexity in larger systems where visibility into prompt structures became essential.
For example, debugging issues related to missing variables or incorrect prompt formatting often required scrolling through multiple abstraction layers, making the system harder to maintain. Over time, I realized that while abstracting prompts is useful for keeping code clean, too much abstraction can hinder debugging and flexibility.
By integrating prompt versioning and leveraging Typescript’s type safety, I found a balance between abstraction and visibility. The versioning system allowed us to manage prompts efficiently while keeping them accessible within the workflow. Typescript’s compile-time checks ensured that any changes to prompts or their expected variables were immediately reflected in the code, reducing the chances of mismatches or errors.
This approach reduced debugging time and improved overall system maintainability. Developers could quickly identify issues related to prompts without wading through unnecessary layers of abstraction, making the system both flexible and robust.
Landing on My Own Approach
After nine months of building and testing prototype apps with Python, FastAPI, LangChain, and Typescript/NestJS, I developed my own Typescript library for interacting with LLMs. This library drastically simplified my backend code and streamlined my approach to building LLM-based components.
Here are the guiding principles I followed:
- Simplicity: The API must be easy to use and understand.
- Type Safety: Enforce type safety wherever possible to prevent bugs and improve reliability.
- Sensible Defaults: Common use cases should work out of the box, with customizations available as needed.
- Scalability: Features like prompt versioning should be available but managed seamlessly and invisibly to the user.
- Concurrency Management: Efficient task execution should be built into the system, simplifying workflow scaling.
- Developer Experience: Writing and using the library should be enjoyable and not burdensome. Clear code and easy debugging are essential.
By focusing on these principles, I built a framework that not only met the technical requirements of our projects but also enhanced the developer experience and contributed to better business outcomes. The integration of Typescript’s type safety, built-in prompt versioning, and a focus on maintainability set this framework apart from existing solutions like LangChain.
Lastly
Choosing the right technology stack and tools is crucial for the success of any project, especially when dealing with complex systems like those involving LLMs. By embracing Typescript’s type safety, integrating built-in prompt versioning, and focusing on developer experience and maintainability, I was able to overcome the limitations I faced with Python and LangChain.
For senior Node.js engineers, these considerations are paramount. The ability to catch errors at compile-time, efficiently manage and version prompts, and maintain a clean, approachable codebase can significantly impact a project’s scalability and reliability.
My journey led me to develop a solution that addresses these challenges head-on. I hope that sharing these experiences helps others facing similar decisions and highlights the benefits of a Typescript-based approach to building scalable, maintainable LLM applications.