Amjad Masad (Replit) Fireside Chat

We were delighted to engage in a dynamic conversation with Amjad Masad, founder and CEO of Replit during our August Chat8VC gathering. As a reminder, Chat8VC is a monthly series hosted at our SF office where we bring together founders and early-phase operators excited about the foundation model ecosystem. We focus on having highly technical, implementation-oriented conversations and give founders and builders the opportunity to showcase what they’re working on through lightning demos and fireside chats. This can be a fun side project, related to a company they’re starting, or a high-leverage use case within a company operating at scale. 

Amjad shed light on the founding story behind Replit, addressing the inherent friction to go from an idea to creating something people can use – Replit’s north star is to speak ideas into existence.

If any of these themes excite you and you’d like to chat more with our team about them or attend future Chat8VC events, please reach out to Vivek Gopalan at vivek@8vc.com and Bela Becerra at bela@8vc.com!

I’ve always been inspired as to how you’ve envisioned the change in the way consumers interact with computers and applications. Thirty years ago, the modal user of a computer was likely a developer and over time the persona has shifted to consumers. An increase in no-code tools has made it easier for consumers to become both creators and developers – co-pilots can help make this even more ubiquitous. How do you anticipate the relationship between users, applications and the computers themselves will change? How is Replit positioned to capitalize on this evolution? 

I coined a term that has stuck, which I refer to as the Steve Jobs Black Pill. In Silicon Valley, rightfully so, we idealize Steve Jobs. He pioneered amazing advancements – we’re all carrying a computer in our pockets that has infinitely improved our lives. As a byproduct, Apple has created a schism between a developer and a consumer of technology. Previously when you bought a computer, you had to learn how to use it – you had to learn how to program, manipulate data, save files etc. You were more sovereign in your use of computers. Over time, we've become consumers of machines. 

Most people use computers as televisions – we sit in front of our screens absorbing incredibly addictive TikToks and reels, aimlessly scrolling to pass time. I believe in a future where people are at minimum, super users of their computers. There shouldn’t be as dramatic of a schism between people who are making and using the software. This divide makes it challenging to do your job. How many times has an application stood in your way? You couldn't get access to the data you're using, so you needed this one feature and you had to go beg the developer to create said feature.

There has been an explosion of SaaS – one very negative view is that SaaS companies are ultimately software consultancy shops. You’re building a long tail of features for every customer to satisfy their specific needs. There's a better option, which is the world of programmability. No-code tools obviously exist, but you don't really need no-code tools. No-code tools are basically the assumption that code is inherently inaccessible, but now with computers being able to generate code, code has become accessible. Let’s look back at history – there’s the shareware era in the seventies, eighties and nineties. The idea is that you can publish a piece of software and get paid for it, but people get the source code and they tinker with it. Open source is an expression of this. 

A recent article highlighted the idea of building custom software – this person was trying to solve their own problems and built a custom piece of software. He soon discovered his friend had the same problem and they decided to fork it. They now have a copy of that application and can edit it to make it their own. The claim that I'm making is that the configuration of the world of software we have today is neither optimal, nor does it have to be this way. There are better ways where we could construct the world of computing and software.

I wanted to touch upon a design question. I used to be a developer myself – like a lot of people, I was very needy and opinionated about my IDE, the tools I leveraged and degree of customizability. Replit is quite general and easy to use. It doesn't focus on a single persona or even language. How did you think about designing an IDE from first principles? How did you take what used to be disparate and make it unified in a way such that millions can seamlessly use it?

At the core, it’s about neat abstractions. The way we try to build a product is to create layers of automation on top of open source abstractions such that you can still peer under the hood and exit the platform easily – we don't want to engineer lock in. An example of this is package management. This tends to be fairly idiosyncratic as it differs across various operating systems, package managers, languages etc. Instead, we aim to unify all of this and determine synergies across systems. 

As an example, we made a big bet on Nix a few years ago, which was fairly unknown. It’s a purely functional package manager such that the operating system is the result of the evaluation of a single Nix expression. The inputs of that Nix expression are the packages as well as associated dependencies – the result is an operating system. The operating system is a byproduct of the evaluation of a purely functional expression. What can you do with this? Well, we cache all the Nix packages in a single 10 TB disc and we mount that disc on every container started in our ecosystem. Every time you run a program, we evaluate that Nix expression. Essentially, every program results in a new operating system. We create all these abstractions around nix and we standardize on nix. Google decided to copy our Nix feature recently, which is great. I'm happy when our features are cloned so they become the standard.

There are ways to create systems that, by default, work, but are also customizable. Larry Wallace, the founder of Pearl, wisely stated that you should “make the easy thing easy, and the hard things possible”. It’s a great saying for designing software – you want the casual user to have the best possible experience without thinking about customizing the system. You want the developer though to be able to pursue customizability. Over the next year or so, people are going to find that Replit is a lot more fun and easy to customize. We're developing an extension system that is more pleasant to customize and will open source a good amount of it. 

I’d love to dive into the topic of AI – I was talking to a founder today and we were brainstorming the areas that are most natural for LLMs. Law stands out as you’re reading, synthesizing information and creating documents. The other key use case is code generation – Ghostwriter obviously is a feature you’ve invested in. Walk us through the evolution of AI in the developer experience – how will it improve overtime? 

Not enough ink has been spilled on the topic of the unreasonable success of LLMs in code. Researchers should spend a lot more time figuring out not only why LLMs became so good at coding, but why training on code makes LLMs better. The launch of Open AI’s GPT 3.5 family of models based on codex marked a step function improvement in LLMs. Codex was the base model for 3.5. There are a handful of papers covering why code models are actually better language models in general. Code models end up performing better at text extraction tasks, reasoning, chain-of-thought and all sorts of logic. Code models are better at theory-of-mind tasks.

I suspect that even for law, you would train on code and actually get a better lawyer model. I’m actually going to test this hypothesis in the next test run. We're trying to build our new generation code model. We're trying to make it the best code model possible and we're going to train on all available code in the world. In addition, we want to train on other corpuses that are related to reasoning or logic – we believe law is one such thing. 

I encourage people, especially if they’re researchers, to dig into this question further. Before we get to the practical application, let’s stick with the abstract. I believe AGI or some kind of a general model can be reached through a coding model. Daniel Kahneman, author of Thinking Fast and Slow, developed this idea of system I and II thinking. System I is gut level, instinctual thinking. You come up with an answer quickly, but don’t know why you came up with said answer – neural networks tend to work this way. System II thinking is a discrete manner of thinking. You actually expend energy, sit down and contemplate the problem at hand. 

Neural networks represent system I thinking. If you want to induce system II thinking, you can generate code and evaluate it. This way, you can introduce that possibility for models. There's a great paper from Nvidia titled Voyager where they build a Minecraft agent that learns over time by writing and editing programs to become better at the game. They refer to this as non-gradient descent learning because it's actually learning, but doing so by updating a library of functions. Perhaps there’s a way to reach more general intelligence via this methodology. 

Now back to the practical…the first generation of copilots (i.e. Ghostwriter) are effectively typing games. They do great things, but are essentially a shortcut, or auto complete system. You can type less and it predicts what you're going to type – this gets you to 30-50% increased coding and productivity. But not everything is coding – during your day to day as an engineer, at best 15-30% of your time is spent coding. The balance is consumed by debugging, figuring out what to build, taking meetings, networking etc. The copilots are typing aids, but what if we can do more than that? Can you make it such that AI can write full features?

I have conviction this is possible. At Replit, we want to embody the agent inside the computing environment. A lot of these systems are currently just operating code. Instead, what if we re-embody them in the computing environment such that they can write to different read and write files and small packages from the internet, evaluate code, iterate on code and have a database for long-term memory. With this, we introduce the next generation of coding with LLMs – you can have a junior developer, maybe there are a handful, or potentially thousands. You choose the task you want to execute on, how much money you’d like to spend and maybe you spin up hundreds of agents to handle said task. 

The act of programming changes – you’re now akin to a project manager. While you’re still capable of coding, you're reviewing a great deal of code and often, you’re directing agents. There's a step beyond this, which will be 5+ years in the making at Replit. We published a manifesto articulating this vision. We believe in Artificial Developer Intelligence (ADI) – you get to a point where it's truly autonomous and AI inherently knows how to develop the system itself and improve upon it. We’re still far from realizing this reality – in the next year or two, we're onboarding more team members who will focus on AI exclusively. 

Replit offers you a very good vantage point on the citizen developer front – I'm curious what you’re seeing and what are the most interesting uses of foundation models that the community is leveraging or that you're seeing via bounties? What are some of the trends that are emerging?

A new class of entrepreneurs has emerged who are engineering adjacent talent. They're product managers and designers, building cool applications. 

There's a designer on Replit that's building a startup called Camp Lingo – reimagine Duolingo from scratch using LLMs. Camp Lingo generates stories on the fly, flashcards and flexible modalities you can leverage to learn languages. 

There are a lot of learning applications around tools for thought that are quite compelling. There was an explosion around this topic in 2019 and 2020 as Notion achieved scale and Roam gained a following. This trend was fairly short lived, but there’s a lot more to do here. LLMs empower engineers to think about tools for thought. Notepad, for example, is a visual canvas, but also harnesses AI to assist with mind mapping. I’m excited about this new class of entrepreneurs because they're typically not engineers and they're bringing diverse ideas to the table. 

How do you think about the opportunities associated with Enterprise customers? Are you interested in going upstream and if so, how do you plan on executing on this motion? 

The enterprise question is an interesting one – in programming there’s an idea of late and early binding. Early binding represents compiling a certain set of execution instructions whereas for dynamic languages like Javascript, there’s a dynamic dispatch such that you could monkeypatch a program on the fly. This gives you a ton of flexibility, which is why people like Python and Javascript because you can do a lot of things at runtime. 

Going into enterprise early is similar to “early binding.” A lot of companies end up set in their ways of doing things. Enterprise is building for the past, most of the time. If you want to build the future, instead of going and building for enterprise needs, there needs to be a pull because they want to be part of the future.

GitHub is a great example of this. They enabled the citizen developers and OS hackers at first – then every company wanted to take part. Companies wanted to hire talent from the GitHub ecosystem and actively contribute to open source. Many companies adopted GitHub in order to be part of the future. We’d like to take this approach to our segue into Enterprise. While it might take longer, trying to sell them an online IDE isn’t the best tactic seeing you’re directly competing with Microsoft, Google and other big players headed into the space.

At Replit, we want to invent a new way of doing software. A lot of the ways people are using Replit is highly untraditional. In the future, we want Replit to be the hub where most software in the world runs and that has to also be within the enterprise. You can call it bottoms up, but there needs to be a pull rather than a push. Developers are great at catalyzing this, as they pull the future within an organization. 

You mentioned the concept of abstractions and abstracting a lot of the technology in a way to make it more accessible. How do you find balance between abstraction and customizability? When you want to create an abstraction, how do you go about this? 

Taste is critical. From a statistical point of view, most abstractions are suboptimal – this is largely because people don't take a lot of time to actually build those abstractions. It took us over a year and a half to release a new file system. Replit used to be capped at one gigabyte project size and now it’s one terabyte. We had to build a dedicated remote NFS – it has copy-on-write semantics, replication semantics as well as multiplayer features. We took a year and a half to design and build this – it's a really great abstraction and is very transparent. Given this, I would disagree with the premise that with abstractions, essential complexity is leaking. Again, it's probably true in the general sense, but I think if you take your time and design things in a way such that it is actually a very reliable abstraction, you can end up with a very maintainable and scalable system.

What is your long term vision for Replit? 

I hope that the moment you have an idea, it’s ingrained within muscle memory to pull up Replit and materialize this idea into existence through our platform. We're building a ton of features on the community side such as bounties. While these features might initially feel disparate, we’re looking to integrate everything in order to create a holistic ecosystem. This ecosystem is not only composed of programmers, but the end users as well as the developers building on top of said programs. 

In five years, the world of software is going to fundamentally change – LLMs will be a key driver of this. Platforms similar to Replit will emerge, which will democratize access to programming. 

I believe that the creator economy and crypto will actually reach full maturity – X’s strategy around creator monetization will inspire many companies. Payments built into software platforms will change how we build software. It’s challenging to predict exactly what the future will hold, but we’re attempting to discover this in hand with whatever technology is emerging. We want to consistently be at the forefront of all these different technologies.