What to know when developing with ChatGPT without any engineering experience in 2024
A hands-on experience on the capabilities of LLMs in code
As someone who had never touched a line of code, ChatGPT felt like a magic gateway into a previously inconceivable world. It made it easy to have code written and explained to me line by line.
Here are some takeaways from building my MVP Leo with ChatGPT!
It’s good at helping you get started when you “don’t know what you don’t know”
Before ChatGPT, as someone who is not a developer, I would’ve faced several knowledge gaps before getting started on any project. For example:
How and which software tools do I need?
What underlying technology do I need to build this?
Do these technologies exist?
The way I would’ve figured out how to build an app would be a combination of various Google searches, talking to several people in the field, and going through trial and error to get to the right technology.
What to know #1
It’s easier to get started on technical projects if you haven’t been in the industry
With ChatGPT, all you have to do is “how do I build something that does x?” For my project, I asked ChatGPT this question:
“I’m building an app where figure sketching students take a picture of a sketch they drew and the app gives feedback on how to make the drawing more realistic. How would I go about building this app using AI?”
From ChatGPT, I learned that I needed the technology for image comparison and analysis, which pointed me to “pose estimation models,” machine learning models that predict the positions of a person's joints and limbs.
Just a few days before asking GPT about this, I had consulted a friend who worked in computer vision for self-driving cars, had a master in ML, and had been working in ML for the past several years, and he never brought up pose estimation models. The ideas we discussed used 3D modeling and generated ML from 3D models. ChatGPT has a much broader database of information that it can source from – things that more specialized engineers might not know about because they are so focused on specific models of thinking, working, and researching.
My S/O had a similar experience when developing his app with his cofounder, who is a developer with 10 years of experience. When facing similar technical challenges, his cofounder would think (as a good engineer would do) from the ground up, really understand the problem and fix it with a complex solution that takes a few days, whereas ChatGPT gives a hacky answer and my S/O (with 0 engineering experience) can fix the problem in a matter of a couple of hours.
For example, when building an app for language learning, they were looking for a technology to parse Chinese. This is a very challenging technical problem if you were to think about it from the ground up. But ChatGPT will tell you that what you want is a “Chinese word parsing library,” and that you need Chinese text segmentation tools that exist in the market like thulac and jieba. Before ChatGPT, you would have had to do some serious digging around on Google and might even conclude that this didn’t already exist and try to build it themselves. (By the way, if you are a Chinese learner – check out Ushabti!)
It’s good at memorizing syntax rules and filling a blank page for you
Gone are the days where you have to translate natural language into code and memorize all the syntax rules of development languages. Ask ChatGPT to give you code to do any specific task, and you got it.
What to know #2
You no longer need to go through a full coding course to write code
This turns weeks of frustration with being constantly blocked by problems, extreme will to get past what feels like dead ends, spartan-like discipline to learn and memorize syntax of developing languages, into just a few hours of conversation with GPT.
When you know that you can make definite progress on each of the bugs and functionalities of your app when you sit down everyday, it is much more enticing to get started on your work.
It’s not good at building multiple functionalities at a time
When ChatGPT generates code, a lot of it produces errors. If you ask it to write code that performs a sequence of functionalities, it would be very inefficient to debug the script. This is because:
It is unclear which part of the code performs what functionalities
Errors from one function cascades into other functions
What to know #3
You need to break down each functionality for ChatGPT and test them individually
I broke down the functionalities for my app into many mini steps for GPT. For example, just for the first step of a user onboarding flow, I broke down the steps for the discord bot to:
Directly messages the user with a welcome message when a user joins
Add a button that says "give me a pose" to the welcome message
Pressing this button triggers the bot to find a random image from the folder of images and send it to the user
Create a log in the directory that saves the name of the image sent to the user temporarily.
I asked the bot to give me 1 or 2 of these functionalities at a time in python script, ran it, and then debugged it.
What to know #4
You need to have a basic understanding of what each line and section of code does
Breaking functionalities down also familiarizes you to each part of the code, and is also helpful as you add more functionalities. When copying your script back and forth on ChatGPT, you want to be careful to only copy over the parts of the script that are new functionalities or specific parts that modify old functionalities. The easiest thing to do is to copy and past an entire script and ask for the changes. But ChatGPT will inevitably break or change the existing features in the script.
Additionally, ChatGPT will sometimes generate errors that it doesn’t know how to fix, and you can’t really know what’s going wrong without having some basic understanding of different functions and what they do, or how different libraries might be used. For example, I was getting consistent errors with the way the discord button was being written. I kept copying and pasting errors to ChatGPT to diagnose the problem but it kept outputting similar solutions that didn’t work. I finally fixed the problem by going through Stack Overflow and the Discord API documentation.
It’s not good at thinking out of the box
What to know #5
Don’t stop using Google and community forums
ChatGPT might not always generate the most optimal solution to a problem. When I first started looking for pose estimation models, I realized that the ones that are popular are all trained off of live-action images, and are very bad at detecting joints and limbs on sketches. When I brought this problem up to ChatGPT, I could not find any model that was built off sketches. I kept asking it for pose estimation models that were more accurate, and it kept outputting the same answers – the popular industry standard models. Until finally, I decided to fine tune my own model.
It took me $350 and about a month to label 500 AI generated sketches to try to fine tune existing pose estimation models using sketches. And then to try to build my own image generation model with a specified style to add to the amount of sketches in training. I realized that I would need over thousands of images labeled to fine tune the model. When I had almost given up, I turned to Discord communities and started looking for other open source projects. Then, I finally found a model that was trained just for sketches. I just wasted 2 months!
What to know #6
Still consult the experts
Another example is setting up the backend. Before I even put all the pieces together for my discord bot on a local server, I asked ChatGPT what I should do to launch this product on a remote server. This was what I asked it:
This is my first time building a discord app and working on a backend.
How would I set this up in a remote server? Where do I store all the files? What kind of remote server do I need? Also, how do I save the files that are user-specific? Each user’s sketch upload and reference image sent by the bot need to be associated with their specific ID so that the bot knows what to compare.
The rabbit hole I got into was how I would optimize running the app, which required a high amount of compute and storage space. After a day’s worth of research and frustration, I talked to my friends and they told me to just build it first on my local server and worry about all this later. Is this painfully obvious? Yes, but I really believed that I needed to put it on a remote server first and ChatGPT was only focused on helping me solve that problem.
What this means: GPT makes easier for non-technical people to launch MVPs
ChatGPT cannot currently do all the code work for you at this stage, but it does make it exponentially easier for someone who has had no experience with code to launch an MVP. It took me about a month to build the discord bot from scratch (and two months trying to train my own model). Now with a better idea of the capabilities of ChatGPT, development could move even faster.
I've seen many friends and peers, including product managers, marketers, and even lawyers, launching MVPs. Previously blocked by development challenges, those who don’t have a technical background can now bring their creative solutions and ideas to market.
This was my experience developing with ChatGPT. I’m curious what other insights people have on developing with GPT and other LLM models. Reach out to me if you’re interested in chatting!
Glad to see you were able to use GPT so effectively with what you’re working on. It’s amazing to wonder what these LLM’s capabilities will be like in 10, 20 years, especially regarding coding.