learn to code with LLMs? i read a bunch of learning science so you don't have to.
also, eating chess pieces and many, many qr codes
i’m currently doing a programming retreat at Recurse Center (which you know since i didn’t shut up about it in the last three issues here).
when my mind is not occupied with devouring pawns, my main goal is to learn Rust.
and booyyy have things changed since i properly learned a programming language for the last time.
What?
LLMs, of course.
i’m using them a lot. But is it worth it? Or am i cheating myself out of this opportunity?
it’s time for some housekeeping and critical reflection.
the capital S Science of AI powered learn to code
since i am a scientist at heart, we start reading some papers about the effects of LLMs on learning to code before i dump some of my own ideas.
here is what i found:
yes, using LLMs (the right way) results in better skill development
luckily for me, there is significantly higher skill development for students who use AI to assisting their learning.
effects are greatest when students ask conceptual design questions and they need and build skills in reading and evaluating code which previously were only becoming important far later in student’s careers. when students outsource the whole code writing to an LLM, they perform far worse than control (duh!) and generally, not paying forward for getting the solution to a problem with solving something similar seems to damage learning.
but you profit most as a newbie
however, it’s the students with lower grades use AI more and they reduce their AI use as they gain proficiency and confidence in their own abilities.
the main benefit of LLMs seems to be their support for self directed learning and the ability to reduce emotional barriers by helping when stuck.
you learn faster, but only deeper if you engage critically
students using LLMs increase the volume of material they can cover in a timeframe but they only increase their understanding if they engage critically by evaluating output and asking questions.
generally, while using LLMs improves programming skill compared to solely teacher led instruction, they don’t change programming knowledge. they also may damage skills in information retrival and critical evaluation.
of course all these studies have limitations and were mostly conducted on novices, high schoolers/students, and suffer from small sample sizes. but all in all, LLMs seem to be an improvement in learning to code.
the most important thing seems to be that you don’t let it become a substitute for your own work but engage critically with what the LLM provides.
what did i build? how? and how did it go?
i’ve built three rust programs until now. one ASCII art generator and two blockchains.
why two?
i built the first incrementally while ping-pong-ing with Claude on design decisions.
of course i know it’s bad to play copy-paste-code-gen slot machine. so i had the hard rule to not copy code from the LLM into my codebase.
still, at some point the project got too big and Claude guided me into the spaghettified mess of an incrementally extended codebase built by someone who learnt as they went.
so i started over using Building Bitcoin in Rust a book that frontloads the theory and builds up the project with a proper structure and code organization in mind.
this was very valuable, but it was more difficult to force myself to not zone out when typing ‘cause i wasn’t as suspicious about the book as i was with Claude.
in both cases i saw and wrote many lines of code.
i definitely learned a lot about Rust and blockchains (everything else would be very, very sad) and verified this by forcing go cold-turkey on LLMs for a bit.
googling for solution like a pre-chatGPT caveman, has two surprising benefits though.
you put more effort into understanding what your problem is so that you know what to search for and you also find a lot of ways that you think should work but don’t.
this is something you don’t learn with an LLM which may get things conceptually wrong but almost always syntactically right.
and you do build intuition by struggling with syntax or the borrow-checker even when it slows you down.
ad-hoc classification of ways to code better
in the end i think there are different benefits to the different ways of (un-)assisted learning with LLMs:
LLM ping pong makes you fast. you see a lot of code and tailored solutions to the problem you are working on right now. it’s really good in the nitty-gritty but gets lost in big conceptual things. it can explain on demand, which is nice…
code-along books and tutorials. if they are of decent quality (many aren’t, especially online tutorials), they explain the necessary theory and build someting in a thought through way. since you know it’s correct, it’s harder to stay critically engaged with the content rather than just zone out and type. you still have a high coding throughput.
foundational books. they only teach you knowledge, not skill. but thou shalt know thy tools, so better read them even if not for immediate benefit… it will pay off, i’m sure.
old fashioned, pre GPT, caveman coding. this is slow and frustrating and you mainly discover a lot of ways that don’t work. but that’s also important to know
i’m pretty happy with where i got my Rust so far, but will write without AI help for a bit to reap the benefits of the struggle and go deep to develop intuition.
i guess, i will go back to the LLM when i’m optimizing for breadth and speed again.
but above all else, i believe one thing holds true: repetition is the mother of didactics.
the people who grow the most at Recurse are the ones who write the most code and ship.
so i better keep my fingers busy.
btw. if you have any experience on how i can accelerate my learning and get better faster, please tell me. i’m on board with building proficiency in a decade rather than 3 months — but if i can become faster, i want to!
NY life update:
the last weeks were fun.
i mostly spent my time building my second blockchain, but i’ll spare you the details.
today there was a QR code exhibition at the Recurse Center and it had some real gems:
there were some other really cool ideas which i sadly did not capture on camera, like a snake game you controlled by waving qr codes in the webcam or a super-mario game you controlled with manipulating a qr code.
there were also a lot of fun easter eggs when you scanned them.