Building a Reinforcement Learning Library in C

Most people learn reinforcement learning through Python, PyTorch, and a stack of abstractions thick enough to insulate you from ever really understanding what's happening. You call env.step(), you call loss.backward(), something trains. It feels like magic because it basically is—you're trusting the framework to handle the hard parts.

Harsh Bhatt's new freeCodeCamp course does the opposite. Over two and a half hours, he builds a complete RL library in C. No PyTorch. No NumPy. No external ML frameworks of any kind. Just C, manual memory management, and a lot of struct definitions. The result is one of the more genuinely educational pieces of RL content I've encountered, and also one of the more honest about what "from scratch" actually costs.

What's actually being built here

The course has three major components, and they're deliberately sequenced to build on each other.

First: an autograd engine. This is the piece most people take for granted when they're working in Python. Autograd—short for automatic differentiation—is the system that figures out gradients automatically during the backward pass of neural network training. Bhatt builds his from scratch, including a computational graph that tracks every operation performed during the forward pass, then traverses that graph in reverse to compute gradients.

The graph traversal is legitimately interesting to watch. He uses a depth-first search algorithm with a deduplication step, explaining: "you will see like you know you probably find that okay when you go in the in this direction you will see you will check the inputs... you have to make sure that okay you are not adding the same parameter same variable." It's one of those implementation details that frameworks handle invisibly, and seeing it done by hand clarifies why it has to be done at all.

Second: a Snake game environment. The agent needs somewhere to live and something to learn. Bhatt builds a grid-based Snake environment from the ground up, including a state vector encoding that represents the snake's position, the food's position, and the snake's current direction of travel—all flattened into a single input vector the neural network can consume. The reward structure is simple: penalty for hitting a wall, reward for reaching food.

Third: the REINFORCE algorithm (a.k.a. policy gradient). This is the training method that actually teaches the snake to navigate. The agent collects trajectories—sequences of states, actions, and rewards—and uses those to compute returns and update the policy network's weights.

Put together, it's a complete end-to-end pipeline: environment, model, and training loop, all in C, all visible.

Why C specifically

The language choice is a deliberate instructional decision, not an exercise in masochism. When Bhatt acknowledges mid-session that it's 5:30 a.m. and he's debugging pointer arithmetic, you get a real sense of what the tradeoff looks like.

The point is exposure. In Python, memory allocation is invisible. In C, Bhatt has to explicitly build an arena allocator—a custom memory management system—before he can even define a variable. Every matrix operation requires implementing row-major indexing by hand. The four transpose variants for matrix multiplication (N-N, N-T, T-N, T-T) have to be written out explicitly with their different indexing formulas.

That's annoying if you just want a working RL agent. It's illuminating if you want to understand why matrix multiplication works the way it does, or what "backward pass" actually means mechanically. Bhatt pitches the course this way himself: "here we can actually go very deep and implement things from very scratch in C." The depth is the point.

The state representation question

Here's the part that genuinely occupied my brain after watching: the state vector.

Bhatt encodes the game state as a flat, one-hot vector—36 values for snake position, 36 for food position, and a few more for direction. It's functional, it's simple, and it teaches the concept. But one-hot encoding of grid positions has a real limitation that matters beyond this toy context: it treats every cell as categorically distinct, with no information about spatial relationships between cells.

A snake at position (2,3) and food at position (2,4) are adjacent. But in the one-hot representation, those two facts live in completely separate, unrelated slots in the input vector. The network has to learn—from scratch, through trial and error—that certain combinations of active slots in the snake-position chunk and active slots in the food-position chunk mean "the food is right next to you." There's no geometric prior baked in.

This isn't a bug in Bhatt's course. It's a tradeoff, and it's the right tradeoff for teaching purposes. You want to show how RL works without making the state representation its own sub-tutorial. But it illustrates something important about the gap between RL that teaches and RL that works: in production, state representation design is often where most of the real work happens. A distance vector, or coordinates normalized to the grid, would give the network geometry for free instead of making it rediscover spatial reasoning from raw reward signals. The one-hot version will learn—it'll just take longer and generalize less cleanly.

This is the kind of decision that's invisible when you're using someone else's environment. Building it yourself forces you to make the call, which is exactly the point of the exercise.

The REINFORCE tradeoff

Bhatt is upfront about the algorithm's limitations. "We don't have a value function in here which is great. Um but it will be very um high variance. So make sure like when we were going to train we'll see like a very high variance and it will train in a very long time."

REINFORCE is the simplest policy gradient algorithm. It works by collecting complete trajectories, computing the total return from each, and using those returns to figure out which actions were good. The problem: returns are noisy. A sequence of mediocre actions might look great if the agent got lucky at the end, and vice versa. This variance makes training slow and unstable.

The standard fix—adding a value function as a baseline, which is the actor-critic architecture—would reduce that variance significantly. Bhatt explicitly considered implementing it and decided against it to keep the scope manageable. That's a reasonable call for a two-hour course. But it means the training you'd see at the end is going to be choppier and slower than it would need to be in a real application.

The micrograd comparison

The obvious parallel here is Andrej Karpathy's micrograd—a tiny Python autograd engine that became a cult educational resource for exactly this reason: understanding backpropagation requires seeing the bones. Bhatt mentions being inspired by an existing ML library implementation video, though he doesn't name it. Whether or not there's a direct line, the intellectual kinship is clear. Both projects bet that building the foundation from scratch—even slowly, even painfully—teaches things that using a framework never will.

The difference is language. Karpathy's micrograd is Python: readable, low-friction, fast to iterate. Bhatt's is C: verbose, unforgiving, and pedagogically uncompromising. You learn more in C. You also spend more time fighting the language. Whether that tradeoff is worth it depends entirely on what gap you're trying to close.

Who this is actually for

Bhatt states his prerequisites directly: "I hope you have some understanding of the reinforcement learning how the agent loop works and how the agent takes action in the environment and environment gives the report to the agent."

That's real. This isn't an intro to RL. The concepts move fast and assume you already have a mental model of what an agent, environment, and reward signal are. The value-add is mechanical depth, not conceptual scaffolding.

If you're comfortable with RL fundamentals and want to understand what's actually happening under the frameworks you use, this course is a local's tour of the infrastructure—the service corridors and load-bearing walls that most tutorials never show you.

By Yuki Okonkwo, AI & Machine Learning Correspondent