TU.Diplomna

Two units on a 960 × 512 px arena divided into 30 × 16 tiles of 32 px. The outer three columns and two rows are lava; the interior is grass, with a one-in-ten chance of ice per tile. One sword lies on the map at a time. Walking over it adds 5 to the mass of your loaded bullet and spawns a new sword elsewhere. Bullets fly at 300 px/s with a 0.6 s cooldown. A hit applies a knockback force of bullet mass × 4000 × the victim's hit factor, then raises that hit factor by 0.2, so every hit makes the next one push harder. Lava drains 15 health per second. You win by pushing the opponent onto the lava.

Physics

• Units: mass 10, health 100, hit factor 1, control speed 200 px/s.
• Forces are composed by vector addition; kinetic friction per tile: grass 40, lava 40, ice 0.5, so ice really slides.
• Fixed timestep of 1/30 s. Training runs the same physics without a frame clock.
• Bullets are units with isFlying set: no friction, no lava damage, removed when they leave the screen.

The environment interface

The environment is written in the shape of OpenAI Gym, which was a year old at the time. env.step(events) returns (state, reward), env.restart() resets, env.checkWin() ends an episode when either unit's health reaches zero. Rendering lives in a separate DrawEngine, so training runs headless at whatever speed the CPU allows and the game is drawn only afterwards. The Gym loop is pasted into game_engine.py as a TODO comment, which is an honest description of how far the resemblance goes: there is no done flag and no info.

The opponent

bot.py is 25 lines. It walks along whichever axis is farther from the sword and fires at the learner's centre on every frame, limited only by the cooldown. It never dodges. Because it monopolises the swords, its bullets get heavier over a game. The learner is UNIT_2, has no shoot action, and can only move.

The state is 13 hand-picked features per unit, for both units, in a fixed order: 26 numbers. Positions are fed as absolute x and y rather than as distances. The README states why:

“I am feeding the NN with (X,Y) values instead of distances so that the NN has to understand the concept of map positions.”

The learner never fires, so its own six bullet features stay at zero. The bot's bullets enter through the bot's block, sorted by distance to the bot, which is where they were just launched from; that is the signal a dodging policy has to work with.

Four actions, encoded as a bitfield (up 8, down 4, left 2, right 1). No diagonals are used and there is no shoot action. The reward for one unit is

reward = 0.7 · (1 − dps/15)
       + 0.3 · (1 − clip(hit_factor, 0, 30)/30)
# health and sword-mass terms exist, weighted 0
# unit_state.py:61-68

which is 0.7 for not standing on lava plus a share that decays as the unit accumulates hits. The environment reward is reward(unit 1) − reward(unit 2). There is no terminal bonus and no done flag.

No framework. nn.py is a numpy class with four weight matrices, four bias rows and a hand-written backward pass. Every layer, including the output, is tanh, so each Q-value is bounded to (−1, 1). Weights start uniform in (−1, 1) under np.random.seed(1), with the comment “Important for debug, random gets deterministic!”; np.seterr(all='raise') turns any numeric warning into an exception. Loss is half the squared error; the optimiser is plain gradient descent.

The per-layer multipliers are the last change made to the network:

# nn.py, lines 97-105
self.wx1 += deltaW0*self.lr
self.w12 += deltaW1*self.lr*10
self.w23 += deltaW2*self.lr*100
self.w3y += deltaW3*self.lr*1000

With one shared learning rate, the backpropagated error shrinks through each tanh layer and the output side barely moves. Scaling the step by 1, 10, 100 and 1000 from input to output compensates by hand. The base rate was cut to 1e-4 and the net narrowed to 26 → 10 → 6 → 6 → 4 in the same commit (11 Sep 2017). The commit message calls it removing weight decay; there was never an explicit decay term in the code, so what was removed is the implicit decay of the step size with depth.

Before the class was trusted with Q-values it was validated as a plain regressor on √(x² + y²), on a sine over [−20, 20] and on x/2; the harnesses survive as comments at the bottom of nn.py. Earlier still, network.py derives backprop for a 2-4-1 net by hand, one partial derivative per line with the chain rule written out in comments, then repeats the computation in matrix form directly beside it and compares the two. That comparison caught an index error in the scalar version.

The variable eps is the probability of taking the greedy action, the opposite of the usual convention: 20 % greedy for the first quarter of training, 30 % after, 100 % once training ends. A random action is held for five frames so that it has a visible physical effect at 1/30 s per step; the network is updated only on those frames. So the behaviour policy is mostly random, the update is toward the greedy value, and the net learns only from exploratory transitions: off-policy in the strict sense.

A 20-sample minibatch was tried on 17 August and dropped two days later. The commit message, in Bulgarian, records the reason: on batches the updates average out, without them the network “corrects on every move”. With only the taken action's entry carrying new information, summing twenty targets does smear the signal across all four outputs.

Cost is printed every 10,000 steps and plotted with matplotlib when training ends; then the loop keeps going with rendering on, AI against bot at 30 FPS. Nothing is saved: the weights live only in that process. No cost curve is reproduced here. 5,898 curve images from runs past 10.8 million steps were committed and deleted the same day (commit 810a5427), and no numbers survive in the repository.

The first learner, in April 2017, was a Q-table. 120 position states aliased and failed. 480 states, one per tile, with a terrain reward and count-based exploration, failed too. What worked on 3 April was collapsing the state to five values: which lava band the unit is in, or the interior (lr 0.1, γ 0.5, ε 0.5). A visualiser coloured every tile by the mean Q of its state, green positive and red negative, which is the display the live demo above re-creates.

Five states cannot express a bullet, an opponent or a sword, which is what pushed the project to function approximation. The README, later:

“Initially I have implemented it with Q-table but wasn't sufficient due to big state space, so I have decided to go for NN to replace the Q-Table.”

A scikit-learn MLPRegressor was tried first, in July, and replaced by the hand-written network in August.

From the README, December 2017:

“My success with the AI is that it learns to avoid bullets. Can't shoot yet.”

A video of the learned agent is linked from the README on Google Drive; the link is reproduced as recorded and was not re-verified for this page. There is a mode to play against the trained agent, switched on by editing main_nn.py.

What the repository does not contain: saved weights, tests, a command-line interface, or a thesis document. Of its 672 MB, almost all is a committed virtualenv and the Dungeon Crawl Stone Soup tileset (3,054 files) used for the sprites; the Python is about 1,500 lines, comments included.

Thirty-five commits between 17 February and 4 December 2017. Messages are paraphrased; dates and hashes are as recorded. The two marked commits are the ones that made the network learn.

A read-through of the final commit, nine years on. The first table is what the code does differently from the 2015 DeepMind paper it cites; the second is what a code review would flag.

None of this changes what the repository shows: a physics engine, an environment, a tabular learner, a network derived by hand and a Q-learning loop around it, built and debugged in one year with the failures left in the history.