Post
22
I trained a chess network with no search. One forward pass, 6.9 ms, no tree, no book.
Then I spent a day finding out most of what I believed about it was wrong.
The instrument: 23,845 Lichess puzzles, sampled per cell of (solution length × rating band), because rating and length are confounded. Lines are forced — the network is queried at every move even after it has erred, so no step is measured on a pre-filtered sample.
My first reading was backwards. I expected accuracy to collapse with solution length. At fixed rating it rises: a 2400-rated mate-in-1 is the hardest cell in the grid. Rating conserves difficulty, length dilutes it.
The real signal was elsewhere. Five-move lines: 66% on the entry move, 91% on the last. Quiet entries at 2400+: 32.8%. The network executes tactics. It does not initiate them.
Then I added Squeeze-and-Excitation. At 96×4 it recovered 62% of what four extra layers had bought, for 34% of their compute. At 128×8 the effect halved and the profile inverted — gains on captures, nothing on checks or quiet moves. Not an attenuated effect. A different one. Both my stated predictions failed, one in sign.
The part that matters: I had fixed Holm-Bonferroni over the family of eight comparisons before looking. It killed two of my four positive results, including a +26 Elo duel at p = 0.028 I would have written up as a win.
The real lesson:
Building a sensitive instrument does not protect you from reading it backwards. What caught the errors was predictions written before measuring, a threshold fixed before looking, and three yardsticks that disagree often enough to keep all three.
Every number, including the ones that contradict me 👇
🤗 RDTvlokip/RDTChess
💻 https://github.com/RDTvlokip/RDTChess
♟️ https://lichess.org/@/RDTChessBot
Then I spent a day finding out most of what I believed about it was wrong.
The instrument: 23,845 Lichess puzzles, sampled per cell of (solution length × rating band), because rating and length are confounded. Lines are forced — the network is queried at every move even after it has erred, so no step is measured on a pre-filtered sample.
My first reading was backwards. I expected accuracy to collapse with solution length. At fixed rating it rises: a 2400-rated mate-in-1 is the hardest cell in the grid. Rating conserves difficulty, length dilutes it.
The real signal was elsewhere. Five-move lines: 66% on the entry move, 91% on the last. Quiet entries at 2400+: 32.8%. The network executes tactics. It does not initiate them.
Then I added Squeeze-and-Excitation. At 96×4 it recovered 62% of what four extra layers had bought, for 34% of their compute. At 128×8 the effect halved and the profile inverted — gains on captures, nothing on checks or quiet moves. Not an attenuated effect. A different one. Both my stated predictions failed, one in sign.
The part that matters: I had fixed Holm-Bonferroni over the family of eight comparisons before looking. It killed two of my four positive results, including a +26 Elo duel at p = 0.028 I would have written up as a win.
The real lesson:
Building a sensitive instrument does not protect you from reading it backwards. What caught the errors was predictions written before measuring, a threshold fixed before looking, and three yardsticks that disagree often enough to keep all three.
Every number, including the ones that contradict me 👇
🤗 RDTvlokip/RDTChess
💻 https://github.com/RDTvlokip/RDTChess
♟️ https://lichess.org/@/RDTChessBot