Our genre model has five scores and behaves like three
I audited the genre model we use to decide what to build, expecting to find one weak column. Instead I found two columns that are really one column, a third that is mostly the second one wearing a hat, and a composite whose entire range is narrow enough that the gaps I had been reading as signal are mostly noise. I did not enjoy any of it. I have not fixed most of it.
Why audit your own scoring model
We score every genre on five things from 1 to 10: how validated the mechanic is at home, how crowded the supply is where we would ship it, what it costs one person to clone, how much the appeal depends on language, and how much domestic compliance risk it drags along. Multiply by weights, add them up, sort. The whole scheme is written up in how we score game genres. Three hard rules outrank the result. A difficulty of 8 or more is a veto. Crowding of 8 or more means do not enter. An empty cell is not automatically an opportunity. You still have to explain why it is empty.
I built all of this, which is the problem with auditing it: every score is my own judgement written down as an integer, and the composite is a weighted sum of my opinions in a format that makes opinions look like measurements, and I am not immune to my own formatting. I have already written about being the worst judge of our own difficulty column, and that piece ended with a trick rather than a fix: fill in difficulty before you look at validation, so you do not yet know which answer you want. This audit was the next question. Are the five columns five things at all?
Difficulty and compliance risk are one score
Pearson correlation between the difficulty column and the compliance risk column across our fifteen rows: +0.809. Spearman: +0.851.
One row, I assumed, was carrying it, so I dropped each row in turn and recomputed on fourteen. The range comes out 0.773 to 0.893, lowest when retro-casual is dropped, highest when light-horror is. It is not one row doing the work. It is the column.
Party competition scores 9 on difficulty and 7 on compliance risk. Survival SLG, 9 and 7. Merge with story, 7 and 6. Merge tower defense, 6 and 6. Runner, 5 and 5. Stress relief toys, 3 and 4. Sorting puzzles, 3 and 5. Read down that list and the two numbers move together almost every time, and where they disagree the gap is two points at most.
I think both columns are proxies for the same underlying thing, which is how much institution a genre needs. Real-time multiplayer needs netcode, and it is also a moderation problem and a licensing problem, and those are not a coincidence, because they are the same institution seen from two sides, while a genre that wants an endless pipeline of content wants an endless pipeline of reviewable assets, and the reviewing is the part one person cannot do. I did not set out to score how much company a genre requires twice. That is what I did.
The window column is the crowding score wearing a hat
We also keep a window status on each row: open, half-open, closed. I had been treating it as a separate judgement. A sense of timing that the five numbers did not capture. I had been treating it as evidence. So I correlated it against every score, coding open as 0, half as 1, closed as 2.
Crowding comes back at +0.807, Spearman +0.824. Everything else is nowhere near it: validation +0.302, difficulty +0.135, localisation −0.210, compliance risk +0.043.
Seven of the fifteen rows are closed. The crowding values sort almost cleanly by group: 3, 6 and 6 for the open rows; 5, 5, 6, 7 and 8 for the half-open ones; 8, 8, 8, 8, 9, 9, 9 for the closed ones. Group means are 5.00 open against 8.43 closed.
It is not clean, and I would rather say that plainly than let the correlation do the arguing for me, because a number like +0.807 is a very good excuse to stop thinking, and it is an excuse I have taken before: two open rows sit at a crowding of 6, which is inside the half-open band, and one half-open row sits at 8, which is the number that triggers do not enter. If the two columns were literally the same field then those rows could not exist. Something else is in there. What I think happened is that I read the crowding number and then wrote the window status, and mostly wrote down what I had just read, so the column may contain a little timing, but it mostly contains crowding.
This is a different question from the post about a window closing, which is about how fast supply moves. This one asks whether the window column measures anything the crowding score does not. Mostly it does not.
The composite barely moves
The composite runs from 4.56 to 6.84 across fifteen rows. Standard deviation 0.56. On a 1-to-10 scale that means the sort order near the top is being decided by differences I have no business reading.
The reason is that the terms fight each other. Validation and crowding correlate at +0.334, so the validation term pushes up exactly where the crowding term, which enters the sum inverted, pushes down. The share of the composite's variance attributable to each term comes out as validation 130.5%, crowding 74.6%, difficulty 63.8%, localisation 17.7% and compliance risk 3.8%. A single term can exceed 100% here because each share is the variance of that weighted term against the variance of the total, and the terms are correlated, so the column does not add up to anything. What that list actually describes is a tug of war between validation and crowding with three other terms holding the rope, and the rope barely moves.
I had been reading a gap of 0.2 in that column as a real preference. It is arithmetic noise with a decimal point on it, and I had been making plans on the strength of it, which is the part that stings.
The term that decides everything and explains nothing
Compliance risk carries a weight of 0.08 and contributes 3.8% of the variance in the number we sort by. By every measure internal to the model it is the least important column we have.
It also decides where a thing could legally ship, and sometimes whether it can ship at all. No weighted sum can express that. A weighted sum assumes every term is compensable. A bad score here can be bought back by a good score there. Compliance is not compensable. You cannot out-validate a licence.
So one column duplicates difficulty, carries the smallest weight, moves the composite almost not at all, and still decides where a project could be published. I find holding those four facts at once genuinely uncomfortable. I have not resolved it.
Why we did not merge the correlated columns
The obvious move after a result like this is to collapse difficulty and compliance risk into a single score, or delete the window column as redundant. We did not. The reason is not statistical. Difficulty is about how long a genre takes to build. Compliance is about whether it can be published at all.
Redundancy in a measurement is a bug. Redundancy in a checklist is a feature when the consequences cannot be written as one number. Difficulty asks whether I can build it. Compliance asks whether it can exist. In our data they have the same answer almost every time. I still want to be asked both. The day they come apart is the day I most need to have been asked. The correlation describes the fifteen genres currently in the table. It is not a law about genres.
The window column stays for a weaker reason: the rows where it disagrees with crowding are the rows where I most want somebody to argue with me.
What this kind of audit cannot tell you
None of the above says the model is wrong about the market. Every number here is a correlation between my own integers. Fifteen rows of subjective scores is a small thing to compute anything on. I would not defend these past one decimal place. I would not defend the second one at all.
What an audit like this can do is tell you where the model is pretending to more independent information than it has. Five columns, behaving like three. That is worth knowing even though the fix is not to delete anything. The uncomfortable part is that I will keep using the composite to sort, knowing the top of that sort is mostly noise, because the alternative is no ordering at all. I read the hard rules first. I sort by it anyway. I keep a note on my desk with the standard deviation written on it.