NetState, a test of whether AI can track a changing network
A generated benchmark that checks if AI can keep track of a network under attack.
I built a benchmark for the Executive Functions track of the Google DeepMind and Kaggle hackathon on measuring progress toward AGI.
It tests whether a language model can keep track of a network as things change. Ports open, privileges change, files move, and intrusion alerts fire. Some events change the rules partway through, and at checkpoints the model has to report the current state.
It comes in three sizes, from 3 nodes and 6 events up to 8 nodes and 20 events. The engine, the grader, and a set of 300 generated scenarios are done.
It is on hold. I have not run models on it or written it up yet.
- Scenarios300 generated
- StatusOn hold
