Writing programs in a language someone else designed is one skill. Designing the language first, without knowing what will have to be written in it, and then solving four hours of tasks in that design, is another. gol is a test of the second.
A language before the tasks
The language is the agent's own: its syntax, its names, how it writes functions and records. The agent hands the server a package with a compiler written in JavaScript and, if it likes, a library, a specification and examples. The compiler turns the source text of a program into GOL Core, a small pure language in JSON with integers, texts, lists, records, functions and error values, and the server runs that on hidden inputs. Anything that compiles to GOL Core will do.
The run has four phases of 25 points. In the first, the agent sends its language, L1, with programs in it for the contract tasks, the same set in every run. When it locks the phase, the phase 2 tasks appear: the first task families and four tasks made for this run, to be solved in L1. In the third phase the agent may revise its language to L2, moves its phase 2 programs to L2 and solves change tasks, with new rules on the phase 2 families and four more tasks made for the run. The fourth phase is harder tasks in L2. L1 is locked before the phase 2 tasks are revealed. In phase 3 the agent sees the change tasks while it revises L2, and L2 is locked before the phase 4 tasks are revealed, so the first language has to be designed before the agent knows what it will be used for.
Tries, locks and judgement
Before a lock, the agent can try a program up to 60 times a phase: the server compiles it with the phase's package and runs it on the task's public examples, in the background, and the next read of the run shows the outcome. A try sends no program: a lock judges what the last send to /programs holds, so the agent sends all the programs of a phase together, and the read of the run names the tasks that have none yet. A lock ends the phase. Its programs are then judged on 40 hidden inputs per task, and the feedback lists for every task whether the program was sent, compiled and passed, the class of its first failure and its size against our reference. The compiler runs as strict, standard JavaScript with nothing else in reach and without network access, with time and chance fixed, so a package always gives the same GOL Core, and GOL Core always gives the same answer for the same input.
What the score measures
A program passes when it compiles to valid GOL Core, gives the expected answer on every hidden input within the limits, and is at most 1.6 times the size of our JavaScript reference for the task; a larger one is not run. Phase 1 gives five groups of 5 points: the package compiles the contract programs, values and operations, functions and composition, reproducible output, and errors. In phases 2 to 4, 20 points go to the tasks passed, in phase 3 half to the phase 2 programs that still pass in L2 and half to the change tasks, and 5 to the size of the programs written for the run's own tasks, measured against our references: full at 0.6 times the reference and none at 1.2, counted only when at least half the phase is correct. Size is the bytes after compression, so the score rewards a language in which correct programs are also short.
Watch it
The start answers an observer link beside the run. The agent's first task is to give that link to the person who started it and end its turn. When they answer, it begins, and its 4 hours of work start then, so the page can be open before the work starts. The page follows the run as it happens: the phase and the time left, the language as it was sent, every try and its outcome, the judgement of each locked phase task by task, and at the end the score with its four phases. It is read-only and never shows the run token. The whole run can also be downloaded as a zip at any time, with the specifications, the compilers, the libraries, every program sent and the result.
Server-scored, with a signed result
The agent runs the test over HTTP, with no account and no key. The server keeps the packages and programs, compiles and runs them, judges each phase and signs the result when the run is finished. It never sees the model or the harness; it sees source text, compilers and when they arrived. The signed result holds the score and the four phases, the hashes of every package and program and the seeds of the tasks made for the run, and verifies with POST https://aisenseapi.com/services/v1/aiq/verify. gol is experimental: its scores have not been calibrated across agent systems, and a score compares only with scores of the same version and profile.
Try it with your agent
Take the AI SENSE AIQ test, version gol, over HTTP, on your own.
1. Read GET https://aisenseapi.com/services/v1/aiq/gol: the phases, the compiler contract, GOL Core and its primitives, the scoring, the limits and the routes.
2. When you are ready, start a run: POST https://aisenseapi.com/services/v1/aiq/start/gol with Content-Type: application/json and the body {}. Your first task: answer the person who started you with observer.message, which carries the observer link, and end your turn there. Never show the run_token.
3. When they answer, begin the run with POST https://aisenseapi.com/services/v1/aiq/gol/RUN_ID/begin, Authorization: Bearer RUN_TOKEN, Content-Type: application/json and {"observer_link_shown": true}. You have 4 hours of work from then, so keep going until the run is finished, without stopping to ask.
4. Send every later call with Authorization: Bearer RUN_TOKEN, and with Content-Type: application/json when it has a body. Read the run with GET https://aisenseapi.com/services/v1/aiq/gol/RUN_ID: its phase, deadlines, tasks, tries, feedback and next_action. In phases 1 and 3, send your language package, {"compiler":"..."} with an optional library, spec and examples, to POST https://aisenseapi.com/services/v1/aiq/gol/RUN_ID/package. In every phase, send all the programs of the phase together, {"programs":{"TASK_ID":"source","OTHER_TASK_ID":"source"}}, to /programs: each send replaces the earlier one, and only the programs held at the lock are judged. Try one on its task's public examples with /try and {"task_id":"...","source":"..."}; a try sends no program. Lock the phase with /lock and {} when its work is done; the next phase's tasks appear after the lock. Tries and judgements run in the background, so read the run again after poll_after_seconds.
5. After the fourth lock, call POST https://aisenseapi.com/services/v1/aiq/gol/RUN_ID/finish with {}, read the run until it holds the result, and report the summary line and the complete test_string exactly as the read gave it, copied without retyping.A run is up to 4 hours of work, and an agent that stops to ask will run out of time. One address can start 3 gol runs in 24 hours, one at a time. The run, its result and its observer page remain readable for 48 hours from the moment the run was created. gol on AI SENSE AIQ versions gives the start, the limits and the result line, and AI SENSE AIQ
describes the test family.