Methodology
How a GoShip Score is computed
20 October 2026
A GoShip Score is a number from 0 to 100 attached to one recorded session on one challenge. This page shows where that number comes from, so a shipper can see what to change and an employer can judge how far to trust it.
What is recorded
The session recorder runs on the developer's own machine, inside the tool they already use. It records events, not source: prompts sent to the AI, the AI's turn boundaries, which files were created or modified (paths and line counts, never contents), terminal commands with their classification (test, build, server, other) and whether they passed, git commits, and a heartbeat every thirty seconds. Files matching .env*, keys, certificates and credentials are never observed. Secret-looking strings are redacted from file events; prompts are sent as typed, so keep secrets out of them.
The recorder installed by the setup command today (version 0.3.3) captures through its MCP tools, file watching and shell history. That means some prompts are only recorded when the model calls the tool, and event batches are not signed. Version 0.4.0, which captures through Claude Code and Cursor hooks on every prompt, tool call and stop and signs each batch, is built and will replace it. Every session records which recorder version produced it.
Six dimensions
| Dimension | Weight | Sub-scores | Model review |
|---|---|---|---|
| AI Fluency | 20% | Prompt specificity, Iteration efficiency, Context management, Tool orchestration, Model selection | yes |
| Output Quality | 25% | Functionality (tests and builds), Code quality, Architecture, Security, Completeness | yes |
| Judgment | 20% | Override quality, Bug catch rate, Security awareness, Manual intervention | yes |
| Speed | 15% | Time to functional, Iteration velocity, Time efficiency | yes |
| Prompt Precision | 10% | First-shot success, Iteration depth, Context provision, Prompt-to-action ratio | no |
| Recovery Speed | 10% | Time from a failing test or build to the next passing one | no |
Each dimension is first scored by rules over the event stream (the sub-scores above, each with its own weight and a written reason). The composite is the weighted sum of the six dimension scores.
The AI blend
For AI Fluency, Output Quality, Judgment and Speed, a model also reads a sample of the session (the first and last interactions and the most informative middle ones) together with the rule-based scores, and returns its own scores and a written summary. The final score for those four dimensions is 70% model and 30% rules. If the model disagrees with the rules by more than 25 points on any dimension, the blend drops to 60/40. Prompt Precision and Recovery Speed are rules only.
When the model is unavailable, the score is rules only and is labelled rule-based rather than hybrid on the score page. No summary is written in that case.
Comprehension check
After scoring, the model writes a short questionnaire about the code the session produced. Answering it shows the shipper understands what was built. It is a comprehension check, not proof of identity or proctoring, and the site does not call it verification.
Integrity report
The recorder is on the developer's machine, so nothing it sends is proof by itself. What the server can check is recorded with every score and shown beside it:
- Events arrived without a client signature
- More than one client version sent events
- Client clock differed from ours by more than five minutes
- Events arrived out of chronological order
- Event rate exceeded what a person produces
- Long stretches without a heartbeat
- No session start was recorded
- No session end was recorded
- Session shorter than three minutes
- No prompts were recorded
A flag is evidence for a reviewer to weigh. No flag changes the score. Sessions with flags are listed for the GoShip team to look at.
Known limits
- Rules reward observable behaviour, such as running tests, rejecting or editing suggestions, and giving the model file paths and acceptance criteria. They cannot see intent. A session that accepts everything and never runs a test is capped, not zeroed.
- The rules were written before real sessions existed and are being re-derived as recorded sessions accumulate. Scoring version is stored with every score so that re-scoring under a newer version is visible.
- Comparisons are fair within a challenge and a difficulty band, not across them.
- A modified recorder can send events that never happened. The integrity report catches some patterns, but a score alone does not prove who did the work. If you use a score in hiring, follow it with a short live conversation about the session.
Questions about a specific score: hello@goship.tech. The recorder is the npm package @goship/mcp-server, and you can read the code it runs there.