How the Pipeline Works

Every leaderboard request runs through three phases. When cached data is fresh enough, phases are skipped.

The three phases

1 — Ingest

runIngest() in lib/leaderboard/ingest.ts

1. Acquires a Redis lock (leaderboard:lock:{scopeId}) — if held, another ingest is in progress, skip
2. Checks ingest_log for the last successful ingest — if within the 24h cooldown, skip
3. Calls fetchOrgScoringDataGraphQL() with an installation token — this is where we talk to GitHub
4. normalizeGitHubData() converts raw commits/PRs/issues/reviews into Signal[] with content hashes
5. upsertSignals() inserts into the partitioned signals table (monthly partitions auto-created)
6. upsertRepositories() updates the repositories table
7. setLastSuccessfulIngestAt() records the timestamp in ingest_log

2 — Score

scoreActivePresetForEntity() in lib/leaderboard/score.ts

1. getSignalsForOrg() reads from the signals table — zero GitHub API calls
2. computeScores() in lib/scoring/engine.ts runs the scoring algorithm per user
3. Scores are written to computed_scores (one row per entity, per period, per preset)
4. An advisory Redis lock prevents concurrent writes to the same materialization
5. All 6 time periods are scored at once — switching presets is instant

3 — Serve

serveLeaderboard() in lib/leaderboard/serve.ts

Two paths:

• Standard request (no from/to): servePrecomputedScores() — single DB query against computed_scores joined to leaderboard_materializations. Instant.
• Custom date range (has from/to): computeCustomDateRange() — reads signals from DB within the range, aggregates in memory. Not persisted.

Freshness enforcement

evaluateIngestFreshness() in lib/leaderboard/ingest-freshness.ts decides whether to trigger a new ingest:

24-hour cooldown

INGEST.COOLDOWN_MS = 86,400,000 (24 hours). Once a successful ingest completes, no new full ingest triggers for 24 hours regardless of staleness. Manual sync during cooldown returns 429.

Per-period TTLs

If ingestAge > periodTTL AND !cooldownActive, an ingest is triggered before scoring and serving proceed. Otherwise, the cached computed scores are used directly.

Force-refresh disabled

A client force-refresh header exists (X-Force-Refresh) and is imported by the score route handler, but forceRefresh is hardcoded to false — the server ignores it. Force-refresh was intentionally disabled to prevent abuse.

Scope resolution

Leaderboard requests accept three scope levels:

resolveReposForScope() in lib/leaderboard/resolve-scope.ts handles resolution. For teams, it calls fetchTeamRepositoriesGraphQL() if the data isn’t already cached. For organizations, it uses fetchOrgRepositoriesGraphQL(). Results are cached in the repositories and team_repositories tables.

Background work

All deferred work uses after() from next/server — code that runs after the HTTP response is sent. There is no external job queue.

Leaderboard routes that use after():

• POST /api/[org]/leaderboard/score — if precomputed data is fresh, scores remaining entity types in background; if stale, triggers full refresh
• POST /api/[org]/leaderboard/recompute — dev-only, triggers full recompute in background

Both routes set maxDuration = 300 (5 minutes). This requires Vercel Pro — the Hobby plan caps at 60 seconds and large orgs will time out.

Concurrent requests

The Redis lock on leaderboard:lock:{stableIngestScopeId} guarantees only one ingest runs at a time per scope. A second concurrent request finds the lock held and skips the ingest phase, proceeding directly to serve with existing data.

setLastSuccessfulIngestAt() handles upsert-race conditions with a retry — if two processes attempt to create the initial ingest_log row simultaneously, one retries after the conflict.