Overview

Every leaderboard request passes through three phases. Phases are skipped when cached data is fresh enough.

Phase 1 — Ingest (runIngest)

Entry point: runIngest(ctx) in lib/leaderboard/ingest.ts

ctx = { octokit, organization, organizationId, installationId, filters }

Steps

1. Acquire Redis lock — key leaderboard:lock:{stableIngestScopeId}. If locked, skip; another ingest is already running for this scope.
2. Fetch — fetchOrgScoringDataGraphQL(octokit, org, { since, repos, forceRefresh }). GraphQL, paginated. Per-repo limits apply (see Reference → Ingest limits).
3. Normalize — normalizeGitHubData(rawData) → Signal[]. Each signal gets a SHA-256 content hash (first 32 chars) derived from event_type + user + repo + timestamp + identifier for deduplication.
4. Store — upsertSignals(installationId, signals, repoNameToId). Writes to the signals table. The unique index on (user_id, type, repository_id, event_timestamp, content_hash) makes this idempotent.
5. Record completion — setLastSuccessfulIngestAt() stores timestamp, preset name, repo_count, and signal_count in ingest_log.

What normalizeGitHubData does

• Converts raw GitHub commits/PRs/issues/reviews/comments into Signal[].
• Splits code additions/deletions proportionally across co-authors.
• Extracts boolean flags: isSelfReview, isSelfMerge, isInMergedPR, hasLinkedIssue, isBot.
• Bot detection: author.type === 'Bot' OR login.endsWith('[bot]').
• Stores per-signal metadata JSONB (max 2048 bytes per row — enforced by DB CHECK constraint).

Phase 2 — Score (scoreActivePresetForEntity)

Entry point: scoreActivePresetForEntity(ctx, repos, teamMemberships, entityType) in lib/leaderboard/score.ts

Steps

1. Load signals — getSignalsForOrg(installationId, repos). Reads entirely from the signals table. Zero GitHub API calls.
2. Loop over all 6 time periods — SCORE_TIME_PERIODS = ['today', 'week', 'month', 'quarter', 'half_year', 'all_time'] (defined in lib/constants/time.ts).
3. Per period:
   • getDateRangeForScoreTimePeriod(period) → { from, to } (see table below)
   • isSignalInRange(signal, from, to) filters signals in-memory
   • aggregateLeaderboard(signals, rules, entityType, teamMemberships) → ranked entries
   • Acquire Redis advisory lock, then writePresetComputedScores(presetId, scopeType, org, team, repo, entityType, period, entries) → writes to computed_scores + leaderboard_materializations
4. Background fill — scoreRemainingEntityTypes(ctx, repos, teamMemberships, primaryEntityType) scores the other two entity types asynchronously.

Time period → date range mapping

all_time is fully implemented. getSignalsForOrg() called without a date filter returns all rows in the signals table. Freshness TTL is 30 minutes.

Phase 3 — Serve (serveLeaderboard)

Entry point: serveLeaderboard(ctx) in lib/leaderboard/serve.ts

Two paths depending on whether the request includes a custom date range:

Standard request (no from/to)

servePrecomputedScores(presetId, timePeriod) — single DB lookup against computed_scores joined to leaderboard_materializations. Returns instantly with no computation.

Active preset: resolved via getActiveScoringRulesPreset() which queries scoring_presets WHERE is_active = true.

Custom date range (from/to present)

computeCustomDateRange(ctx):

1. getSignalsForOrg(installationId, repos, { since: from, until: to }) — reads signals from DB within the date range.
2. aggregateLeaderboard() — aggregates in-memory.
3. Returns immediately. No write to computed_scores.
4. Result is still written to the Redis response cache.

Custom ranges are not persisted, so repeated identical requests re-compute each time (though the Redis response cache may serve them).

Freshness enforcement (evaluateIngestFreshness)

Module: lib/leaderboard/ingest-freshness.ts

evaluateIngestFreshness(lastSuccessfulIngestAt, filters):
  ingestAgeMs = Date.now() - Date.parse(lastSuccessfulIngestAt)
  cooldownActive = (ingestAgeMs != null) AND (ingestAgeMs < 86_400_000)
  ingestStale = (ingestAgeMs > ttlMs) OR forceRefresh OR (never ingested)

24-hour cooldown

INGEST.COOLDOWN_MS = 86_400_000 (24 hours). Once a successful ingest completes, no new full ingest is triggered for 24 hours regardless of staleness. This prevents hammering the GitHub API.

• Manual sync request during cooldown → 429 Too Many Requests.
• Non-manual request during cooldown → skip INGEST, serve from cache or precomputed scores.

Per-period freshness TTLs (INGEST.TTL_MS in lib/constants/time.ts)

If ingestStale AND !cooldownActive, INGEST is triggered before SCORE and SERVE proceed.

Ingest limits

These constants are defined in lib/github/ingest-limits.ts: