Ingest — Fetching Data

The ingest phase pulls raw activity from GitHub, normalizes it into signals, and writes them to the database. This is the only phase that touches the GitHub API.

Entry point

runIngest() in lib/leaderboard/ingest.ts. Called from the pipeline orchestrator in pipeline.ts.

Fetching from GitHub

lib/github/fetch-graphql.ts (31KB, the largest module in the codebase) is responsible for all GraphQL calls during ingest.

It uses an installation token — not the user’s OAuth token — via createLeaderboardOctokit(). This ensures the ingest sees all repos the app is installed on, not just the repos the signed-in user can see.

What gets fetched

For each repository in the scope:

• Commits — author, date, message, additions, deletions, co-authors
• Pull requests — open/merge/close events, linked issues, author
• Issues — open/close events, author
• Reviews — reviewer, state (approved/changes_requested/commented)
• Review comments — reviewer, associated PR

Pagination and concurrency

• PAGE_SAFETY_CAP = 20 — max pages per GraphQL query type, regardless of total results
• REPO_CONCURRENCY = 6 — at most 6 repositories fetched in parallel (mapPool() in fetch-graphql.ts)
• All enrichment steps (PR metadata, issue links, review details) always run — there are no toggle flags to skip them
• Ingest limits are defined in lib/github/ingest-limits.ts

Incremental fetching

resolveIncrementalSince() in ingest-limits.ts computes the effective since timestamp for a fetch. If a previous successful ingest exists, it adds a 5-minute overlap (INGEST.INCREMENTAL_OVERLAP_MS) to catch late-arriving events:

effectiveSince = max(requestedSince, lastFetchedAt - 5min)

If no from parameter is provided, ingest defaults to 90 days ago (INGEST.DEFAULT_LOOKBACK_DAYS).

Signal normalization

normalizeGitHubData() in lib/scoring/normalize.ts converts the raw GraphQL response into Signal[] — the canonical unit of activity in the system.

What a signal is

{
  user_id: number;          // GitHub user ID
  repository_id: number;    // GitHub repo ID
  type: 'commit' | 'pr_open' | 'pr_merge' | ...;
  value: number;            // Default 1, may be fractional for co-authors
  event_timestamp: string;  // ISO 8601
  content_hash: string;     // SHA-256 deduplication key (first 32 chars)
  metadata: {               // JSONB, max 2048 bytes
    sha?: string;
    additions?: number;
    deletions?: number;
    pr_number?: number;
    issue_number?: number;
    review_state?: string;
    isBot?: boolean;
    // ...
  }
}

Content hash deduplication

Every signal gets a SHA-256 hash built from:

user_id | type | repository_id | event_timestamp | message/body/state

Only the first 32 characters are stored (the DB column is char(64) — over-wide but harmless). The signals table has a unique index on (user_id, type, repository_id, event_timestamp, content_hash). Re-ingesting the same GitHub activity produces zero duplicate rows — upsertSignals() is fully idempotent.

Co-author splitting

Commits with co-authors are split proportionally. If a commit has 340 additions and 18 deletions with 2 co-authors, each author gets a signal with 170 additions and 9 deletions. The content hash differs because the user_id component changes.

Bot detection

A signal is flagged as bot activity (isBot = true) when:

• author.type === 'Bot'
• author.login ends with [bot] (GitHub’s convention for automated accounts like Dependabot)

The flag is stored in both the signal metadata JSONB and the users.is_bot column. When the bot_activity zero-point condition is enabled (default), bot signals score 0 points.

Writing to the database

upsertSignals()

lib/supabase/signals.ts handles the write. Before inserting, it calls ensure_signals_monthly_partitions(p_start, p_end) — an RPC that creates the signals_YYYY_MM partition for every month in the date range if it doesn’t already exist. Rows landing in signals_default (the catch-all partition) indicate a partition wasn’t pre-created.

upsertRepositories()

Repository metadata (name, owner, visibility, default branch) is upserted from the GraphQL response. This keeps repositories.full_name consistent even if a repo is renamed on GitHub.

ingest_log

After a successful ingest, setLastSuccessfulIngestAt() records:

• last_successful_ingest_at — timestamp used by the 24h cooldown
• repo_count — how many repos were fetched
• signal_count — how many signal rows were inserted
• ingest_preset — the time period this ingest covers

The log row is scoped to (installation_id, organization_id, [team_id | repository_id]) — ingests for different scopes don’t block each other.