Back to Portfolio

Case Study: Securing Sticky Summary Authorship & Identity Pinning

Open Source Contribution to alibaba/open-code-review

1. Executive Summary & Problem Statement

alibaba/open-code-review (OCR) is an automated AI code-review tool that runs inside GitHub Actions and posts inline code comments alongside a top-level "sticky" Pull Request summary comment. To avoid spamming a PR timeline on every commit push, OCR searches existing PR comments for a hidden HTML marker (<!-- ocr-summary -->) and updates that comment in place. Furthermore, OCR uses a cross-push checkpoint system (<!-- ocr-checkpoint:v1 ... --> embedded inside the sticky summary) to record the last reviewed commit SHA so subsequent pushes only review the new delta.

The Vulnerability / Bug (Issue #1557)

Previously, the comment lookup functions (findExistingSummaryComment and findSummaryIssueComment) located the sticky summary solely by checking if the comment body contained the string <!-- ocr-summary -->. It did not verify who authored the comment during the summary update phase.

This created two critical problems:

  1. Comment Hijacking / Overwriting Human Comments: If a human developer or external fork contributor wrote a PR comment that quoted or included <!-- ocr-summary -->, OCR would mistake the human's comment for its own sticky summary and overwrite the human's text on the next run.
  2. Inconsistent Trust Boundaries Across Token Types: OCR can be run under three different authentication identities:
    • The default GitHub Actions GITHUB_TOKEN (posts as github-actions[bot] via the github-actions App).
    • A custom GitHub App installation token (posts as <custom-app>[bot]).
    • A Personal Access Token / PAT (posts as a human User account matching the PAT owner's login).

Because github.rest.users.getAuthenticated() (GET /user) returns 403 Resource not accessible by integration when called by a GITHUB_TOKEN or GitHub App installation token, simple login matching fails for bots, whereas strict bot-only checks fail for human PAT users. The posting path needed a unified, zero-regression authorship verification model that aligned with the repository's checkpoint trust boundary.

2. Architectural Solution & Design Decisions

A. Unified Authorship Verification (isSummaryAuthorOurs)

Instead of relying on a single fragile check, we introduced isSummaryAuthorOurs(comment, botLogin, appSlug):

  • PAT Support: If botLogin is resolved (meaning GET /user succeeded, which only happens under a user PAT), and comment.user.login === botLogin, the comment is trusted as ours.
  • App & Bot Fallback: Otherwise, it delegates to isCheckpointAuthorOurs(comment, appSlug). This function immediately rejects any User-typed account (preventing a fork contributor from spoofing a marker even if they post via a third-party CLI GitHub App, where performed_via_github_app is populated). It then verifies that the author is a Bot and, if appSlug is pinned (e.g., 'github-actions'), strictly requires the comment's App slug to match.

B. Lazy & Memoized Identity Resolution (authenticatedLogin)

In runPostReviewComments, resolving the PAT user identity requires calling getAuthenticatedLogin(github, log).

  • Why Lazy Evaluation? Many runs (such as clean PRs with no existing comments, or standard runs that don't enable incremental deduplication or outdated-thread resolution) never need to inspect comment authorship if no marker is found. Firing an eager API call at the top of the script wastes quota and triggers unnecessary 403 warnings in logs.
  • Implementation: We created a memoized closure authenticatedLogin initialized before any early exits (preventing Temporal Dead Zone errors on malformed JSON or zero-finding exits) and passed the function reference itself down into postSummary, ensureSummaryAnchor, and finalizeSummary. The API call is only awaited inside the comment loop after a comment containing <!-- ocr-summary --> is actually encountered.

C. Preserving the 12-Reason Checkpoint Diagnostic Contract (ignoreAuthor)

The checkpoint reader (readCheckpointComment) shares findSummaryIssueComment to locate the sticky summary. However, readCheckpointComment has a strict diagnostic contract tested by testCheckpointTwelveReasonsAreDistinct: if a human plants a fake summary comment, readCheckpointComment must find the comment and explicitly return { reason: "author_unverified" } rather than { reason: "no_summary_comment" }.

  • Why ignoreAuthor = true? If findSummaryIssueComment filtered out unverified authors during the checkpoint read, readCheckpointComment would think no comment existed at all. By adding an ignoreAuthor = false default parameter to findSummaryIssueComment and setting ignoreAuthor: true specifically inside readCheckpointComment, the posting path safely ignores human comments while the checkpoint reader still surfaces the exact author_unverified telemetry reason.

D. Defensive API Mocking (getAuthenticatedLogin)

Several unit tests in the suite pass stripped-down mock github objects that omit the rest.users namespace entirely. We hardened getAuthenticatedLogin to check for the existence of github?.rest?.users?.getAuthenticated before invoking it, returning null cleanly instead of throwing a TypeError that would trip the checkpoint resolver's catch block into a resolver_error.

3. File-by-File Code Breakdown

File 1: scripts/github-actions/post-review-comments.js

This is the core runtime engine for posting reviews, managing sticky summaries, and evaluating cross-push checkpoints.

  1. Function Signature (runPostReviewComments): Added appSlug = "" to the destructured options object so the caller can pin the expected GitHub App identity ('github-actions' when running under the default token).
  2. Early Memoized Closure (authenticatedLogin): Moved the lazy promise memoizer above the early-exit branches (checkpointNoop, malformed JSON catch block, and comments.length === 0 zero-findings exit) so all summary-posting paths have access to it without hitting a JavaScript Temporal Dead Zone (TDZ) error:
    let authenticatedLoginPromise = null;  
                            const authenticatedLogin = () => {  
                            if (!authenticatedLoginPromise) authenticatedLoginPromise = getAuthenticatedLogin(github, log);  
                            return authenticatedLoginPromise;  
                            };
  3. Threading botLogin and appSlug Through Summary Call Sites: Updated the malformed-JSON error summary path, the zero-findings summary path, ensureSummaryAnchor, and finalizeSummary to pass botLogin: authenticatedLogin (as a lazy function reference) and appSlug.
  4. Updating findExistingSummaryComment and findSummaryIssueComment: Updated both lookup loops to lazily resolve botLogin only when a matching marker (SUMMARY_MARKER or tag) is found, and gated the return on isSummaryAuthorOurs(comments[i], login, appSlug):
    async function findSummaryIssueComment({ github, owner, repo, prNumber, sticky, tag, log, botLogin, appSlug, ignoreAuthor = false }) {  
                            const comments = await readAllPages("listIssueComments", (page, per_page) =>  
                                github.rest.issues.listComments({ owner, repo, issue_number: prNumber, per_page, page }), log  
                            );  
                            for (let i = comments.length - 1; i >= 0; i--) {  
                                const body = comments[i].body || "";  
                                if (sticky ? body.includes(SUMMARY_MARKER) : body.includes(tag)) {  
                                const login = typeof botLogin === "function" ? await botLogin() : botLogin;  
                                if (ignoreAuthor || isSummaryAuthorOurs(comments[i], login, appSlug)) {  
                                    return comments[i];  
                                }  
                                }  
                            }  
                            return null;  
                            }
  5. Implementing isSummaryAuthorOurs & Hardening getAuthenticatedLogin: Added isSummaryAuthorOurs to bridge PAT logins and App/Bot verification, and added defensive null-checks to getAuthenticatedLogin:
    function isSummaryAuthorOurs(comment, botLogin, appSlug) {  
                            if (!comment || !comment.user) return false;  
                            if (botLogin && comment.user.login === botLogin) return true;  
                            return isCheckpointAuthorOurs(comment, appSlug);  
                            }

File 2: action.yml

This composite GitHub Action manifest defines the environment variables and inline actions/github-script execution blocks.

  1. Exposing OCR_CHECKPOINT_APP_SLUG in the Posting Step: In GitHub Actions, step-level env: blocks do not carry over to subsequent steps. Previously, OCR_CHECKPOINT_APP_SLUG was only defined on the Resolve review range step. We added the exact same expression to the Post review comments step's env: block:
    OCR_CHECKPOINT_APP_SLUG: ${{ inputs.github_token == github.token && 'github-actions' || '' }}

    Why this expression? When the user uses the default github.token, we know with 100% certainty the writer is the github-actions App, so we pin it. If they pass a custom token (PAT or custom App), it evaluates to '' because an installation token cannot query GET /app without a JWT.

  2. Forwarding appSlug into runPostReviewComments: Inside the with: script: | block of Post review comments, we wired the environment variable into the function call:
    resolveOutdated: process.env.OCR_RESOLVE_OUTDATED,  appSlug: process.env.OCR_CHECKPOINT_APP_SLUG || '',

File 3: scripts/github-actions/post-review-comments.test.js

The unit and integration test suite for the review poster. Added targeted regression tests covering:

  • Human Comment Protection: Ensuring a human comment containing <!-- ocr-summary --> is ignored and never overwritten.
  • PAT Authorship: Ensuring a sticky summary authored by the authenticated PAT user is recognized and updated in place.
  • Custom GitHub App Authorship: Ensuring comments written by a custom [bot] account are recognized when appSlug is unpinned, while foreign bots are rejected when appSlug is pinned to 'github-actions'.
  • Malformed JSON / Stderr Fallback: (testMalformedJsonWithStderrUsesAuthenticatedLogin) Verifying that early-exit error summaries properly resolve identity without throwing reference or TDZ errors.

File 4: scripts/github-actions/action-contract.test.js

The static and behavioral contract test suite that parses action.yml without external dependencies.

  1. Upgrading parseSteps() to Extract actions/github-script Bodies: Previously, the custom YAML parser in action-contract.test.js only extracted run: | bash blocks and env: maps, leaving actions/github-script steps (with: script: |) invisible to contract assertions. We extended finish() inside parseSteps() to detect script: | and capture its 10-space-indented JavaScript body as step.script:
    let script;  
                            const scriptMarker = rawLines.findIndex((line) => /^        script:\s*\|\s*$/.test(line));  
                            if (scriptMarker >= 0) {  
                            const body = [];  
                            for (let index = scriptMarker + 1; index < rawLines.length; index += 1) {  
                                const line = rawLines[index];  
                                if (line.trim() === "") {  
                                body.push("");  
                                } else if (/^          /.test(line)) {  
                                body.push(line.slice(10));  
                                } else {  
                                break;  
                                }  
                            }  
                            script = body.join("\n");  
                            }
  2. Adding testPostReviewCommentsWiresAppSlug: Added a dedicated contract test verifying both halves of the action.yml wiring so they can never drift apart in future refactors:
    function testPostReviewCommentsWiresAppSlug() {  
                            const post = stepNamed("Post review comments");  
                            assert.ok(post, "action.yml must retain the Post review comments step");
    
                            // 1. The step exposes the env value.  
                            assert.strictEqual(  
                                post.env.OCR_CHECKPOINT_APP_SLUG,  
                                "${{ inputs.github_token == github.token && 'github-actions' || '' }}",  
                                "Post review comments must resolve and expose OCR_CHECKPOINT_APP_SLUG in its env block"  
                            );
    
                            // 2. The github-script body forwards it into the runPostReviewComments({...}) call.  
                            assert.ok(post.script, "Post review comments must define a github-script `with: script:` body");  
                            const call = post.script.match(/runPostReviewComments\(\{\[\s\S\]*?\n\}\);/);  
                            assert.ok(call, "Post review comments script must call runPostReviewComments({...})");  
                            assert.match(  
                                call[0],  
                                /^\s*appSlug:\s*process\.env\.OCR_CHECKPOINT_APP_SLUG\s*\|\|\s*(\['"\])\1\s*,?\s*$/m,  
                                "runPostReviewComments must be passed the appSlug from process.env"  
                            );  
                            }

4. The Code Review Lifecycle & Maintainer Collaboration

A major highlight of this contribution was the iterative, rigorous code review process with community reviewer @Qiyuanqiii. Here is the chronological breakdown of the review rounds and how each technical challenge was resolved:

Round 1: Initial Regressions (TDZ, Zero-Findings Path, & App Trust)

  • Reviewer Feedback: @Qiyuanqiii pointed out three edge-case regressions in the initial approach:
    1. authenticatedLogin was initialized after the early exits (malformed JSON and zero-findings), causing a Temporal Dead Zone (ReferenceError) when those paths tried to post a summary.
    2. The zero-findings path did not carry the authenticated identity.
    3. Relying solely on GET /user broke GitHub App and GITHUB_TOKEN runs because GET /user returns HTTP 403 for integration tokens; the summary check needed to preserve the App/Bot trust check (isCheckpointAuthorOurs).
  • Our Resolution: Hoisted authenticatedLogin above all early exits, created isSummaryAuthorOurs to combine PAT login matching with isCheckpointAuthorOurs, and added regression tests for human, PAT, and custom-App comments.

Round 2: Scoping & Environment Wiring (b1f8cf7 -> 4bd31bd)

  • Reviewer Feedback: Upon re-reviewing b1f8cf7, @Qiyuanqiii noted that the three architectural concerns were solved, but caught a blocking integration issue:
    1. appSlug was passed to helper functions inside runPostReviewComments, but appSlug = "" was missing from runPostReviewComments's own parameter destructuring, causing ReferenceError: appSlug is not defined.
    2. OCR_CHECKPOINT_APP_SLUG was only scoped to the Resolve review range step in action.yml, not the Post review comments step.
    3. Pointed out that make test only runs Go tests, whereas npm run test:github-actions runs the JS suite.
  • Our Resolution: Added appSlug = "" to runPostReviewComments, added OCR_CHECKPOINT_APP_SLUG to the Post review comments step's env: block in action.yml, made botLogin lazy so testAuthenticatedLoginIsResolvedOncePerRun passed with 0 unnecessary API calls, and hardened getAuthenticatedLogin against stripped test mocks.
  • Our Reply: Explained the four fixes clearly and confirmed npm run test:github-actions passed locally.

Round 3: Final action.yml Script Forwarding & Contract Testing (4bd31bd -> Final Approval)

  • Reviewer Feedback: @Qiyuanqiii confirmed all runtime issues and author-check logic were fixed, noting only one final wiring gap before approval:
    • While Post review comments now exposed OCR_CHECKPOINT_APP_SLUG in its env: block, the inline JS call runPostReviewComments(...) inside action.yml ended at resolveOutdated: process.env.OCR_RESOLVE_OUTDATED without passing appSlug: process.env.OCR_CHECKPOINT_APP_SLUG || ''.
    • Requested adding that line and writing an action-contract.test.js regression test asserting both the env exposure and the runPostReviewComments argument forwarding.
  • Our Resolution & Reply: Added appSlug: process.env.OCR_CHECKPOINT_APP_SLUG || '' to action.yml, upgraded parseSteps() in action-contract.test.js to parse with: script: | blocks, and added testPostReviewCommentsWiresAppSlug. Replied with a detailed breakdown showing that removing either the env line or the forwarding line caused the contract test to fail, and that all 53 action contract tests passed.
  • Outcome: @Qiyuanqiii formally approved the changes!

5. Current Status & Open-Source Governance Lesson

After receiving @Qiyuanqiii's approval, the PR showed:

  • 1 approval (@Qiyuanqiii)
  • 3 workflows awaiting approval (GitHub Actions security gate for fork PRs)
  • Merging is blocked: At least 1 approving review is required by reviewers with write access.

Ask AI

Hi! I'm Irvallen's AI assistant. Ask me anything about his projects or experience.