Operations and recovery

Start with the failing boundary: timer, ESPN, private state, public publication, email, or browser fetch. The Function’s logs intentionally avoid printing provider exception bodies and recipient details.

Routine checks

Inspect the check_pool invocation in Function monitoring/Application Insights.
Confirm that the configured season is active and DRY_RUN is false for real runs.
Check whether results.json reflects expected names, dates, and scores.
If a game is eligible but incomplete, inspect the cached schedule and final status.
Review delivery markers alongside ACS records when email is missing.
Review Sentry browser errors in the production environment and match the event release to the deployed commit.

as_of_utc is the last published snapshot timestamp, not an hourly heartbeat. Unchanged public content is intentionally not rewritten. The browser fetches once on load; refresh it to fetch a newly published result.

Troubleshooting by symptom

Symptom	First checks	Relevant behavior
No work on an hourly invocation	Season window, `DRY_RUN`, configured pool dates	Off-season and dry-run exits avoid all external work
No halftime email during the game	ESPN final status	Both periods are notified after the game finishes
Final game missing	Cached kickoff, NFL week, ESPN status, score parsing	First check is kickoff + 3 hours; old unresolved games retry daily
Roster change not visible	Function deployed, active-season invocation, browser reload	Public content refresh is independent of email completion
Browser says results cannot load	Network panel, Blob URL, anonymous read, Blob CORS	Azure website hosting and Blob endpoint permissions differ
JSON is current but open page is stale	Reload the page	The minute timer changes the featured week without refetching JSON
State fails to load	Results connection string, container access, state version	`pool-state` must be private and state version must be 1
Write fails due to concurrency	Overlapping execution or manual edits	ETag conditions reject stale writers
Email failure or delivery requires review	`sending` marker and ACS submission records	Ambiguous submissions are not automatically resent
Function deploy cannot authenticate	Federated subject, audience, variable scope, Azure role	The OIDC credential must match the branch/project exactly
Function deploy succeeds but code does not load	Remote-build logs, Python version, required templates	CLI success alone is not an application smoke check
Docs are missing on Pages	`docs:build`, `docs:pages`, Pages access and URL	Use the URL shown by GitLab, including any unique-domain setting

Verify Sentry reporting

After a website deployment, open the public site and run this in the browser developer console to send a deliberate test event:

Sentry.captureException(new Error("Public site Sentry test"));

Confirm the event appears in the configured Sentry project with environment production and the deployed commit SHA as its release. For a local preview, expect development; the checked-in configuration has no release value. Successful configuration checks alone do not verify delivery to Sentry.

If the event is missing, check that sentry-config.js contains a nonempty DSN, the SDK loads from browser.sentry-cdn.com, and the browser’s Network panel shows the event request to the configured Sentry ingest host. Browser extensions or network filtering may block these requests. Check the project’s environment filters as well. An explicitly empty GitLab SENTRY_DSN disables reporting; redeploy after changing deployment variables.

For the site’s results error message, look for operation:load-results in Sentry and inspect the browser’s Blob request and CORS response. Sentry does not monitor the Python Function; use Azure logs for publishing, scoring, and email failures.

Ambiguous email delivery

Each delivery key starts with a configured game date followed by a colon and one of summary, winner:<player-id>, season:halfway, or season:season-end. Season summaries use the last configured game date in their reporting period. The scheduler saves a claim before contacting ACS. Milestone claims use the same recovery procedure, even when every game is already marked completed.

stateDiagram-v2
    [*] --> Unclaimed
    Unclaimed --> Sending: Conditional state write succeeds
    Sending --> Sent: ACS succeeds and state save succeeds
    Sending --> Review: Timeout, process stops, or uncertain response
    Review --> Sent: Operator confirms acceptance
    Review --> Unclaimed: Operator confirms no send and removes only that claim
    Sent --> [*]: Subsequent runs skip delivery

A retained sending value is deliberately blocking. It could mean ACS accepted an email but the worker stopped before recording success. Do not remove all state to force another run.

Stop the Function before manually editing its state.
Back up the private pool-state/<season>.json blob.
Inspect ACS submission/delivery records for the affected attempt.
If accepted, change only the relevant delivery marker to sent.
If definitely not submitted and a retry is wanted, remove only that delivery entry.
Preserve cached results and unrelated delivery, publication, and completion markers.
Restart the Function and monitor the next eligible invocation.

The normal statuses persisted are sending and sent; Review in the diagram is an operator action, not another persisted value. This design reduces duplicate emails but is not a transactional exactly-once guarantee.

State and history

State entry	Purpose
`version`	State format, currently 1
`schedule`, `schedule_refreshed`	ESPN kickoff cache and UTC refresh date
`checked`	Last UTC-day check for long-delayed games
`results`	Verified completed-game scores and winners
`snapshot_hash`	Detects changes to public content
`published`	Games whose result publication was recorded
`deliveries`	Weekly, winner, and season-summary send claims
`completed`	Games whose processing and delivery handling finished

A missing season-state file begins a fresh catch-up workflow. It does not import emails sent by older implementations and can resend prior notifications. Public daily snapshots cannot restore delivery markers. Keep a private backup before any intentional state repair or migration.

Roster or season updates

Use the pool data checklist, run the tests, and deploy the Function. An active-season invocation publishes changed public configuration even if no newly completed game exists. This refresh preserves existing email and completion markers.

For emergency suspension, stop the Function or set DRY_RUN=true; the latter skips the scheduled workflow entirely. Resume only after confirming configuration and state.