Product Engineer dashboard
How Product Engineers are evaluated
Three signals drive every PE row on the Dashboard: Justification (did you earn your seat this week?), Score (a 1–5 quality grade), and Plan completion (how much of your committed weekly plan you actually delivered). This page defines all three so the rules are transparent and reproducible — there are no hidden judgments.
Jump to: Justification · Score · Plans · Why commits might not show · Domain throughput
Justification — did this PE earn their seat this week?
Justified ✓ Yes
The PE put in enough authoring effort this week, spread across enough of their expected workdays, on substantive (non-chore) work, to defend their compensation at this domain's weight.
- Authored at least one commit this week — release-pipeline re-exports of older work do not count.
- Authoring covered a reasonable share of expected workdays (already excludes weekends, PTO, future days, and pre-domain-start days; scaled by part-time weight).
- Work was a real feature, improvement, bug fix, or refactor — not only chores or trivia.
- Effort wasn't a single concentrated burst masking an otherwise empty week.
Not justified ✗ No
Any one of these conditions makes the week Not justified:
- Zero authored commits. Hard rule — release-only weeks never justify a slot.
- Coverage below 50% of expected workdays (when ≥2 days were expected). Catches the "concentrated burst masking an otherwise empty week" pattern — a single big day doesn't replace a week of expected effort.
- All commits were chores or trivia (whitespace, dep bumps, formatting).
- Pipeline activity here reflects work authored in prior weeks being promoted; the PE didn't actually work on this domain this week.
- Slack presence was high but no code was authored — Slack alone doesn't rescue an empty authoring week.
What it does not mean
- It's not about what shipped this week. A giant production deploy authored last month is not this-week effort.
- It's not about overall domain health or pipeline throughput (see below).
- It's not a judgment of the PE overall — it's a one-week snapshot.
- It's not about the absolute value of the work — a one-line fix shipped to thousands of users is still trivial as a week of effort.
Score — how the 1–5 number is calculated
Every PE row carries a single Overall score (1–5) made of two independent sub-scores. The rule is intentionally simple: Overall = min(Meaningfulness, Engagement). The "weakest-link" model means a great week of meaningful work that only happened on one day still scores low — and consistent showing-up on trivial work also scores low. Both have to be there. A third factor — plan completion — sets a hard ceiling: see the Plans section for how a missing plan caps the score at 4 and how the top tier splits into 5 vs 5★.
Meaningfulness (1–5) — what was actually built
LLM-judged from the diffs and commit categorization. Ignores how the work was distributed across the week — that's Engagement's job. Looks only at what shipped or was authored.
- 5 ★ 5★ — Exceptional and on-plan. Major feature delivery or significant product advancement, AND plan completion ≥ 90%. The star recognises engineers who delivered almost exactly what they said they would deliver.
- 5 5 — Exceptional, off-plan. Same calibre of delivery as 5★, but plan completion is 0–89% — substantial work landed, just not all of the work that was committed to in the weekly plan. Numerically still a 5; the star is withheld.
- 4 4 — Solid. Clear progress with tangible new capabilities at any pipeline stage.
- 3 3 — Moderate. Some real features or meaningful fixes, even if not yet shipped.
- 2 2 — Minor. Small fixes or cosmetic improvements.
- 1 1 — Trivial. Whitespace, dependency bumps, formatting, or other chore-only work.
Engagement (1–5) — how consistently the PE showed up
Deterministic floor computed from coverage
(meaningful_days / expected_workdays) and the
longest gap between active days. The LLM may adjust the floor by at most ±1
with reasoning. The denominator is already clamped — it excludes weekends, PTO, future days,
pre-domain-start days, and is scaled by part-time weight.
- 5 — 100% coverage. Meaningful work on every expected workday, no gaps. A single missed expected day disqualifies a 5.
- 4 — ≥80% coverage. Longest gap ≤ 1 day, active on at least
expected − 1days. - 3 — ~60–80% coverage. Gaps ≤ 2 days. Solid but not consistent.
- 2 — ~40–60% coverage or one longer gap. Partial engagement.
- 1 — <40% coverage, or only trivial chore commits sprinkled across the week.
reducing coefficient = max(0, presence% − fine%), and the engagement score is
multiplied by it. Example: presence 100% − fine 50% → ×0.50. Hover the admin × pill
on a row for the per-day breakdown.
Overall = min(Meaningfulness, Engagement)
- Meaningfulness 5, Engagement 5 → Overall 5. Best possible week.
- Meaningfulness 5, Engagement 1 → Overall 1. Big-burst week with most days empty — the burst doesn't rescue the week.
- Meaningfulness 1, Engagement 5 → Overall 1. Showed up every day but only on trivia — consistency doesn't rescue the work.
- Meaningfulness 0 (no commits) → Overall 0. Engagement is irrelevant when nothing was authored.
Plans — what you committed to, and how it scores
Every Monday a new week begins, and every PE submits one short plan per domain describing what they intend to deliver that week. The analyzer reads each plan when scoring the week and produces a Plan Completion percentage (0–100%) by checking how many of the listed plan items are reflected in the week's commits. Both the plan text and the resulting percentage are visible on the Dashboard's Plan column and inside each week's expanded panel on the PE Deep Dive.
Where to submit
The Plans tab. You'll see two cards per domain you're on:
- Next week (required). The standard submission. Edit freely until the deadline; resubmit as many times as you like.
- Current week. Visible for reference, editable Monday and Tuesday for late catch-up, locked from Wednesday 00:00 UTC onward (see Edit windows below).
Deadlines & the late flag
- Next-week plan: Sunday 23:59 UTC of the week before. A reminder is sent by Slack/email on Sunday afternoon if a plan for any of your domains is still missing.
- Late submission. If your first submission lands on or after the Monday the week starts (i.e. you missed Sunday EOD UTC), the plan is accepted and analysed normally but the card is marked Submitted · late and the row keeps that flag for the rest of the week.
- Missing plan = score capped at 4. A week with no plan submitted at all is the only condition that mechanically lowers your Score — see the rule below.
Edit windows
- Future weeks (next week, two weeks out, etc.): always editable by you.
- Current week, Monday & Tuesday UTC: editable by you — fixes and late catch-up are fine.
- Current week, Wednesday 00:00 UTC onward: locked for self-edits. This prevents retroactively rewriting "what I'd planned" to match what actually shipped later in the week. An admin can still edit on your behalf if a genuine correction is needed.
- Past weeks: always locked for self-edits. The analyzer also takes a frozen snapshot of the plan text at scoring time, so even an admin edit to a past plan won't change a cached delivery score.
How the analyzer scores your plan
For each engineer with a submitted plan, the LLM produces a plan_delivery block with four parts:
- percent (0–100): the share of plan items reflected in this week's commits. Partially-done items count proportionally (e.g. "implement and ship X" with implementation done but not yet merged ≈ 60%).
- delivered: each plan item the analyzer matched to actual commits, tagged with where the work landed (shipped / staging / testing / review).
- missing: plan items NOT found in the commits.
- summary: one or two sentences. Out-of-plan work is mentioned here as context but does not raise the percent — only the listed plan items count.
How plans affect the Score
There are exactly two places where plans touch the Score number:
- No plan submitted → Meaningfulness capped at 4. Even an exceptional week of authoring scores at most 4 if the corresponding plan is missing. The justification reasoning will note this explicitly (e.g. "Capped at 4 because no weekly plan was submitted; otherwise this would have been a 5.").
- 5 splits into 5 vs 5★. 5 is awarded for an exceptional week with plan completion 0–89%; 5 ★ is awarded for the same calibre of delivery with plan completion ≥ 90%. Both count as numeric 5 in averages, trends, and stats — the star is a visual recognition, not a different number.
Practically: write plans you can actually deliver, keep them concrete (one ticket / one outcome per line), and update them mid-week if scope genuinely shifts (Monday/Tuesday). A short, honest plan beats an ambitious plan you can't deliver — Plan Completion rewards accuracy of forecasting, not ambition.
The Plan Completion column on the Dashboard
Each PE row now shows a compact bar with the percent. A few states you may see:
- 95% — plan submitted, ≥ 90% delivered. Pairs with a 5★ score when Meaningfulness and Engagement are at 5.
- 55% — plan submitted, partial delivery. Score not capped, but no star at the top tier.
- No plan — no plan was submitted for this week. Meaningfulness is capped at 4.
Why your commits might not be showing
The Dashboard does not stream commits in real time. Commit data, the Score, the heatmap, the Plan Completion percent, and the engagement sparkline are all the output of a discrete analysis run — a server-side job that pulls every author's commits across the configured repositories, classifies them, and asks the LLM to score the week. If you push commits after the most recent analysis run for a given week, those commits will not appear until the next analysis runs.
What "analysis" means here
- An admin clicks Analyze on a domain-week (or a scheduled job runs across all domains for the current week).
- The analyzer collects every authoring commit for that week, snapshots the plan text, runs the LLM, computes the Score, Plan Completion, engagement metrics, etc., and persists the result.
- The Dashboard reads from that persisted analysis. It does not query Bitbucket on every page load.
Common reasons your commit isn't visible
- You pushed after the most recent analysis. The commit will appear when the week is re-analysed. Ask an admin to re-run analysis for the affected domain-week, or wait for the next scheduled run.
- You committed locally but didn't push. The analyzer only sees pushed commits — local work in progress is invisible to the system.
- The commit is on a branch the analyzer doesn't track. Only branches configured for the domain (development / staging / production / release branches) are scanned. A commit on a personal experimental branch that never merges in won't surface.
-
Author email mismatch. Commits are tied to your engineer record by
email. If your local
git config user.emaildoesn't match a verified email on your profile, the commit is attributed to "Unassigned" instead of you. Check your Profile tab and your local git config. - The week hasn't been analysed yet. A brand-new domain or a week that no one has clicked Analyze on shows the placeholder state, not your commits.
The commit-and-push-daily rule
The single most important habit for keeping your Dashboard accurate is:
Commit and push at least once every working day — and don't let work sit unpushed for more than ~24 hours.
This is not bureaucracy. Three concrete things break when you batch a week's work into a single Friday push:
-
Engagement drops. Engagement is computed from
meaningful_days / expected_workdays. A week of real work pushed all at once on Friday looks like four empty days plus one busy day — which scores as concentrated and pulls Engagement down even though you worked the whole week. - Justification can flip to "Not justified". The deterministic rule "< 50% coverage of expected workdays" is applied after the LLM, and the LLM cannot see commits that aren't pushed yet. A weekly burst pattern is one of the documented ways to get marked Not Justified despite real work happening.
- The week's analysis can run before your push. If the scheduled analysis runs Friday morning and you push Friday evening, the persisted analysis for that week shows zero commits from you. Re-analysing fixes it but it's avoidable friction.
Forcing a refresh
- The Dashboard auto-refreshes the underlying analysis state on tab switches and on a periodic poll, but it does not auto-trigger new analyses.
- To bring in newly pushed commits, an admin must click Analyze on the affected domain-week, or wait for the next scheduled run.
- If a commit you pushed yesterday still doesn't show up after a re-analysis, it's almost certainly an attribution problem — check your engineer record's emails on the Profile tab.
Domain throughput — is the pipeline moving?
Domain throughput is a domain-level signal independent of any individual PE. It describes what landed across the pipeline (In Review → Testing → Staging → Production) this week, including release events. A domain can be healthy while an individual PE on it is Not justified, and vice versa.
Healthy healthy
Normal flow: production output landed this week, or there's clear progression of substantive work moving stage-to-stage. The domain is shipping.
Stuck stuck
Substantive work piled up in Testing or Staging with nothing reaching Production. Work is accumulating but not crossing the finish line — usually a release-cadence or QA-bottleneck signal.
No production no production
There is Testing or Staging activity this week but zero commits made it to Production. Lighter than stuck — the pipeline is alive, just not delivering.
Idle idle
Little or no activity at any stage this week. The domain didn't visibly move.
User management has moved
Manage users and per-dashboard access from the global Settings page.
HelpDesk Agent dashboard
Per-Agent This Week
| Agent | Resolved / Open | First Response | Avg Resolution | Productivity | Quality | Engagement | CSAT | Overall | Justified? |
|---|
Compliance Check Matrix
Click a failing row to see violating tickets| # | Check | Category | Status | Trend (last 8 weeks) | Violations |
|---|
⏰ Upcoming SLA breaches
Recent Actions
| Time | Action | Target | Outcome | Notes |
|---|---|---|---|---|
| Loading… | ||||
| ID | Status | HD Ticket | Requester | Channel / Thread | Started | Duration | Instruction | |
|---|---|---|---|---|---|---|---|---|
| Loading… | ||||||||
Add QA adjustment
Compliance checks — current-week status
| # | Rule | Category | Status | Violations |
|---|
DevOps Engineer dashboard
Learn more
docs/DevOps/decisions-locked.md §5.Settings
Users & Access
Manage who can access each dashboard and at what role level. Owners can set any role; admins can only assign contributor-level access on dashboards they manage.
Presence report
Per-day online time and in-schedule time for selected people over a date range (up to 84 days). Export to Excel for offline review.
AI model
Choose which provider answers every LLM call across the dashboard. Options whose token isn't set in environment variables are shown but can't be selected.
Public holidays
Per-country holiday calendars. A public holiday in a user's country counts as a day off (like PTO) so they aren't flagged idle on it. Applies wherever a user's profile Country is set.
Reassigning — in Jira. Current assignee: —.
712020:…) for contractors not on the roster.
Each row reflects a live event from POST /api/devops/sync-stream.
Rows mark themselves stale on error or unconfigured and the rest of
the sync continues — DOR is the only required source.
Your message will be emailed to the dashboard owner along with your account, current page, browser info, and any attachments. Use this for bug reports, feature ideas, or anything you'd like to flag.