Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 11 additions & 4 deletions skills/slop-test-detector/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
name: slop-test-detector
description: "Static analyzer that detects 15 slop patterns in test code β€” tests that compile and pass but catch zero bugs."
description: "Static analyzer that detects 18 slop patterns in test code β€” tests that compile and pass but catch zero bugs."
---

# Slop Test Detector
Expand Down Expand Up @@ -36,6 +36,9 @@ Before auditing or generating tests, identify which slop patterns apply:
| Tests vary their inputs | Do sibling tests exercise different code paths? | `no_input_variation`, `duplicate_assertion_set` |
| Assertions test computed values | Are assertions checking results, not echoing construction literals? | `literal_roundtrip`, `schema_success_only` |
| Assertions always execute | Can the test pass without any assertion running? | `conditional_assertion` |
| Property tests are meaningful | Do fc.property callbacks assert on all paths with varied inputs? | `vacuous_property` |
| Tests exercise production code | Does the test call an imported function, not just builtins? | `no_production_call` |
| Assertions can actually fail | Is the assertion mathematically capable of failing? | `impossible_assertion` |

---

Expand Down Expand Up @@ -64,6 +67,7 @@ import {
formatReport,
formatReportJSON,
getPreset,
parseImports,
parseTestFile,
} from './slop-detector.ts';
```
Expand All @@ -76,9 +80,9 @@ Three built-in presets control which rules are active:

| Preset | Rules | Default threshold | Use case |
|--------|-------|-------------------|----------|
| `balanced` | 10 rules (no defect-comment rules) | 80 | **Default.** Conservative, low noise |
| `strict` | All 12 rules | 90 | Teams that enforce `// Defect:` comments |
| `advisory` | All 12 rules | 0 | Report everything, fail nothing |
| `balanced` | 16 rules (no defect-comment rules) | 80 | **Default.** Conservative, low noise |
| `strict` | All 18 rules | 90 | Teams that enforce `// Defect:` comments |
| `advisory` | All 18 rules | 0 | Report everything, fail nothing |

```typescript
import { getPreset } from './slop-detector.ts';
Expand Down Expand Up @@ -188,6 +192,9 @@ score = max(0, round(100 Γ— (1 - weightedFindings / testCount)))
| `literal_roundtrip` | Assertion compares `obj.field` to the same literal used to construct `obj` | should-fail | on |
| `schema_success_only` | `safeParse()` result checked for `.success` but never `.data` or `.error.issues` | should-fail | on |
| `conditional_assertion` | All assertions are inside `if`/`switch` blocks β€” test may silently pass | must-fail | on |
| `vacuous_property` | `fc.property` callback has `return true` path with zero assertions, or all generators are `fc.constant` | should-fail | on |
| `no_production_call` | Test body calls no imported production function β€” only builtins or language guarantees | should-fail | on |
| `impossible_assertion` | Assertion is mathematically impossible to fail (e.g., `.length >= 0`) | should-fail | on |

**opt-in** rules are only active in the `strict` preset. Use `getPreset('strict')` or add them to a custom `enabledRules` set.

Expand Down
100 changes: 99 additions & 1 deletion skills/slop-test-detector/references/patterns.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Slop Patterns Reference

All 15 slop patterns with before (slop) and after (fixed) examples.
All 18 slop patterns with before (slop) and after (fixed) examples.

---

Expand Down Expand Up @@ -352,3 +352,101 @@ it("handles results", () => {
assert.equal(results[0].status, "ok");
});
```

---

### 16. vacuous_property

**Before (slop)** β€” `return true` path with zero assertions (2a):
```typescript
it('priority correctness', () => {
fc.assert(
fc.property(gameKindGen, eventTypeGen, (gameKind, eventType) => {
if (hasGameSpecific && isPlatformEvent) {
expect(result).toEqual(gameSpecificMeta);
return true;
}
return true; // most inputs hit this path β€” 0 assertions
}),
);
});
```

**Before (slop)** β€” zero-variation generators (2b):
```typescript
it('default constructor', () => {
fc.assert(
fc.property(fc.constant(undefined), (_ignored) => {
const chain = new EventHashChain();
expect(chain.getSemanticHash()).toBeUndefined();
// fc.constant = same test every run
}),
);
});
```

**After (fixed)** β€” assert on all paths with real generators:
```typescript
it('priority correctness', () => {
fc.assert(
fc.property(gameKindGen, eventTypeGen, (gameKind, eventType) => {
const result = resolvePriority(gameKind, eventType);
expect(result).toBeDefined();
expect(typeof result.priority).toBe('number');
}),
);
});
```

---

### 17. no_production_call

**Before (slop)** β€” test only exercises builtins, no production function called:
```typescript
it('expert-level weights must always yield negative net score', () => {
fc.assert(
fc.property(
fc.integer({ min: -100, max: -50 }),
fc.integer({ min: 20, max: 49 }),
(breakingPenalty, creationBonus) => {
const netScore = breakingPenalty + creationBonus;
return netScore < 0; // pure arithmetic β€” no production function
},
),
);
});
```

**After (fixed)** β€” call the actual scoring function:
```typescript
it('expert-level weights must always yield negative net score', () => {
fc.assert(
fc.property(
fc.integer({ min: -100, max: -50 }),
fc.integer({ min: 20, max: 49 }),
(breakingPenalty, creationBonus) => {
const score = computeNetScore(breakingPenalty, creationBonus);
return score < 0;
},
),
);
});
```

---

### 18. impossible_assertion

**Before (slop)** β€” assertion that is mathematically impossible to fail:
```typescript
expect(Object.keys(aliasToCanonical).length).toBeGreaterThanOrEqual(0);
// .length is always >= 0 β€” this CANNOT fail
```

**After (fixed)** β€” assert on a specific expected length:
```typescript
expect(Object.keys(aliasToCanonical).length).toBeGreaterThanOrEqual(1);
// or assert the exact count:
expect(Object.keys(aliasToCanonical)).toHaveLength(5);
```
Loading