Separate crawling from indexing controls
- Structural
- IF
- OR
- THEN
- ELSE
The decision
- IF
- the goal is to keep a page out of search results
- THEN
- use an appropriate noindex, authentication, removal, redirect, or deletion method
- ELSE
- use robots.txt only to manage crawler requests, not as a deindex guarantee
- Primary metric
- intended index-state accuracy
- Guardrail
- content leakage
- Reason
- Makes the decision explicit, measurable, and reversible where possible.
- LAKA levels
- Structural
- Operators
- IF
- OR
- THEN
- ELSE
- Governs gates
- technical feasibility
- Source
- 04-IF-ELSE-AND-OR-RULEBOOK.md — TEC-004 — Separate crawling from indexing controls
The other eleven rules in Technical Eligibility
- TEC-001 Require public accessibility IF a page is intended for search visibility Baseline
- TEC-002 Use correct response states IF a canonical page is healthy and available Structural
- TEC-003 Validate rendered main content IF essential content or links depend on JavaScript Structural
- TEC-005 Maintain canonical coherence IF multiple URLs represent substantially the same content Structural
- TEC-006 Keep sitemaps canonical and current IF a URL is important, canonical, and intended for indexing Structural
- TEC-007 Redirect to the closest valid successor IF a URL permanently moves or is consolidated Structural
- TEC-008 Treat not-found pages intentionally IF a resource no longer exists and has no valid replacement Structural
- TEC-009 Control faceted and parameter volume IF filters, sorting, search, tracking, or session parameters create many near-duplicate URLs Structural
- TEC-010 Preserve content parity IF mobile, desktop, locale, or rendered versions differ Structural
- TEC-011 Prioritize real-user performance IF field performance or business outcomes show latency, instability, or interaction problems Minor
- TEC-012 Scale crawl work to site size and change rate IF the site is very large, rapidly changing, or shows crawl/index waste Structural
This rule as JSON: GET /v1/rules/TEC-004