01 · ATLAS
model-agnostic knowledge foundation
A versioned dataset system for training and evaluating language models — Qwen, Llama, DeepSeek, Mistral, Gemma, and whatever comes next.
02
Atlas is a governed, deterministic, versioned dataset factory. It starts with raw knowledge — documentation, conversations, code examples, research — and transforms it into structured, machine-readable training data through a scripted pipeline with quality gates.
It is model-agnostic by design. The canonical format is plain JSONL; model-specific chat templates (ChatML, Alpaca, ShareGPT, etc.) are produced by downstream converters and never stored as source of truth. The dataset outlives any single model.
03
I keep building AI tools, and I kept running into the same problem: every time I wanted to experiment with a new model or retrain, the dataset was either locked to one model's format, undocumented in its processing, or impossible to reproduce from scratch.
Most public datasets share the same failure modes — model coupling, weak provenance, no versioning, no audit trail. I wanted something that would survive model transitions and let me reason about what's actually in the data.
The dataset is the long-term asset. Models are replaceable. I built Atlas to make that idea operational.
04
External data, documentation extracts, anonymised conversations,
and personal knowledge exports live under raw/.
Original sources are never modified.
Cleaners, deduplication, and validators transform raw material into canonical Atlas records — structured JSONL with source, license, tags, and quality metadata.
A state machine runs quality checks, provenance resolution,
content revision, and validation. Every transition is persisted
to metadata/.
No dataset reaches release without a signed approval. The gate is mandatory and auditable.
Versioned releases with manifests, SHA-256 checksums, and changelogs. Model-specific formats are generated on demand.
All pipeline scripts are stdlib-only Python. No pip installs, no external dependencies. Deterministic, reproducible, CI-friendly.
05
| Category | Records |
|---|---|
| 01_foundation | 1,000,613 |
| 02_software_engineering | 997,144 |
| 03_system_engineering | 1,039,979 |
| 04_ai_machine_learning | 1,066,501 |
| 05_hardware_engineering | 1,090,289 |
| 06_science_engineering | 1,249,899 |
| 07_business_knowledge | 1,066,944 |
| 08_creative_knowledge | 1,004,557 |
| 09_personal_assistant | 1,000,012 |
v1.0 release — metadata/releases/v1.0_release.json
Every record carries a difficulty classification (L1–L5) and a confidence score. The classifier is deterministic and versioned — same input always produces the same output.
Classification runs are crash-safe: each source is appended to the unified output file immediately after completion, then the per-source file is deleted. A restarted run skips finished sources automatically.
Training views are generated per model family (Qwen, Llama, DeepSeek) from the same canonical dataset. No format is stored as the source of truth.
06
The automation layer is a finite state machine with 11 states and a mandatory human approval gate before any release. It was built as v1.0 across 7 commits, 12 files, 5,122 lines of production Python.
quality_score ≥ 7 for automatic passage.
Every release candidate passes through RC validation before promotion. Promoted releases are immutable — no in-place edits. Manifests include per-file SHA-256 checksums so any tampering is detectable.
07
The Atlas Research Protocol v1.0 governs all future training and evaluation experiments. Every experiment must be reproducible from recorded inputs alone: git commit, dataset checksum, model revision, training config, evaluation version, and random seed.
Qwen2.5-7B, QLoRA on the math training view. The initial delta was evaluator-artifact-driven — a corrected re-score shows no reasoning gain, only answer-format consistency. Directional, not conclusive.
Same model, code domain. Eval set was N=2 — below the research protocol minimum. Results are inconclusive; the eval expansion (Phase 6.2) brought both math and code to N=100.
Qwen/Qwen2.5-7B-Instruct, 4-bit NF4 quantization, bfloat16 compute,
on an RTX 5070 12GB. Inference completed on 29 evaluation samples.
Full results documented in docs/evaluation/.
The Quality Evaluation Engine (QEE) shows a systematic positive bias of +2.14 points compared to human reviewers across 100 matched records from v0.2. Exact agreement is 0%. The QEE never scores below 7 on reviewed records, meaning it would never block a human-rejected item at the current threshold. This is a known calibration gap — the engine is being refined, not used for automated approval decisions yet.
08
The biggest open question is evaluation fidelity. The QEE is useful for filtering obviously bad records, but it's not reliable for approving good ones at the current calibration. I'm exploring whether fine-tuning the scorer on human review signals helps, or whether a hybrid approach (QEE pass + human sample) is the right middle ground.
The cross-domain transfer question is still early — Phase 8 plans a controlled probe to measure whether LoRA trained on math data moves code performance and vice versa. No results yet.
Source licensing is a practical constraint I'm still working
through. About 84% of v1.0 records carry an unknown
license. The license gate blocks NC and proprietary sources at
the curated boundary, but the provenance resolution pipeline
needs more coverage to push that number down.
And model training itself is paused. Not because the dataset isn't ready — the pipeline works — but because I want to get the evaluation story solid before investing compute on full-finetune runs. The pilots gave direction, not answers.
09