apps/web/css_tokenizer.sg (sigil-apps 2ae5bda) is the CSS byte-stream tokenizer for sigilOS browser, following the same pattern as html_tokenizer.sg (de47df3). Source CSS in src_buf at 0x400000; token array in tok_buf at 0x500000 — interop-identical layout with the HTML tokenizer: 12 bytes per token, (type:i32, off:i32, len:i32). State machine: INIT→SELECTOR→BLOCK→PROP→VALUE→COMMENT→ATRULE. Four token types: Selector, Property, Value, AtRule. Handles class selectors (.class), ID selectors (#id), tag selectors, @import/@media/@font-face, and block comments (/* ... */ skipped, previous state restored). Ready for style resolver / render_page integration.
Token model
Identical memory layout to html_tokenizer.sg — the style resolver and render_page.sg read CSS tokens from the same tok_buf format as HTML tokens:
| Token type | Value | What it represents |
|---|---|---|
| Selector (1) | Content of selector before { | h1, .lead, #logo, *, a:hover (trimmed) |
| Property (2) | Name before : | color, font-size, margin-top (trim_end whitespace) |
| Value (3) | Content after : until ; or } | #ffffff, 16px, bold (trim_end whitespace) |
| AtRule (4) | @ keyword + params until ; or { | @import "style.css", @media (max-width:640px) |
Token: 12 bytes at tok_buf + i * 12: type (i32, bytes 0–3), off (i32, bytes 4–7, offset into src_buf), len (i32, bytes 8–11, byte count).
State machine
@: go to ATRULE. On /*: go to COMMENT (save INIT as prev_state). On any selector char: start accumulating, go to SELECTOR.{: emit Selector token (trimmed content), go to BLOCK. On ,: emit Selector token for current group, reset accumulator, continue SELECTOR (comma-separated selector groups each become a token). Handles .class (starts with .), #id (starts with #), tag name, *.{ ... }. Skip whitespace. On }: go back to INIT. On /*: go to COMMENT (save BLOCK as prev_state). On a property name char: start accumulating, go to PROP.:: emit Property token (trim_end trailing spaces), go to VALUE. On /*: go to COMMENT (save PROP as prev_state).;: emit Value token (trim_end), go back to BLOCK. On }: emit Value token, go back to INIT (shorthand last-property with no semicolon). On /*: go to COMMENT (save VALUE as prev_state)./* ... */. Consume bytes until */ found. On */: restore prev_state, continue. No token emitted for comments.@. Accumulate until ; (emit AtRule token, go to INIT) or { (emit AtRule token, go to BLOCK for the at-rule's block). Covers @import (ends at ;), @media (ends at {), @font-face (ends at {).Test document — all 4 token types exercised
Test CSS:
@import "base.css";
h1, h2 { color: #fff; font-size: 24px; }
p.lead { margin-top: 8px; }
#logo { display: block; }
a { text-decoration: none; }
@media (max-width: 640px) { h1 { font-size: 16px; } }Token stream highlights (showing token type, selector/property/value):
| Token | Type | Content |
|---|---|---|
| 0 | AtRule | @import "base.css" |
| 1 | Selector | h1 |
| 2 | Selector | h2 (comma group) |
| 3 | Property | color |
| 4 | Value | #fff |
| 5 | Property | font-size |
| 6 | Value | 24px |
| 7 | Selector | p.lead |
| 8 | Property | margin-top |
| 9 | Value | 8px |
| 10 | Selector | #logo |
| 11 | Property | display |
| 12 | Value | block |
| 13 | Selector | a |
| 14 | Property | text-decoration |
| 15 | Value | none |
| 16 | AtRule | @media (max-width: 640px) |
| 17 | Selector | h1 (inside @media block) |
| 18 | Property | font-size |
| 19 | Value | 16px |
All 4 token types hit. @import ends at ; (AtRule). @media ends at { and its inner h1 block is tokenized inside the at-rule block. Comma-group selectors (h1, h2) each emit a separate Selector token.
Interop with html_tokenizer + render_page integration
The identical (type:i32, off:i32, len:i32) layout at tok_buf 0x500000 is intentional. The style resolver (next) can consume HTML tokens and CSS tokens through the same reader — differentiated by type codes and the source buffer pointer. render_page.sg integration: the style resolver will walk the HTML token stream (html_tokenizer output), look up matching CSS rules (css_tokenizer output), and apply color/font-size/margin values to layout_box nodes before layout_flow. Ready now — 10864B arm-el0, standalone, no deps.
src_buf 0x400000 ← raw CSS bytes
tok_buf 0x500000 ← token array: (type:i32, off:i32, len:i32) × N
html_tokenizer.sg tok_buf 0x300000 type 1–8 (HTML token types)
css_tokenizer.sg tok_buf 0x500000 type 1–4 (CSS token types)
↓
style_resolver.sg (next)
walks both streams → layout_box attrs
↓
layout_flow → render_page.sg