Carlos Salgado commited on
Commit
e832d44
·
1 Parent(s): e9a4841

refactor: 2×2 CVD grid, route protanopia to VLM (#18)

Browse files

* refactor: 2×2 CVD grid, route protanopia to VLM, rename branch to demo-wip

- Replace 10-variant CVD gallery with fixed 2×2 grid (Normal/Protanopia/Deuteranopia/Tritanopia)
- Send Protanopia-simulated image to VLM endpoint (not original screenshot)
- Add image_to_bytes() helper for CVD→VLM serialization
- Rename branch DemoWeep → demo-wip

* fix: send all 4 CVD perspectives to VLM, fix stale tests

- analyze_all_perspectives() calls MiniCPM endpoint for each CVD variant
(Normal, Protanopia, Deuteranopia, Tritanopia) with type-specific prompts
- _merge_cvd_results() deduplicates findings across perspectives
- Removed URL capture / Playwright code; file-upload only
- Removed MODELS dict; single SUPPORTED_MODEL constant
- Updated TestCVDGallery -> TestCVDGrid, TestMODELS -> TestSupportedModel

* add modal deploy script and test, remove duplicate test, update dependencies

Files changed (7) hide show
  1. .pr-body.md +11 -0
  2. app.py +213 -334
  3. pyproject.toml +3 -0
  4. tests/test_app_space.py +25 -27
  5. tests/test_smoke.py +2 -9
  6. uv.lock +17 -0
  7. vlm/accessibility_report.py +0 -379
.pr-body.md ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ## Summary
2
+
3
+ - **Multi-perspective VLM analysis**: The VLM endpoint now receives **all four CVD perspectives** (Normal, Protanopia, Deuteranopia, Tritanopia), each with a type-specific prompt so the model understands which color deficiency it's simulating. Findings are deduplicated into a single aggregated report — simulating a full panel of colorblind testers.
4
+
5
+ - **2×2 CVD grid**: Fixed layout showing Normal (top-left), Protanopia (top-right), Deuteranopia (bottom-left), Tritanopia (bottom-right)
6
+
7
+ - **Only file upload, no URL capture**: Removed all Playwright/URL capture code to comply with HF Spaces constraints. Only user-uploaded screenshots accepted.
8
+
9
+ - **Single model, no dropdown clutter**: Model registry simplified to just `minicpm-v-4.6` since that's the only working endpoint. No unsupported model choices in the UI.
10
+
11
+ - **Stale tests updated**: Renamed `TestCVDGallery` → `TestCVDGrid` with 4-item assertions; replaced `TestMODELS` → `TestSupportedModel`
app.py CHANGED
@@ -2,56 +2,38 @@
2
  Color-UX-Access — Gradio application
3
  =====================================
4
  Single-file Gradio app for colorblind accessibility testing.
5
-
6
- Usage:
7
- # HF Spaces (file upload only — no Playwright needed):
8
- # app_file: app.py in Space settings → runs automatically
9
-
10
- # Local development (file upload OR URL capture):
11
- python app.py # file upload mode
12
 
13
  Architecture:
14
  Screenshot (file upload)
15
-
16
-
17
- Stage 1: CVD Simulation (CPU) 10 variants
18
-
19
-
20
- Stage 2: VLM Inference (GPU via Modal endpoint) WCAG 2.1 JSON
21
-
22
-
23
  Stage 3: Report (Markdown)
24
 
25
  Requirements:
26
  - Python 3.12
27
- - gradio>=6.0, spaces (for HF Space deployment)
28
- - torch with CUDA libs
29
- - openai, pillow, daltonlens, requests, python-dotenv
30
- - huggingface_hub==0.25.2 (HfFolder removed in 0.26)
31
- - playwright (optional, for URL capture mode — not needed on Space)
32
 
33
  Local setup:
34
  uv sync --python 3.12
35
- playwright install chromium # only for --url mode
36
 
37
  HF Space deploy:
38
  1. Push to GitHub
39
  2. Create HF Space (SDK: Gradio, hardware: T4/mega or A10G)
40
- 3. Add MODAL_URL secret in Space settings (or MODAL_INFERENCE_BASE_URL + MODAL_INFERENCE_API_KEY)
41
  4. Link to GitHub repo
42
-
43
- Note: HF_TOKEN in Space secrets is for Space management only.
44
- Inference goes through the Modal endpoint — no HF_TOKEN needed here.
45
-
46
- Two VLM modes:
47
- A) Legacy — MODAL_URL (Gradio API endpoint) — falls back automatically
48
- B) Provider — MODAL_INFERENCE_BASE_URL + MODAL_INFERENCE_API_KEY (OpenAI-compatible)
49
  """
50
 
51
  import os
52
  import io
53
  import json
54
- import sys
55
 
56
  import gradio as gr
57
  from PIL import Image
@@ -59,9 +41,17 @@ import numpy as np
59
  from daltonlens import simulate
60
  import requests
61
  import base64
62
- from openai import OpenAI
63
 
64
- # ── CVD Simulators ────────────────────────────────────────────────────────────
 
 
 
 
 
 
 
 
 
65
 
66
  simulator = simulate.Simulator_Machado2009()
67
  severe_simulator = simulate.Simulator_Vienot1999()
@@ -78,28 +68,8 @@ deficiency_config = {
78
  'tritanomaly': {'simulator': tritan_simulator, 'severity': 0.4, 'deficiency': simulate.Deficiency.TRITAN},
79
  }
80
 
81
- # ── Swappable VLM Model Registry ──────────────────────────────────────────────
82
- # One-line swap for different sponsor prize eligibility:
83
- # - aya-vision-32b → CohereLabs/aya-vision-32b (default, Cohere prize)
84
- # - minicpm-v-4.6 → openbmb/mini-cpm-v-4_6 (OpenBMB $5K prize)
85
- # - nemotron-15b → nvidia/Nemotron-4-15B-base (NVIDIA prize, if required)
86
- MODELS = {
87
- "aya-vision-32b": {
88
- "provider": "cohere",
89
- "model_id": "CohereLabs/aya-vision-32b",
90
- "description": "Default — 32B vision model via HF Router",
91
- },
92
- "minicpm-v-4.6": {
93
- "provider": "openbmb",
94
- "model_id": "openbmb/mini-cpm-v-4_6",
95
- "description": "OpenBMB prize — MiniCPM-V 4.6 (~4B params, under 32B cap)",
96
- },
97
- "nemotron-15b": {
98
- "provider": "nvidia",
99
- "model_id": "nvidia/Nemotron-4-15B-base",
100
- "description": "NVIDIA prize — confirm Nemotron requirement with organizers",
101
- },
102
- }
103
 
104
 
105
  def simulate_cvd(image: Image.Image, sim, deficiency, severity) -> Image.Image:
@@ -118,35 +88,51 @@ def simulate_achromatopsia(image: Image.Image, severity: float) -> Image.Image:
118
  return gray_rgb
119
 
120
 
 
 
 
 
 
 
121
  def generate_cvd_gallery(original: Image.Image) -> list[tuple[Image.Image, str]]:
122
- """Generate all 10 CVD simulation variants for the gallery."""
123
- results = []
124
- for name, cfg in deficiency_config.items():
125
- img = simulate_cvd(
126
- original, cfg['simulator'], cfg['deficiency'], cfg['severity']
127
- )
128
- label = name.replace('_', ' ').title()
129
- results.append((img, label))
130
- results.append((simulate_achromatopsia(original, 1.0), 'Achromatopsia'))
131
- results.append((simulate_achromatopsia(original, 0.5), 'Achromatomaly'))
132
- return results
 
 
 
 
 
 
 
 
 
 
133
 
134
 
135
  def format_wcag_report(vlm_result: dict) -> str:
136
  """Convert VLM JSON output into a formatted markdown report."""
137
  if 'error' in vlm_result:
138
- return f"⚠️ **Error:** {vlm_result['error']}"
139
 
140
  findings = vlm_result.get('findings', [])
141
  if not findings:
142
  if vlm_result.get('passes', False):
143
- return "Pass No accessibility issues detected."
144
- return "✅ **No accessibility issues detected.**"
145
 
146
  report = "## WCAG Accessibility Report\n\n"
147
- report += f"**Overall:** {'Pass' if vlm_result.get('passes', False) else 'Fail'}\n\n"
148
 
149
- severity_icons = {'critical': '🔴', 'serious': '🟠', 'moderate': '🟡'}
150
  wcag_links = {
151
  '1.1.1': 'https://www.w3.org/WAI/WCAG21/Understanding/non-text-content',
152
  '1.4.1': 'https://www.w3.org/WAI/WCAG21/Understanding/use-of-color',
@@ -155,12 +141,15 @@ def format_wcag_report(vlm_result: dict) -> str:
155
  }
156
 
157
  for i, f in enumerate(findings, 1):
158
- icon = severity_icons.get(f.get('severity', 'moderate'), '⚪')
159
  wcag = f.get('wcag_criterion', 'N/A')
160
  link = wcag_links.get(wcag, '#')
 
161
  report += f"### {icon} Issue {i}: {f.get('type', 'Unknown')}\n\n"
162
  report += f"- **WCAG:** [{wcag}]({link})\n"
163
  report += f"- **Severity:** {f.get('severity', 'N/A').capitalize()}\n"
 
 
164
  report += f"- **Description:** {f.get('description', 'N/A')}\n"
165
  report += f"- **Location:** {f.get('location', 'N/A')}\n\n"
166
 
@@ -170,195 +159,146 @@ def format_wcag_report(vlm_result: dict) -> str:
170
  return report
171
 
172
 
173
- # ── URL Capture (optional — Playwright, local dev only) ───────────────────────
174
-
175
- def _get_playwright():
176
- """Lazy-import Playwright only when URL mode is requested."""
177
- try:
178
- from playwright.sync_api import sync_playwright
179
- return sync_playwright
180
- except ImportError:
181
- raise RuntimeError(
182
- "Playwright not installed. Run: uv pip install playwright && playwright install chromium"
183
- )
184
-
185
-
186
- def _capture_url(url: str) -> bytes:
187
- """Capture a screenshot of a URL using Playwright (headless Chromium)."""
188
- pw = _get_playwright()
189
- with pw() as p:
190
- browser = p.chromium.launch(
191
- headless=True,
192
- args=['--no-sandbox', '--disable-setuid-sandbox'],
193
- )
194
- page = browser.new_page(viewport={'width': 1280, 'height': 720})
195
- page.goto(url, wait_until='networkidle', timeout=60000)
196
- page.wait_for_timeout(3000)
197
- screenshot = page.screenshot(full_page=True, timeout=60000)
198
- browser.close()
199
- return screenshot
200
-
201
-
202
- # ── Modal Endpoint Helper ──────────────────────────────────────────────────────
203
- # Inference runs via the deployed Modal app, not HF Router directly.
204
- # The Modal app (color_ux_access/modal_app.py) handles GPU/VLM internally.
205
- #
206
- # Two modes (selectable by env var):
207
- # Legacy mode (default): MODAL_URL
208
- # Provider mode (OpenAI): MODAL_INFERENCE_BASE_URL + MODAL_INFERENCE_API_KEY
209
- # If MODAL_INFERENCE_API_KEY is not set, analyze_with_vlm falls back to legacy.
210
-
211
- _MODAL_URL = os.environ.get('MODAL_URL', 'https://narwall-tech--color-ux-access-ui.modal.run')
212
-
213
- # Modal Inference Provider configuration
214
- _MODAL_INFERENCE_BASE_URL = os.environ.get('MODAL_INFERENCE_BASE_URL', 'https://inference.modal.com/v1')
215
- _MODAL_INFERENCE_API_KEY = os.environ.get('MODAL_INFERENCE_API_KEY') # from modal secret modal-inference-key
216
-
217
 
218
- def _call_modal_analyze(image_bytes: bytes, timeout: int = 120) -> dict:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
219
  """
220
- Call the Modal Gradio endpoint's /analyze_screenshot API.
221
- Upload image POST predict poll SSE for result.
222
-
223
- Args:
224
- image_bytes: PNG/JPEG bytes of the screenshot.
225
- timeout: Max seconds to wait for VLM inference result.
226
-
227
- Returns:
228
- WCAG JSON dict with keys: findings, passes, summary.
229
-
230
- Raises:
231
- RuntimeError: If upload, predict, or polling fails.
232
  """
233
- gradio_api = f"{_MODAL_URL}/gradio_api"
234
-
235
- upload_resp = requests.post(
236
- f"{gradio_api}/upload",
237
- files={'files': ('screenshot.png', image_bytes, 'image/png')},
238
- timeout=30,
239
- )
240
- if upload_resp.status_code != 200:
241
- raise RuntimeError(f"Modal file upload failed: {upload_resp.status_code} {upload_resp.text[:100]}")
242
 
243
- file_paths = upload_resp.json()
244
- if not file_paths:
245
- raise RuntimeError(f"Modal file upload returned no paths: {upload_resp.text[:100]}")
246
 
247
- file_path = file_paths[0]
 
 
 
 
 
 
 
248
 
249
- predict_resp = requests.post(
250
- f"{gradio_api}/call/analyze_screenshot",
251
- json={'data': [{'path': file_path, 'meta': {'_type': 'gradio.FileData'}}]},
252
- timeout=10,
253
- )
254
- if predict_resp.status_code != 200:
255
- raise RuntimeError(f"Modal predict call failed: {predict_resp.status_code} {predict_resp.text[:100]}")
256
 
257
- event_id = predict_resp.json().get('event_id')
258
- if not event_id:
259
- raise RuntimeError(f"Modal predict returned no event_id: {predict_resp.text[:100]}")
260
 
261
- poll_url = f"{gradio_api}/call/analyze_screenshot/{event_id}"
262
- with requests.get(poll_url, timeout=timeout, stream=True) as poll_resp:
263
- if poll_resp.status_code != 200:
264
- raise RuntimeError(f"Modal poll failed: {poll_resp.status_code}")
265
 
266
- full_data = ''
267
- for line in poll_resp.iter_lines():
268
- if line:
269
- decoded = line.decode('utf-8')
270
- if decoded.startswith('data: '):
271
- full_data = decoded[6:]
272
 
273
- if not full_data:
274
- raise RuntimeError("Modal poll returned no data")
 
275
 
276
- result = json.loads(full_data)
277
- if isinstance(result, list):
278
- result = result[0]
279
- return result
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
280
 
281
 
282
- # ── VLM Inference (via Modal Inference Provider) ──────────────────────────────────────
283
- def analyze_with_vlm(image_bytes: bytes, model: str = "aya-vision-32b") -> dict:
284
  """
285
- Analyze a screenshot for WCAG color-accessibility issues via Modal Inference Provider.
286
- Falls back to the legacy Gradio endpoint (_call_modal_analyze) if the new
287
- MODAL_INFERENCE_API_KEY environment variable is not configured.
288
- """
289
- if not _MODAL_INFERENCE_API_KEY:
290
- return _call_modal_analyze(image_bytes)
291
 
292
- try:
293
- # Prepare OpenAI client pointing to Modal inference endpoint
294
- client = OpenAI(
295
- base_url=_MODAL_INFERENCE_BASE_URL,
296
- api_key=_MODAL_INFERENCE_API_KEY,
297
- )
298
-
299
- # Encode image to base64
300
- image_b64 = base64.b64encode(image_bytes).decode('utf-8')
301
-
302
- # System prompt from modal_app.py (adjusted)
303
- system_prompt = (
304
- "You are an accessibility expert specializing in colorblind user experience. "
305
- "Analyze screenshots for WCAG 2.1 compliance issues. "
306
- "For each finding, cite the specific success criterion (1.1.1, 1.4.1, 1.4.3, or 1.4.11). "
307
- "Output a JSON object with this structure:\n"
308
- "{\n"
309
- " \"findings\": [\n"
310
- " {\n"
311
- " \"type\": \"Low Contrast | Color Only Information | Missing Text Alternative | Insufficient Non-Text Contrast\",\n"
312
- " \"wcag_criterion\": \"1.4.1 | 1.4.3 | 1.1.1 | 1.4.11\",\n"
313
- " \"description\": \"...\",\n"
314
- " \"severity\": \"critical | serious | moderate\",\n"
315
- " \"location\": \"Top-left, center, etc.\",\n"
316
- " }\n"
317
- " ],\n"
318
- " \"summary\": \"Overall assessment\",\n"
319
- " \"passes\": true/false\n"
320
- "}\n"
321
- )
322
-
323
- # Determine actual model ID from MODELS dict
324
- model_info = MODELS.get(model, {})
325
- model_id = model_info.get("model_id", model) # fallback to model key
326
-
327
- response = client.chat.completions.create(
328
- model=model_id,
329
- messages=[
330
- {
331
- "role": "user",
332
- "content": [
333
- {"type": "text", "text": system_prompt},
334
- {
335
- "type": "image_url",
336
- "image_url": {"url": f"data:image/png;base64,{image_b64}"},
337
- },
338
- ],
339
- }
340
- ],
341
- max_tokens=1024,
342
- temperature=0.1,
343
- )
344
-
345
- content = response.choices[0].message.content
346
- if content.startswith("```"):
347
- # Strip code fences if present
348
- lines = content.split("\n")
349
- content = "\n".join(lines[1:-1])
350
- return json.loads(content)
351
- except Exception as e:
352
- return {"error": str(e), "findings": [], "passes": False}
353
-
354
- # ── URL Mode Flag ──────────────────────────────────────────────────────────────
355
- # Set --url on the command line to enable URL capture input.
356
- # On HF Spaces, __file__ is set and --url is not passed, so URL mode stays off.
357
 
358
- _url_mode = '--url' in sys.argv
359
 
360
 
361
- # ── Gradio App ────────────────────────────────────────────────────────────────
362
 
363
  _theme_css = """
364
  :root { --color-primary: #1E88E5; }
@@ -386,117 +326,60 @@ with gr.Blocks(
386
  '1. Capture your screen (OS/Browser screenshot tool)\n'
387
  '2. Upload the screenshot below\n'
388
  '3. Get CVD simulations + WCAG 2.1 accessibility report\n\n'
389
- '⚠️ First analysis takes ~60–90s (model loads once, then stays cached).'
 
 
390
  )
391
 
392
  with gr.Row():
393
- if _url_mode:
394
- url_input = gr.Textbox(label='Website URL', placeholder='https://example.com', scale=1)
395
- submit_btn = gr.Button('Capture & Analyze', variant='primary', scale=0)
396
- else:
397
- file_input = gr.File(
398
- label='Screenshot',
399
- file_types=['.png', '.jpg', '.jpeg', '.webp'],
400
- type='binary',
401
- height=80,
402
- )
403
- submit_btn = gr.Button('Analyze', variant='primary', scale=0)
404
-
405
- model_select = gr.Dropdown(
406
- choices=list(MODELS.keys()),
407
- value="aya-vision-32b",
408
- label='VLM Model',
409
- info='Switch models for different sponsor prize eligibility',
410
  )
 
411
 
412
  with gr.Row():
413
- with gr.Column(scale=1):
414
- original_output = gr.Image(label='Original', type='pil')
415
- with gr.Column(scale=2):
416
- cvd_output = gr.Gallery(
417
- label='CVD Simulations (10 types)',
418
- columns=4,
419
- rows=3,
420
- object_fit='contain',
421
- height='auto',
422
- )
423
 
424
  report_output = gr.Markdown(label='Accessibility Report')
425
 
426
- # ── Event Handlers ─────────────────────────────────────────────────────────
427
-
428
- def run_analysis_from_file(file_obj, model: str = "aya-vision-32b"):
429
- """File upload mode — used on HF Spaces."""
430
  if file_obj is None:
431
- return None, [], '⚠️ Please upload a screenshot first.'
432
 
433
  image_bytes = file_obj if isinstance(file_obj, bytes) else file_obj.read()
434
 
435
  try:
436
  original = Image.open(io.BytesIO(image_bytes)).convert('RGB')
437
  except Exception as e:
438
- return None, [], f'⚠️ Could not open image: {e}'
439
 
440
- cvd_gallery = generate_cvd_gallery(original)
441
 
 
 
442
  try:
443
- vlm_result = analyze_with_vlm(image_bytes, model=model)
444
  except Exception as e:
445
  vlm_result = {'error': str(e), 'findings': [], 'passes': False}
446
 
447
  report_md = format_wcag_report(vlm_result)
448
- return original, cvd_gallery, report_md
449
-
450
- def run_analysis_from_url(url: str, model: str = "aya-vision-32b"):
451
- """URL capture mode — Playwright local dev only."""
452
- if not url:
453
- return None, [], '⚠️ Please enter a URL first.'
454
-
455
- try:
456
- image_bytes = _capture_url(url)
457
- except Exception as e:
458
- return None, [], f'⚠️ Could not capture URL: {e}'
459
 
460
- try:
461
- original = Image.open(io.BytesIO(image_bytes)).convert('RGB')
462
- except Exception as e:
463
- return None, [], f'⚠️ Could not open screenshot: {e}'
464
-
465
- cvd_gallery = generate_cvd_gallery(original)
466
-
467
- try:
468
- vlm_result = analyze_with_vlm(image_bytes, model=model)
469
- except Exception as e:
470
- vlm_result = {'error': str(e), 'findings': [], 'passes': False}
471
-
472
- report_md = format_wcag_report(vlm_result)
473
- return original, cvd_gallery, report_md
474
-
475
- if _url_mode:
476
- submit_btn.click(
477
- fn=run_analysis_from_url,
478
- inputs=[url_input, model_select],
479
- outputs=[original_output, cvd_output, report_output],
480
- )
481
- gr.Examples(
482
- examples=[
483
- ["https://www.google.com"],
484
- ["https://www.wikipedia.org"],
485
- ["https://www.apple.com"],
486
- ],
487
- inputs=url_input,
488
- outputs=[original_output, cvd_output, report_output],
489
- fn=run_analysis_from_url,
490
- cache_examples=False,
491
- )
492
- else:
493
- submit_btn.click(
494
- fn=run_analysis_from_file,
495
- inputs=[file_input, model_select],
496
- outputs=[original_output, cvd_output, report_output],
497
- )
498
- gr.Markdown('---')
499
- gr.Markdown('*Upload a screenshot or use URL capture mode (`python app.py --url`).*')
500
 
501
 
502
  if __name__ == '__main__':
@@ -504,11 +387,7 @@ if __name__ == '__main__':
504
  demo.launch(
505
  server_name='0.0.0.0',
506
  server_port=7860,
507
- theme=gr.themes.Base(
508
- primary_hue='blue',
509
- secondary_hue='gray',
510
- neutral_hue='gray',
511
- ),
512
  css=_theme_css,
513
  )
514
  else:
 
2
  Color-UX-Access — Gradio application
3
  =====================================
4
  Single-file Gradio app for colorblind accessibility testing.
5
+ Only accepts user-uploaded screenshots (no URL capture / browser automation).
 
 
 
 
 
 
6
 
7
  Architecture:
8
  Screenshot (file upload)
9
+ |
10
+ v
11
+ Stage 1: CVD Simulation (CPU) -> 3-type comparison grid
12
+ |
13
+ v
14
+ Stage 2: VLM Inference (GPU via Modal endpoint) -> WCAG 2.1 JSON
15
+ |
16
+ v
17
  Stage 3: Report (Markdown)
18
 
19
  Requirements:
20
  - Python 3.12
21
+ - gradio>=6.0, pillow, daltonlens, requests, python-dotenv
 
 
 
 
22
 
23
  Local setup:
24
  uv sync --python 3.12
25
+ cp .env.example .env # set MODAL_INFERENCE_URL
26
 
27
  HF Space deploy:
28
  1. Push to GitHub
29
  2. Create HF Space (SDK: Gradio, hardware: T4/mega or A10G)
30
+ 3. Add MODAL_INFERENCE_URL secret in Space settings
31
  4. Link to GitHub repo
 
 
 
 
 
 
 
32
  """
33
 
34
  import os
35
  import io
36
  import json
 
37
 
38
  import gradio as gr
39
  from PIL import Image
 
41
  from daltonlens import simulate
42
  import requests
43
  import base64
 
44
 
45
+ from custom_theme import color_ux_access_theme
46
+
47
+ # -- Load .env if available ---------------------------------------------------
48
+ try:
49
+ from dotenv import load_dotenv
50
+ load_dotenv()
51
+ except ImportError:
52
+ pass
53
+
54
+ # -- CVD Simulators -----------------------------------------------------------
55
 
56
  simulator = simulate.Simulator_Machado2009()
57
  severe_simulator = simulate.Simulator_Vienot1999()
 
68
  'tritanomaly': {'simulator': tritan_simulator, 'severity': 0.4, 'deficiency': simulate.Deficiency.TRITAN},
69
  }
70
 
71
+ SUPPORTED_MODEL = "minicpm-v-4.6"
72
+ _MODAL_INFERENCE_URL = os.environ.get('MODAL_INFERENCE_URL')
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
73
 
74
 
75
  def simulate_cvd(image: Image.Image, sim, deficiency, severity) -> Image.Image:
 
88
  return gray_rgb
89
 
90
 
91
+ def image_to_bytes(img: Image.Image, fmt: str = 'PNG') -> bytes:
92
+ """Serialize a PIL Image to bytes for VLM transmission."""
93
+ buf = io.BytesIO()
94
+ img.save(buf, format=fmt)
95
+ return buf.getvalue()
96
+
97
  def generate_cvd_gallery(original: Image.Image) -> list[tuple[Image.Image, str]]:
98
+ """Alias for generate_cvd_grid."""
99
+ return generate_cvd_grid(original)
100
+
101
+ def generate_cvd_grid(original: Image.Image) -> list[tuple[Image.Image, str]]:
102
+ """Generate the 2x2 CVD comparison grid.
103
+
104
+ Fixed layout:
105
+ Top-left: Normal vision (original design)
106
+ Top-right: Protanopia (red-blind)
107
+ Bottom-left: Deuteranopia (green-blind)
108
+ Bottom-right: Tritanopia (blue-blind)
109
+ """
110
+ protan = simulate_cvd(original, simulator, simulate.Deficiency.PROTAN, 0.8)
111
+ deuter = simulate_cvd(original, simulator, simulate.Deficiency.DEUTAN, 0.8)
112
+ tritan = simulate_cvd(original, tritan_simulator, simulate.Deficiency.TRITAN, 0.8)
113
+ return [
114
+ (original, "Normal vision (original design)"),
115
+ (protan, "Protanopia (red-blind)"),
116
+ (deuter, "Deuteranopia (green-blind)"),
117
+ (tritan, "Tritanopia (blue-blind)"),
118
+ ]
119
 
120
 
121
  def format_wcag_report(vlm_result: dict) -> str:
122
  """Convert VLM JSON output into a formatted markdown report."""
123
  if 'error' in vlm_result:
124
+ return f"Warning: {vlm_result['error']}"
125
 
126
  findings = vlm_result.get('findings', [])
127
  if not findings:
128
  if vlm_result.get('passes', False):
129
+ return "Pass -- No accessibility issues detected."
130
+ return "No accessibility issues detected."
131
 
132
  report = "## WCAG Accessibility Report\n\n"
133
+ report += f"**Overall:** {'Pass' if vlm_result.get('passes', False) else 'Fail'}\n\n"
134
 
135
+ severity_icons = {'critical': ':red_circle:', 'serious': ':orange_circle:', 'moderate': ':yellow_circle:'}
136
  wcag_links = {
137
  '1.1.1': 'https://www.w3.org/WAI/WCAG21/Understanding/non-text-content',
138
  '1.4.1': 'https://www.w3.org/WAI/WCAG21/Understanding/use-of-color',
 
141
  }
142
 
143
  for i, f in enumerate(findings, 1):
144
+ icon = severity_icons.get(f.get('severity', 'moderate'))
145
  wcag = f.get('wcag_criterion', 'N/A')
146
  link = wcag_links.get(wcag, '#')
147
+ cvd_perspective = f.get('cvd_perspective', '')
148
  report += f"### {icon} Issue {i}: {f.get('type', 'Unknown')}\n\n"
149
  report += f"- **WCAG:** [{wcag}]({link})\n"
150
  report += f"- **Severity:** {f.get('severity', 'N/A').capitalize()}\n"
151
+ if cvd_perspective:
152
+ report += f"- **CVD Perspective:** {cvd_perspective}\n"
153
  report += f"- **Description:** {f.get('description', 'N/A')}\n"
154
  report += f"- **Location:** {f.get('location', 'N/A')}\n\n"
155
 
 
159
  return report
160
 
161
 
162
+ # -- VLM Inference ------------------------------------------------------------
163
+
164
+ _VLM_CVD_PROMPTS = {
165
+ "Normal vision (original design)": (
166
+ "You are an accessibility expert viewing this page with normal color vision. "
167
+ "Analyze it for WCAG 2.1 compliance issues. "
168
+ "Focus on contrast, color usage, and text readability as a fully sighted user."
169
+ ),
170
+ "Protanopia (red-blind)": (
171
+ "You have protanopia (red-blind CVD). Analyze this page as it appears to you. "
172
+ "Focus on WCAG 2.1 compliance issues that affect a protanope: "
173
+ "red-green color confusion, information conveyed solely by red, "
174
+ "and contrast problems specific to your condition."
175
+ ),
176
+ "Deuteranopia (green-blind)": (
177
+ "You have deuteranopia (green-blind CVD). Analyze this page as it appears to you. "
178
+ "Focus on WCAG 2.1 compliance issues that affect a deuteranope: "
179
+ "green-red color confusion, information conveyed solely by green, "
180
+ "and contrast problems specific to your condition."
181
+ ),
182
+ "Tritanopia (blue-blind)": (
183
+ "You have tritanopia (blue-blind CVD). Analyze this page as it appears to you. "
184
+ "Focus on WCAG 2.1 compliance issues that affect a tritanope: "
185
+ "blue-yellow color confusion, information conveyed solely by blue, "
186
+ "and contrast problems specific to your condition."
187
+ ),
188
+ }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
189
 
190
+ _ACCESSIBILITY_SYSTEM_PROMPT = (
191
+ "Output a JSON object with this structure:\n"
192
+ "{\n"
193
+ ' "findings": [\n'
194
+ " {\n"
195
+ ' "type": "Low Contrast | Color Only Information | Missing Text Alternative | Insufficient Non-Text Contrast",\n'
196
+ ' "wcag_criterion": "1.4.1 | 1.4.3 | 1.1.1 | 1.4.11",\n'
197
+ ' "description": "...",\n'
198
+ ' "severity": "critical | serious | moderate",\n'
199
+ ' "location": "Top-left, center, etc."\n'
200
+ " }\n"
201
+ " ],\n"
202
+ ' "summary": "Overall assessment from your perspective",\n'
203
+ ' "passes": true/false\n'
204
+ "}\n"
205
+ "Return ONLY valid JSON -- no markdown fences, no commentary."
206
+ )
207
+
208
+
209
+ def _call_minicpm_endpoint(image_bytes: bytes, system_prompt: str) -> dict:
210
  """
211
+ Call the MiniCPM vLLM endpoint on Modal directly.
212
+ POST to MODAL_INFERENCE_URL with base64 image + accessibility prompt.
 
 
 
 
 
 
 
 
 
 
213
  """
214
+ if not _MODAL_INFERENCE_URL:
215
+ return {"error": "MODAL_INFERENCE_URL not set. Configure in .env or Space secrets.", "findings": [], "passes": False}
 
 
 
 
 
 
 
216
 
217
+ image_b64 = base64.b64encode(image_bytes).decode('utf-8')
218
+ payload = {"prompt": system_prompt, "image_base64": image_b64}
 
219
 
220
+ try:
221
+ resp = requests.post(_MODAL_INFERENCE_URL, json=payload, timeout=180)
222
+ resp.raise_for_status()
223
+ data = resp.json()
224
+ except requests.exceptions.Timeout:
225
+ return {"error": "MiniCPM endpoint timed out (cold-start may need ~90s). Try again.", "findings": [], "passes": False}
226
+ except Exception as e:
227
+ return {"error": f"MiniCPM endpoint call failed: {e}", "findings": [], "passes": False}
228
 
229
+ raw = data.get("response", "")
230
+ if not raw:
231
+ return {"error": "MiniCPM returned empty response", "findings": [], "passes": False}
 
 
 
 
232
 
233
+ if raw.startswith("```"):
234
+ lines = raw.split("\n")
235
+ raw = "\n".join(lines[1:-1])
236
 
237
+ try:
238
+ return json.loads(raw)
239
+ except json.JSONDecodeError:
240
+ return {"error": f"MiniCPM returned non-JSON: {raw[:500]}", "findings": [], "passes": False}
241
 
 
 
 
 
 
 
242
 
243
+ def _merge_cvd_results(results: dict[str, dict]) -> dict:
244
+ """
245
+ Merge VLM results from multiple CVD perspectives into a single report.
246
 
247
+ Deduplicates findings by description (same issue flagged by multiple
248
+ CVD types only appears once) and aggregates summaries.
249
+ """
250
+ all_findings = []
251
+ summaries = []
252
+ overall_passes = True
253
+
254
+ for cvd_label, result in results.items():
255
+ if "error" in result:
256
+ summaries.append(f"{cvd_label}: {result['error']}")
257
+ continue
258
+ if not result.get("passes", False):
259
+ overall_passes = False
260
+ for finding in result.get("findings", []):
261
+ finding["cvd_perspective"] = cvd_label
262
+ all_findings.append(finding)
263
+ if result.get("summary"):
264
+ summaries.append(f"{cvd_label}: {result['summary']}")
265
+
266
+ # Deduplicate by description hash
267
+ seen = set()
268
+ unique = []
269
+ for f in all_findings:
270
+ key = f.get("description", "")[:80]
271
+ if key and key not in seen:
272
+ seen.add(key)
273
+ unique.append(f)
274
+
275
+ return {
276
+ "findings": unique,
277
+ "summary": " | ".join(summaries) if summaries else "Multi-perspective analysis complete.",
278
+ "passes": overall_passes,
279
+ }
280
 
281
 
282
+ def analyze_all_perspectives(cvd_grid: list) -> dict:
 
283
  """
284
+ Run VLM analysis on each CVD perspective sequentially.
 
 
 
 
 
285
 
286
+ Each CVD variant gets a type-specific prompt so the model understands
287
+ which color deficiency it's simulating. Results are merged into a
288
+ single deduplicated report simulating a full panel of colorblind testers.
289
+ """
290
+ results = {}
291
+ for img, label in cvd_grid:
292
+ role_prompt = _VLM_CVD_PROMPTS.get(label, _VLM_CVD_PROMPTS["Normal vision (original design)"])
293
+ full_prompt = f"{role_prompt}\n\n{_ACCESSIBILITY_SYSTEM_PROMPT}"
294
+ img_bytes = image_to_bytes(img)
295
+ result = _call_minicpm_endpoint(img_bytes, full_prompt)
296
+ results[label] = result
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
297
 
298
+ return _merge_cvd_results(results)
299
 
300
 
301
+ # -- Gradio App ---------------------------------------------------------------
302
 
303
  _theme_css = """
304
  :root { --color-primary: #1E88E5; }
 
326
  '1. Capture your screen (OS/Browser screenshot tool)\n'
327
  '2. Upload the screenshot below\n'
328
  '3. Get CVD simulations + WCAG 2.1 accessibility report\n\n'
329
+ 'The VLM analyzes **all four CVD perspectives** (Normal, Protanopia, Deuteranopia, '
330
+ 'Tritanopia) -- simulating a full panel of colorblind testers.\n\n'
331
+ 'Note: First analysis takes ~60-90s (MiniCPM cold-start on Modal GPU).'
332
  )
333
 
334
  with gr.Row():
335
+ file_input = gr.File(
336
+ label='Screenshot',
337
+ file_types=['.png', '.jpg', '.jpeg', '.webp'],
338
+ type='binary',
339
+ height=80,
 
 
 
 
 
 
 
 
 
 
 
 
340
  )
341
+ submit_btn = gr.Button('Analyze', variant='primary', scale=0)
342
 
343
  with gr.Row():
344
+ cvd_grid = gr.Gallery(
345
+ label='Color-Vision Comparison (2x2 grid)',
346
+ columns=2,
347
+ rows=2,
348
+ object_fit='contain',
349
+ height=600,
350
+ )
 
 
 
351
 
352
  report_output = gr.Markdown(label='Accessibility Report')
353
 
354
+ def run_analysis(file_obj):
355
+ """File upload mode -- receives screenshot bytes, returns CVD grid + report."""
 
 
356
  if file_obj is None:
357
+ return [], 'Please upload a screenshot first.'
358
 
359
  image_bytes = file_obj if isinstance(file_obj, bytes) else file_obj.read()
360
 
361
  try:
362
  original = Image.open(io.BytesIO(image_bytes)).convert('RGB')
363
  except Exception as e:
364
+ return [], f'Could not open image: {e}'
365
 
366
+ grid = generate_cvd_grid(original)
367
 
368
+ # Send ALL CVD perspectives to the VLM endpoint
369
+ # Each variant gets a role-specific prompt (Normal, Protanopia, Deuteranopia, Tritanopia)
370
  try:
371
+ vlm_result = analyze_all_perspectives(grid)
372
  except Exception as e:
373
  vlm_result = {'error': str(e), 'findings': [], 'passes': False}
374
 
375
  report_md = format_wcag_report(vlm_result)
376
+ return grid, report_md
 
 
 
 
 
 
 
 
 
 
377
 
378
+ submit_btn.click(
379
+ fn=run_analysis,
380
+ inputs=file_input,
381
+ outputs=[cvd_grid, report_output],
382
+ )
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
383
 
384
 
385
  if __name__ == '__main__':
 
387
  demo.launch(
388
  server_name='0.0.0.0',
389
  server_port=7860,
390
+ theme=color_ux_access_theme,
 
 
 
 
391
  css=_theme_css,
392
  )
393
  else:
pyproject.toml CHANGED
@@ -26,6 +26,9 @@ dependencies = [
26
  "requests>=2.34.2",
27
  "modal>=1.4.3",
28
  "openai>=1.0",
 
 
 
29
  ]
30
 
31
  [project.optional-dependencies]
 
26
  "requests>=2.34.2",
27
  "modal>=1.4.3",
28
  "openai>=1.0",
29
+ "pytest>=9.0.3",
30
+ "dotenv>=0.9.9",
31
+ "colorspacious>=1.1.2",
32
  ]
33
 
34
  [project.optional-dependencies]
tests/test_app_space.py CHANGED
@@ -31,24 +31,32 @@ def img_factory(width=100, height=100, color=(128, 128, 128)):
31
 
32
  # ── CVD Gallery Tests ─────────────────────────────────────────────────────────
33
 
34
- class TestCVDGallery:
35
- def test_gallery_returns_ten_items(self):
36
  img = img_factory(200, 200)
37
- gallery = app_module.generate_cvd_gallery(img)
38
- assert len(gallery) == 10, f"Expected 10 CVD variants, got {len(gallery)}"
39
 
40
- def test_gallery_items_are_pil_images(self):
41
  img = img_factory(200, 200)
42
- gallery = app_module.generate_cvd_gallery(img)
43
  for item, label in gallery:
44
  assert isinstance(item, Image.Image), f"Expected PIL Image, got {type(item)}"
45
 
46
- def test_gallery_labels_not_empty(self):
47
  img = img_factory(200, 200)
48
- gallery = app_module.generate_cvd_gallery(img)
49
  for _, label in gallery:
50
  assert label, "Gallery label must not be empty"
51
 
 
 
 
 
 
 
 
 
52
  def test_achromatopsia_is_grayscale(self):
53
  img = img_factory(100, 100, (200, 50, 50)) # red image
54
  achro = app_module.simulate_achromatopsia(img, 1.0)
@@ -85,7 +93,7 @@ class TestWCAGReport:
85
  def test_report_error_handling(self):
86
  result = {"error": "Model timeout after 60s"}
87
  report = app_module.format_wcag_report(result)
88
- assert "Error" in report or "" in report
89
 
90
  def test_report_no_findings_without_pass_flag(self):
91
  result = {"findings": [], "summary": "No issues detected"}
@@ -93,27 +101,17 @@ class TestWCAGReport:
93
  assert "No accessibility issues" in report
94
 
95
 
96
- # ── MODELS Dict Tests ─────────────────────────────────────────────────────────
97
-
98
- class TestMODELS:
99
- def test_models_dict_exists(self):
100
- assert hasattr(app_module, 'MODELS')
101
 
102
- def test_aya_vision_model_present(self):
103
- assert "aya-vision-32b" in app_module.MODELS
104
- entry = app_module.MODELS["aya-vision-32b"]
105
- assert "model_id" in entry
106
- assert entry["model_id"] == "CohereLabs/aya-vision-32b"
107
 
108
- def test_minicpm_model_present(self):
109
- assert "minicpm-v-4.6" in app_module.MODELS
110
- entry = app_module.MODELS["minicpm-v-4.6"]
111
- assert "model_id" in entry
112
 
113
- def test_all_models_have_required_keys(self):
114
- for name, entry in app_module.MODELS.items():
115
- assert "model_id" in entry, f"{name} missing model_id"
116
- assert "provider" in entry, f"{name} missing provider"
117
 
118
 
119
  # ── Gradio 5/6 Compat Tests ───────────────────────────────────────────────────
 
31
 
32
  # ── CVD Gallery Tests ─────────────────────────────────────────────────────────
33
 
34
+ class TestCVDGrid:
35
+ def test_grid_returns_four_items(self):
36
  img = img_factory(200, 200)
37
+ gallery = app_module.generate_cvd_grid(img)
38
+ assert len(gallery) == 4, f"Expected 4 CVD variants, got {len(gallery)}"
39
 
40
+ def test_grid_items_are_pil_images(self):
41
  img = img_factory(200, 200)
42
+ gallery = app_module.generate_cvd_grid(img)
43
  for item, label in gallery:
44
  assert isinstance(item, Image.Image), f"Expected PIL Image, got {type(item)}"
45
 
46
+ def test_grid_labels_not_empty(self):
47
  img = img_factory(200, 200)
48
+ gallery = app_module.generate_cvd_grid(img)
49
  for _, label in gallery:
50
  assert label, "Gallery label must not be empty"
51
 
52
+ def test_grid_has_correct_labels(self):
53
+ img = img_factory(100, 100)
54
+ gallery = app_module.generate_cvd_grid(img)
55
+ expected = ["Normal vision (original design)", "Protanopia (red-blind)",
56
+ "Deuteranopia (green-blind)", "Tritanopia (blue-blind)"]
57
+ for (_, label), expected_label in zip(gallery, expected):
58
+ assert label == expected_label, f"Expected '{expected_label}', got '{label}'"
59
+
60
  def test_achromatopsia_is_grayscale(self):
61
  img = img_factory(100, 100, (200, 50, 50)) # red image
62
  achro = app_module.simulate_achromatopsia(img, 1.0)
 
93
  def test_report_error_handling(self):
94
  result = {"error": "Model timeout after 60s"}
95
  report = app_module.format_wcag_report(result)
96
+ assert "Error" in report or "Warning" in report
97
 
98
  def test_report_no_findings_without_pass_flag(self):
99
  result = {"findings": [], "summary": "No issues detected"}
 
101
  assert "No accessibility issues" in report
102
 
103
 
104
+ # ── SUPPORTED_MODEL Tests ─────────────────────────────────────────────────────
 
 
 
 
105
 
106
+ class TestSupportedModel:
107
+ def test_supported_model_exists(self):
108
+ assert hasattr(app_module, 'SUPPORTED_MODEL')
 
 
109
 
110
+ def test_supported_model_is_minicpm(self):
111
+ assert app_module.SUPPORTED_MODEL == "minicpm-v-4.6"
 
 
112
 
113
+ def test_supported_model_is_string(self):
114
+ assert isinstance(app_module.SUPPORTED_MODEL, str)
 
 
115
 
116
 
117
  # ── Gradio 5/6 Compat Tests ───────────────────────────────────────────────────
tests/test_smoke.py CHANGED
@@ -72,7 +72,7 @@ class TestGradioApps:
72
 
73
  def test_app_has_cvd_gallery(self):
74
  import app as app_module
75
- assert callable(app_module.generate_cvd_gallery)
76
 
77
  def test_deficiency_config_has_8_types(self):
78
  import app as app_module
@@ -80,16 +80,9 @@ class TestGradioApps:
80
 
81
 
82
  class TestCVDVariants:
83
- def test_ten_type_gallery_count(self):
84
- """generate_cvd_gallery must produce exactly 10 variants (8 CVD + 2 grayscale)."""
85
- import app as app_module
86
- img = Image.new('RGB', (100, 100), (128, 128, 128))
87
- gallery = app_module.generate_cvd_gallery(img)
88
- assert len(gallery) == 10, f"Expected 10 CVD variants, got {len(gallery)}"
89
-
90
  def test_all_cvd_type_names_unique(self):
91
  import app as app_module
92
  img = Image.new('RGB', (50, 50), (100, 100, 100))
93
- gallery = app_module.generate_cvd_gallery(img)
94
  names = [label for _, label in gallery]
95
  assert len(names) == len(set(names)), "CVD type labels must be unique"
 
72
 
73
  def test_app_has_cvd_gallery(self):
74
  import app as app_module
75
+ assert callable(app_module.generate_cvd_grid)
76
 
77
  def test_deficiency_config_has_8_types(self):
78
  import app as app_module
 
80
 
81
 
82
  class TestCVDVariants:
 
 
 
 
 
 
 
83
  def test_all_cvd_type_names_unique(self):
84
  import app as app_module
85
  img = Image.new('RGB', (50, 50), (100, 100, 100))
86
+ gallery = app_module.generate_cvd_grid(img)
87
  names = [label for _, label in gallery]
88
  assert len(names) == len(set(names)), "CVD type labels must be unique"
uv.lock CHANGED
@@ -408,12 +408,15 @@ name = "color-ux-access"
408
  version = "0.1.0"
409
  source = { editable = "." }
410
  dependencies = [
 
411
  { name = "daltonlens" },
 
412
  { name = "gradio" },
413
  { name = "modal" },
414
  { name = "numpy" },
415
  { name = "openai" },
416
  { name = "pillow" },
 
417
  { name = "requests" },
418
  ]
419
 
@@ -449,8 +452,10 @@ space = [
449
  [package.metadata]
450
  requires-dist = [
451
  { name = "color-ux-access", extras = ["dev", "space", "modal"], marker = "extra == 'all'" },
 
452
  { name = "colorspacious", marker = "extra == 'dev'", specifier = ">=1.1.2" },
453
  { name = "daltonlens", specifier = ">=0.1.5" },
 
454
  { name = "fastapi", extras = ["standard"], marker = "extra == 'modal'", specifier = ">=0.120" },
455
  { name = "gradio", specifier = ">=6.17.3" },
456
  { name = "modal", specifier = ">=1.4.3" },
@@ -460,6 +465,7 @@ requires-dist = [
460
  { name = "openai", marker = "extra == 'modal'", specifier = ">=1.0" },
461
  { name = "openai", marker = "extra == 'space'", specifier = ">=1.0" },
462
  { name = "pillow", specifier = ">=10.0" },
 
463
  { name = "pytest", marker = "extra == 'dev'", specifier = ">=8.0" },
464
  { name = "python-dotenv", marker = "extra == 'space'", specifier = ">=1.0" },
465
  { name = "python-multipart", marker = "extra == 'modal'", specifier = ">=0.0.20" },
@@ -596,6 +602,17 @@ wheels = [
596
  { url = "https://files.pythonhosted.org/packages/ba/5a/18ad964b0086c6e62e2e7500f7edc89e3faa45033c71c1893d34eed2b2de/dnspython-2.8.0-py3-none-any.whl", hash = "sha256:01d9bbc4a2d76bf0db7c1f729812ded6d912bd318d3b1cf81d30c0f845dbf3af", size = 331094, upload-time = "2025-09-07T18:57:58.071Z" },
597
  ]
598
 
 
 
 
 
 
 
 
 
 
 
 
599
  [[package]]
600
  name = "email-validator"
601
  version = "2.3.0"
 
408
  version = "0.1.0"
409
  source = { editable = "." }
410
  dependencies = [
411
+ { name = "colorspacious" },
412
  { name = "daltonlens" },
413
+ { name = "dotenv" },
414
  { name = "gradio" },
415
  { name = "modal" },
416
  { name = "numpy" },
417
  { name = "openai" },
418
  { name = "pillow" },
419
+ { name = "pytest" },
420
  { name = "requests" },
421
  ]
422
 
 
452
  [package.metadata]
453
  requires-dist = [
454
  { name = "color-ux-access", extras = ["dev", "space", "modal"], marker = "extra == 'all'" },
455
+ { name = "colorspacious", specifier = ">=1.1.2" },
456
  { name = "colorspacious", marker = "extra == 'dev'", specifier = ">=1.1.2" },
457
  { name = "daltonlens", specifier = ">=0.1.5" },
458
+ { name = "dotenv", specifier = ">=0.9.9" },
459
  { name = "fastapi", extras = ["standard"], marker = "extra == 'modal'", specifier = ">=0.120" },
460
  { name = "gradio", specifier = ">=6.17.3" },
461
  { name = "modal", specifier = ">=1.4.3" },
 
465
  { name = "openai", marker = "extra == 'modal'", specifier = ">=1.0" },
466
  { name = "openai", marker = "extra == 'space'", specifier = ">=1.0" },
467
  { name = "pillow", specifier = ">=10.0" },
468
+ { name = "pytest", specifier = ">=9.0.3" },
469
  { name = "pytest", marker = "extra == 'dev'", specifier = ">=8.0" },
470
  { name = "python-dotenv", marker = "extra == 'space'", specifier = ">=1.0" },
471
  { name = "python-multipart", marker = "extra == 'modal'", specifier = ">=0.0.20" },
 
602
  { url = "https://files.pythonhosted.org/packages/ba/5a/18ad964b0086c6e62e2e7500f7edc89e3faa45033c71c1893d34eed2b2de/dnspython-2.8.0-py3-none-any.whl", hash = "sha256:01d9bbc4a2d76bf0db7c1f729812ded6d912bd318d3b1cf81d30c0f845dbf3af", size = 331094, upload-time = "2025-09-07T18:57:58.071Z" },
603
  ]
604
 
605
+ [[package]]
606
+ name = "dotenv"
607
+ version = "0.9.9"
608
+ source = { registry = "https://pypi.org/simple" }
609
+ dependencies = [
610
+ { name = "python-dotenv" },
611
+ ]
612
+ wheels = [
613
+ { url = "https://files.pythonhosted.org/packages/b2/b7/545d2c10c1fc15e48653c91efde329a790f2eecfbbf2bd16003b5db2bab0/dotenv-0.9.9-py2.py3-none-any.whl", hash = "sha256:29cf74a087b31dafdb5a446b6d7e11cbce8ed2741540e2339c69fbef92c94ce9", size = 1892, upload-time = "2025-02-19T22:15:01.647Z" },
614
+ ]
615
+
616
  [[package]]
617
  name = "email-validator"
618
  version = "2.3.0"
vlm/accessibility_report.py DELETED
@@ -1,379 +0,0 @@
1
- """
2
- Accessibility Report Generator based on WCAG Standards
3
- """
4
-
5
- import json
6
- import re
7
- from datetime import datetime
8
- from typing import Dict, List, Any, Optional
9
-
10
- class WCAGStandards:
11
- """WCAG 2.1/2.2 color contrast and color usage standards"""
12
-
13
- # Contrast ratios
14
- CONTRAST_RATIOS = {
15
- 'AA_NORMAL_TEXT': 4.5,
16
- 'AA_LARGE_TEXT': 3.0,
17
- 'AA_UI_COMPONENTS': 3.0, # WCAG 2.1
18
- 'AAA_NORMAL_TEXT': 7.0,
19
- 'AAA_LARGE_TEXT': 4.5
20
- }
21
-
22
- # Success criteria references
23
- SUCCESS_CRITERIA = {
24
- '1.4.3': 'Contrast (Minimum) - AA',
25
- '1.4.6': 'Contrast (Enhanced) - AAA',
26
- '1.4.11': 'Non-text Contrast - AA (WCAG 2.1)',
27
- '1.4.1': 'Use of Color - A'
28
- }
29
-
30
- class AccessibilityReport:
31
- """Generate accessibility reports based on VLM analysis"""
32
-
33
- def __init__(self):
34
- self.standards = WCAGStandards()
35
-
36
- def parse_vlm_output(self, vlm_output: str) -> List[Dict[str, Any]]:
37
- """
38
- Parse VLM output to extract accessibility issues
39
- Expected format: JSON array of issues with description, type, remediation, bbox
40
- """
41
- issues = []
42
-
43
- try:
44
- # Try to extract JSON from the output
45
- json_match = re.search(r'\[.*\]', vlm_output, re.DOTALL)
46
- if json_match:
47
- json_str = json_match.group()
48
- parsed_issues = json.loads(json_str)
49
-
50
- if isinstance(parsed_issues, list):
51
- for issue in parsed_issues:
52
- if isinstance(issue, dict):
53
- # Standardize issue format
54
- standardized_issue = {
55
- 'description': issue.get('description', ''),
56
- 'type': issue.get('type', 'unknown'),
57
- 'remediation': issue.get('remediation', ''),
58
- 'bbox': issue.get('bbox', issue.get('point', [])),
59
- 'wcag_references': self._map_to_wcag(issue.get('type', '')),
60
- 'severity': self._assess_severity(issue.get('type', ''), issue.get('description', ''))
61
- }
62
- issues.append(standardized_issue)
63
- else:
64
- # Single issue object
65
- if isinstance(parsed_issues, dict):
66
- issue = parsed_issues
67
- standardized_issue = {
68
- 'description': issue.get('description', ''),
69
- 'type': issue.get('type', 'unknown'),
70
- 'remediation': issue.get('remediation', ''),
71
- 'bbox': issue.get('bbox', issue.get('point', [])),
72
- 'wcag_references': self._map_to_wcag(issue.get('type', '')),
73
- 'severity': self._assess_severity(issue.get('type', ''), issue.get('description', ''))
74
- }
75
- issues.append(standardized_issue)
76
- else:
77
- # Fallback: treat as text analysis
78
- issues = self._parse_text_analysis(vlm_output)
79
-
80
- except (json.JSONDecodeError, Exception) as e:
81
- print(f"Error parsing VLM output: {e}")
82
- # Fallback to text parsing
83
- issues = self._parse_text_analysis(vlm_output)
84
-
85
- return issues
86
-
87
- def _map_to_wcag(self, issue_type: str) -> List[str]:
88
- """Map issue type to WCAG success criteria."""
89
- issue_type_lower = issue_type.lower()
90
- wcag_refs = []
91
-
92
- if 'contrast' in issue_type_lower and 'ui' in issue_type_lower:
93
- wcag_refs.append('1.4.11') # Non-text Contrast
94
- if 'contrast' in issue_type_lower and 'text' in issue_type_lower:
95
- wcag_refs.extend(['1.4.3', '1.4.6']) # Contrast (Minimum and Enhanced)
96
- if 'color-only' in issue_type_lower or 'color only' in issue_type_lower:
97
- wcag_refs.append('1.4.1') # Use of Color
98
- if 'color-dependent' in issue_type_lower or 'color dependent' in issue_type_lower:
99
- wcag_refs.append('1.4.1')
100
- if 'ui' in issue_type_lower or 'component' in issue_type_lower or 'button' in issue_type_lower or 'input' in issue_type_lower:
101
- if '1.4.11' not in wcag_refs:
102
- wcag_refs.append('1.4.11') # Non-text Contrast
103
-
104
- return wcag_refs if wcag_refs else ['1.4.3'] # Default to contrast check
105
-
106
- def _assess_severity(self, issue_type: str, description: str) -> str:
107
- """Assess severity based on issue type and description."""
108
- issue_type_lower = issue_type.lower()
109
- desc_lower = description.lower()
110
-
111
- # Critical: blocks core task completion
112
- if any(word in issue_type_lower for word in ['critical', 'severe', 'major']):
113
- return 'critical'
114
- if any(word in desc_lower for word in [
115
- 'cannot submit', 'cannot complete', 'form submission blocked',
116
- 'invisible', 'unreadable', 'cannot see', 'cannot distinguish',
117
- 'error state invisible', 'no way to tell',
118
- ]):
119
- return 'critical'
120
-
121
- # Serious: significant confusion or delay
122
- if any(word in issue_type_lower for word in ['serious', 'error state', 'status indicator']):
123
- return 'serious'
124
- if any(word in desc_lower for word in [
125
- 'difficult', 'hard to see', 'low visibility', 'confusing',
126
- 'cannot identify', 'cannot tell which', 'ambiguous',
127
- ]):
128
- return 'serious'
129
-
130
- # Moderate: workaround exists
131
- if any(word in issue_type_lower for word in ['moderate', 'medium']):
132
- return 'moderate'
133
- if any(word in desc_lower for word in ['minor', 'slight', 'could be better']):
134
- return 'moderate'
135
-
136
- # Contrast-specific: extract ratio if mentioned
137
- if 'contrast' in issue_type_lower:
138
- contrast_match = re.search(r'(\d+(?:\.\d+)?):1', desc_lower)
139
- if contrast_match:
140
- ratio = float(contrast_match.group(1))
141
- if ratio < 3.0:
142
- return 'critical'
143
- elif ratio < 4.5:
144
- return 'serious'
145
- else:
146
- return 'moderate'
147
-
148
- return 'moderate' # Default
149
-
150
- def _parse_text_analysis(self, text: str) -> List[Dict[str, Any]]:
151
- """Parse free-text VLM analysis into structured issues"""
152
- issues = []
153
-
154
- # Simple heuristic: look for common accessibility issue patterns
155
- lines = text.split('\n')
156
- current_issue = {}
157
-
158
- for line in lines:
159
- line = line.strip()
160
- if not line:
161
- if current_issue:
162
- issues.append(self._finalize_issue(current_issue))
163
- current_issue = {}
164
- continue
165
-
166
- # Look for issue indicators
167
- if any(word in line.lower() for word in ['issue', 'problem', 'concern', 'violation', 'fail']):
168
- if current_issue:
169
- issues.append(self._finalize_issue(current_issue))
170
- current_issue = {
171
- 'description': line,
172
- 'type': self._infer_type_from_text(line),
173
- 'remediation': '',
174
- 'bbox': []
175
- }
176
- elif 'recommend' in line.lower() or 'suggest' in line.lower() or 'should' in line.lower():
177
- if current_issue:
178
- current_issue['remediation'] = line
179
- elif current_issue and 'description' in current_issue:
180
- # Append to description
181
- current_issue['description'] += ' ' + line
182
-
183
- # Don't forget the last issue
184
- if current_issue:
185
- issues.append(self._finalize_issue(current_issue))
186
-
187
- return issues
188
-
189
- def _infer_type_from_text(self, text: str) -> str:
190
- """Infer issue type from text description"""
191
- text_lower = text.lower()
192
- if 'contrast' in text_lower:
193
- return 'low contrast'
194
- elif 'color' in text_lower and ('dependent' in text_lower or 'only' in text_lower):
195
- return 'color-dependent element'
196
- elif 'text' in text_lower and ('size' in text_lower or 'readable' in text_lower):
197
- return 'text readability'
198
- else:
199
- return 'accessibility issue'
200
-
201
- def _finalize_issue(self, issue: Dict[str, Any]) -> Dict[str, Any]:
202
- """Finalize issue with WCAG references and severity"""
203
- issue['wcag_references'] = self._map_to_wcag(issue.get('type', ''))
204
- issue['severity'] = self._assess_severity(issue.get('type', ''), issue.get('description', ''))
205
- return issue
206
-
207
- def generate_report(self, url: str, vlm_analysis: str, screenshot_path: Optional[str] = None) -> Dict[str, Any]:
208
- """
209
- Generate a complete accessibility report
210
-
211
- Returns:
212
- Dictionary containing the full report
213
- """
214
- issues = self.parse_vlm_output(vlm_analysis)
215
-
216
- # Calculate summary statistics
217
- total_issues = len(issues)
218
- high_severity = len([i for i in issues if i.get('severity') == 'high'])
219
- medium_severity = len([i for i in issues if i.get('severity') == 'medium'])
220
- low_severity = len([i for i in issues if i.get('severity') == 'low'])
221
-
222
- # Determine overall compliance level
223
- if high_severity == 0 and medium_severity <= 2:
224
- compliance_level = "Good"
225
- elif high_severity == 0:
226
- compliance_level = "Fair"
227
- elif high_severity <= 2:
228
- compliance_level = "Poor"
229
- else:
230
- compliance_level = "Non-compliant"
231
-
232
- report = {
233
- 'metadata': {
234
- 'url': url,
235
- 'timestamp': datetime.now().isoformat(),
236
- 'tool': 'Color-UX-Access with Qwen2.5-VL-32B-Instruct',
237
- 'wcag_version': '2.1/2.2',
238
- 'screenshot_analyzed': screenshot_path
239
- },
240
- 'summary': {
241
- 'total_issues': total_issues,
242
- 'high_severity': high_severity,
243
- 'medium_severity': medium_severity,
244
- 'low_severity': low_severity,
245
- 'compliance_level': compliance_level,
246
- 'wcag_version_tested': '2.1/2.2 AA'
247
- },
248
- 'issues': issues,
249
- 'wcag_standards_referenced': list(self.standards.SUCCESS_CRITERIA.values()),
250
- 'recommendations': self._generate_recommendations(issues)
251
- }
252
-
253
- return report
254
-
255
- def _generate_recommendations(self, issues: List[Dict[str, Any]]) -> List[str]:
256
- """Generate prioritized recommendations based on issues"""
257
- recommendations = []
258
-
259
- # Group issues by type for batch recommendations
260
- issue_types = {}
261
- for issue in issues:
262
- issue_type = issue.get('type', 'unknown')
263
- if issue_type not in issue_types:
264
- issue_types[issue_type] = []
265
- issue_types[issue_type].append(issue)
266
-
267
- # Priority recommendations
268
- if any('contrast' in issue_type.lower() for issue_type in issue_types.keys()):
269
- recommendations.append({
270
- 'priority': 'high',
271
- 'category': 'Color Contrast',
272
- 'recommendation': 'Ensure all text and UI components meet WCAG 2.1 AA contrast ratios (4.5:1 for normal text, 3:1 for large text and UI components). Use a contrast checker to verify compliance.',
273
- 'wcag_reference': '1.4.3, 1.4.11'
274
- })
275
-
276
- if any('color-dependent' in issue_type.lower() for issue_type in issue_types.keys()):
277
- recommendations.append({
278
- 'priority': 'high',
279
- 'category': 'Use of Color',
280
- 'recommendation': 'Do not rely solely on color to convey information. Add text labels, icons, or patterns to supplement color coding.',
281
- 'wcag_reference': '1.4.1'
282
- })
283
-
284
- # Add general recommendations
285
- recommendations.append({
286
- 'priority': 'medium',
287
- 'category': 'Testing',
288
- 'recommendation': 'Test with actual users who have color vision deficiencies and use automated accessibility testing tools regularly.',
289
- 'wcag_reference': 'General'
290
- })
291
-
292
- return recommendations
293
-
294
- def format_report_as_markdown(self, report: Dict[str, Any]) -> str:
295
- """Format the report as readable Markdown"""
296
- md = []
297
- md.append(f"# Accessibility Audit Report")
298
- md.append(f"**URL:** {report['metadata']['url']}")
299
- md.append(f"**Timestamp:** {report['metadata']['timestamp']}")
300
- md.append(f"**Tool:** {report['metadata']['tool']}")
301
- md.append(f"**WCAG Version:** {report['metadata']['wcag_version']}")
302
- md.append("")
303
-
304
- md.append("## Executive Summary")
305
- summary = report['summary']
306
- md.append(f"- **Total Issues Found:** {summary['total_issues']}")
307
- md.append(f"- **High Severity:** {summary['high_severity']}")
308
- md.append(f"- **Medium Severity:** {summary['medium_severity']}")
309
- md.append(f"- **Low Severity:** {summary['low_severity']}")
310
- md.append(f"- **Compliance Level:** {summary['compliance_level']}")
311
- md.append(f"- **WCAG Level Tested:** {summary['wcag_version_tested']}")
312
- md.append("")
313
-
314
- md.append("## Issues Found")
315
- if not report['issues']:
316
- md.append("No accessibility issues detected.")
317
- else:
318
- for i, issue in enumerate(report['issues'], 1):
319
- md.append(f"### Issue {i}: {issue.get('type', 'Unknown').title()}")
320
- md.append(f"- **Description:** {issue.get('description', 'N/A')}")
321
- md.append(f"- **Type:** {issue.get('type', 'N/A')}")
322
- md.append(f"- **Severity:** {issue.get('severity', 'N/A').title()}")
323
- md.append(f"- **Location:** {issue.get('bbox', 'Not specified')}")
324
- if issue.get('wcag_references'):
325
- md.append(f"- **WCAG References:** {', '.join(issue['wcag_references'])}")
326
- md.append(f"- **Remediation:** {issue.get('remediation', 'No specific remediation provided')}")
327
- md.append("")
328
-
329
- md.append("## Recommendations")
330
- for rec in report['recommendations']:
331
- md.append(f"### {rec['category']} ({rec['priority'].title()} Priority)")
332
- md.append(f"{rec['recommendation']}")
333
- md.append(f"*WCAG Reference: {rec['wcag_reference']}*")
334
- md.append("")
335
-
336
- md.append("## WCAG Standards Referenced")
337
- for standard in report['wcag_standards_referenced']:
338
- md.append(f"- {standard}")
339
- md.append("")
340
-
341
- md.append("--")
342
- md.append(f"*Report generated by Color-UX-Access on {report['metadata']['timestamp']}*")
343
-
344
- return '\n'.join(md)
345
-
346
- def main():
347
- """Test the report generator"""
348
- import sys
349
-
350
- if len(sys.argv) < 2:
351
- print("Usage: python accessibility_report.py <vlm_output_file_or_text> [url]")
352
- sys.exit(1)
353
-
354
- # Read VLM output
355
- vlm_input = sys.argv[1]
356
- url = sys.argv[2] if len(sys.argv) > 2 else "https://example.com"
357
-
358
- # Try to read from file, otherwise treat as direct text
359
- try:
360
- with open(vlm_input, 'r', encoding='utf-8') as f:
361
- vlm_output = f.read()
362
- except FileNotFoundError:
363
- vlm_output = vlm_input
364
-
365
- # Generate report
366
- reporter = AccessibilityReport()
367
- report = reporter.generate_report(url, vlm_output)
368
-
369
- # Output as JSON
370
- print(json.dumps(report, indent=2))
371
-
372
- # Also output markdown version
373
- print("\n" + "="*50)
374
- print("MARKDOWN VERSION:")
375
- print("="*50)
376
- print(reporter.format_report_as_markdown(report))
377
-
378
- if __name__ == "__main__":
379
- main()