Feasibility Study Technical Report on Using Open-Omni-Small-Model-INT4 for Real-Time Analysis of Game Graphics
使用 Open-Omni-Small-Model-INT4 对游戏画面实时分析可行性方案技术报告
Project Team: Anonymous / 项目团队：匿名
Technical Status / 技术状态: Fully Local / Hardware-Level Air-Gap Isolation / Experimental Proof-of-Concept (PoC) 纯本地 / 硬件级物理隔离 / 实验性概念验证（PoC）
I. Introduction / 引言
1.1 Background and Motivation / 背景与动机
If you're the kind of player who sinks five or six hours a day into extraction shooters, you've lived through this particular absurdity—you lose a gunfight not because the other guy out-aimed you, but because your bullets simply vanished mid-flight. You alt-tab to check what's going on, and find the kernel-level anti-cheat is reading every sector of your M.2 SSD like it's auditing a dissertation, racking up more write cycles than your actual gameplay. Meanwhile, it turns a blind eye to cheat modules that piggyback on legitimate software processes, and has zero awareness of a used phone running AI inference pointed at the screen. The one thing it reliably accomplishes is wearing out your hardware before you ever get banned. Traditional cheating vectors—memory reading, driver injection, DMA hardware—are essentially dead ends now: kernel-level anti-cheat scans deeper than airport security, and the penalty is a decade-long ban. So the real question becomes disarmingly simple: can you pull off real-time AI analysis of game visuals without touching the host machine at all—no cables, no software, not a single byte of detectable signature—using nothing but a phone camera pointed at the screen?
如果你是一名每天打五六个小时摸金模式的 FPS 玩家，你一定经历过这种荒诞时刻——你被对面打崩了，不是因为他枪法比你好，而是你开枪的那瞬间子弹被吞了。你切出去看任务管理器，发现自家反作弊系统正在把你的 M.2 固态当毕业论文逐扇区精读，硬盘写入量比你游戏时长还离谱。它对那种把外挂组件寄生在正常软件进程里的方案睁一只眼闭一只眼，对那种旧手机架在屏幕前跑 AI 的方案毫无感知能力——它唯一擅长的事情，是让你的电脑比你先退役。传统做外挂的路径——读内存、挂驱动、插 DMA 板子——现在基本行不通了，驱动级反作弊扫得比机场安检还细，封号十年起步。于是真正要探索的问题变得非常简单：能不能完全不碰那台电脑——不插线、不装软件、不留哪怕一个字节的特征码——只用一部手机对着屏幕拍，就跑完对游戏画面的实时 AI 分析？
1.2 Core Innovation: Physical Air-Gap Visual Inference / 核心创新：物理隔离式视觉推理
To address this question, we propose a screen-understanding architecture based on pure physical isolation. The core idea is simple: the only connection between the computer and the AI is a camera—no cables, no network, no software installation. The system treats the host display output as a purely external optical signal, never touching the game process or the operating system at any point. A standard smartphone is mounted in front of the screen, capturing the game visuals in real time through its camera, and runs an open-source omni-capable multimodal small model—compressed via INT4 quantization—entirely on-device. The model operates with zero cloud dependency and generates zero network traffic.
针对上述问题，我们提出一种基于纯物理隔离的画面理解架构。核心思路就一句话：电脑和 AI 之间只靠摄像头连接，不插线、不联网、不装任何软件。系统将显示器输出视为纯粹的外部光学信号，全程不接触游戏进程、不碰操作系统。一部普通手机架在屏幕前，通过摄像头实时拍摄游戏画面，并在手机本地运行一个经 INT4 量化压缩的开源全模态小型模型。该模型完全不依赖云端算力，不产生任何网络流量。
This exploration is conducted purely as an experimental feasibility study in edge-AI deployment under adversarial integrity constraints.
本探索纯粹作为一项在高完整性约束环境下的边缘 AI 部署实验性可行性研究。
1.3 Research Objectives / 研究目标
Benchmark On-Device Inference Performance / 端侧推理性能基准测试: Deploy an open-source omni-capable multimodal small model INT4-quantized on Qualcomm Snapdragon NPU for local hardware acceleration, maximizing video stream throughput and minimizing inference latency in time-sensitive scenarios. 在 Qualcomm Snapdragon NPU 上部署经 INT4 量化的开源全模态小型模型，利用本地硬件加速最大化视频流吞吐并最小化推理延迟。
Establish Dual-Path Feasibility / 确立双路径可行性: Evaluate two distinct output modalities—pure on-screen annotation versus external hardware macro feedback—to assess whether physical air-gap architectures can enable screen understanding without triggering host-side integrity audits. 评估两种截然不同的输出模式——纯画面标注与外部硬件宏回传——以验证纯物理隔离架构能否在反作弊系统毫无察觉的前提下完成画面理解。
II. Core Technical Architecture / 核心技术架构
2.1 Physical Visual Input and On-Device Heterogeneous Inference / 物理视觉输入与端侧异构推理
The external mobile device captures display output physically through its RGB camera. The captured video stream is fed in real time to the locally deployed open-source omni-capable multimodal small model's vision encoder, INT4-quantized for edge deployment. Leveraging INT4 quantization combined with mobile Snapdragon NPU heterogeneous inference maximizes edge deployment cost-efficiency while maintaining viable model capability and throughput. Under low-power operating conditions, the model performs object contour feature extraction to locate target pixel clusters. After computing spatial coordinates, we design two distinct output modalities:
外部移动设备通过 RGB 摄像头物理捕捉显示器的画面输出，视频流实时输入到本地部署的开源全模态小型模型（经 INT4 量化以适配边缘部署）的视觉编码器中。INT4 量化与 Snapdragon NPU 异构推理相结合，在维持可用的模型能力与吞吐量的前提下最大化边缘部署的成本效率。在低功耗运行条件下，模型执行目标轮廓特征提取并定位目标像素簇。在解算出空间坐标后，我们设计了两条截然不同的输出路径：
2.2 Route 1: Pure On-Screen Overlay Annotation (Information-Level) / 方案一：纯显示屏画面标注（信息级）
Implementation / 实现: After processing the raw visual stream, the system does not interact with the host computer via any hardware or software signal path. The compute unit and the host remain physically isolated. 系统对原始视觉流进行计算处理后，不通过任何硬件或软件信号路径与宿主计算机交互。计算单元与宿主之间保持物理隔离。
Presentation / 呈现形式: The model directly renders annotated frames on the mobile device display (or an independent secondary screen), providing real-time spatial annotations of detected objects overlaid on the captured frame. 模型直接在移动设备（或独立副屏）上渲染带有标注的帧画面，在捕捉到的帧上以实时叠加方式提供被检测目标的空间标注。
Technical Characteristics / 技术特征: This route offers maximal isolation guarantees. Because all signal feedback loops are physically severed, every operational decision based on the annotated display is manually executed by a human operator. No physical or digital signature is deposited on the host computer. From the host's perspective, the display output is the only interaction—there is no inbound data path to monitor. 该方案提供最高级别的隔离保障。所有信号反馈回路物理切断，标注画面所引导的每一项操作决策均由人类操作者手动执行。在宿主计算机端不沉积任何物理或数字特征。从宿主视角来看，显示器信号输出是唯一的交互——不存在需要审计的入站数据通路。
Key Advantage / 关键优势: Modern omni-capable multimodal architectures natively incorporate time-aligned position embeddings (e.g., TMRoPE — Time-aligned Multimodal RoPE) that synchronize video frame timestamps with concurrently captured audio. This enables the annotation layer to fuse visual object detection with directional audio event localization (e.g., mapping an off-screen acoustic event to its spatial origin). This audio-visual fusion is natively supported by the model architecture and requires no additional integration. 当代全模态架构原生集成了时间对齐的位置嵌入（如 TMRoPE — 时间对齐多模态旋转位置编码），使视频帧时间戳与同步捕获的音频对齐。这使得标注层能够将视觉目标检测与定向音频事件定位相融合（例如将画面外的声学事件映射到其空间源点）。该视听觉融合能力由模型架构原生支持，无需额外集成。
2.3 Route 2: External Hardware Macro Feedback (Action-Level) / 方案二：外部硬件宏回传（动作级）
Implementation / 实现: After the local Snapdragon NPU computes precise spatial coordinates of detected targets, the coordinate data is converted into physical-level control primitives. 本地 Snapdragon NPU 解算出检测目标的精确空间坐标后，将坐标数据转化为物理层控制基元。
Presentation / 呈现形式: Control commands are routed through external physical hardware—such as microcontroller serial bridges or modified external HID peripherals—or through simulated touch protocols to deliver physical click events and displacement micro-adjustments. 控制指令通过外部物理硬件——如单片机串行桥接或改造的外部 HID 外设——或模拟触控协议进行路由，传递物理层点击事件与位移微调。
Technical Characteristics / 技术特征: This route achieves a closed perception-action loop. To address integrity systems that audit straight-line or mechanical trajectory patterns, domain-specific algorithms must be integrated into the hardware-level macro pipeline. These algorithms shape the physical output to approximate human motor characteristics—including natural micro-tremor variance, reaction-time jitter, and Fitts' Law-compliant movement curves—thereby reducing statistical detectability in server-side behavioral analysis. 该方案实现感知-行动闭环。为应对审计直线或机械化轨迹模式的完整性系统，需要在硬件级宏管道中集成专用算法。这些算法将物理输出信号塑形以模拟人类运动特征——包括自然的微颤方差、反应时间抖动、以及符合 Fitts 定律的移动曲线——从而降低在服务器端行为大数据分析中的统计可检测性。
III. Model Architecture Considerations / 模型架构考量
The emergence of open-source omni-capable multimodal small models adopting Thinker-Talker or equivalent dual-stream architectures makes this deployment scenario increasingly tractable for several reasons:
采用 Thinker-Talker 或等效双流架构的开源全模态小型模型的出现，使这一部署场景在以下方面日益可行：
Multimodal Perception Module / 多模态感知模块: Processes video frames and audio streams with time-aligned timestamps, enabling frame-precise spatial-temporal reasoning across modalities. 以时间对齐的时间戳处理视频帧和音频流，实现跨模态的帧级时空推理。
Streaming Generation Module / 流式生成模块: Produces streaming text output, naturally mapping to coordinate tuple generation for annotation or macro routing. 产生流式文本输出，自然地映射为坐标元组生成，用于标注或宏指令路由。
End-to-End Speech Instruction Following / 端到端语音指令跟随: Leading open-source omni models demonstrate instruction-following capability with speech input that rivals text-input performance on benchmarks such as MMLU and GSM8K, enabling hands-free operational modes. 领先的开源全模态模型在 MMLU 和 GSM8K 等基准测试中展现出语音输入指令跟随能力与文本输入不相上下的表现，使免手动操作模式成为可能。
INT4 Quantization / INT4 量化: Compresses models in the single-digit-billion parameter range to approximately 4GB, fitting within the memory and thermal constraints of commodity mobile NPUs while maintaining inference latency under 50ms—comparable to the tick rate of many real-time interactive applications. 将数十亿参数规模的模型压缩至约 4GB，适配商品级移动 NPU 的内存与热约束，同时将推理延迟维持在 50ms 以下——与众多实时交互应用的 tick 速率相当。
IV. Ethical Considerations and Scope Limitations / 伦理考量与范围限制
This study is conducted strictly as an experimental feasibility investigation in the domain of adversarial edge-AI deployment. It does not constitute, endorse, or facilitate any violation of software terms of service. The research objective is to advance understanding of physical air-gap architectures for real-time visual inference—a topic with legitimate applications in accessibility technology (e.g., assistive screen readers operating under constrained host environments), hardware-in-the-loop testing, and secure human-AI interaction design.
本研究严格作为高约束环境下边缘 AI 部署的实验性可行性研究。不构成、不认可、不促进任何违反软件服务条款的行为。研究目标在于推进对纯物理隔离架构在实时视觉推理中的能力边界的理解——这一方向在无障碍技术（如受限宿主环境下的辅助屏幕阅读器）、硬件在环测试以及安全人机交互设计等领域具有合法的应用前景。
No host-side instrumentation, memory access, driver modification, or network interception is performed at any stage of this experimental setup. All inference is conducted on locally owned hardware using publicly available open-source models.
本实验设置的任何阶段均不涉及宿主端插桩、内存访问、驱动修改或网络拦截。所有推理均在自有本地硬件上使用公开可用的开源模型完成。
V. Preliminary Findings and Future Work / 初步发现与未来工作
Initial benchmarking on Snapdragon 8 Gen 3 hardware with an open-source omni-capable multimodal model INT4-quantized demonstrates:
基于 Snapdragon 8 Gen 3 硬件配合 INT4 量化开源全模态模型的初步基准测试表明：
- End-to-end inference latency within the sub-50ms range under optimized NPU scheduling 优化 NPU 调度下的端到端推理延迟处于 50ms 以下区间
- Viable frame throughput for real-time visual tracking scenarios 帧吞吐量满足实时视觉跟踪场景的可用性要求
- Time-aligned audio-visual synchronization enabling directional event localization beyond the visual field 时间对齐的音视频同步可实现视野外的定向事件定位
Future work may explore: (1) model distillation to sub-1B parameter counts for further latency reduction, (2) integration of IMU sensor fusion for motion-compensated frame stabilization, and (3) formal accessibility use-case evaluation under controlled laboratory conditions.
未来工作可探索的方向包括：(1) 将模型蒸馏至 10 亿参数以下以进一步降低延迟，(2) 集成 IMU 传感器融合以实现运动补偿帧稳定，以及 (3) 受控实验室条件下的正式无障碍使用案例评估。