Arnab Sanyal

A very popular way to enable image generation on memory-constrained accelerators using modern spatiotemporal generative models is post-training quantization (PTQ). However, every existing diffusion PTQ scheme shares an unstated commitment: the number format is fixed in advance, and only the scale, zero point, or the per-layer bit-width is allowed to move. This is unfortunate, because at a fixed bit-width the best format depends on the distribution being encoded, and that distribution differs from tensor to tensor: across weight channels, across layers, and — in a denoising model — along the diffusion timestep, where activations slide from heavy-tailed and noise-dominated at t → T to tightly clustered and structured at t → 0. One format for the whole network is therefore mispriced for most of its tensors, and re-tuning a scale cannot repair a format mismatch. In this work, we propose a PTQ framework, which we call Chameleon, that holds the bit-width fixed and treats the number format itself as a discrete variable, chosen per weight channel and per (layer, timestep bucket) activation tensor. Chameleon draws activation formats from a palette of {INT8, FP8 E4M3, FP8 E5M2, MXFP8, MXINT8} and weight formats from {INT8, MXINT8} at 8 bits or {INT4, NF4, FP4 E2M1, MXINT4, MXFP4} at 4 bits, choosing activation formats ahead of time from two cheap statistics — empirical activation kurtosis and closed-form diffusion SNR ᾱ_t/(1−ᾱ_t) — stored in a per-(layer, timestep bucket) lookup table, and weight formats offline by reconstruction error. An architectural fork adapts the same selection layer to each model family, so each model exercises the palette as far as its own statistics warrant: weights and per-(layer, bucket) activations on multi-step SDXL, weights alone where a single sampler step leaves no timestep axis. We evaluate Chameleon across three diffusion regimes — multi-step SDXL, single-step SDXL-Turbo, and the PixArt-α Diffusion Transformer — on COCO-2014. Chameleon achieves the best FID for all six backbone × bit-width settings, while preserving semantic alignment with the text prompts (CLIP within 0.24 of the FP16 reference in every setting, and the best of all quantized methods at W₄A₈). Adapting the format, rather than only its scale, recovers distributional fidelity at 4-bit weights while holding prompt adherence at the FP16 level.

Index terms: Post-Training Quantization, Diffusion Models, Number Formats, Microscaling (MX) Formats, FP8

Large Language Models (LLMs) achieve strong performance across tasks, but face storage and compute challenges on edge devices. We propose EntroLLM, a compression framework combining mixed quantization and entropy coding to reduce storage while preserving accuracy. We use a combination of unsigned and asymmetric quantization. Tensor-level quantization produces an entropy-reducing effect, increasing weight compressibility, and improving downstream Huffman encoding by 7× (8-bit) and 11.3× (4-bit) over state-of-the-art methods. Huffman coding further reduces memory bandwidth demands, while a parallel decoding strategy enables efficient weight retrieval with minimal latency. Experiments on edge-scale LLMs (smolLM-1.7B, phi3-mini-4k, mistral-7B) show up to 30% storage savings over uint8 and 65% over uint4 models, with 31.9–146.6% faster inference on memory-limited devices like the NVIDIA JETSON P3450. EntroLLM requires no retraining and is compatible with existing post-training quantization pipelines, making it practical for edge LLM deployment.

Index terms: Large Language Models (LLMs), quantization, entropy coding, Huffman coding, parallel decoding

While small LLMs in the range of 100M–500M parameters have enabled on-device inference using edge GPUs, mobile SoCs, and even Pi-class systems, bare metal microcontroller-based deployment remains open to this day for all models beyond a trivial size. The challenges include up to 1–2 MB of on-chip SRAM, no external DRAM, single in-order cores, low-power budgets of 10–100 mW, and an autoregressive loop causing a large KV cache and full vocabulary LM-head matmul every step. Deployments of LLMs on MCUs have been published so far only for synthetic children’s stories trained with sub-15M-parameter models. In this paper we introduce three specific areas of research leading to deployment of models like Gemma 3 (270M) onto Cortex-M55-class commodity chips. The ideas proposed in this paper cover a ternary GEMM kernel with entropy-encoded weight transport, KV-aware memory management of SRAM/HyperRAM/Flash, and the graph manipulations required in ExecuTorch to put the pieces together.

Index terms: Large Language Models (LLMs), Microcontrollers, Ternary Quantization, Entropy Coding, KV-cache Memory Management

In-sensor computing, which integrates computation directly within the sensor, has emerged as a promising paradigm for machine vision applications such as AR/VR and smart home systems. By processing data on-chip before transmission, it alleviates the bandwidth bottleneck caused by high-resolution, high-frame-rate image transmission. We propose a system architecture that integrates a CMOS image sensor (CIS) with a logic chip via advanced packaging, where the logic chip executes a lightweight encoder that aggressively compresses activations before off-chip transmission. However, conventional deep neural network (DNN) partitioning strategies require multiple layers before achieving meaningful dimensionality reduction, limiting bandwidth savings. To address this, we propose a dual-branch autoencoder-based vision architecture trained with a triple loss function combining task-specific, entropy, and reconstruction objectives to produce compact, Huffman-codable representations. The encoder is prototyped on a FPGA to provide hardware-validated energy measurements, complementing circuit-simulated CIS front-end characterization. Our approach achieves a four-order-of-magnitude reduction in transmitted data volume compared to raw input images, resulting in 2×–4.5× energy savings. We evaluate on CNN and ViT-based models across diverse tasks, achieving state-of-the-art accuracy with energy efficiency of up to 22.7 TOPS/W.

Long-form video question answering (LVQA) requires answering natural-language queries about videos spanning minutes to hours, demanding temporal reasoning across thousands of frames. Standard vision-language models (VLMs) struggle with this task: their fixed frame budgets force aggressive downsampling that misses the temporal structure that complex queries depend on. Neuro-symbolic approaches address this by decomposing queries into atomic propositions, translating them into temporal logic specifications, and applying formal model checking to retrieve segments satisfying the specification. This yields up to 10% higher accuracy on temporally complex benchmarks, with interpretability and formal guarantees. However, constructing the video automaton requires grounding every proposition at every frame window via VLM calls, resulting in up to 130× slower inference than standard VLM prompting. We present RT-NeuS, a framework that preserves the accuracy and formal guarantees of temporal-logic-guided LVQA while closing this latency gap. RT-NeuS introduces coarse-to-fine adaptive sampling to identify the small set of query-relevant, visually distinct frames, and batched proposition detection with KV-cache reuse to evaluate all propositions per window in a single forward pass. We derive latency upper bounds as a function of video length, proposition count, and sampling density. Experiments on LongVideoBench, Video-MME, and MLVU reduce inference latency by up to 13× on a single NVIDIA H200 GPU, while matching or exceeding prior neuro-symbolic accuracy.

In-sensor computing reduces the cost of transmitting high-resolution image data by performing early-stage processing near the sensor. However, the logic chip integrated with a CMOS image sensor (CIS) is tightly constrained in compute and memory, limiting conventional deep neural network partitioning. We present OASIS, a distributed in-sensor vision framework that uses a lightweight encoder to generate compact, task-relevant representations before off-chip transmission. The encoder is trained end-to-end using task, entropy, and reconstruction objectives, while the decoder is used only during training. OASIS supports two complementary deployment paths. The first applies 4-bit quantization and Huffman coding while preserving the spatial structure required by classification and dense-prediction tasks. The second uses Sobol-based hyperdimensional computing (HDC) to transform the encoder latent into a fixed-dimensional binary hypervector for associative-memory classification. For the SwinViT-based VWW model, mapping a 3×3×8 latent to a 64-dimensional hypervector provides an additional 1.77× communication reduction with less than one percentage point of accuracy loss relative to the 128-dimensional configuration, yielding an overall 18,816× reduction compared with raw 8-bit image transmission. We implement the digital near-sensor pipeline on an AMD Xilinx Zynq UltraScale+ FPGA and characterize it using direct board-level power measurements and Vivado post-implementation analysis, together with circuit-simulated CIS models and a 7-nm ASIC projection. Across visual wake-word classification, hand tracking, and eye tracking, OASIS reduces total system energy by approximately 2×–4.5× while maintaining competitive accuracy, demonstrating a practical hardware-algorithm co-design path for communication-efficient in-sensor vision.

The high computational complexity associated with training deep neural networks limits online and real-time training on edge devices. This paper proposed an end-to-end training and inference scheme that eliminates multiplications by approximate operations in the log-domain which has the potential to significantly reduce implementation complexity. We implement the entire training procedure in the log-domain, with fixed-point data representations. This training procedure is inspired by hardware-friendly approximations of log-domain addition which are based on look-up tables and bit-shifts. We show that our 16-bit log-based training can achieve classification accuracy within approximately 1% of the equivalent floating-point baselines for a number of commonly used datasets.

Index terms: Logarithmic Number System, Deep Neural Networks, Approximate Computation