http://raufibishov.com/tags/huggingface-accelerate/Recent content in HuggingFace Accelerate on Rauf IbishovHugo -- gohugo.ioen© 2026 Rauf IbishovThu, 01 Jan 2026 00:00:00 +0000http://raufibishov.com/projects/az-neobert/Thu, 01 Jan 2026 00:00:00 +0000http://raufibishov.com/projects/az-neobert/<p><em>Status: <strong>In training</strong> · Phase 1 active on 8× NVIDIA H200 · Component 2 of the AzBERT pipeline</em></p> <h2 class="relative group">TL;DR <div id="tldr" class="anchor"></div> <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none"> <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#tldr" aria-label="Anchor">#</a> </span> </h2> <p>I am training AzNEOBERT — an Azerbaijani encoder language model from scratch, based on the NeoBERT architecture. The model runs on 8× NVIDIA H200 GPUs via SLURM, trained on ~12.2B tokens of Azerbaijani text across 11 corpus collections. Infrastructure stack: DeepSpeed ZeRO-2, Flash Attention 3, and <code>torch.compile</code>, reaching 1.24M tokens/sec throughput. Loss dropped from 11.1 (random init) to 2.35 within the first 4,100 steps.</p>