--- license: other license_name: lfm1.0 license_link: LICENSE tags: - llm - generative_ai - embedded - sima - liquidai - lfm2 - return-logits - accuracy-benchmarking pipeline_tag: text-generation base_model: LiquidAI/LFM2-350M --- # LFM2-350M-a16w4_return_logits: Optimized for SiMa.ai Modalix Accuracy Benchmarking ## Overview This repository contains the **LFM2-350M-a16w4_return_logits** model, optimized and compiled for the **SiMa.ai Modalix** platform. This model is compiled with `--return_logits` so that it can be used for Modalix accuracy benchmarking without recompiling the source model. - **Model Architecture:** LFM2 (350M parameters) - **Quantization:** Hybrid - **Prompt Processing:** A16W8 (16-bit activations, 8-bit weights) - **Token Generation:** A16W4 (16-bit activations, 4-bit weights) - **Maximum context length:** 2048 - **Source Model:** [LiquidAI/LFM2-350M](https://huggingface.co/LiquidAI/LFM2-350M) - **Return logits:** Enabled ## Accuracy Benchmark Results The table below compares accuracy results produced by SiMa.ai on the Modalix backend against Hugging Face backend reference results. | Task | Version | Filter | n-shot | Metric | Direction | HF Backend Value | HF Stderr | Modalix Backend Value | Modalix Stderr | Abs. Diff | |---|---:|---|---:|---|---|---:|---:|---:|---:|---:| | hellaswag | 1.0 | none | 0 | acc | higher | 0.3852 | 0.0049 | 0.369150 | 0.004816 | 0.016050 | | hellaswag | 1.0 | none | 0 | acc_norm | higher | 0.4892 | 0.0050 | 0.476399 | 0.004984 | 0.012801 | | piqa | 1.0 | none | 0 | acc | higher | 0.6910 | 0.0108 | 0.674646 | 0.010931 | 0.016354 | | piqa | 1.0 | none | 0 | acc_norm | higher | 0.6986 | 0.0107 | 0.675190 | 0.010926 | 0.023410 | | triviaqa | 3.0 | remove_whitespace | 0 | exact_match | higher | 0.1220 | 0.0024 | 0.106554 | 0.002303 | 0.015446 | | wikitext | 2.0 | none | 0 | bits_per_byte | lower | 1.1750 | N/A | 1.304415 | N/A | 0.129415 | | wikitext | 2.0 | none | 0 | byte_perplexity | lower | 2.2579 | N/A | 2.469835 | N/A | 0.211935 | | wikitext | 2.0 | none | 0 | word_perplexity | lower | 77.8836 | N/A | 125.826566 | N/A | 47.942966 | | winogrande | 1.0 | none | 0 | acc | higher | 0.5627 | 0.0139 | 0.554854 | 0.013968 | 0.007846 | ### Evaluation Sample Counts | Task | sample_len | |---|---:| | hellaswag | 10042 | | piqa | 1838 | | triviaqa | 17944 | | wikitext | 62 | | winogrande | 1267 | ## Prerequisites To benchmark accuracy with this model, you need: 1. **SiMa.ai Modalix Device** 2. **SiMa.ai CLI**: Installed on your Modalix device. 3. **SiMa.ai Neat Runtime**: Install or update the Neat Library on Modalix. The LLiMa runtime is installed as part of the Neat runtime. 4. **LLiMa benchmark CLI**: Installed on the host machine used to launch accuracy benchmarking. 5. **Hugging Face CLI**: Optional, for downloading the model on a host before copying it to Modalix. ## Installation & Deployment Follow these steps to deploy the model to your Modalix device. ### 1. Install or Update Neat Runtime > **Note:** This is a **one-time setup**. If the Neat Library is already installed on your Modalix device, you can skip this step and continue with model download. Follow the [SiMa.ai Neat getting started guide](https://developer.sima.ai/software/getting-started/) to install or update the Neat Library on your Modalix device. The `llima` CLI is available on Modalix after the Neat runtime is installed. It manages precompiled GenAI models under `/media/nvme/llima/models` by default. Set `LLIMA_MODELS_PATH` to use a different model directory. ### 2. Download the Model Download the compiled model assets from this repository directly to your device. ```bash # Download the model to a local directory llima pull LFM2-350M-a16w4_return_logits ``` Alternatively, you can download the compiled model to a Host and copy it to the Modalix device: ```bash hf download simaai/LFM2-350M-a16w4_return_logits --local-dir LFM2-350M-a16w4_return_logits scp -r LFM2-350M-a16w4_return_logits sima@:/media/nvme/llima/models/ ``` *Replace \ with the IP address of your Modalix device.* **Expected Directory Structure:** ```text /media/nvme/llima/ `-- models/ `-- LFM2-350M-a16w4_return_logits/ # The compiled model ``` ## Usage ### Modalix Backend Accuracy Run the benchmark from a host machine. The positional `model_id` must be the source Hugging Face model because MOLE uses its tokenizer and configuration; `--board_model` selects this compiled artifact on Modalix. ```bash MODALIX_IP="" llima-benchmark accuracy LiquidAI/LFM2-350M \ --backend modalix \ --output ./results/modalix \ --board_ip "$MODALIX_IP" \ --board_model LFM2-350M-a16w4_return_logits \ --task hellaswag piqa triviaqa wikitext winogrande ``` The compiled `--board_model` must already exist on Modalix and contain both `devkit/` and `elf_files/`. The CLI starts the benchmark server by default. Add `--board_venv_path /path/to/venv` when `llima` is installed in a non-default virtual environment, or `--no-board_start_server` when connecting to a server that is already running. ### Hugging Face Backend Accuracy Run the same task suite against the source model on the host: ```bash llima-benchmark accuracy LiquidAI/LFM2-350M \ --backend hf \ --output ./results/hf \ --task hellaswag piqa triviaqa wikitext winogrande ``` See [MOLE accuracy benchmarking](https://developer.sima.ai/software/genai-llima/mole) for the full accuracy benchmarking workflow. ## Limitations - **Quantization**: This model is quantized (A16W4/A16W8) for Modalix execution. Minor deviations from the full-precision source model may occur. - **Return logits**: This model is compiled with `--return_logits` for accuracy benchmarking workflows. For standard text generation deployments, use the corresponding SiMa.ai model from the Large Language Models collection. ## Troubleshooting - **`sima-cli` not found**: Ensure that `sima-cli` is installed on your Modalix device. - **`llima-benchmark` not found**: Ensure that the LLiMa benchmark CLI is installed on the host machine. - **`llima` not found**: Install or update the Neat Library. See [Getting Started](https://developer.sima.ai/software/getting-started/). - **Modalix accuracy benchmark fails**: Verify that the model was compiled with `--return_logits` and that the model directory contains both `devkit/` and `elf_files/`. - **Model can't be found on Modalix**: Verify the model directory is exactly inside `/media/nvme/llima/models/` and not nested (e.g., `/media/nvme/llima/models/LFM2-350M-a16w4_return_logits/LFM2-350M-a16w4_return_logits`). - **Permission Denied**: Ensure you have read/write permissions for the `/media/nvme` directory. ## Resources - [GenAI with LLiMa](https://developer.sima.ai/software/genai-llima/) - [LLiMa compilation](https://developer.sima.ai/software/genai-llima/compilation_genai) - [MOLE accuracy benchmarking](https://developer.sima.ai/software/genai-llima/mole)