MPS Backend Not Working – CPU is Slow on Apple Silicon

[theseriouspdx]

Title: MPS Backend Not Working – CPU is Slow on Apple Silicon

---

Description:
While trying to run Insanely Fast Whisper on a Mac with Apple Silicon (using the MPS backend), I encountered a NotImplementedError related to sparse tensor operations. The error suggested that the operation aten::_sparse_coo_tensor_with_dims_and_tensors is not implemented for MPS.

I resolved the issue by forcing the --device-id to cpu, but it seems that MPS compatibility is either incomplete or not gracefully handled in this tool.

---

Error Details:
The specific error message when using MPS was:

NotImplementedError: Could not run 'aten::_sparse_coo_tensor_with_dims_and_tensors' with arguments from the 'SparseMPS' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build). 'aten::_sparse_coo_tensor_with_dims_and_tensors' is only available for these backends: [MPS, Meta, SparseCPU, SparseMeta, BackendSelect, Python, ...]

---

Environment Details:

• OS: macOS (e.g., Ventura 13.6 / Sonoma, whichever you have)
• Chip: Apple Silicon (e.g., M1 Pro/M2 Max)
• Python Version: 3.12.x
• PyTorch Version: 2.5.1
• Insanely Fast Whisper Version: [latest version installed]
• Diarization Model: pyannote/speaker-diarization-3.1
• Command Executed:

  insanely-fast-whisper --file-name star69_interview.wav \
      --transcript-path star69_diarized_transcript.txt \
      --diarization_model pyannote \
      --num-speakers 3 \
      --device-id mps

---

Steps to Reproduce:

1. Install insanely-fast-whisper using pip.
2. Execute the command with the --device-id mps flag on a Mac with Apple Silicon.
3. Observe the NotImplementedError related to sparse tensors.

---

Steps Taken to Fix:

1. Switched --device-id to cpu, and the tool successfully ran in real-time transcription mode.
2. Edited cli.py to force device = "cpu" for all runs. This resolved the issue.

---

Expected Behavior:
The MPS backend should be supported, as it’s designed for Apple Silicon acceleration. Alternatively, if sparse tensor operations are unsupported on MPS, the tool should gracefully handle the error by defaulting to cpu.

---

Suggestions:

1. Add a fallback mechanism to gracefully default to cpu if MPS backend raises errors.
2. Consider a warning message if MPS is detected but incompatible operations are being requested.
3. Verify MPS compatibility with PyTorch and address sparse tensor limitations, if feasible.

---

Performance Context:

• The tool works as expected on CPU, but the performance is slower than ideal. For example, transcribing a 60-minute audio file with three participants took approximately 75 minutes on the CPU.
• MPS support would likely improve transcription speeds significantly for Mac users on Apple Silicon, reducing time-to-completion.

---

Additional Context:

• Sparse tensor operations are known to have limited support on MPS (per PyTorch documentation). This may require specific implementation changes.
• The tool itself is fantastic and very user-friendly—just needs better MPS compatibility for Apple Silicon users.

---

Feel free to reach out if you need further details or additional testing logs. I’d be happy to assist!

[eustlb]

Hey, thanks for raising this issue :) !
We've had indeed an MPS compatibility issue with the 4.47.0 release of Transformers. Nevertheless you should have run in this issue (#255) before this one. Installing from the patch (huggingface/transformers#35295) to solve the former cited isssue, I am then unable to reproduce yours. Maybe I am not using to correct audio input. Could you please provide it ?