Finetuning ASR model

Project Description

This project was for a hackathon. I finetuned OpenAI's ASR model (whisper-medium.en), and accelerated it using FasterWhisper, a library designed to increase inference speed of Whisper models up to 4x. I chose that model in particular to finetune because it gave a good balace between model size, speed, and accuracy, which was crucial when integrated with the other modules for the hackathon. My team (h3althydr0plet) unfortunately did not win, but I did build the 5th best ASR model among 60 other groups.

Project Showcase

Previous Next

My Reflections & Learnings

This was actually my first time dealing with ASR models,and even fine-tuning so there was a steep learning curve, especially for what I can do in 2 weeks. I initially attempted to finetune Nvidia's ParakeetV2, but ran into the problem, that it is built on Nvidia's NeMo framework. After a few days, I realised that though the NeMo framework is open source, to meaningfully use it to finetune Nvidia models like Parakeet, I require access to Nvidia microservices which was beyond my budget, hence i switched over to Hugging Face's framework for fine-tuning.