Ruchao Fan

Ruchao Fan, Ph.D. Senior Applied Scientist at Microsoft Email: fanruchao at g dot ucla dot edu [LinkedIn] [Github] [Google Scholar] Prior Ph.D. Student @UCLA Speech Processing and Auditory Perception Laboratory Ph.D. Advisor: Abeer Alwan

Bio

Ruchao Fan is currently a Senior Applied Scientist at Microsoft Azure AI Speech Team. He obtained his Ph.D. degree from the Department of Electrical and Computer Engineering, UCLA. His research interests are multi-modal representation learning, speech and language modeling, and so on.

During his Ph.D., Rucho Fan worked with Prof. Abeer Alwan on speech processing problems. He was focusing on children's speech recognition from a low-resource perspective and end-to-end speech recognition. Prior to that, He received his Master's and Bachelor's degree from Beijing University of Posts and Telecommunications (BUPT) in 2019 and 2016, respectively.

Ruchao has interned at Sogou Inc. (2018-2019), PAII Inc. (2020), and Microsoft Corporation (2021, 2022), Amazon Web Service (2023), conducting research on speech recognition and language modeling. He was awarded UCLA-Amazon Fellowship during 2021-2022, and was honored as Amazon Fellow.

News

• [Feb 2025] We released the first Microsoft Multimodal LLM, Phi-4-MultiModal!

  • Phi4-MM Huggingface Weights

  • Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs

• [Dec 2024] Checkout our new Arxiv paper, AlignFormer!

  • When applying the idea of modality matching in speech-LLM, we unlock the zero-shot capability for speech-LLM when training with ASR data only.

  • AlignFormer: Modality Matching Can Achieve Better Zero-shot Instruction-Following Speech-LLM

• [Dec 2024] One co-authored paper was presented at SLT 2024.

  • The paper talks about how to leaverge extensive text machine translation data for speech translation with modality matching.

  • CTC-GMM: CTC guided modality matching for fast and accurate streaming speech translation

• [Jun 2024] Check out our paper on bechmarking children's ASR with speech foundation models. The paper was accpeted to Interspeech 2024.

  • Benchmarking Children's ASR with Supervised and Self-supervised Speech Foundation Models

  • Code, models are released in Huggingface!

• [May 2024] I joined Microsoft Azure Speech Team as an Applied Scientist.

• [Mar 2024] I successfully defended my Ph.D. thesis and graduated from UCLA.

  • Improving the Accuracy and Inference Efficiency for Low-resource Automatic Speech Recognition

• [Feb 2024] The Last work for CASS-NAT seris was accepted to IEEE Signal Processing Letters. Check out the paper here.

  • UniEnc-CASSNAT: An Encoder-only Non-autoregressive ASR for Speech SSL Models

• [Jun 2023] Summer intern at Amazon Web Service, working on long-context large language modeling.

• [Mar 2023] My second Journal Paper was accepted to IEEE Transactions on Audio, Speech,and Language Processing (TASLP), and is available for early access:

  • A CTC Alignment-based Non-autoregressive Transformer for End-to-end Automatic Speech Recognition

  • Codes for our CASS-NAT work are available at CASS-NAT Code

• [Feb 2023] My summer intern work CTCBERT was accepted to ICASSP 2023!

  • CTCBERT: Advancing Hidden-unit BERT with CTC Objectives

  • ICASSP Poster Presentation

• [Oct 2022] My first Journal paper was accpeted to JSTSP (IEEE Journal of Selected Topics in Signal Processing)!

  • Towards Better Domain Adaptation for Self-Supervised Models: A Case Study of Child ASR

  • ICASSP Presentation of Journal Papers

  • Code for the DRAFT!

• [Jun 2022] A wonderful summer intern @Microsoft, working with Dr. Yiming Wang, Dr. Yashesh Gaur, Dr. Guoli Ye, and Dr. Jinyu Li. This is also my second intern at Microsoft.

• [Jun 2022] One paper has been accepted by Interspeech 2022!

  • DRAFT: A Novel Framework to Reduce Domain Shifting in Self-supervised Learning and Its Application to Children's ASR

• [Feb 2022] I was awarded with UCLA-Amazon Fellowship!

• [Jun 2022] Two co-authored papers have been accepted by ICASSP 2022!

  • Towards Better Meta-Initialization with Task Augmentation for Kindergarten-aged Speech Recognition

  • LPC Augment: An LPC-Based ASR Data Augmentation Algorithm for Low and Zero-Resource Children's Dialects

• [Dec 2021] I passed my Oral Qualifying Exam and became a Ph.D. Candiate.

• [Dec 2021] Our journal paper Fundamental frequency feature warping for frequency normalization and data augmentation in child automatic speech recognition has been published on Speech Communication.

• [Jun 2021] I started my summer intern @Microsoft, working with Dr. Guoli Ye and Dr. Jinyu Li.

• [Jun 2021] Two papers have been accepted by Interspeech 2021!

  • An Improved Single Step Non-autoregressive Transformer for Automatic Speech Recognition

  • Low Resource German ASR with Untranscribed Data Spoken by Non-native Children – INTERSPEECH 2021 Shared Task SPAPL System

• [Jan 2021] Three papers have been accepted by ICASSP 2021!

  • CASS-NAT: CTC Alignment-based Single Step Non-autoregressive Transformer for Speech Recognition

  • Bi-APC: Bidirectional Autoregressive Predictive Coding for Unsupervised Pre-training and Its Application to Children's ASR

  • Fundamental Frequency Feature Normalization and Data Augmentation for Child Speech Recognition

• [Sep 2020] I had a fruitful summer @PAII, working with Dr. Wei Chu and Dr. Peng Chang.

• [Jul 2020] Our challenge paper Exploring the Use of an Unsupervised Autoregressive Model as a Shared Encoder for Text-Dependent Speaker Verification has been accepted by Interspeech 2020.

• [Sep 2019] I started my Ph.D. at UCLA.

• [Jun 2019] I got my Master's degree in Beijing University of Posts and Telecommunications.

• [Jun 2019] Our Paper An Online Attention-based Model for Speech Recognition has been accepted by Interspeech 2019.

Peer Review

• Top speech conference and journal

  • Interspeech, ICASSP

  • Speech Communication

  • IEEE Signal Processing Letter

  • IEEE Journal of Selected Topics in Signal Processing, JSTSP

  • IEEE Transactionon Audio, Speech, and Language Processing, TASLP

• Others

  • IEEE Transactions on Pattern Analysis and Machine Intelligence, TPAMI

  • NeurIPS 2024, 2025

  • ICLR 2023, 2024, 2025

  • IEEE International Conference on Multimedia and Expo, ICME (2021)

  • Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC (2020, 2021)

Education

• Ph.D. University of California, Los Angeles, Los Angeles, CA, U.S.A., Sept. 2019 - Mar. 2024

• M.S. & B.E., Beijing University of Posts and Telecommunications, Beijing, China

  • M.S., Sept. 2016 - Jun. 2019

  • B.E., Sept. 2012 - Jun. 2016