Xingjian Shi

施行健

Co-founder, Boson AI

: 4677 Old Ironsides Drive, Santa Clara; CA 95054, United States
: xshiab AT connect DOT ust DOT hk
: My Github
: My Google Scholar

About Me

I'm leading model development at Boson AI, founded by Dr. Mu Li and Dr. Alex Smola. We created the Higgs series of LLM and multimodal LLMs, like Higgs-Llama and Higgs-Audio, designed to advance the state of the art in language and audio understanding and generation.

Before that, I was a senior applied scientist at Amazon that leads two projects: AutoGluon Multimodal and DeepEarth. AutoGluon Multimodal goes beyond traditional AutoML tools by leveraging foundation models. DeepEarth builds foundation models for earth. I obtained Ph.D degree from the Hong Kong University of Science and Technology in 2018. My Ph.D supervisor was Prof. Dit-Yan Yeung. I received my B.E. degree from Shanghai Jiao Tong University in 2014. I love open source and participated in projects like Apache/MXNet.

Research Interest

Multimodal generative model (Vision & Audio & Language)
Multimodal AutoML
Deep learning for earth science

Selected Publications by Topic

Deep Learning for Earth Science

Bridging Remote Sensors with Multisensor Geospatial Foundation Models

Boran Han, Shuai Zhang, Xingjian Shi, Markus Reichstein.

IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.

[paper]
PreDiff: Precipitation Nowcasting with Latent Diffusion Models.

Zhihan Gao, Xingjian Shi, Boran Han, Hao Wang, Xiaoyong Jin, Danielle Maddix Robinson, Yi Zhu, Mu Li, Yuyang Bernie Wang.

Thirty-Sixth Annual Conference on Neural Information Processing Systems (NeurIPS), 2023.

[paper] [project page] [Amazon Science]
Towards Geospatial Foundation Models via Continual Pretraining.

Matias Mendieta, Boran Han, Xingjian Shi, Yi Zhu, Chen Chen.

International Conference on Computer Vision (ICCV), 2023.

[paper] [project page]
Earthformer: Exploring Space-Time Transformers for Earth System Forecasting.

Zhihan Gao, Xingjian Shi*, Hao Wang, Yi Zhu, Yuyang Wang, Mu Li, Dit-Yan Yeung (* Contact person)

Thirty-Fifth Annual Conference on Neural Information Processing Systems (NeurIPS), 2022.

[paper] [project page] [Amazon Science]
Deep learning and the weather forecasting problem -- precipitation nowcasting.

Zhihan Gao, Xingjian Shi*, Hao Wang, Dit-Yan Yeung, Wang-chun Woo, and Wai-Kin Wong (* indicates corresponding author)

Deep learning for the Earth Sciences: A Comprehensive Approach to Remote Sensing, Climate Science and Geosciences, G. Camps-Valls, D. Tuia, X.X. Zhu, and M. Reichstein (eds.), Wiley & Sons, 2021.

[Book Preview]
Deep Learning for Precipitation Nowcasting: A Benchmark and A New Model.

Xingjian Shi, Zhihan Gao, Leonard Lausen, Hao Wang, Dit-Yan Yeung, Wai-kin Wong, Wang-chun Woo

Thirty-First Annual Conference on Neural Information Processing Systems (NeurIPS), 2017. (Accepted as Spotlight)

[paper] [project page] [poster]
Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting.

Xingjian Shi, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-kin Wong, Wang-chun Woo

Twenty-Ninth Annual Conference on Neural Information Processing Systems (NeurIPS), 2015.

[pdf] [supplementary] [code and data]

Multimodal AutoML

Tailoring Instructions to Student's Learning Levels Boosts Knowledge Distillation.

Yuxin Ren*, Zihan Zhong*, Xingjian Shi, Yi Zhu, Chun Yuan, Mu Li (* indicates equal contribution)

The 61st Annual Meeting of the Association for Computational Linguistics (ACL), 2023.

[paper] [project page]
XTab: Cross-table Pretraining for Tabular Transformers.

Bingzhao Zhu, Xingjian Shi, Nick Erickson, Mu Li, George Karypis, Mahsa Shoaran

Fortieth International Conference on Machine Learning (ICML), 2023.

[paper] [project page]
Parameter-Efficient Fine-Tuning Design Spaces.

Jiaao Chen, Aston Zhang, Xingjian Shi, Mu Li, Alex Smola, Diyi Yang

The Eleventh International Conference on Learning Representations (ICLR), 2023.

[paper] [project page]
Learning Multimodal Data Augmentation in Feature Space.

Zichang Liu, Zhiqiang Tang, Xingjian Shi, Aston Zhang, Mu Li, Anshumali Shrivastava, Andrew Gordon Wilson

The Eleventh International Conference on Learning Representations (ICLR), 2023.

[paper] [project page]
Benchmarking Multimodal AutoML for Tabular Data with Text Fields.

Xingjian Shi*, Jonas Mueller*, Nick Erickson, Mu Li, Alexander J. Smola (* indicates equal contribution)

Proceedings of the Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks, 2021.

[paper] [project page]

Generative Models

L3Ms--Lagrange Large Language Models.

Guneet S Dhillon, Xingjian Shi, Yee Whye Teh, Alexander J. Smola

The Thirteenth International Conference on Learning Representations (ICLR), 2025.

[paper]
LayoutDiffuse: Adapting Foundational Diffusion Models for Layout-to-Image Generation.

Jiaxin Cheng, Xiao Liang, Xingjian Shi, Tong He, Tianjun Xiao, Mu Li

The AI for Content Creation (AI4CC) workshop at CVPR, 2023.

[paper] [project page]
Symbolic Music Generation with Transformer-GANs.

Aashiq Muhamed*, Liang Li*, Xingjian Shi, Suri Yaddanapudi, Wayne Chi, Dylan Jackson, Rahul Suresh, Zachary C. Lipton, Alexander J. Smola (* indicates equal contribution)

Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI), 2021.

[paper] [project page]

Deep Learning System

GluonCV and GluonNLP: Deep Learning in Computer Vision and Natural Language Processing.

Jian Guo, He He, Tong He, Leonard Lausen, Mu Li, Haibin Lin, Xingjian Shi, Chenguang Wang, Junyuan Xie, Sheng Zha, Aston Zhang, Hang Zhang, Zhi Zhang, Zhongyue Zhang, Shuai Zheng, Yi Zhu

Journal of Machine Learning Research (JMLR), 2020.

[paper]

Spatiotemporal Modeling

GaAN: Gated Attention Networks for Learning on Large and Spatiotemporal Graphs.

Xingjian Shi*, Jiani Zhang*, Junyuan Xie, Hao Ma, Irwin King, Dit-Yan Yeung (* indicates equal contribution)

Thirty-Fourth Conference on Uncertainty in Artificial Intelligence (UAI), 2018.

[paper] [project page]
Spatiotemporal Modeling for Crowd Counting in Videos.

Feng Xiong, Xingjian Shi, Dit-Yan Yeung

Sixteenth IEEE International Conference on Computer Vision (ICCV), 2017.

[paper] [project page]

Academic Service

PC Member: IJCAI 2015, AAAI 2018-now
Senior PC Member: IJCAI 2021
Conference Reviewer: ICML (2019-now), NeurIPS (2019-now), ICLR (2021-now)
Journal Reviewer:
- ACM Transactions on Intelligent Systems and Technology
- ACM Transactions on Knowledge Discovery from Data
- Computational Intelligence
- Data Mining and Knowledge Discovery
- Frontiers of Computer Science
- IEEE Journal of Biomedical and Health Informatics
- IEEE Transactions on Big Data
- IEEE Transactions on Image Processing
- IEEE Transactions on Information Forensics and Security
- IEEE Transactions on Knowledge and Data Engineering
- IEEE Transactions on Neural Networks and Learning Systems
- International Journal of Data Science and Analytics
- Neural Computation
- Sustainable Cities and Society
- Weather and Forecasting

Working Experiences

- Member of Technical Staff at Boson AI
- May, 2023-Now
- Senior Applied Scientist at AWS
- Apr, 2021-Mar, 2023
- Applied Scientist at AWS
- Jul, 2019-Apr, 2021
- Applied Scientist Intern at AWS
- Nov, 2017-Apr, 2018

Selected Awards

- Hong Kong PhD Fellowship
- 2014-2018
- NeurIPS Travel Award
- 2015, 2017
- National Scholarship, Shanghai Jiao Tong University (Top 1%)
- 2011-2013

Presentations

AutoGluon: Empowering (Multimodal) AutoML for the Next 10 Million Users, NeurIPS Workshop, 2022

[Project Page]
Practical Automated Machine Learning with Tabular, Text, and Image Data, KDD Tutorial, 2020

[Project Page]
Deep Learning for Precipitation Nowcasting: A Benchmark and A New Model, NeurIPS Spotlight Presentation, 2017

[Download Slide][Video]
Dynamic Key-Value Memory Networks for Knowledge Tracing, VALSE, 2017

[Download Slide]
모두를 위한 MXNET, AWS Seoul Summit, 2017

[Download Slide] [Project Page]
A Practitioner's Guide to MXNet, HKUST CSE Seminar, 2017

[Download Slide]
Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting, VALSE Seminar, 2016

[Download Slide]