Zhize Li (李志泽)

I am now a Research Scientist at Carnegie Mellon University since 2022, working with Prof. Yuejie Chi. Before that, I was a Research Scientist at King Abdullah University of Science and Technology (KAUST).

Previously, I was a Postdoctoral Research Fellow at KAUST (hosted by Prof. Peter Richtárik), a Visiting Scholar at Duke University (hosted by Prof. Rong Ge), and a Visiting Scholar at Georgia Institute of Technology (hosted by Prof. Guanghui (George) Lan).

I received my PhD in Computer Science from the Institute for Interdisciplinary Information Sciences at Tsinghua University in July 2019. My Advisor was Prof. Jian Li.

My research interests lie in mathematical optimization and machine learning, in particular convex/nonconvex/distributed optimization, federated learning, private machine learning, algorithms and data structures.

PhD thesis: Simple and Fast Optimization Methods for Machine Learning. June 2019.
2019 Tsinghua Outstanding Doctoral Dissertation Award.

News

• 1 paper was accepted to SIMODS, 2022.
• 1 paper was accepted to ICML 2022.
• I gave a talk at Foundations Of Data Science (FODS) seminar on May 13, 2022.
• I was named a Rising Star in AI, organized by KAUST AI Initiative (headed by Prof. Jürgen Schmidhuber) in 2022.
• 1 paper was accepted to NeurIPS 2021, and 3 papers were accepted to NeurIPS Workshop on Optimization (OPT 2021).
• 1 paper was accepted to Information and Computation, 2021.
• 2 papers were accepted to ICML 2021.

Remote Interns/Students

1. Haoyu Zhao (PhD student at Princeton University)

   • BEER: Fast O(1/T) Rate for Decentralized Nonconvex Optimization with Communication Compression

     Haoyu Zhao, Boyue Li, Zhize Li\(^\dagger\), Peter Richtárik, Yuejie Chi.   (\(^\dagger\)corresponding author)

     arXiv:2201.13320, 2022.

     In this work, we propose a fast algorithm BEER for decentralized nonconvex optimization. We show that BEER converges at a faster rate of \(O(1/T)\), significantly improving over the state-of-the-art rate \(O((G/T)^{2/3})\), where \(T\) is the number of communication rounds and \(G\) measures the data heterogeneity/bounded gradient assumption. In sum, BEER removes the strong assumptions required in prior works and also enjoys a faster convergence rate.

   • Faster Rates for Compressed Federated Learning with Client-Variance Reduction

     Haoyu Zhao, Konstantin Burlachenko, Zhize Li\(^\dagger\), Peter Richtárik.   (\(^\dagger\)corresponding author)

     arXiv:2112.13097, 2021.

     In this work, we design two algorithms, which successfully incorporate communication compression (deal with communication bottleneck) with client-variance reduction (deal with large number of heterogeneous clients), for federated learning.
     Both algorithms do not need to communicate with all clients and provide the first/faster convergence rates for convex and nonconvex federated learning, while previous works either require full clients communication (thus not practical) or obtain worse convergence results.

   • FedPAGE: A Fast Local Stochastic Gradient Method for Communication-Efficient Federated Learning

     Haoyu Zhao, Zhize Li\(^\dagger\), Peter Richtárik.   (\(^\dagger\)corresponding author)

     arXiv:2108.04755, 2021.

     This is Haoyu's first work on optimization and federated learning. In this work, we propose a new federated learning algorithm, FedPAGE, providing much better state-of-the-art communication complexity for both federated convex and nonconvex optimization.
     In particular, Haoyu will give a long talk at Federated Learning One World Seminar on August 11, 2021. [slides]

Short Notes

1. A Short Note of PAGE: Optimal Convergence Rates for Nonconvex Optimization

   Zhize Li

   Note (4 pages), 2021.

Publications (by year) [google scholar] [dblp]

1. SoteriaFL: A Unified Framework for Private Federated Learning with Communication Compression

   Zhize Li, Haoyu Zhao, Boyue Li, Yuejie Chi

   arXiv:2206.09888, 2022.

2. DESTRESS: Computation-Optimal and Communication-Efficient Decentralized Nonconvex Finite-Sum Optimization

   Boyue Li, Zhize Li, Yuejie Chi

   SIAM Journal on Mathematics of Data Science (SIMODS), 2022.

   Previously appeared in NeurIPS Workshop on Optimization for Machine Learning (NeurIPS-OPT 2021)

3. 3PC: Three Point Compressors for Communication-Efficient Distributed Training and a Better Theory for Lazy Aggregation

   Peter Richtárik, Igor Sokolov, Ilyas Fatkhullin, Elnur Gasanov, Zhize Li, Eduard Gorbunov

   39th International Conference on Machine Learning (ICML 2022)

4. CANITA: Faster Rates for Distributed Convex Optimization with Communication Compression

   Zhize Li, Peter Richtárik

   35th Conference on Neural Information Processing Systems (NeurIPS 2021) [slides]

5. ANITA: An Optimal Loopless Accelerated Variance-Reduced Gradient Method

   Zhize Li

   NeurIPS 2021 Workshop on Optimization for Machine Learning (NeurIPS-OPT 2021)

6. EF21 with Bells & Whistles: Practical Algorithmic Extensions of Modern Error Feedback

   Ilyas Fatkhullin, Igor Sokolov, Eduard Gorbunov, Zhize Li\(^\dagger\), Peter Richtárik

   NeurIPS 2021 Workshop on Optimization for Machine Learning (NeurIPS-OPT 2021)

7. Optimal In-Place Suffix Sorting

   Zhize Li, Jian Li, Hongwei Huo

   Information and Computation, 2021. (CCF rank A journal in theory)

8. PAGE: A Simple and Optimal Probabilistic Gradient Estimator for Nonconvex Optimization

   Zhize Li, Hongyan Bao, Xiangliang Zhang, Peter Richtárik

   38th International Conference on Machine Learning (ICML 2021). (long talk 166/5513 = 3%)

   Previously appeared in NeurIPS Workshop on Optimization for Machine Learning (NeurIPS-OPT 2020). (oral)

9. MARINA: Faster Non-Convex Distributed Learning with Compression

   Eduard Gorbunov, Konstantin Burlachenko, Zhize Li, Peter Richtárik

   38th International Conference on Machine Learning (ICML 2021)

10. A Unified Analysis of Stochastic Gradient Methods for Nonconvex Federated Optimization

    Zhize Li, Peter Richtárik

    Submitted to Journal of Machine Learning Research (JMLR), under revision.

    NeurIPS Workshop on Scalability, Privacy, and Security in Federated Learning (NeurIPS-SpicyFL 2020). (spotlight)

11. Acceleration for Compressed Gradient Descent in Distributed and Federated Optimization

    Zhize Li, Dmitry Kovalev, Xun Qian, Peter Richtárik

    37th International Conference on Machine Learning (ICML 2020) [slides]

12. A Fast Anderson-Chebyshev Acceleration for Nonlinear Optimization

    Zhize Li, Jian Li

    23rd International Conference on Artificial Intelligence and Statistics (AISTATS 2020)

13. A Unified Variance-Reduced Accelerated Gradient Method for Convex Optimization

    [\(^*\)alphabetical order] Guanghui Lan\(^*\), Zhize Li\(^*\), Yi Zhou\(^*\)

    33rd Conference on Neural Information Processing Systems (NeurIPS 2019) [slides]

14. SSRGD: Simple Stochastic Recursive Gradient Descent for Escaping Saddle Points

    Zhize Li

    33rd Conference on Neural Information Processing Systems (NeurIPS 2019)

15. Stochastic Gradient Hamiltonian Monte Carlo with Variance Reduction for Bayesian Inference

    Zhize Li, Tianyi Zhang, Shuyu Cheng, Jun Zhu, Jian Li

    Machine Learning, 2019, journal.

16. Stabilized SVRG: Simple Variance Reduction for Nonconvex Optimization

    [\(^*\)alphabetical order] Rong Ge\(^*\), Zhize Li\(^*\), Weiyao Wang\(^*\), Xiang Wang\(^*\)

    32nd Conference on Learning Theory (COLT 2019)

17. Learning Two-layer Neural Networks with Symmetric Inputs

    [\(^*\)alphabetical order] Rong Ge\(^*\), Rohith Kuditipudi\(^*\), Zhize Li\(^*\), Xiang Wang\(^*\)

    7th International Conference on Learning Representations (ICLR 2019)

18. Gradient Boosting With Piece-Wise Linear Regression Trees

    Yu Shi, Jian Li, Zhize Li

    28th International Joint Conference on Artificial Intelligence (IJCAI 2019)

19. A Simple Proximal Stochastic Gradient Method for Nonsmooth Nonconvex Optimization

    Zhize Li, Jian Li

    32nd Conference on Neural Information Processing Systems (NeurIPS 2018). (spotlight 198/4856 = 4%)

20. A Fast Polynomial-time Primal-Dual Projection Algorithm for Linear Programming

    Zhize Li, Wei Zhang, Kees Roos

    arXiv:1810.04517, 2018. A talk presented at the 23rd International Symposium on Mathematical Programming.

21. A Two-Stage Mechanism for Ordinal Peer Assessment

    Zhize Li, Le Zhang, Zhixuan Fang, Jian Li

    11th International Symposium on Algorithmic Game Theory (SAGT 2018)

22. Optimal In-Place Suffix Sorting

    Zhize Li, Jian Li, Hongwei Huo

    25th International Symposium on String Processing and Information Retrieval (SPIRE 2018) (invited) [slides]
    Previous one-page [summary paper] appeared in 28th IEEE Data Compression Conference (DCC 2018)

    Suffix array (as a space-efficient alternative to suffix tree) is a fundamental data structure for many applications that involve string searching and data compression. Designing time/space-efficient suffix array construction algorithms has attracted significant attention and considerable advances have been made for the past 20 years. We give the first linear time in-place suffix array construction algorithm which is optimal both in time and space. In particular, our algorithm solves the important open problem posed by Franceschini and Muthukrishnan in ICALP 2007. Unfortunately, our paper has been prevented from being published since 2017 due to a malicious conflict of interest [a disclaimer].

23. On Top-k Selection in Multi-Armed Bandits and Hidden Bipartite Graphs

    Wei Cao, Jian Li, Yufei Tao, Zhize Li

    29th Conference on Neural Information Processing Systems (NIPS 2015)

Experience/Services

• Conference Program Committee/Reviewer:

    2022 (NeurIPS, ICML, MLSys, ALT, ICLR)
    2021 (NeurIPS, ICML, COLT, STOC, ICLR)
    2020 (NeurIPS, ICML, COLT, AAAI)
    2019 (NeurIPS, ICML, COLT, STOC, ECML)
    COLT'18, ICML'17, AAAI'17, ICALP'16
  

• Conference Area Chair/Senior Program Committee: IJCAI 2021

• Journal Referee:

  ​  Operations Research
    Mathematics of Operations Research
    SIAM Journal on Optimization (SIOPT)
    Journal of Machine Learning Research (JMLR)
    Journal of the Royal Statistical Society (JRSS)
    Computational Optimization and Applications (COAP)
    IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)
    IEEE Transactions on Signal Processing (TSP)
    IEEE Transactions on Parallel and Distributed Systems (TPDS)
    IEEE Transactions on Neural Networks and Learning Systems (TNNLS)
    IEEE/ACM Transactions on Networking (ToN)

• Research Assistant (RA):

    Tsinghua-MIT-CUHK Research Center for Theoretical Computer Science (Sep, 2014 - Jun, 2019)

• Teaching Assistant (TA):

    i) Advanced Theoretical Computer Science for graduate students (Spring 2017)
    ii) Algorithm Design for undergraduate students (Yao Class) (Fall 2015)

• The 2016 National Combinatorial Optimization Summer School

    Courses: Computational Complexity, Approximation Algorithms, Randomized Algorithms
    Outstanding Student (top 10/110)

• Algorithm Engineer Intern: Alibaba (Jul - Aug, 2013), NetEase (Jun - Aug, 2017)

Conference Talks/Posters

NeurIPS 2021, ICML 2021, NeurIPS 2020, ICML 2020, AISTATS 2020, NeurIPS 2019, ECML 2019, COLT 2019, ICLR 2019, IJCAI 2019, NeurIPS 2018, SPIRE 2018, SAGT 2018, DCC 2018

Other Talks/Posters

1. China Theory Week 2018 (founded by Prof. Andrew Chi-Chih Yao in 2007), 2018
2. The 16th China Symposium on Machine Learning and Applications (MLA'18), 2018
3. The 23rd International Symposium on Mathematical Programming (ISMP), 2018
4. KAUST-Tsinghua-Industry Workshop on AI, 2019
5. SIAM Conference on Optimization (OP20), 2020. cancelled due to Covid-19
6. Carnegie Mellon University, 2021
7. Foundations Of Data Science (FODS) seminar, 2022
8. International Conference on Continuous Optimization (ICCOPT), 2022

Selected Awards

• Rising Star in AI
• Tsinghua Outstanding Doctoral Dissertation Award
• National Scholarship (top 1%), Tsinghua Scholarship, Kwang-Hua Scholarship
• Pacesetter of Outstanding Graduates (highest, top 1%)
• The First Prize in Collegiate Advanced Mathematics Contest (highest, top 13/15000+)

Contact

Zhize Li

Email: zhizeli DOT thu AT gmail DOT com

zz-li14 AT mails DOT tsinghua DOT edu DOT cn

Address: Room 4-609, FIT Building, Tsinghua University, Beijing 100084, China