Jacob Zhiyuan Fang

I may not maintain this website in future. Check new website.

I obtained my Ph.D. degree from the Department of Computer Science at Arizona State University (ASU) as a member of APG Lab led by Dr. Yezhou Yang. Before joining ASU, I obtained my BEng degree from Department of Computer Science and Engineering at Southern University of Science and Technology. I've also spent great times collaborating with researchers in Microsoft, UC, Irvine, Chinese Academy of Sciences as research intern.



Amazon Alexa-AI, Sunnyvale, USA

       Applied Scientist,  June 2022.

Mirosoft, Azure Florence - Vision and Language, Redmond, USA

       Research Intern,  Summer 2021.

       Mentors: Jianfeng Wang, Zhe Gan, Xiaowei Hu, Zicheng Liu, Lijuan Wang

Mirosoft, AI Cloud & Platform, Redmond, USA

       Research Intern,  Summer 2020.

       Mentors: Jianfeng Wang, Lei Zhang, Lijuan Wang, Zicheng Liu

MM Lab, Chinese Academy of Science, Shenzhen, China

       Research Intern,  2017

       Mentors: Yu Qiao, Zhifeng Li

IGB, University of California, Irvine, USA

       Research Intern,  Summer 2016

       Mentors: Juan Wang, Pierre Baldi

Southern University of Science and Technology, Shenzhen, China

       Bachelor of Engineering,  2013-2017

       Academic Mentors: Qi Hao, Kun Chen

Selected Publications & Preprints

Google Scholar for full publication list
<⋆>: equal contribution.
Huiliang Shao, Zhiyuan Fang, Yezhou Yang,
CAVAN: Commonsense Knowledge Anchored Video Captioning
International Conference on Pattern Recognition, Montreal, Aug, 2022
Arnav Chakravarthy, Zhiyuan Fang, Yezhou Yang,
Tragedy Plus Time: Capturing Unintended Human Activities from Weakly-labeled Videos
Robustness in Sequential Data Workshop (ROSE) @ CVPR 2022, Oral Presentation, Louisiana, June, 2022
Zhiyuan Fang
Building Vision and Language Models with Implicit Supervision and Increased Efficiency
Doctorate Dissertation, April, 2022
Zhiyuan Fang, Jianfeng Wang, Xiaowei Hu, Lin Liang, Zhe Gan, Lijuan Wang, Yezhou Yang, Zicheng Liu
Injecting Semantic Concepts into End-to-End Image Captioning
Conference on Computer Vision and Pattern Recognition, (CVPR), New Orleans-Louisiana, June, 2022
Zhiyuan Fang, Jianfeng Wang, Xiaowei Hu, Lijuan Wang, Yezhou Yang, Zicheng Liu
Compressing Visual-linguistic Model via Knowledge Distillation
International Conference on Computer Vision (ICCV), Virtual, Aug, 2021
Zhiyuan Fang, Jianfeng Wang, Lijuan Wang, Lei Zhang, Yezhou Yang, Zicheng Liu
SEED: Self-supervised Distillation For Visual Representation
International Conference on Representation Learning (ICLR), Virtual, May, 2021
Zhiyuan Fang, Shu Kong, Zhe Wang, Charless Fowlkes, Yezhou Yang
Weakly-Supervised Temporal-Language Association with Referring Attention
arXiv preprint arXiv:2006.11747 [ Rejected 3 times by: ICCV19, CVPR20, WACV20 Smiley face ]
[PDF] [Video Demo][Website]
Zhe Wang⋆, Zhiyuan Fang⋆, Jun Wang, Yezhou Yang
ViTAA: Visual-Textual Attributes Alignment in Person Search by Natural Language
European Conference on Computer Vision (ECCV), Virtual, July, 2020
Zhiyuan Fang⋆,Tejas Gokhale⋆, Pratyay Banerjee, Chitta Baral, Yezhou Yang
Video2Commonsense: Generating Commonsense Descriptions to Enrich Video Captioning
Conference on Empirical Methods in Natural Language Processing,
EMNLP, Long&Oral Presentation, Nov 2020
[PDF][Website & Data][Code]
Tejas Gokhale, Shailaja Sampat, Zhiyuan Fang, Yezhou Yang, Chitta Baral
Cooking With Blocks : A Recipe for Visual Reasoning on Image-Pairs
CVPR 2019 Vision Meets Cognition Workshop
Zhiyuan Fang, Shu Kong, Charless Fowlkes, Yezhou Yang
Modularized Textual Grounding for Counterfactual Resilience
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, June 2019.
[Website][PDF][Poster][Code] [Slide]
Zhiyuan Fang, Shu Kong, Tianshu Yu, Yezhou Yang
Weakly Supervised Attention Learning for Textual Phrases Grounding
CVPR 2018 Language And Vision Workshop
Xiao Zhang, Zhiyuan Fang, Yandong Wen, Zhifeng Li, Yu Qiao
Range Loss for Deep Face Recognition with Long tail
International Conference on Computer Vision (ICCV), Venice, Italy, 2017.
Juan Wang, Zhiyuan Fang, Ning Lang, Huishu Yuan, Min-Ying Su, and Pierre Baldi
A Multi-Resolution Approach for Spinal Metastasis Detection using Deep Siamese Neural Networks
Computers in Biology and Medicine, 2017 (Honor Paper, 5%)


I am fortunate to have worked with some great researchers and I sincerely appreciate all the helps from them:
Shu Kong (CMU) Lijuan Wang (Microsoft) Xiaowei Hu (Microsoft) Zicheng Liu(Microsoft) Jianfeng Wang (Microsoft) Zhe Gan (Microsoft)
Tianshu Yu (CUHK) Yu Qiao (MMLAB) Chengxi Ye (Amazon) Xiao Zhang (CUHK) Tejas Gokhale (ASU) Pratyay Banerjee (ASU)
Zhifeng Li (Tencent) Charless Fowlkes (UCI) Shailaja Sampat (ASU) Pierre Baldi (UCI) Lei Zhang (IDEA) Lingqi Zhang (Upenn)
Yandong Wen (CMU) Chitta Baral (ASU) Juan Wang (IIT) Kun Chen (SUSTech) Huiliang Shao (ASU) Zhe Wang (MSRA)

Also shout out to my Ph.D. Committee Supervisors:
Prof. Yezhou Yang (ASU) Prof. Chitta Baral (ASU) Prof. Huan Liu (ASU) Prof. Zicheng Liu (Microsoft)

Professional activities

  • Conference reviewer & Programme Committee:
    CVPR-19/20/21;   Neurips-21;   ICLR-22;   ICCV-21;   AAAI-21;   WACV-20/21;   BMVC-19/20/21;   ACCV-20;   ICRA-21;   ...

  • Journal reviewer:
    Transactions on Image Processing (TIP)
    Pattern Recognition, Elsevier
    IEEE Transactions on Circuits and Systems for Video Technology

  • Teaching associative:
      FIN208: Data Mining, 2017 Spring, SusTech
      CSE205: Object Oriented Programming, 2017 Fall, 2018 Spring, ASU
      CSE230: Computer Organization and Assembly Programming, 2018 Spring, ASU
      CSE310: Data Structures and Algorithms, 2017 Fall, 2018 Fall, 2021 Fall, ASU
      CSE510: Artificial Intelligence, 2019 Spring, ASU & Coursera Online Master Degrees

  • Academic Seminar Organizer:
      Frontier topics in Vision and/or Language, Spring 2021

  • Conference Workshop Organizer:
      O-DRUM: Open-Domain Retrieval Under Multi-Modal Setting, June 2022


Designed via Bootstrap. Credits to my friend Jian Kang.