👨🏻‍💻 About Me

I am a postdoctoral research associate in the Knowledge Technology Group, University of Hamburg.

I am interested in building robust neural architectures that generalize to new tasks while not forgetting previously learned knowledge by encouraging them to learn world models (i.e., causal relationships underlying the input data) and discouraging them from learning spurious correlations. The target domains are multimodal language learning and spatio-temporal reasoning and learning, where I employ explainability and neuro-symbolic integration to elucidate and facilitate learning world models.

🗞 News

• Our paper Enhancing Zero-Shot Chain-of-Thought Reasoning in Large Language Models through Logic has been accepted to LREC-COLING 2024.
• Program committee member, ECAI 2024
• Our paper Causal State Distillation for Explainable Reinforcement Learning has been accepted to Causal Learning and Reasoning (CLeaR) 2024.
• Program committee member, LREC-COLING 2024
• Our paper Visually Grounded Continual Language Learning with Selective Specialization has been accepted to the Findings of EMNLP 2023.
• MSc student Björn Plüster, whom I advise, developed a German LLM called LeoLM with LAION and hessian.AI.
• Our paper Internally Rewarded Reinforcement Learning has been accepted to ICML 2023.
• I am co-organizing the 2nd International Workshop on Spatio-Temporal Reasoning and Learning.
• Received ~10K € research funding for a student project on “Explainable Visual Question Answering” that I am advising. Check out our paper and code.

🔬 Research Experience

• Neurocognitive Models of Crossmodal Language Learning (2020–present)
  • University of Hamburg 🇩🇪 (PIs: Stefan Wermter and Cornelius Weber)
  • Topics: Multimodal Language Learning, XAI, Neuro-Symbolic AI
• Formal Lexically Informed Logics for Searching the Web (2018–2020)
  • Cardiff University 🇬🇧 (PI: Steven Schockaert)
  • Topic: Natural Language Processing
• Feodor Lynen Research Fellowship by the Humboldt Foundation (2016–2017)
  • University of Technology Sydney 🇦🇺 (Host: Sanjiang Li)
  • Topics: Spatio-Temporal Reasoning and Multiagent Systems
• Artificial Intelligence Meets Wireless Sensor Networks (2015)
  • The Australian National University 🇦🇺 (PI: Jochen Renz)
  • Topic: Spatio-Temporal Reasoning and Learning
• Reasoning about Paths, Shapes, and Configuration (2009–2014)
  • University of Bremen 🇩🇪 (PIs: Christian Freksa^† and Diedrich Wolter)
  • Topic: Spatio-Temporal Reasoning
• The International Research Training Group “Semantic Integration of Geospatial Information” (2010-2013)
  • University of Bremen 🇩🇪
  • Topic: Spatio-Temporal Reasoning

🎓 Education

• Dr. rer. nat. in Computer Science, University of Bremen 🇩🇪 (2009–2013)
• Visiting PhD student, North Carolina State University 🇺🇸 (2012)
• Visiting PhD student, University at Buffalo 🇺🇸 (2011)
• Diplom in Mathematics, University of Bremen 🇩🇪 (2003–2009)
• DAAD exchange student, Seoul National University 🇰🇷 (2008)

📜 Selected Publications

ℹ Categorized by topics. Multiple appearances in different categories possible.
ℹ Full publication list available on DBLP, Google Scholar, and Semantic Scholar.
ℹ : code : arXiv : project page.

(Multimodal) Language Learning:

• K. Ahrens, L. Bengtson, J. H. Lee , S. Wermter, Visually Grounded Continual Language Learning with Selective Specialization, EMNLP Findings 2023.
• J. H. Lee, M. Kerzel, K. Ahrens, C. Weber, S. Wermter, What is Right for Me is Not Yet Right for You: A Dataset for Grounding Relative Directions via Multi-Task Learning, IJCAI 2022.
• B. Plüster, J. Ambsdorf, L. Braach, J. H. Lee, S. Wermter, Harnessing the Power of Multi-Task Pretraining for Ground-Truth Level Natural Language Explanations, arXiv preprint, 2022.
• J. H. Lee, Y. Yao, O. Özdemir, M. Li, C. Weber, Z. Liu, S. Wermter, Spatial relation learning in complementary scenarios with deep neural networks, Frontiers in Neurorobotics, 2022.
• O. Özdemir, M. Kerzel, C. Weber, J. H. Lee, and S. Wermter, Language Model-Based Paired Variational Autoencoders for Robotic Language Learning, IEEE Transactions on Cognitive and Developmental Systems, 2022.
• C. Volquardsen, J. H. Lee, C. Weber, and S. Wermter, More Diverse Training, Better Compositionality! Evidence from Multimodal Language Learning, ICANN 2022.
• M. Li, C. Weber, M. Kerzel, J. H. Lee, Z. Zeng, Z. Liu, S. Wermter, Robotic Occlusion Reasoning for Efficient Object Existence Prediction IROS 2021.
• A. Eisermann, J. H. Lee, C. Weber, S. Wermter, Generalization in Multimodal Language Learning from Simulation, IJCNN 2021.
• J. H. Lee, J. Camacho-Collados, L. Espinosa Anke, S. Schockaert, Capturing Word Order in Averaging Based Sentence Embeddings, ECAI 2020.

Explainable Artificial Intelligence:

• W. Lu, X. Zhao, T. Fryen, J. H. Lee, M. Li, S. Magg, S. Wermter, Causal State Distillation for Explainable Reinforcement Learning, accepted to CLeaR 2024.
• J. H. Lee, S. Lanza, S. Wermter, From Neural Activations to Concepts: A Survey on Explaining Concepts in Neural Networks, arXiv preprint, 2023.
• B. Plüster, J. Ambsdorf, L. Braach, J. H. Lee, S. Wermter, Harnessing the Power of Multi-Task Pretraining for Ground-Truth Level Natural Language Explanations, arXiv preprint, 2022.

Continual Learning

• K. Ahrens, H. H. Lehmann, J. H. Lee, S. Wermter, Read Between the Layers: Leveraging Intra-Layer Representations for Rehearsal-Free Continual Learning with Pre-Trained Models, arXiv preprint, 2023.
• K. Ahrens, L. Bengtson, J. H. Lee , and S. Wermter, Visually Grounded Continual Language Learning with Selective Specialization, EMNLP Findings 2023.

Neuro-Symbolic Artificial Intelligence:

• X. Zhao, M. Li, W. Lu, C. Weber, J. H. Lee, K. Chu, S. Wermter, Enhancing Zero-Shot Chain-of-Thought Reasoning in Large Language Models through Logic, accepted to LREC-COLING 2024.
• J. H. Lee, S. Lanza, S. Wermter, From Neural Activations to Concepts: A Survey on Explaining Concepts in Neural Networks, arXiv preprint, 2023.
• J. H. Lee, M. Sioutis, K. Ahrens, M. Alirezaie, M. Kerzel, S. Wermter, Neuro-Symbolic Spatio-Temporal Reasoning, in Compendium of Neurosymbolic Artificial Intelligence, IOS Press, 2023, pp. 410–429.
• J. H. Lee, Y. Yao, O. Özdemir, M. Li, C. Weber, Z. Liu, S. Wermter, Spatial relation learning in complementary scenarios with deep neural networks, Frontiers in Neurorobotics, 2022.

Other Machine Learning Topics:

• M. Li, X. Zhao, J. H. Lee, C. Weber, S. Wermter, Internally Rewarded Reinforcement Learning, ICML 2023.
• S. Kong, J. Bai, J. H. Lee, D. Chen, A. Allyn, M. Stuart, M. Pinsky, K. Mills, C. Gomes, Deep Hurdle Networks for Zero-Inflated Multi-Target Regression: Application to Multiple Species Abundance Estimation, IJCAI 2020.

Spatio-Temporal Reasoning and Learning:

• J. H. Lee, M. Sioutis, K. Ahrens, M. Alirezaie, M. Kerzel, S. Wermter, Neuro-Symbolic Spatio-Temporal Reasoning, in Compendium of Neurosymbolic Artificial Intelligence, IOS Press, 2023, pp. 410–429.
• J. H. Lee, M. Kerzel, K. Ahrens, C. Weber, S. Wermter, What is Right for Me is Not Yet Right for You: A Dataset for Grounding Relative Directions via Multi-Task Learning, IJCAI 2022.
• J. H. Lee et al. Spatial relation learning in complementary scenarios with deep neural networks, Frontiers in Neurorobotics, 2022.
• S. Kong, J. H. Lee, S. Li, Multiagent Simple Temporal Problem: The Arc-Consistency Approach AAAI 2018.
• F. Dylla, J. H. Lee, T. Mossakowski, T. Schneider, A.V. Delden, J.V.D. Ven, D. Wolter, A Survey of Qualitative Spatial and Temporal Calculi: Algebraic and Computational Properties, ACM Computing Surveys, 2017.
• J. H. Lee, S. Li, Z. Long, M. Sioutis, On Redundancy in Simple Temporal Networks, ECAI 2016.
• D. Wolter, J. H. Lee, Connecting Qualitative Spatial and Temporal Representations by Propositional Closure, IJCAI 2016.
• X. Ge, J. H. Lee, J. Renz, P. Zhang, Trend-Based Prediction of Spatial Change, IJCAI 2016.
• S. Schockaert, J. H. Lee, Qualitative Reasoning About Directions in Semantic Spaces, IJCAI 2015.
• P. Zhang, J. H. Lee, J. Renz, From Raw Sensor Data to Detailed Spatial Knowledge, IJCAI 2015.
• J. H. Lee, The Complexity of Reasoning with Relative Directions, ECAI 2014.
• J. H. Lee, J. Renz, D. Wolter, StarVars: Effective Reasoning About Relative Directions, IJCAI 2013.
• D. Wolter, J. H. Lee, Qualitative reasoning with directional relations, Artificial Intelligence, 2010.

Multiagent Systems:

• S. Kong, J. H. Lee, S. Li, Multiagent Simple Temporal Problem: The Arc-Consistency Approach AAAI 2018.
• S. Kong, J. H. Lee, S. Li, A new distributed algorithm for efficient generalized arc-consistency propagation, Autonomous Agent Multi-Agent Systems 32, 2018.
• S. Kong, J. H. Lee, S. Li, A Deterministic Distributed Algorithm for Reasoning with Connected Row-Convex Constraints, AAMAS 2017.

🏫 Recent Teaching Activities

• Lecture on “Transformers and Crossmodal Learning”, University of Hamburg (2023) [Slides]
• Supervisor, Neural Networks Seminar, University of Hamburg (2023)
• Organizer, WISDUM Reading Group, Knowledge Technology, University of Hamburg (2023)
• Supervisor, Bio-inspired Artificial Intelligence Seminar, University of Hamburg (2022)

🧑‍🎓 Thesis Supervision

• Continual Learning for Language-Conditioned Robotic Manipulation, Lennart Bengtson, MSc (2023)
• Multivariate Normal Methods in Pre-trained Models for Continual Learning, Hans Hergen Lehmann, BSc (2023) improved version submitted to a conference
• Generalization of Transformer-Based Models on Visual Question Answering Tasks, Frederic Voigt, MSc (2023)
• Improving Compositional Generalization By Learning Concepts Individually, Ramtin Nouri, MSc (2023)
• Learning Concepts a Developmental Lifelong Learning Approach to Visual Question Answering, Ramin Farkhondeh, BSc (2022)
• Benchmarking Faithfulness: Towards Accurate Natural Language Explanations in Vision-Language Tasks, Jakob Ambsdorf, MSc (2022), now PhD student at University of Copenhagen
• Tackling The Binding Problem And Compositional Generalization In Multimodal Language Learning, Caspar Volquardsen, BSc (2021), ICANN 2022
• Learning Bidirectional Translation Between Robot Actions and Linguistic Descriptions, Markus Heidrich, BSc (2021)
• Using the Reformer for Efficient Summarization, Yannick Wehr, BSc (2020)
• Generalization in Multi-Modal Language Learning from Simulation, Aaron Eisermann, BSc (2020), IJCNN 2021
• Commonsense Validation and Explanation, Christian Rahe, BSc (2020)

✍🏼 Blog Posts

• Deep Learning Courses and Books for Beginners
• Common LaTeX Mistakes by Newbies

📮 Contact

Address

Knowledge Technology Research Group
Vogt-Koelln-Str. 30
22527 Hamburg, Germany
Google Maps

Email

ed.eeleeheaj@eaj