InES Seminar Winter Semester 2024

In reinforcement learning (RL), an agent learns the optimal sequence of actions through trial and error in an unknown environment. In this context, model-based RL entails constructing a world model which serves as an internal representation of the environment. This capability enhances the efficiency and generalization by allowing agents to plan and simulate actions without direct interaction in the external world. Recently, various ways of learning such world models have been proposed. The focus of this seminar paper is on summarizing recent advancements and exploring open challenges where further research is needed to overcome existing limitations.

[1] Hafner, D., Lillicrap, T., Ba, J., & Norouzi, M. (2019). Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603.
[2] Chen, C., Wu, Y. F., Yoon, J., & Ahn, S. (2022). Transdreamer: Reinforcement learning with transformer world models. arXiv preprint arXiv:2202.09481.
[3] Micheli, V., Alonso, E., & Fleuret, F. (2022). Transformers are sample-efficient world models. arXiv preprint arXiv:2209.00588.
[4] Gumbsch, C., Sajid, N., Martius, G., & Butz, M. V. (2023, October). Learning Hierarchical World Models with Adaptive Temporal Abstractions from Discrete Latent Dynamics. In The Twelfth International Conference on Learning Representations.

In reinforcement learning (RL), an agent learns the optimal sequence of actions through trial and error in an unknown environment. The integration of diffusion models presents a novel approach to enhancing various aspects of RL by capturing the underlying distribution of states, actions, and rewards. For example, as a generative method, they can generate diverse state-action pairs for better exploration, create synthetic training data to enhance robustness or help in adapting learned distributions to new tasks for faster convergence. The goal of this seminar is to provide a comprehensive overview of how diffusion models can enhance RL. This includes recent advancements and challenges that require further research.

[1] Chen, J., Ganguly, B., Xu, Y., Mei, Y., Lan, T., & Aggarwal, V. (2024). Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directions. arXiv preprint arXiv:2402.13777.

[2] Ajay, A., Du, Y., Gupta, A., Tenenbaum, J., Jaakkola, T., & Agrawal, P. (2022). Is conditional generative modeling all you need for decision-making?. arXiv preprint arXiv:2211.15657.

[3] Wang, Z., Hunt, J. J., & Zhou, M. (2022). Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.06193.

The research project “MediCar 4.0” seeks to create an advanced logistics platform for self-driving vehicles on clinic premises. The goal is to develop robust AI-based routing strategies that integrate world-knowledge from large language models (LLM). In the field of natural language processing, LLMs have shown remarkable performance in various tasks. However, understanding and reasoning about graphs remains a challenging problem. Recent methods have been developed to enhance the ability of LLMs to comprehend and process graph structured data. These methods involve integrating graph neural networks with LLMs, developing graph-aware architectures, and designing specialized pretraining tasks that enable the models to better capture the relationships and structures within graphs. The focus of this seminar paper is on summarizing these recent advancements and finding applications within routing problems.

[1] Jin, B., Liu, G., Han, C., Jiang, M., Ji, H., & Han, J. (2023). Large language models on graphs: A comprehensive survey. arXiv preprint arXiv:2312.02783.

[2] Chen, Z., Mao, H., Li, H., Jin, W., Wen, H., Wei, X., ... & Tang, J. (2024). Exploring the potential of large language models (llms) in learning on graphs. ACM SIGKDD Explorations Newsletter, 25(2), 42–61.

[3] Huang, C., Ren, X., Tang, J., Yin, D., & Chawla, N. (2024, May). Large Language Models for Graphs: Progresses and Directions. In Companion Proceedings of the ACM on Web Conference 2024 (pp. 1284-1287).

Time series segmentation is a crucial technique in the analysis of time series data, which is data recorded or measured at successive points in time [1]. The goal of time series segmentation is to divide a time series into meaningful, homogenous segments that represent distinct patterns, behaviors, or regimes within the data [1]. This segmentation helps in understanding the underlying structure and dynamics of the time series, facilitating better analysis, prediction, and decision-making. Challenges include noise and outliers, complexity and scalability, parameter selection [2]. Furthermore, key concepts include pattern recognition, anomaly detection, change point detection [3], [4], etc. To connect this seminar to a contemporary subject—specifically Physical Guards—at the Institute for Enterprise Systems (InES) and the Chair of Practical Computer Science IV (Prof. Armknecht), we propose to concentrate on change point detection following a comprehensive overview of time series segmentation. The objective of this seminar is to offer an in-depth exploration of time series segmentation, with a particular focus on change point detection. This topic is highly relevant to our ongoing project, as we aim to identify and classify “transient components” [5], such as power ramp-ups during the initiation of a transmission in Wireless Communication Systems, like WiFi. These ramp-ups represent abrupt changes in a continuous signal and should thus be detectable by change point detection models [6].

Short non-exclusive references:
[1] Aghabozorgi, S., Shirkhorshidi, A. S., & Wah, T. Y. (2015). Time-series clustering–a decade review. Information systems, 53, 16–38.
[2]  Bagnall, A., Bostrom, A., Large, J., & Lines, J. (2016). The great time series classification bake off: An experimental evaluation of recently proposed algorithms. Extended version. arXiv 2016. arXiv preprint arXiv:1602.01711.
[3]  Aminikhanghahi, S., & Cook, D. J. (2017). A survey of methods for time series change point detection. Knowledge and information systems, 51(2), 339–367.
[4]  Namoano, B., Starr, A., Emmanouilidis, C., & Cristobal, R. C. (2019, June). Online change detection techniques in time series: An overview. In 2019 IEEE international conference on prognostics and health management (ICPHM) (pp. 1–10). IEEE.
[5] Köse, M., Taşcioğlu, S., & Telatar, Z. (2019). RF fingerprinting of IoT devices based on transient energy spectrum. IEEE Access, 7, 18715-18726.
[6]  Hall, J., Barbeau, M., & Kranakis, E. (2003). Detection of transient in radio frequency fingerprinting using signal phase. Wireless and optical communications, 9, 13.

Rare event classification refers to the task of identifying events that occur infrequently within a dataset [1]. These events are characterized by their low prevalence compared to the more common, or majority, class instances. This imbalance poses significant challenges for machine learning models, which tend to be biased towards the majority class and often fail to accurately detect rare events [2]. Key challenges include data imbalance, choice of appropriate evaluation metrics, overfitting to majority class(es), etc. The goal of this seminar is to give a perspective of different rare event tasks, challenges and shed a light on different strategies to address the identified challenges. Additionally, describe applications and typical use cases of rare event classification (e.g., [3]).

Short non-exclusive references:

[1] Abubakar, Y. I., Othmani, A., Siarry, P., & Sabri, A. Q. M. (2024). A Systematic Review of Rare Events Detection Across Modalities using Machine Learning and Deep Learning. IEEE Access.

[2] Shyalika, C., Wickramarachchi, R., & Sheth, A. (2023). A comprehensive survey on rare event prediction. arXiv preprint arXiv:2309.11356.

[3] Saputri, P. D., Prastyo, D. D., Oktaviana, P. P., Azmi, U., & Siswono, G. O. (2023, November). Rare Event Classification Based on Binary Generalized Extreme Value-Additive Models. In 2023 6th International Conference on Information and Communications Technology (ICOIACT) (pp. 269–274). IEEE.

In the rapidly evolving field of artificial intelligence, the architecture of a neural network is pivotal to its performance. Traditional approaches require retraining from scratch with any modification in network size, leading to significant resource consumption and inefficiency. To address this, self-expanding neural networks (SENN) offer a dynamic approach, starting with a small architecture and expanding only as needed, without interfering with previous optimization. This method is particularly advantageous for continual learning, where maintaining plasticity—the ability to adapt to new information while retaining previous knowledge—is crucial.

The seminar is motivated by a practical study. This includes a short summary of the most recent advances in SENNs. The main focus lies on a practical demonstration of a SENN, as presented by [1], in a chosen domain (i.e. Reinforcement Learning).

[1] Mitchell, Rupert, Robin Menzenbach, Kristian Kersting, und Martin Mundt. „Self-Expanding Neural Networks“. arXiv, http://arxiv.org/abs/2307.04526.

In the context of AI-driven education, Retrieval Augmented Generation (RAG) and contextual learning are state-of-the-art methods that can be used to automate the grading process. RAG is the use of retrieval mechanisms that allow LLMs to access relevant learning materials, such as lecture scripts or exercises, during the grading process. This is essential to verify that a given answer on an exam is directly or indirectly related to the course material, thus ensuring academic correctness. Contextual learning refines this by enabling LLMs to understand the relationships between course content and answers given on a test, improving the accuracy of grading and feedback.

Recent developments in AI have introduced different methods for implementing RAG and context learning. The focus of this seminar paper is to review the advances and to analyze their application in automated grading systems. To explore the open challenges to improve the reliability and transparency of AI-based grading.