Deliberate Reasoning in Language Models as Structure-Aware Planning with an Accurate World Model
Siheng Xiong, Ali Payani, Yuan Yang, Faramarz Fekri

Project

• SWAP is a structure-aware planning framework for multi-step reasoning with LMs. At each step, given the current state, represented as a graph, and an action, the enhanced world model predicts the next state as an updated graph. The policy model is guided by this graph to propose next action.

CausalEval: Towards Better Causal Reasoning in Language Models
Longxuan Yu*, Delin Chen*, Siheng Xiong*, Qingyang Wu, Dawei Li, Zhikai Chen, Xiaoze Liu, Liangming Pan

Project

• We provide a comprehensive review of research aimed at enhancing LMs for causal reasoning. We evaluate the performance of different LMs and methods on various causal reasoning tasks, providing key findings and in-depth analysis.

Can LLMs Reason in the Wild with Programs?
Yuan Yang, Siheng Xiong, Ali Payani, Ehsan Shareghi, Faramarz Fekri

Project

• Tiger is a TactIc-Guided ReasonER designed to tackle reasoning-in-the-wild tasks by generating and refining programs. It learns from previous trajectories to iteratively improve program generation, enabling more effective reasoning (like OpenAI o1 model).

Large Language Models Can Learn Temporal Reasoning
Siheng Xiong, Ali Payani, Ramana Kompella, Faramarz Fekri

Project

• TG-LLM performs temporal reasoning in two steps: 1) Text-to-Temporal Graph translation: generate (relevant) temporal graph given the context and keyword (extracted from questions); 2) Temporal Graph Reasoning: perform deliberate Chain-of-Thought reasoning over the temporal graph.

Harnessing the power of large language models for natural language to first-order logic translation
Yuan Yang, Siheng Xiong, Ali Payani, Ehsan Shareghi, Faramarz Fekri

Project

• LogicLLaMA can be used standalone or to correct previously generated rules by other models for the NL-FOL translation task.

Long-Context Modeling with Dynamic Hierarchical Sparse Attention for On-Device LLMs
Siheng Xiong, Joe Zou, Faramarz Fekri, Yae Jee Cho

Project

• We introduce Dynamic Hierarchical Sparse Attention (DHSA), a plug-in module integrated into each Transformer layer that enables multi-head attention to skip unimportant token pairs. The key idea is to exploit chunk-level similarity to guide token-level sparsity prediction, enhanced with dynamic chunking and length-normalized chunk representations. DHSA attains the performance of dense attention while significantly reducing both latency and memory usage.

The Compressor-Retriever Architecture for Language Model OS
Yuan Yang, Siheng Xiong, Ehsan Shareghi, Faramarz Fekri

Project

• We introduce compressor-retriever, a model-agnostic architecture designed for life-long context management. Our approach exclusively uses the base model’s forward function to compress and retrieve context, ensuring end-to-end differentiability.

TILR: Temporal Inductive Logic Reasoning over Hypergraphs
Yuan Yang, Siheng Xiong, Ali Payani, James C Kerce, Faramarz Fekri

Project

• TILR is a reasoning framework that detects inductive patterns in temporal data via neural-logic methodology. The framework aims to assist the training of modern ML models by inducing patterns for accurate grounding with fewer data.

TEILP: Time prediction over knowledge graphs via logical reasoning
Siheng Xiong, Yuan Yang, Ali Payani, James C Kerce, Faramarz Fekri

Project

• TEILP is a follow-up work of TILP. We convert TKGs into a temporal event knowledge graph (TEKG) which equips us to develop a differentiable random walk approach. We also introduce conditional probability density functions, associated with the logical rules involving the query interval, using which we arrive at the time prediction.

TILP: Differentiable learning of temporal logical rules on knowledge graphs
Siheng Xiong, Yuan Yang, Faramarz Fekri, James Clayton Kerce

Project

• TILP is the first differentiable framework for temporal logical rules learning. We introduce constrained random walk mechanism and temporal operators with temporal features modeling, e.g., recurrence, temporal order, interval between pair of relations, and duration.