NextBigFuture

Analog in-memory Computing Attention Mechanism for Fast and Energy-efficient Large Language Models

A Nature paper describes an innovative analog in-memory computing (IMC) architecture tailored for the attention mechanism in large language models (LLMs). They want to drastically reduce latency and ...
IEEE

ASTRA: Reconfigurable Training Architecture Design for Nonlinear Softmax and Activation Functions in Transformers

Abstract: The efficient training of Transformer-based neural networks on resource-constrained personal devices is attracting continuous attention due to domain adaptions and privacy concerns. However, ...
C&EN

Integrating ESM-2 and Graph Neural Networks with AlphaFold-2 Structures for Enhanced Protein Function Prediction

Creative Commons (CC): This is a Creative Commons license. Attribution (BY): Credit must be given to the creator. Protein function prediction is essential for elucidating biological processes and ...
GitHub

F.gumbel_softmax returns NaN on MPS device

The reason seems to be that the exponential_() method sometimes produces actual zeros, which the log() method turns into infinities. Maybe similar to #2561? As a workaround, I've copied the function ...
Frontiers

MoSViT: a lightweight vision transformer framework for efficient disease detection via precision attention mechanism

1 School of Mechanical Engineering, Xijing University, Xi'an, China 2 School of Electronic Information, Xijing University, Xi'an, China Maize, a globally essential staple crop, suffers significant ...
Frontiers

A hypergraph transformer method for brain disease diagnosis

Objective: To address the high-order correlation modeling and fusion challenges between functional and structural brain networks. Method: This paper proposes a hypergraph transformer method for ...
VentureBeat

Microsoft’s Differential Transformer cancels attention noise in LLMs

Want smarter insights in your inbox? Sign up for our weekly newsletters to get only what matters to enterprise AI, data, and security leaders. Subscribe Now Improving the capabilities of large ...
GitHub

TypeError: Tensor Out of Scope in TensorFlow Function Graph When Using CardBench

First, I want to express my sincere gratitude for your contribution of CardBench to the field of Cardinality Estimation—it has been incredibly helpful in my work. I'm replicating the Instance based ...