Machine Learning

490 readers

1 users here now

A community for posting things related to machine learning

Icon base by Lorc under CC BY 3.0 with modifications to add a gradient

founded 1 year ago

MODERATORS

Ategon

Akisamb

ericjmorey

Has anybody replaced attention with Hyena Hierarchy (self.machine_learning)

submitted 1 year ago by Akisamb to c/machine_learning

1 comments fedilink hide all child comments

Hyena Hierarchy seems to aim to be a drop-in replacement for attention : https://arxiv.org/pdf/2302.10866.pdf

It looks good on paper, but I haven't been able to find anybody using it in a model. Does anyone have an example of a code or implementation ? Is there really a big improvement on long context lengths ?

you are viewing a single comment's thread
view the rest of the comments

[–] kraegar 2 points 1 year ago

My research area has been in time series forecasting and unsupervised anomaly detection, but it is SOMEWHAT related to NLP.

Papers with code had a few potential implementations: https://paperswithcode.com/paper/hyena-hierarchy-towards-larger-convolutional

I am always skeptical of papers. They could have good results, but how much did they adjust their experiment to look good on paper?