FNet: Haɗa Alamomin Bayanai tare da Canjin Fourier a cikin Masu Rikodin Transformer

Table of Contents

1. Gabatarwa & Bayyani

Tsarin gine-ginen Transformer, tun lokacin da Vaswani da sauransu suka gabatar da shi a cikin 2017, ya zama ma'auni na ainihi don mafi kyawun Sarrafa Harshe na Halitta (NLP). Babban ƙirƙirarsa, tsarin kulawar kai, yana ba da damar ƙirar ta auna muhimmancin duk alamomin bayanai a cikin jerin lokacin da ake sarrafa kowane alama. Duk da haka, wannan tsarin yana da tsadar lissafi mai yawa, yana ƙaruwa da murabba'i ($O(N^2)$) tare da tsawon jerin ($N$), wanda ke iyakance ingancinsa don dogayen takardu ko aikace-aikace masu yawan aiki.

Wannan takarda, "FNet: Haɗa Alamomin Bayanai tare da Canjin Fourier," tana gabatar da sauƙaƙe mai tsattsauran ra'ayi. Marubutan sun bincika ko za a iya maye gurbin ƙaramin sashen kulawar kai mai tsadar lissafi gaba ɗaya da mafi sauƙi, tsarin haɗa alamomin bayanai na layi. Babban abin mamakin da suka gano shi ne cewa amfani da daidaitaccen Canjin Fourier na Discrete 2D (DFT) mara sigogi ya kai kashi 92-97% na daidaiton ƙirar BERT akan ma'auni na GLUE yayin da ake horar da shi da sauri kashi 80% akan GPUs da kashi 70% akan TPUs don daidaitattun jerin alamomin bayanai 512.

2. Hanyoyi & Tsarin Gine-gine

2.1. Maye gurbin Kulawar Kai

Babban hasashe shine cewa za a iya kusanta ko maye gurbin rikitarwa, haɗin bayanai da kulawar kai ke yi ta hanyar gyare-gyaren layi na ƙayyadaddun sigogi. Marubutan sun fara gwada ƙananan sassan haɗawa na layi masu sigogi (matakan cikakku). Da ganin sakamako mai ban sha'awa, sun bincika saurin gyare-gyaren layi na tsari, a ƙarshe sun tsaya kan Canjin Fourier.

2.2. Ƙaramin Sashen Canjin Fourier

A cikin FNet, an maye gurbin ƙaramin sashen kulawar kai a cikin daidaitaccen toshe na rikodin Transformer da Canjin Fourier 2D. Don wakilcin shigarwa $X \in \mathbb{R}^{N \times d}$ (inda $N$ shine tsawon jerin kuma $d$ shine girman ɓoyayye), ana yin haɗawa kamar haka:

$\text{FNet}(X) = \mathcal{F}_{\text{seq}}(\mathcal{F}_{\text{hidden}}(X))$

Inda $\mathcal{F}_{\text{hidden}}$ ke amfani da Canjin Fourier 1D tare da girman ɓoyayye ($d$) kuma $\mathcal{F}_{\text{seq}}$ ke amfani da shi tare da girman jerin ($N$). Ana kiyaye kawai ainihin sassan sakamakon da aka canza. Muhimmanci, wannan ƙaramin sashen ba shi da sigogi masu koyo.

2.3. Tsarin Gine-ginen FNet

Toshen rikodin FNet yana riƙe da sauran daidaitaccen tsarin gine-ginen Transformer: cibiyar sadarwar ciyarwa (FFN) ƙaramin sashe tare da rashin layi (misali, GeLU), haɗin ragowar, da daidaita sashe. Tsari shine: Ƙaramin sashen haɗawar Fourier → haɗin ragowar & daidaita sashe → Ƙaramin sashen FFN → haɗin ragowar & daidaita sashe.

3. Cikakkun Bayanai na Fasaha & Tsarin Lissafi

An ayyana Canjin Fourier na Discrete 1D (DFT) don jerin $x$ mai tsayi $N$ kamar haka:

$X_k = \sum_{n=0}^{N-1} x_n \cdot e^{-i 2\pi k n / N}$

Don canjin 2D da aka yi amfani da shi akan matrix shigarwa $X$, ana ƙididdige shi azaman canje-canje 1D guda biyu a jere. Amfani da algorithm na Fast Fourier Transform (FFT) yana rage rikitarwar wannan aikin zuwa $O(Nd \log N)$ don canjin girman jerin, wanda ya fi na daidaitaccen kulawar kai na $O(N^2 d)$ gaba ɗaya don babban $N$.

Babban fahimta shine cewa Canjin Fourier yana yin haɗin duniya na duk alamomin bayanai na shigarwa a cikin yankin mitar, wanda zai iya ɗaukar irin wannan dogaro na duniya kamar kulawar kai amma ta hanyar ƙayyadaddun tushen lissafi, maimakon wanda aka koya, wanda ya dogara da bayanai.

4. Sakamakon Gwaji & Aiki

4.1. Sakamakon Ma'auni na GLUE

An kimanta ƙirar FNet (girman Base da Large) daidai da takwarorinsu na BERT. Sakamakon yana da ban mamaki:

• FNet-Base ya kai kashi 92.2% na matsakaicin maki GLUE na BERT-Base.
• FNet-Large ya kai kashi 97.3% na matsakaicin maki GLUE na BERT-Large.

Wannan yana nuna cewa yawancin daidaiton ƙirar kulawar kai da aka daidaita a hankali ana iya dawo dasu tare da sauƙaƙan tsarin haɗawar Fourier.

4.2. Ma'auni na Long Range Arena (LRA)

A kan ma'auni na LRA, wanda aka tsara don gwada aikin ƙirar akan dogayen jerin (alamomin bayanai 1k zuwa 4k), FNet ya yi daidai da daidaiton mafi daidaitattun ƙirar "Transformer masu inganci". Muhimmanci, ya kasance da sauri sosai fiye da mafi saurin ƙirar a kan duk tsayin jerin akan GPUs.

4.3. Binciken Sauri & Ingantacciyar Aiki

Ribobin aikin suna da girma:

• Saurin Horarwa: Sauri kashi 80% fiye da BERT akan GPUs, sauri kashi 70% akan TPUs a tsayin jerin 512.
• Ƙafar Ƙwaƙwalwar Ajiya: Mai sauƙi fiye da daidaitattun Transformers, musamman mai fa'ida a ƙananan girman ƙira.
• Ma'auni: Ma'aunin $O(N \log N)$ na FFT yana ba FNet fa'ida mai ƙarfi fiye da ko da kusancin kulawar lokaci-layin ($O(N)$) akan GPUs don dogayen jerin, saboda waɗannan hanyoyin sau da yawa suna da manyan abubuwan ɓoyayye masu girma.

5. Tsarin Bincike & Misalin Lamari

Lamari: Rarraba Rubutu akan Dogayen Takardu
Yi la'akari da aiki kamar rarraba kwangilolin shari'a ko labaran kimiyya, inda takardu suka wuce alamomin bayanai 2000 akai-akai. Daidaitaccen ƙirar Transformer zai yi wahala tare da murabba'in ƙwaƙwalwar ajiya da farashin lissafi. "Mai inganci" Transformer na layi zai iya taimakawa amma yana iya yin jinkiri a aikace saboda yawan aikin kernel.

Aikace-aikacen FNet: Ƙirar FNet na iya sarrafa waɗannan dogayen jerin cikin inganci. Ƙaramin sashen Fourier yana haɗa wakilcin alamomin bayanai a duniya cikin lokacin $O(N \log N)$. Sassan FFN masu zuwa za su iya gina fasali akan waɗannan wakilcin da aka haɗa. Don ƙayyadaddun kasafin jinkiri, mutum zai iya tura ƙirar FNet mafi girma fiye da Transformer mai kwatankwacinsa, mai yuwuwar dawo da ɗan guntun tazarar da aka lura akan gajerun jerin.

Fahimtar Tsarin: FNet yana canza karkatar da hankali daga "auna alaƙar da bayanai suka motsa" (kulawa) zuwa "haɗin yanayin duniya na ƙayyadadden mitar". Nasarar FNet tana nuna cewa ga yawancin ayyukan NLP, ikon haɗa bayanai a duniya ya fi mahimmanci fiye da takamaiman, hanyar haɗawa da aka koya.

6. Babban Fahimta & Bincike Mai Muhimmanci

Babban Fahimta: Sarki na iya samun ƙarancin tufafi fiye da yadda muke tunani. Nasarar FNet kalubale ce mai tayar da hankali ga koyarwar NLP. Ya nuna cewa saniya mai tsarki ta kulawar kai—wanda sau da yawa ake ɗauka azaman tushen ƙarfin Transformer da ba za a iya rabuwa da shi ba—za a iya maye gurbinsa da aikin lissafi na shekaru 150 mara sigogi tare da ɗan rauni kaɗan na aiki amma riba mai yawa na inganci. Wannan yana nuna cewa wani muhimmin sashi na iyawar Transformer ya samo asali ne daga gaba ɗayan tsarin gine-ginensa (ragowa, FFNs, daidaita sashe) da ikonsa na kwararar bayanai na duniya, maimakon rikitarwa, koyon motsi na kulawar kanta.

Kwararar Hankali: Hankalin takardar yana da gamsarwa. Fara da matsalar tsada (kulawar murabba'i). Yi hasashen cewa haɗawa mafi sauƙi zai iya aiki. Gwada sassan layi (yana aiki lafiya). Gane cewa canji mai tsari kamar FFT yana da sauri kuma yana da ma'auni mai kyau. Gwada shi—abin mamaki, yana aiki kusan daidai. Kwararar daga matsala zuwa mafita ta maimaitawa zuwa ganowa mai ban mamaki a bayyane kuma yana da inganci a kimiyance.

Ƙarfi & Kurakurai:
Ƙarfi: Ribobin inganci ba za a iya musantawa ba kuma suna da mahimmanci a aikace. An kimanta takardar cikin tsauri akan daidaitattun ma'auni (GLUE, LRA). Ra'ayin yana da sauƙi mai kyau kuma yana da ƙarfin "me ya sa ban yi tunanin haka ba?" yana buɗe sabon sararin ƙira don ingantattun gine-gine.
Kurakurai: Tazarar daidaito, ko da yake ƙanƙanta, gaskiya ce kuma mai yiwuwa tana da mahimmanci ga aikace-aikacen bin SOTA. Takardar ba ta bincika sosai dalilin da ya sa Fourier ke aiki da kyau ko kuma wane irin kaddarorin harshe aka rasa. Akwai zato cewa aikinsa na iya tsayawa a kan ayyukan da ke buƙatar tunani mai zurfi, na nahawu ko rikitarwa, ƙididdiga masu matakai da yawa inda kulawar motsi ke da mahimmanci. Dogaro akan GPUs/TPUs tare da ingantattun ƙwayoyin FFT shine dogaro na ɓoye don da'awar sauri.

Fahimta Mai Aiki:
1. Ga Masu Aiki: Yi la'akari da FNet sosai don turawa na samarwa inda kwarara, jinkiri, ko farashi suke manyan ƙuntatawa, kuma raguwar daidaito kashi 3-8% ya yarda. Shine babban ɗan takara don "ya isa" babban sikelin sarrafa rubutu.
2. Ga Masu Bincike: Kada ku tsaya a Fourier. Wannan takarda ce kore don bincika dukan gidan namun daji na gyare-gyaren layi (Wavelets, Hartley, DCT) da matakan tsari azaman maye gurbin kulawa. Babban tambayar bincike ta zama: "Menene mafi ƙarancin, mafi saurin tsarin haɗawa wanda ya isa don fahimtar harshe?"
3. Ga Fannin: Wannan aikin, tare da takwarorinsa kamar MLP-Mixer don hangen nesa, yana nuna alamar yiwuwar motsi "koma ga tushe". Bayan shekaru da ƙaruwar rikitarwar gine-gine, muna iya shiga cikin zaman sauƙaƙe mai tsattsauran ra'ayi, yana tambayar waɗanne sassa ne ainihin mahimmanci. Yana aiki azaman tunatarwa mai mahimmanci don ƙalubalantar hasashe na asali lokaci-lokaci.

7. Aikace-aikace na Gaba & Hanyoyin Bincike

• Ƙirar Haɗaɗɗu: Haɗa sassan FNet tare da sassan kulawa masu yawa ko na gida zai iya ƙirƙirar ƙirar waɗanda duka suna da inganci kuma suna riƙe da babban daidaito don matakan tunani masu mahimmanci.
• Ƙaddamar da Yanayi: Yin amfani da ƙa'idodin FNet ga masu canzawa masu yanayi (hangen nesa, sauti). Farkon haɗa siginonin tsaka-tsakin yanayi ta hanyar canjin Fourier zai iya zama mai inganci sosai.
• Haɗin Kayan Aiki-Software: Ƙirƙirar na'urorin haɓaka AI na musamman waɗanda aka inganta don aikin FFT zai iya sa ƙirar kamar FNet su fi rinjaye a cikin yanayin da inganci ke da mahimmanci.
• Fahimtar Ka'idar: Bincike mai zurfi kan abin da Aikin Canjin Fourier ke yi na ayyukan harshe da kuma yadda sassan FFN ke rama rashin kulawar da aka koya yanki ne mai wadata don aikin gaba.
• Ƙirar Mahallin Dogon Lokaci: FNet ɗan takara ne na halitta don turawa iyakokin tsawon mahallin a cikin ƙirar harshe, yana ba da damar sarrafa dukan littattafai ko dogayen tattaunawa tare da lissafi mai sarrafawa.

8. Nassoshi

1. Vaswani, A., da sauransu. (2017). Kulawa Shine Duk Abin da Kuke Bukata. Ci gaba a cikin Tsarin Bayanai na Neural.
2. Devlin, J., da sauransu. (2019). BERT: Horon Farko na Masu Canzawa Masu Gudana Biyu Masu Zurfi don Fahimtar Harshe. NAACL-HLT.
3. Tolstikhin, I., da sauransu. (2021). MLP-Mixer: Duk Tsarin Gine-ginen MLP don Hangon Nesa. Ci gaba a cikin Tsarin Bayanai na Neural.
4. Tay, Y., da sauransu. (2020). Masu Canzawa Masu Inganci: Bincike. Binciken Kwamfuta na ACM.
5. Wang, S., da sauransu. (2020). Linformer: Kulawar Kai tare da Rikitarwa na Layi. arXiv preprint arXiv:2006.04768.
6. Katharopoulos, A., da sauransu. (2020). Masu Canzawa RNNs ne: Masu Canzawa na Autoregressive Sauri tare da Kulawar Layi. Taron Duniya akan Koyon Injiniya.
7. Google Research. Rumbun Code na FNet na Hukuma. https://github.com/google-research/google-research/tree/master/f_net