공부/AI

ChatGPT API 사용해보기 + 추천서비스 구현을 위한 임베딩(Embedding)

JangGiraffe 2024. 5. 29. 21:16

ChatGPT API 키 발급

QuickStart (튜토리얼)

  • https://platform.openai.com/docs/quickstart
  • ChatGPT API 사용법을 익히기 위해 공식문서에 있는 튜토리얼을 따라해보자.
  • 파이썬,curl,Node.js 이렇게 3가지 환경의 예제를 제공하는데 파이썬으로 결정

튜토리얼 - 파이썬

1. OpenAI 파이썬 라이브러리 설치 (파이썬 설치 생략)

pip install --upgrade openai

2. API키 셋업

  • 환경변수에 설정하는것(추천), 프로젝트에 설정하는것 두가지 방식이 있었는데 계속 쓸 API key는 아니기때문에 프로젝트에 설정하기로 함.
  • 프로젝트용 환변변수 설정하고 사용하기
  • python-dotenv 모듈 설치
  • pip install python-dotenv
  • .env파일을 생성 후 내 API키 넣기
![](@attachment/Clipboard_2024-05-29-20-21-57.png)

```
chatGpt.env
OPENAI_API_KEY="내 API 키"
```

3. API 날려보기

  • 3가지 예제를 제공하는데, 임베딩때문에 시작한거니깐 임베딩 예제를 실행해봤음
  • from dotenv import load_dotenv from openai import OpenAI import os # load .env load_dotenv("chatGpt.env") client = OpenAI(api_key=os.environ.get('OPENAI_API_KEY'),) response = client.embeddings.create( model="text-embedding-ada-002", input="The food was delicious and the waiter..." ) print(response)

- 요청 시 다음과 같은 에러가 발생한다.

openai.RateLimitError: Error code: 429 - {'error': {'message': 'You exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors.', 'type': 'insufficient_quota', 'param': None, 'code': 'insufficient_quota'}}
  • 나는 무료 크레딧도 없고 결제 카드도 연결되어있지 않아서 돈내라고 하는거같음
  • )
  • 결제하구 다시 시도하니 잘된다.
  • The food was delicious and the waiter... 의 임베딩 값은 이렇다.
CreateEmbeddingResponse(data=\[Embedding(embedding=\[-0.029828643426299095, 0.0008259337046183646, -0.08130206167697906, -0.06542624533176422, 0.035150397568941116, -0.034412506967782974, -0.011045989580452442, 0.04091935604810715, -0.03409946337342262, -0.05156285688281059, 0.01160499732941389, -0.02095160447061062, -0.0038627423346042633, -0.030275849625468254, 0.0229193102568388, 0.0034406916238367558, 0.011593816801905632, -0.013069597072899342, 0.001780439168214798, 0.03588828444480896, 0.024998819455504417, -0.0007239148253574967, -0.02430564910173416, 0.013896928168833256, -0.007826105691492558, 0.00331491488032043, -0.01996774971485138, 0.0014303606003522873, -0.012197544798254967, -0.018637312576174736, 0.025244781747460365, -0.006613059435039759, 0.02250564470887184, -0.01907333917915821, -0.049013782292604446, -0.026139194145798683, 0.05876287445425987, 0.004100320395082235, 0.024842295795679092, 0.011739159002900124, -0.02421620860695839, 0.06945110112428665, 0.05903119966387749, 0.015406249091029167, 0.04972931370139122, 0.006422996986657381, -0.04856657609343529, 0.02056029811501503, -0.00230311113409698, -0.033518094569444656, 0.014925502240657806, 0.01831308752298355, -0.0014534196816384792, 0.06126723065972328, -0.002368794521316886, 0.010078906081616879, 0.02681000344455242, 0.008977661840617657, 0.040539227426052094, -0.018659673631191254, 0.07007718831300735, 0.014981403015553951, -0.0018349423771724105, 0.00908387266099453, -0.01701619103550911, -0.03215411677956581, -0.03532927855849266, -0.0057465978898108006, 0.014098171144723892, -0.03624605014920235, 0.03324976935982704, 0.037252265959978104, -0.04700135812163353, -0.03752058744430542, 0.05585603788495064, -0.02649695985019207, -0.01112425047904253, -0.0586734339594841, -0.01751929707825184, -0.00935219693928957, -0.030141688883304596, -0.0032673992682248354, -0.01884973607957363, -0.0029627401381731033, 0.015014943666756153, -0.07522005587816238, -0.02247210405766964, 0.007009955123066902, 0.020236074924468994, 0.06220636144280434, -0.003968953620642424, -0.00303820613771677, 0.06547096371650696, -0.015819914638996124, 0.03926469013094902, -0.024126766249537468, -0.03206467255949974, 0.04065103083848953, -0.015115564689040184, 0.031237343326210976, 0.04548085480928421, -0.06193803995847702, 0.00921244453638792, -0.033428654074668884, 0.029918085783720016, -0.012286986224353313, 0.042350415140390396, -0.032869644463062286, -0.03664853796362877, -0.024663414806127548, 0.027525532990694046, -0.010732945054769516, -0.00687020318582654, -0.0012626583920791745, 0.018637312576174736, 0.02779385633766651, 0.033518094569444656, -0.07924491167068481, -0.017027370631694794, 0.004114295821636915, -0.00912859383970499, 0.028308143839240074, -0.05616908147931099, -0.044765327125787735, -0.033764056861400604, -0.06202748045325279, -0.03461374714970589, 0.016132958233356476, -0.022460924461483955, -0.02508825995028019, 0.057734303176403046, -0.006596289575099945, -0.01385220792144537, -0.02289695106446743, -0.006210573948919773, -0.001879662973806262, 0.00936896726489067, 0.013293200172483921, 0.0016672400524839759, -0.011359034106135368, -0.021622413769364357, 9.276031778426841e-05, 0.03119262307882309, -0.025356583297252655, -0.012633571401238441, -0.019822409376502037, -0.002909634495154023, 0.03485971316695213, -0.004712434019893408, 0.005209950730204582, -0.043915633112192154, -0.001134785357862711, -0.013393821194767952, -0.016054697334766388, -0.05746597796678543, 0.0014003139222040772, 0.028487026691436768, -0.02390316314995289, 0.0033708156552165747, 0.049684591591358185, -0.026250995695590973, 0.024171488359570503, -0.033540453761816025, 0.006143493112176657, -0.01227580662816763, 0.0458386205136776, -0.008234181441366673, 0.06193803995847702, -0.04440756142139435, -0.04281998053193092, 0.017250973731279373, 0.01679258793592453, 0.012577670626342297, 0.019308121874928474, -0.01374040637165308, -0.006115542724728584, 0.040293265134096146, -0.006165853701531887, -0.002143793972209096, -0.023724282160401344, -0.01450065616518259, 0.06301133334636688, 0.0685119703412056, 0.007820515893399715, 0.0018614951986819506, -0.04013674333691597, -0.007937907241284847, -0.04344606772065163, -0.019498184323310852, 0.04682247340679169, 0.023970244452357292, 0.02953796088695526, -0.03944357484579086, -0.025893229991197586, -0.009514308534562588, 0.0028313733637332916, -0.03170691058039665, 0.028106899932026863, -0.02064974047243595, 0.027682054787874222, 0.02441745065152645, 0.0030270260758697987, 0.05527466908097267, -0.011616176925599575, 0.04700135812163353, -0.032668404281139374, -0.04107587784528732, 0.03470319136977196, 0.038884565234184265, -0.05858399346470833, 0.04722496122121811, -0.0057298275642097, -0.029381437227129936, 0.0135168032720685, -0.010274559259414673, 0.009944744408130646, -0.02443981170654297, 0.010375180281698704, 0.014690718613564968, 0.009234804660081863, -0.014523016288876534, 0.017910603433847427, 0.05903119966387749, -0.01805594377219677, 0.010710584931075573, 0.004505601245909929, 0.0019215885549783707, 0.008513685315847397, -0.04109823703765869, -0.004234482534229755, 0.03445722535252571, 0.009374557062983513, -0.012197544798254967, -0.00954784918576479, -0.04219389334321022, 0.011616176925599575, 0.004704048857092857, 0.032131753861904144, 0.0011389779392629862, 0.04156780242919922, -0.004896906204521656, 0.0049248565919697285, 0.009184494614601135, -0.0022416203282773495, 0.055453550070524216, 0.017910603433847427, -0.0008196448325179517, 0.02430564910173416, 0.011196921579539776, 0.003619574010372162, 0.026228634640574455, -0.021499430760741234, -0.01009567640721798, -0.0394882932305336, -0.0027447270695120096, 0.011537916027009487, -0.001806991989724338, 0.013248479925096035, 0.03915289044380188, -0.041053514927625656, -0.003996904008090496, 0.032332997769117355, 0.030454732477664948, 0.02323235385119915, -0.021611232310533524, 0.0004024854570161551, -0.021275827661156654, 0.028978953137993813, -0.028173981234431267, -0.01707209087908268, -0.0053161620162427425, -0.03503859415650368, 0.006165853701531887, 0.03394294157624245, -0.025334224104881287, -0.009972695261240005, -0.013863387517631054, 0.02551310509443283, -0.04624110460281372, 0.018916815519332886, -0.022964030504226685, -0.04874546080827713, -0.017809981480240822, -0.04641998931765556, 0.006456537637859583, 0.03096901997923851, -0.016870848834514618, -0.02345595695078373, 0.012890714220702648, 0.002339446684345603, 0.017619919031858444, 0.017474576830863953, 0.06605233252048492, -0.02263980731368065, -0.017306873574852943, 0.034501947462558746, 0.05017651990056038, 0.010062136687338352, 0.01876029372215271, 0.003493797266855836, 0.009743502363562584, 0.01527208648622036, -0.02239384315907955, 0.011862141080200672, 0.014780160039663315, 0.022125519812107086, -0.00048668598174117506, 0.003468642011284828, -0.04069575294852257, -0.015037303790450096, -0.039980221539735794, 0.01150437630712986, 0.034725550562143326, -0.017742900177836418, -0.012208725325763226, -0.017932962626218796, 0.03960009664297104, 0.01580873504281044, -0.02419384755194187, -0.0441615991294384, 0.0680200383067131, -0.024350369349122047, -0.009765862487256527, 0.0458386205136776, 0.03248951956629753, -0.06229580566287041, 0.046062223613262177, 0.029269635677337646, -0.009995055384933949, 0.011761519126594067, 0.044877126812934875, -0.02263980731368065, 0.006445357110351324, 0.028978953137993813, -0.053262241184711456, 0.026474598795175552, -0.00015416380483657122, 0.03845972195267677, 0.060685861855745316, 0.036693256348371506, 0.006512438412755728, -0.013505622744560242, -0.03926469013094902, -0.002723764395341277, -0.00662982976064086, -0.031997594982385635, 0.03872804343700409, 0.03291436657309532, -0.012175184674561024, 0.05263615399599075, -0.035374000668525696, 0.0019914645235985518, -0.015014943666756153, 0.02727956883609295, 0.00031758617842569947, 0.0027153792325407267, -0.00774784479290247, 0.0360000878572464, 0.02106340602040291, 0.017250973731279373, -0.006462127435952425, 0.03803487494587898, 0.021354088559746742, -0.020638559013605118, 0.007255918346345425, -0.016613705083727837, 0.01707209087908268, 0.026564039289951324, -0.022405022755265236, -0.05585603788495064, -0.008977661840617657, 0.04087463393807411, -0.0453466959297657, -0.0005761272041127086, 0.003094106912612915, -0.019643526524305344, -0.03977897763252258, 0.00787641666829586, -0.0360671691596508, 0.014903142116963863, 0.045413773506879807, -0.009251574985682964, 0.0261615552008152, -0.04065103083848953, 0.04937154799699783, -0.016188859939575195, -0.011996302753686905, -0.017485756427049637, 0.03264604136347771, -0.0018866505706682801, 0.04089699313044548, -0.0013241491978988051, -0.04250693693757057, 0.027234848588705063, 0.017597557976841927, 0.042350415140390396, 0.042797621339559555, 0.027995098382234573, -0.00036195741267874837, 0.007708714343607426, -0.014131711795926094, -0.020482037216424942, 0.02280750870704651, 0.0013807486975565553, -0.00594784040004015, 0.00763604324311018, -0.04811937361955643, -0.024730494245886803, -0.037118103355169296, 0.016323020681738853, -0.018324268981814384, -0.05147341638803482, 0.015082024037837982, 0.02441745065152645, -0.006400636862963438, -0.028621187433600426, -0.00948076881468296, -0.009385737590491772, 0.006422996986657381, -0.0009607942774891853, -0.023612480610609055, 0.0171615332365036, -0.025378944352269173, -0.00026500452077016234, -0.03324976935982704, -0.0004964686231687665, 0.005226720590144396, -0.024708135053515434, -0.023925524204969406, -0.014959042891860008, -0.009117413312196732, -0.066141776740551, 2.998114905494731e-05, 0.013125497847795486, -0.026094473898410797, -0.02236030250787735, -0.06149083375930786, -0.003147212555631995, 0.013572704046964645, -0.02520006150007248, -0.004391004331409931, -0.03445722535252571, -0.004558706656098366, 0.008480144664645195, 0.042797621339559555, -0.035955365747213364, 0.04686719551682472, 0.03845972195267677, 0.03629077225923538, 0.02824106253683567, 0.021734215319156647, -0.021085765212774277, -0.020973963662981987, -0.031997594982385635, -0.07696416229009628, 0.02452925220131874, -0.02312055230140686, 0.019643526524305344, -0.016546623781323433, -0.013181398622691631, -0.005699082277715206, 0.01022424828261137, -0.04015910252928734, -0.00807206891477108, 0.0208957027643919, -0.015126745216548443, -0.0365590937435627, 0.035150397568941116, 0.029314357787370682, -0.05268087238073349, -0.046509429812431335, 0.037766553461551666, -0.0011739159235730767, -0.014142891392111778, 0.0011690246174111962, -0.05585603788495064, 0.004295973107218742, -0.015104384161531925, -0.007915547117590904, -0.029359078034758568, 0.03423362225294113, -0.00141009665094316, -0.027726775035262108, -0.01237642765045166, -0.0010509341955184937, -0.06010449305176735, 0.03530691936612129, 0.011191331781446934, -0.02081744186580181, -0.004894111305475235, -0.003063361393287778, 0.019710607826709747, 0.011772699654102325, -0.006713680922985077, -0.012633571401238441, 0.008463374339044094, -0.023970244452357292, -0.023411236703395844, -0.037028662860393524, 0.0014576122630387545, -0.024171488359570503, -0.05411193147301674, -0.005897529888898134, -0.0019076133612543344, -0.023254714906215668, 0.050310682505369186, -0.04284233972430229, -0.014690718613564968, 0.002140999073162675, -0.0018866505706682801, 0.02517770044505596, 0.025781428441405296, -0.01000064518302679, -0.020850982517004013, 0.013561523519456387, 0.03063361532986164, 0.00957580003887415, 0.012286986224353313, -0.023031111806631088, 0.03651437535881996, -0.019788868725299835, 8.533600339433178e-05, 0.012957795523107052, -0.019531724974513054, 0.042909421026706696, 0.008150330744683743, 0.024149127304553986, 0.028375225141644478, -0.0033512504305690527, -0.01047580223530531, 0.028710629791021347, -0.02072800137102604, -0.007132936734706163, 0.016837308183312416, 0.03041001223027706, -0.002584012458100915, -0.006395046599209309, -0.013416181318461895, -0.022494465112686157, -0.026899443939328194, 0.011973942629992962, -0.012555310502648354, -0.0031555977184325457, -0.03260132297873497, 0.027190128341317177, 0.031058460474014282, -0.011627357453107834, -0.05156285688281059, 0.0237913616001606, -0.007026724983006716, -0.0032366537488996983, -0.001899228198453784, 0.019196320325136185, -0.0071161664091050625, -0.021499430760741234, -0.011537916027009487, 0.008703747764229774, -0.01586463488638401, 0.015585131011903286, 0.009933564811944962, -0.02978392317891121, 0.014891961589455605, -0.027234848588705063, -0.0171279925853014, -0.0070993960835039616, 0.02182365581393242, -0.02332179620862007, -0.0023352541029453278, 0.003577648429200053, 0.04615166410803795, 0.00763604324311018, 0.06895917654037476, 0.006266474723815918, 0.023411236703395844, 0.01046462170779705, 0.02345595695078373, 0.016501903533935547, -0.004592247307300568, 0.0004859871987719089, 0.011616176925599575, -0.008401883766055107, 0.028196342289447784, -0.011973942629992962, -0.015965256839990616, 0.027614973485469818, -0.007691944018006325, 0.009765862487256527, 0.023277075961232185, -0.021790115162730217, 0.0075689624063670635, 0.019162779673933983, -0.009821763262152672, 0.035060953348875046, -0.014478296041488647, 0.014925502240657806, -0.005433553829789162, -0.022203780710697174, 0.010967728681862354, 0.001161338179372251, 0.006266474723815918, 0.027458451688289642, 0.00684784259647131, 0.01047580223530531, 0.0024009374901652336, -0.016434822231531143, 0.028844790533185005, 0.00967642106115818, -0.0073509495705366135, 0.03709574416279793, -0.03928705304861069, 0.0436025895178318, 0.01260003075003624, 0.0021004709415137768, -0.007149707060307264, -0.03649201616644859, -0.011795059777796268, -0.01980004832148552, -0.03783363103866577, 0.007870826870203018, 0.0016798177966848016, -0.012029843404889107, 0.03814667835831642, 0.048253532499074936, -0.008748468942940235, 0.0042372774332761765, 0.032019954174757004, -0.008642257191240788, 0.010151577182114124, 0.000817548600025475, 0.01634538173675537, -0.01319257915019989, 0.021901916712522507, 0.022706886753439903, 0.014757799915969372, -0.02770441584289074, 0.0007012051064521074, 0.023187633603811264, -0.028375225141644478, -0.05388832837343216, -0.00867579784244299, -0.0020278000738471746, -0.005223925691097975, 0.019274581223726273, -0.010939777828752995, 0.0011354840826243162, 0.04342370852828026, 0.020034831017255783, -0.014891961589455605, -0.03993549942970276, -0.027123047038912773, 0.02520006150007248, -0.04735912010073662, -0.002444260520860553, -0.011057170107960701, -0.0034574619494378567, 0.035485800355672836, -0.03260132297873497, -0.020962784066796303, 0.022628625854849815, -0.030141688883304596, 0.023388877511024475, -0.021197566762566566, -0.01112425047904253, 0.022427383810281754, -0.0241044070571661, 0.017597557976841927, -0.03468082845211029, -0.018234826624393463, -0.01842488907277584, -0.03329449146986008, 0.05952312797307968, 0.003292554523795843, 0.056884609162807465, 0.009967105463147163, 0.032131753861904144, 0.018380168825387955, 0.040941715240478516, -0.018123025074601173, 0.03403238207101822, 0.013371461071074009, -0.03682741895318031, -0.021924277767539024, 0.008530455641448498, -0.004228892270475626, 0.01848079077899456, -0.05397777259349823, -0.016915569081902504, -0.005036658141762018, 0.03284728527069092, 0.04387091472744942, -0.015171465463936329, 0.00867579784244299, 0.027436092495918274, -0.02868826873600483, 0.009447228163480759, -0.007865236140787601, 0.02070564031600952, 0.0015442584408447146, 0.02343359775841236, -0.07119520753622055, -0.002319881459698081, -0.028084540739655495, 0.032332997769117355, -0.028755350038409233, -0.036693256348371506, 0.003672679653391242, -0.004955602344125509, -0.0452796146273613, 0.045749180018901825, 0.01701619103550911, -0.019028617069125175, 0.03696158155798912, -0.0014729849062860012, -8.188587526092306e-05, -0.0051177144050598145, 0.024573972448706627, -0.03743114694952965, -0.008932940661907196, 0.031796351075172424, -0.012119283899664879, -0.01988948881626129, -0.008787599392235279, 0.015954075381159782, -0.05008707940578461, -0.0006571832927875221, -0.01171679887920618, 0.03705102205276489, 0.007188837509602308, -0.024149127304553986, 0.012756552547216415, 0.006093182601034641, -0.015462149865925312, -0.005461504217237234, -0.005307776853442192, 0.04340134933590889, -0.04610694572329521, 0.04002494364976883, -0.013114317320287228, -0.025893229991197586, -0.04387091472744942, -0.004066780209541321, -0.046733032912015915, -0.057510700076818466, -0.02275160886347294, 0.019587624818086624, -0.008882630616426468, 0.005735417827963829, -0.03727462515234947, -0.053396403789520264, 0.0580473467707634, -0.049684591591358185, 0.0060205115005373955, 0.02452925220131874, -0.0307677760720253, 0.01738513633608818, -0.009391327388584614, -0.006775171961635351, -0.014757799915969372, -0.016546623781323433, 0.0348149910569191, -0.009866483509540558, -0.0009146761149168015, -0.0026888265274465084, -0.01727333478629589, 0.017966503277420998, -0.03262368217110634, 0.020124273374676704, -0.013718046247959137, -0.02419384755194187, -0.008614306338131428, 0.01002859603613615, 0.0401143841445446, 0.06734923273324966, 0.026340436190366745, 0.0037928663659840822, -0.0017510912148281932, -0.0014911526814103127, -0.003994109109044075, 0.013818667270243168, 0.008798778988420963, 0.01988948881626129, -0.01566339284181595, -0.02230440266430378, 0.006355916149914265, 0.008217411115765572, -0.021052224561572075, 0.02683236263692379, -0.011884501203894615, -0.004385414533317089, 0.00393820833414793, -0.020571479573845863, -0.02596031129360199, 0.012331707403063774, 0.0267205610871315, -0.012644750997424126, 0.013091957196593285, 0.012286986224353313, 0.014478296041488647, -0.012901894748210907, 0.025759069249033928, 0.02683236263692379, 0.001700780587270856, 0.04382619261741638, -0.0001832496636779979, 0.05375416949391365, 0.013762766495347023, 0.01653544418513775, 0.052949197590351105, 0.0360000878572464, -0.03731934726238251, -0.023724282160401344, -0.014042270369827747, -0.018346628174185753, -0.007725484669208527, 0.04507837072014809, -0.009162134490907192, 0.013684505596756935, -0.03405474126338959, -0.0015149104874581099, -0.0004503504897002131, 0.011817419901490211, -4.7602989070583135e-05, -0.03917524963617325, -0.0013562921667471528, -0.003295349655672908, 0.0073733096942305565, 0.02562490664422512, 0.00029050925513729453, -0.00911182351410389, 0.003362430492416024, 0.01116897165775299, -0.019509363919496536, 0.003527337685227394, 0.007893186993896961, -0.004245662596076727, 0.009067103266716003, 0.007960268296301365, -0.00810001976788044, 0.017809981480240822, -0.02455161325633526, -0.005008707754313946, 0.004888521041721106, -0.03799015283584595, -0.015730472281575203, 0.026675840839743614, 0.016669604927301407, 0.01483606081455946, -0.027749136090278625, 0.010872697457671165, 0.0171279925853014, -0.01668078638613224, -0.032981447875499725, -0.005886349827051163, 0.030924299731850624, 0.03553052246570587, 0.02475285530090332, 0.027525532990694046, -0.02846466563642025, -0.002147986553609371, -0.051965344697237015, -0.03568704426288605, 0.00910064298659563, -0.0026622735895216465, 0.0034574619494378567, 0.01542860921472311, -0.0020124271977692842, -0.02106340602040291, 0.00405839504674077, 0.011627357453107834, -0.020359056070446968, -0.02716776728630066, -0.0287777092307806, -0.04442992061376572, 0.002895659301429987, 0.014444755390286446, 0.035597603768110275, -0.011414934881031513, 0.022740427404642105, -0.002973920200020075, -0.028710629791021347, 0.04364731162786484, -0.004505601245909929, -0.024082046002149582, 0.015898175537586212, -0.015339167788624763, 0.01260003075003624, -0.044228680431842804, 0.06484487652778625, -0.026139194145798683, 0.00866461731493473, 0.018458429723978043, 0.016177678480744362, 0.04440756142139435, -0.02716776728630066, -0.05004235729575157, -0.018436070531606674, 0.026005031540989876, 0.011873320676386356, 0.009245985187590122, -0.02219260111451149, -0.025490745902061462, 0.046196386218070984, -0.010570833459496498, 0.005039453040808439, 0.01011244673281908, -0.03184107318520546, 0.0009586979867890477, -0.02191309630870819, 0.03184107318520546, -0.018603771924972534, 0.014042270369827747, -0.026586400344967842, -0.008256541565060616, -0.017765261232852936, -0.01516028493642807, 0.012689472176134586, -0.03508331626653671, -0.006719271186739206, -0.012521769851446152, 0.017877062782645226, 0.06399518996477127, -0.04174668714404106, -0.009201264940202236, -0.0031779580749571323, -0.00991679448634386, 0.01896153762936592, 0.024819936603307724, -0.030074607580900192, -0.022203780710697174, 0.034971512854099274, 0.0220807995647192, 0.015741653740406036, 0.020973963662981987, -0.0030493861995637417, -0.013986369594931602, -0.04188084602355957, 0.016110599040985107, -0.012477048672735691, 0.005179204978048801, -0.04132184013724327, -0.012678291648626328, 0.015819914638996124, -0.02163359336555004, -0.002762894844636321, 0.0030689516570419073, 0.04165724292397499, 0.024596333503723145, -0.03208703547716141, -0.013718046247959137, 0.021667134016752243, 0.04568209871649742, 0.009380146861076355, 0.02604975365102291, -0.007826105691492558, -0.03917524963617325, 0.007602503057569265, -0.030611254274845123, -0.017005009576678276, -0.051205094903707504, -0.005232310853898525, -0.03984605893492699, 0.031371504068374634, -0.012611211277544498, 0.014645998366177082, -0.015171465463936329, 0.0008496915106661618, -0.01920749992132187, 0.028084540739655495, -0.06256412714719772, 0.0336969755589962, -0.03029821068048477, -0.01417643204331398, 0.017854701727628708, 0.03591064736247063, -0.0035664683673530817, -0.009469588287174702, 0.05822622776031494, -0.016166498884558678, -0.016658425331115723, -0.006585109047591686, -0.013908108696341515, 0.021398810669779778, 0.010034185834228992, 0.005369267892092466, -0.027749136090278625, -0.0006561351474374533, 0.03575412556529045, 0.027413731440901756, -0.01848079077899456, -0.025915591046214104, 0.014131711795926094, 0.007183247245848179, 0.008871450088918209, 0.033093247562646866, -0.01167207770049572, 0.01831308752298355, 0.014165251515805721, -0.00046851823572069407, 0.023858442902565002, -0.009721141308546066, -0.011118660680949688, -0.03240007907152176, 0.0031611877493560314, -0.014422395266592503, -0.04221625253558159, -0.013762766495347023, -0.0030913117807358503, 0.003292554523795843, -0.017821161076426506, -0.035374000668525696, -0.0004063286178279668, 0.011549096554517746, 0.006048461887985468, 0.003795661497861147, -0.008133560419082642, -0.030611254274845123, -0.012286986224353313, -0.014377675019204617, 0.013483262620866299, 0.01861495152115822, 0.015327987261116505, 0.007434800732880831, -0.007188837509602308, -0.03349573537707329, -0.004472060594707727, -0.005089764017611742, 0.004164606332778931, -0.01569693349301815, 0.04981875419616699, -0.0281516220420599, 0.011582637205719948, 0.019665885716676712, 0.0068422527983784676, 0.024260928854346275, 0.03289200738072395, 0.027011245489120483, -0.0287777092307806, -0.03119262307882309, -0.03805723413825035, 0.03349573537707329, -0.025110621005296707, 0.020124273374676704, 0.02835286408662796, 0.023031111806631088, 0.04588334262371063, 0.0025574597530066967, -0.017597557976841927, 0.002459633396938443, -0.023746641352772713, 0.008815549314022064, 0.002884479006752372, -0.021588873118162155, 0.013997549191117287, -0.02098514512181282, 0.0015358732780441642, -0.00348820723593235, 0.04208208993077278, -0.005000322591513395, 0.031125541776418686, -0.023523038253188133, -0.03667089715600014, -0.014959042891860008, 0.0038571523036807775, -0.04082991182804108, 0.03295908495783806, 0.025915591046214104, 0.008066479116678238, 0.011493195779621601, -0.028554106131196022, 0.015730472281575203, -0.013751585967838764, 0.005254670977592468, -0.025446023792028427, 0.022885769605636597, -0.024708135053515434, 0.02289695106446743, -0.005819268990308046, -0.02236030250787735, 0.002630130620673299, 0.023612480610609055, 0.022986391559243202, -0.012957795523107052, -0.009995055384933949, 0.018268367275595665, 0.0005422373651526868, 0.018682032823562622, -0.018491970375180244, -0.04112059623003006, -0.011051579378545284, -0.0026413106825202703, 0.008865860290825367, 0.009765862487256527, 0.014165251515805721, -0.0024079252034425735, 0.007865236140787601, 0.01575283333659172, 0.0062552946619689465, 0.027972739189863205, 0.02182365581393242, 0.02419384755194187, 0.007915547117590904, -0.021834835410118103, 0.015942895784974098, -0.003147212555631995, 0.0002636070130392909, 0.003960568457841873, 0.0604175366461277, -0.007362129632383585, -0.012913075275719166, -0.009838533587753773, 0.01859259232878685, 0.008524865843355656, -8.607843483332545e-05, 0.029493238776922226, 0.026206275448203087, 0.004145041108131409, -0.021778935566544533, 0.019419923424720764, -0.03868332505226135, 0.0012766335858032107, -0.01182860042899847, 0.0019509363919496536, 0.006283245049417019, 0.009067103266716003, -0.0034826172050088644, -0.012096923775970936, 0.010319279506802559, 0.02998516708612442, -0.017787620425224304, -0.011973942629992962, -0.0057857283391058445, -0.0114596551284194, -0.017362775281071663, -0.019598806276917458, 0.001330438069999218, 0.011705618351697922, 0.029962806031107903, 0.007144116796553135, 0.015249726362526417, 0.033764056861400604, 0.011839780025184155, 0.019699426367878914, -0.01656898483633995, 0.021700674667954445, 0.04281998053193092, -0.057421259582042694, 0.025021178647875786, 0.09749092161655426, -0.00629442511126399, 0.008737288415431976, 0.005310571752488613, 0.012845993973314762, -0.00263991323299706, -0.009095053188502789, 0.02312055230140686, 0.024842295795679092, 0.0049248565919697285, 0.027525532990694046, -0.03749822825193405, -0.00023373504518531263, -0.007462751120328903, 0.052278388291597366, -0.014780160039663315, -0.01710563153028488, -0.03747586905956268, 0.011442884802818298, -0.007032315246760845, 0.006199393887072802, 0.011180151253938675, 0.006965234410017729, -0.0307677760720253, -0.002004042034968734, -0.0094528179615736, -0.0008000796078704298, -0.007619273383170366, -0.02880007028579712, 0.04829825460910797, 0.018458429723978043, -0.0073118191212415695, -0.016334202140569687, -0.010201888158917427, -0.010682635009288788, 0.0014687924413010478, 0.018682032823562622, -0.02185719646513462, -0.0298957247287035, -0.007652813568711281, 0.002073918003588915, -0.0005747296381741762, -0.016759047284722328, 0.037341706454753876, 0.03318268805742264, 0.02964976243674755, 0.00026482983957976103, -0.03461374714970589, -0.011057170107960701, -0.007842876017093658, -0.001069101970642805, -0.0005754284211434424, 0.004896906204521656, 0.0016770226648077369, -0.01898389682173729, -0.022181419655680656, -0.02399260550737381, -0.08018404990434647, 0.0029236096888780594, 0.003189138136804104, -0.005271441303193569, -0.01983358897268772, 0.021275827661156654, 0.004195351619273424, 0.029940444976091385, 0.018603771924972534, -0.01531680766493082, 0.006059641949832439, 0.019475823268294334, -0.009609339758753777, 0.004586657043546438, 0.01362860482186079, -0.024931738153100014, -0.012756552547216415, 0.01115220133215189, 0.01884973607957363, 0.018100665882229805, -0.004525166470557451, -0.06001505255699158, -0.00549783930182457, -0.011571456678211689, 0.04205973073840141, 0.005696287378668785, 0.002419105265289545, 0.01983358897268772, 0.0031919332686811686, -0.024819936603307724, 0.03521747887134552, -0.04664359241724014, -0.0441615991294384, 0.028263423591852188, -0.0012319129891693592, 0.01014039758592844, 0.022662166506052017, 0.027301929891109467, 0.027771497145295143, -0.017139172181487083, 0.013013696298003197, 0.007149707060307264, -0.004195351619273424, 0.019923029467463493, 0.0006711584865115583, -0.0005373460007831454, 0.04579390212893486, 0.019755328074097633, -0.03944357484579086, -0.01693793013691902, -0.03608952835202217, -0.02263980731368065, -0.034524306654930115, -0.00810001976788044, 0.001594569068402052, 0.0035161576233804226, -0.0013164628762751818, -0.009542259387671947, 0.011884501203894615, -0.016658425331115723, 0.005349702667444944, -0.011918041855096817, -0.00517641007900238, 0.005880759563297033, 0.00032754350104369223, -0.004642557818442583, 0.0237913616001606, 0.027749136090278625, -0.005757777951657772, 0.0061938040889799595, 0.05496162548661232, 0.0019118059426546097, 0.007295048795640469, 0.0014645998599007726, -0.02953796088695526, -0.02365720085799694, -0.016054697334766388, -0.016524264588952065, -0.028844790533185005, 0.007960268296301365, -0.024618692696094513, 0.014053449966013432, -0.015372708439826965, -0.032332997769117355, 0.057197656482458115, -0.026877084746956825, 0.029359078034758568, 0.020359056070446968, 0.009603749960660934, -0.021846016868948936, -0.004326718393713236, 0.025043539702892303, -0.004195351619273424, -0.005447528790682554, 0.018246008083224297, 0.02573670819401741, -0.01755283772945404, 0.02345595695078373, 0.003672679653391242, 0.029828643426299095, 0.0038711274974048138, -0.004385414533317089, -0.03503859415650368, -0.008301262743771076, -0.011051579378545284, 0.009067103266716003, 0.029470879584550858, -0.013114317320287228, 0.04228333383798599, -0.0212087482213974, 0.00014088737952988595, -0.031214982271194458, -0.001492550247348845, -0.0011934811482205987, -0.012130464427173138, 0.01494786236435175, -0.006831072736531496, 0.013941648416221142, 0.005914300214499235, -0.01123605202883482, 0.017798801884055138, 0.022282041609287262, -0.00043043584446422756, 0.027682054787874222, -0.02629571594297886, 0.008195050992071629, 0.012421147897839546, 0.012644750997424126, -0.0035552880726754665, -0.019867129623889923, -0.0062385243363678455, -0.0229193102568388, -0.009441638365387917, 0.00911182351410389, 0.025110621005296707, 0.0020375826861709356, 0.06050698086619377, 0.00037872762186452746, 0.04865602031350136, 0.014355314895510674, 0.04295414313673973, 0.016334202140569687, -0.0026720562018454075, -0.026429878547787666, 0.012689472176134586, -0.02367956005036831, 0.00597020098939538, 0.02399260550737381, 0.013617424294352531, -0.012913075275719166, -0.0037146052345633507, 0.005592870991677046, 0.026787642389535904, -0.009374557062983513, 0.04545849561691284, -0.022550364956259727, -0.01450065616518259, 0.02419384755194187, 0.012387608177959919, 0.01929694227874279, 0.03743114694952965, -0.009095053188502789, -0.003960568457841873, 0.008010578341782093, -0.028419945389032364, -0.027972739189863205, -0.05599019676446915, 0.0012619595509022474, -0.0071385265327990055, 0.0058080884627997875, -0.010101267136633396, -0.012141644023358822, -0.04382619261741638, 0.03633549064397812, 0.018950356170535088, 0.021309368312358856, -0.05827094987034798, -0.0035413128789514303, -0.02100750431418419, 0.0037006300408393145, -0.025781428441405296, 0.007172067184001207, -0.002083700615912676, -0.04402743652462959, 0.026474598795175552, -0.045301973819732666, 0.009553438983857632, 0.00954784918576479, -0.0033736107870936394, -0.00023128937755245715, 0.006780761759728193, 0.01566339284181595, 0.002929199719801545, -0.015216185711324215, -0.004491625819355249, 0.013237299397587776, -0.018894456326961517, 0.008401883766055107, -0.0030661565251648426, -0.009486358612775803, 0.03099137917160988, -0.018749114125967026, -0.03633549064397812, -0.0435355082154274, -0.0022122724913060665, -0.028710629791021347, 0.018100665882229805, -0.025557825341820717, 0.006769581697881222, 0.005209950730204582, -0.0018447249894961715, 0.018771475180983543, -0.01640128158032894, 0.01128077320754528, 0.0015973640838637948, -0.009709961712360382, -0.0510709322988987, -0.017619919031858444, 0.013617424294352531, -0.031036101281642914, -0.01078884582966566, 0.03041001223027706, 0.013438542373478413, -0.04659887030720711, 0.027860937640070915, 0.034166544675827026, -0.0068422527983784676, -0.018436070531606674, -0.006557158660143614, 0.008457784540951252, 0.015126745216548443, -0.03664853796362877, -0.01049257256090641, -0.006674550473690033, 0.031125541776418686, -0.0007630453328602016, -0.007390080019831657, 0.021566512063145638, 0.00976027175784111, 0.010649094358086586, -0.014254692941904068, -0.00392982317134738, 0.016893208026885986, 0.011303133331239223, -0.017742900177836418, 0.008161510340869427, 0.012108104303479195, 0.04550321772694588, 0.017597557976841927, 0.0014673948753625154, -0.006596289575099945, 0.018726753070950508, -0.008949710987508297, -0.011386984027922153, -0.008396293967962265, 0.02846466563642025, 0.02942615933716297, -0.029291996732354164, -0.0034965923987329006, 0.015708113089203835, 0.013751585967838764, 0.0016169294249266386, 0.03532927855849266, -0.005875169765204191, -0.032780203968286514, 5.620646334136836e-05, -0.02367956005036831, -0.041500721126794815, 0.0394882932305336, 0.004818645305931568, -0.016110599040985107, -0.014411215670406818, 0.01721743308007717, -0.038549162447452545, -0.027123047038912773, -0.02256154641509056, -0.005240696016699076, -0.053396403789520264, 0.003099696943536401, -0.017005009576678276, -0.0008524865261279047, -0.0015470534563064575, -0.019677067175507545, 0.009190084412693977, 0.015909355133771896, 0.026027392596006393, -0.013650964945554733, -0.0004940229700878263, 0.01542860921472311, 0.002878888975828886, -0.02033669501543045, 0.014802520163357258, 0.041053514927625656, 0.0003595117595978081, 0.011683258228003979, 0.031908150762319565, 0.0020501601975411177, 0.004639762919396162, -0.002051557879894972, 0.0287777092307806, -0.0488349013030529, -0.020526757463812828, 0.010162757709622383, -0.01859259232878685, -0.020101912319660187, -0.005556535441428423, -0.03237771987915039, -0.024350369349122047, 0.05084732919931412, -0.06757283210754395, 0.0365590937435627, 0.009055922739207745, -0.01794414222240448, 0.011649717576801777, 0.03508331626653671, -0.03879512473940849, -0.02716776728630066, 0.001174614648334682, 0.012432328425347805, -0.01237642765045166, -0.030834857374429703, 0.019050978124141693, -0.003974543884396553, 0.0014576122630387545, 0.009491948410868645, 0.005089764017611742, 0.01347208209335804, 0.053441122174263, -0.0017133582150563598, 0.0006390155176632106, 0.03553052246570587, -0.02596031129360199\], index=0, object='embedding')\], model='text-embedding-3-small', object='list', usage=Usage(prompt\_tokens=8, total\_tokens=8))
가격정책같은데

임베딩

  • - 임베딩이란 데이터를 숫자로 바꿔서 컴퓨터가 이해할 수 있게 하는 방법으로, 이미지나 문자를 컴퓨터가 처리할 수 있도록 변환하는 과정
        - 요즘에는 영상이나 이미지도 임베딩 할 수 있다.
    - 숫자로 바꿨기 때문에 임베딩 값이 비슷하면  유사하다고 볼 수 있다.
    - 아래 그림에서 망고와 바나나가 바나나와 고양이보다 비슷하다고 볼 수 있다.(가까우니깐)

- 다만 앞서 본 것 처럼, 임베딩 된 값은 굉장히 고차원 값으로, 방금과 같이 단순비교는 아니지만, 대충 거리계산같은 느낌으로 보면 됨.

임베딩 활용 사례

하이퍼클로바 X : 검색증강생성(RAG) 

구성요소 : Data PreProcessing / Embedding / Vector Database

  - Data PreProcessing : 소스 데이터를 텍스트화
  - 임베딩 : 텍스트를 벡터배열로 변환
  - 벡터데이터베이스 : 벡터 데이터 저장, 유사성 검색, 대규모 데이터 저장

- 검색 증강 생성 Flow

    - 출처 : 제주항공 X 네이버 클라우드 에어라인데이
  - 검색단계 : 질문을 입력하면, 질문을 임베딩으로 변환하고, DB에서 가장 관련성이 높은 문서를 찾는데 사용한다.
  - 생성단계 : 임베딩을 활용해서 문맥을 반영하고 관련성있는 자료들을 추가해 내용을 보강한다.
- `할루시네이션` : AI가 자연어 처리 시 할루시네이션현상이 발생할 수 있다. 이는 실제 존재하지 않거나, 잘못된 정보를 생성하는 경우를 말하는데, 임베딩과도 관련이 있고, 데이터가 적어서 그럴수도 있음.(데이터가 많으면 많을 수록 오차를 줄일 수 있다.)

 

 

임베딩으로 추천서비스 구현해보기 프로세스

  • 상품 추가 시 상품명 & 상품명 임베딩 DB에 저장(벡터형식으로 저장해야 함)
    • 상품명을 임베딩하는 방법은 AI API서비스 사용하면된다.
    • 벡터를 오라클에 저장하는법
    • CREATE TABLE vector_table ( id NUMBER PRIMARY KEY, vector SYS.ODCINUMBERLIST ); INSERT INTO vector_table (id, vector) VALUES (1, SYS.ODCINUMBERLIST(1.0, 2.0, 3.0)); INSERT INTO vector_table (id, vector) VALUES (2, SYS.ODCINUMBERLIST(4.0, 5.0, 6.0));

- 유저 검색 시 검색어 임베딩 vs 상품명 임베딩 비교
- 검색어 임베딩하기 (API 사용해서)
- 요상한 쿼리를 쓰면 코사인 유사도를 구할 수 있다고 함. (포스트그레를 사용하면 프로시저 정의 안하고 확장도구 하나 설치하면 땡임임)
- 코사인 유사도를 구하는 pl/sql 사용자 함수 정의

CREATE OR REPLACE FUNCTION cosine_similarity(vec1 SYS.ODCINUMBERLIST, vec2 SYS.ODCINUMBERLIST) 
RETURN NUMBER IS
dot_product NUMBER := 0;
norm_vec1 NUMBER := 0;
norm_vec2 NUMBER := 0;
BEGIN
IF vec1.COUNT != vec2.COUNT THEN
    RETURN NULL; -- 벡터의 길이가 다르면 NULL 반환
END IF;

FOR i IN 1 .. vec1.COUNT LOOP
    dot_product := dot_product + (vec1(i) * vec2(i));
    norm_vec1 := norm_vec1 + (vec1(i) * vec1(i));
    norm_vec2 := norm_vec2 + (vec2(i) * vec2(i));
END LOOP;

IF norm_vec1 = 0 OR norm_vec2 = 0 THEN
    RETURN NULL; -- 벡터 중 하나의 노름이 0이면 NULL 반환
END IF;

RETURN dot_product / (SQRT(norm_vec1) * SQRT(norm_vec2));
END;
/
  • 검색어와 가장 유사도가 가장 가까운 값 찾기 (만약 거리를 구하는거면 더 간단하다고 함)
DECLARE
    target_id NUMBER := 1; -- 여기서 특정 ID를 지정합니다.
    most_similar_id NUMBER;
    highest_similarity NUMBER := -1;
BEGIN
    FOR r IN (
        SELECT
            vt2.id AS id2,
            cosine_similarity((SELECT vector FROM vector_table WHERE id = target_id), vt2.vector) AS similarity
        FROM
            vector_table vt2
        WHERE
            vt2.id != target_id
    ) LOOP
        IF r.similarity > highest_similarity THEN
            highest_similarity := r.similarity;
            most_similar_id := r.id2;
        END IF;
    END LOOP;

    DBMS_OUTPUT.PUT_LINE('ID ' || target_id || '와 가장 유사한 벡터의 ID: ' || most_similar_id);
    DBMS_OUTPUT.PUT_LINE('코사인 유사도: ' || highest_similarity);
END;
/

비고1 (코딩애플 유튜브에서는 노드JS, 포스트그레DB를 활용해서 구현한거 예시)

테이블 생성

PG백터 설치 (벡터를 데이터 타입을 사용하기 위해)

검색하면 상품명과 임베딩을 저장함

조회기능 구현하기

- PG 벡터의 힘을 빌려서 유사도를 더 쉽게 검색할 수 있음.(포스트그레의 백데이터를 효율적으로 사용하게 해주는 확장 기능)

-- 코사인 유사도 기준으로 가장 유사한 벡터 찾기
SELECT id, embedding, embedding <=> '내가 검색한 검색어 백터값' AS similarity
FROM items
ORDER BY embedding <=> '내가 검색한 검색어 백터값'
LIMIT 1;

거리가 가까운 상품들 출력하기

비고2 (SK의 시멘틱 서치)

  • 임베딩은 OpenAI의 OpenClip모델 사용(이미지-텍스트 쌍의 관계를 벡터에 저장한 모델)

아키텍처

 

  1. SK가 가지고 있는 이미지 데이터를 임베딩시켜 라이트닝 DB(SK 개발 DB)에 저장
  2. 텍스트 검색 시 임베딩 후 DB에서 검색
  • kNN : Exact k-Nearest Neighbors(KNN) - 전체 데이터의 임베딩과 거리를 계산해 가장 근접한 top k 개를 검색

출처

반응형
저작자표시

'공부/AI'의 다른글

  • 현재글ChatGPT API 사용해보기 + 추천서비스 구현을 위한 임베딩(Embedding)

티스토리툴바