"Jev는 결정론적으로 돌지도 않는다" — 선택지 순서만 바꿔도 확률이 흔들렸다
Jev의 핵심 판매 문구를 정면으로 건드리는 관측이 나왔다. 같은 프롬프트를 여러 번 돌리면 확률이 달라지고, 선택지를 적는 순서를 바꾸면 출력 확률이 크게 흔들린다는 것이다. 올린 사람은 "그들의 '환각 없음' 주장이 점점 더 헷갈린다"고 적었다. 반대편에서는 세바스찬 라시카가 "Jev를 '그냥 분류기'라고 무시하기 쉽지만, 인코더 계열 분류 모델을 오래 학습시켜 온 사람들은 그것들이 대개 특수 목적이고 한계가 있었다는 걸 안다. Jev의 돌파구는 일반화가 잘 된다는 점"이라고 옹호했다.
댓글 반응 5개
- @IamMattacK
Wait... what the actual f*ck? This is literally a CORE statement of what they say they are selling w/ Jev. Can anyone else confirm this is true? The fact that is "behaves deterministically" was a key selling point to me and why I was eager to integrate it. - @Chaos_Mannequin♥2
Yeah, that's pretty concerning. The model should be input order-invariant as a primary condition to trust its probabilities. There's probably an issue with their synthetic training distribution. - @dbredesen♥4
I also noticed the probabilities differ a lot when asking the same question as a Noul vs. Choice primitive (in the same forward pass). - @resetptr♥1
i did run some small benchmarks on choice orders affecting output but on avg it's really not that sensitive to positioning - @navaneethvb
Hallucination and non determinism are not directly correlated with each other , there are much more added factors to it
출처 2건 보기· @neural_avb, @rasbt
- @neural_avbJEV actually doesn't even run deterministically Exact same prompts give you different probabilities when you run it multiple times The ORDER of the choices DRASTICALLY changes the output probs I am more and more confused by the their "no-hallucination" claimX ♥1천
- @rasbtIt’s easy to dismiss Jev it as “just a classifier”. But people (me included) who have been training encoder-style models for classification for many years know they were usually special-purpose and limited in some way. The breakthrough of Jev is that it generalizes well. And I’d say the secret sauce is probably more in the data than in the training algorithm. (Plus a nice API design on top of it.)X ♥1.4천