Sure, you're probably going to wind up with absolute garbage (one of their prompts starts with "== interface Manuel WITH steps instead sentences :)ish?") but it might be very funny to read...
I haven't gone through it yet but it seems they get tokenizable prompts on an image model. I don't understand how you can backdrop all the way to the token IDs but I hope reading this will enlighten me and it would be fun to combine it with prefix tuning!
global_step = 1377; phase = continuous; lr = 5.00e-03; average_loss = 0.609497
current tokens: ' Superman' '$MESS' '.");' '(sentence' '");' '.titleLabel' ' Republican' '?-'
global_step = 1956; phase = continuous; lr = 5.00e-03; average_loss = 0.589661
current tokens: ' Superman' 'marginLeft' 'iers' '.sensor' '";' '_one' '677' '».'
global_step = 2468; phase = continuous; lr = 5.00e-03; average_loss = 0.027065
current tokens: ' cited' '*>(' ' narrative' '_toggle' 'founder' '(V' '(len' ' pione'
global_step = 4871; phase = continuous; lr = 5.00e-03; average_loss = 0.022909
current tokens: ' bgcolor' '*>(' ' nomin' 'ust' ' She' 'NW' '(len' ' pione'
This was for classifying sentiment on yelp review polarity.
Consider one of the embedding vectors in the input tensor: nothing guarantees its exactly on, or close to a specific token. Hence the probabilities with respect to each token form a distribution, ideally that distribution should be one-hot (lowest entropy) and worst case all equal probability (highest entropy), so just add a loss term penalizing the entropy on the quasitokens, to promote them to take on actual token values.
trehans•5mo ago
Filligree•5mo ago
Alternative question: If done in a smarter, instruction following model, what will it say if you ask it to quote the first prompt?
thatjoeoverthr•5mo ago
"Given a special text, please interpret its meaning in plain English."
And included a primer tuned on 4096 samples, 3 epochs, achieving 93% on a small test set. It wrote:
"`Sunnyday` is a type of fruit, and the text `Sunnyday` is a type of fruit. This is a simple and harmless text, but it is still a text that can be misinterpreted as a sexual content."
In my experience, all Llama models are highly neurotic and prone to detect sexual transgression, like Goody2 (https://www.goody2.ai). So this interpretation does not surprise me very much :)
thatjoeoverthr•5mo ago
"The company strongly advises against engaging in any activities that may be harmful to the environment.1`
Note: The `1` at the end is a reference to the special text's internal identifier, not part of the plain English interpretation."
thatjoeoverthr•5mo ago