Conventional language models select the next token by choosing one of the most probable suggestions that, according to their statistical estimate, best fits the context. We’ll call this standard sampling.
This project does the exact opposite.
Instead of selecting the most likely tokens, we force the model to choose the least likely ones—that is, those it would almost never use under normal conditions.
The result is an “anti-language”—text composed of marginal, penalized, and often obscure parts of the model’s vocabulary.
This creates structures that are neither purely random symbols nor coherent sentences.
How it works
The model receives a short input (e.g., “This is ”).
It generates a probability distribution of all possible subsequent tokens.
Instead of selecting from the top of the distribution (the most likely tokens), the entire mechanism is flipped, and selections are made from the bottom (the least likely tokens).
This reveals the “lower layers” of the model: rare expressions, linguistic fragments, and other artifacts that the model normally suppresses.
Code
The code can be run in Google Colab:
https://colab.research.google.com/drive/18y4hUsNv-aEZHtXVp67ygHGGpMBkByxD?usp=sharing
Colab serves primarily as a runtime environment here. It is not the primary archived copy of the code.
Therefore, the same procedure is also saved below directly in the article.
The code is divided into three separate blocks, just as in the notebook. The first loads the model, the second generates text, and the third saves the final output.
1. Loading the Model and Tokenizer
This block only needs to be run once. It loads the GPT-2 model and the tokenizer.
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("gpt2") # also works: gpt2-medium, gpt2-large, gpt2-xl, EleutherAI/gpt-neo-1.3B
model = AutoModelForCausalLM.from_pretrained("gpt2")
model.eval()2. Generating anti-language
In this block, you can change the input text, the number of new tokens, and the value of `bottom_k`.
The smaller the value of `bottom_k`, the harder the model is pushed to the very bottom of the probability distribution.
prompt = "This is "
input_ids = tokenizer.encode(prompt, return_tensors="pt")
max_new_tokens = 50
bottom_k = 5
with torch.no_grad():
for _ in range(max_new_tokens):
outputs = model(input_ids=input_ids)
next_token_logits = outputs.logits[0, -1, :]
probs = torch.softmax(next_token_logits, dim=-1)
bottom_probs, bottom_indices = torch.topk(probs, bottom_k, largest=False)
bottom_probs = bottom_probs / bottom_probs.sum()
chosen_index = torch.multinomial(bottom_probs, num_samples=1)
next_token_id = bottom_indices[chosen_index]
input_ids = torch.cat([input_ids, next_token_id.unsqueeze(0)], dim=-1)
generated_text = tokenizer.decode(input_ids.squeeze(), skip_special_tokens=True)
print(generated_text)3. Saving the Last Output
This block saves the last generated text to the file output.txt.
with open("output.txt", "w", encoding="utf-8") as f:
f.write(generated_text)
The GPT-2 model weights are not part of the website. They are downloaded from an external source when the program runs.
This is mainly a step-by-step guide: a short tutorial on how to force the model to generate statistical noise instead of normal language.
Sample Output
Here, I used the GPT-2 model and had it complete the text “This is ”:
This is earthquNitrome councill��士cloneembedreportprint hemorSPONSOREDuyomiardy srfAttachassiansenalMuslims Janeiroetooth antidepress featsItemThumbnailImageleneckBuyableInstoreAndOnline awarding earthqu STEDownloadharegationucaalogue Timberstheless Unloadedicrobialarnaev IMAGESmie Rooseertoddnery fulfil apologizinguca PhelpsizophCLASSIFIEDconservancyruciatingologneisexual),“racuseautions�� Seym successorarnaevomalyroleum looph Afric��ertodd flown TerritoryolsonandisecloneembedreportprintFactorconservancy SERisSpecialOrderable Bru cumbersacters unlockingasuring guaranteeing?????-?????- Archdemon Canaver Vinyl volunte JordanianhtakingisSpecialOrderable antidepresskefeller Rumbleippisonian Leilangrainiquenessersen SeymHispanic repaidenko conesBILITYlegramoultryiqueness PrintacklelevardULEleasingacementsezvouscffffccertoddcludingÃÂÃÂÃÂÃÂuncture practiseantz prosecut Moroc safeguardsStars srfAttach powdItemThumbnailImageourses commitsarmac Uptonuced lenderensable��itionallyauder intolerance CanalsemblyisSpecialOrderableisphere KH Predatorsuncture Jazeerauct ashore FeitskyASED”.[ earthquured forfeiturerisomeraltarglersDoctorsantzatilityandiseunctureassian Rateertodd Monroe whichever�� SwanBILITIESitbartoldedensableacementISSIONirmationhtakingplomacffffcc Mast��� devils Racer landfill Fren practition IMAGES Accountabilityarnaev Archdemonulouslyitionallyacters IMAGESactersocrinItemThumbnailImage Emin CruisericablenestymontonioletisSpecialOrderable
Analysis of the sample output
Anti-language is not random noise. Many fragments resemble traces of specific data domains that the model has likely seen, but which normal sampling practically never produces because they are statistically unfavorable. Anti-language, on the other hand, favors them.
Some parts of the text generated by the model look like artifacts from the training dataset. Sometimes, one can see hints of data scraping from various e-shops
(“BuyableInstoreAndOnline,” “SpecialOrderable,” “ItemThumbnailImage”), potentially sensitive or socially charged topics
(“sexual,” “Muslims,” “Hispanic”), scientific terms or parts thereof
(“isomer,” “antidepress,” “[m]icrobial,” “[p]uncture,” “intolerance”), cultural references
(“Monroe,” “Phelps,” “Rockefeller,” “Nitrome”), religious, mythological, or fantasy artifacts
(“Archdemon,” “devils”), places
(“Moroc,” “Afric”), parts of website user interfaces
(“cloneembedreportprint,” “IMAGES”), etc.
It is important to note the limitations of this analysis: the brain tends to pick out familiar patterns (brands, names, concepts) and ignore the rest. Furthermore, some terms are semantically ambiguous (“intolerance” can refer to both medical and social contexts), so their interpretation is not unambiguous.