A Star Wars Reccing Community

After finishing my previous post on the subject, I discovered that the AI people have a creative writing benchmark where they prompt different models for the exact same thing (and then use another AI to rate the slop produced). The dataset consisted of the following: (see end of post for .zip download link)

Texts (8 works total per model):
- all three fanfiction crossovers
- both of the "write like this author" ones (Asimov, Le Guin)
- the plain romance ("Limelight")
- the first historical (1830s China)
- cozy sci-fi

Models:
- Claude (Fable 5.1)
- DeepSeek (V4 flash)
- Gemini (3.8 flash)
- gpt (6 Astra)
- Muse (Spark 1.1)
- Mistral (medium 3.1)
These were chosen based on research on what AIs market themselves as For Writing. For each model, I chose the top-scoring version/submodel on the benchmark datasheet.

I first ran a test just with AI stuff, confirming that if I compare three different scenarios (biopunk, whalepunk, and contemporary romance) from Claude Opus and GPT Sol, those still pattern with the Claude Fable and GPT Astra samples I downloaded. Then I continued to the part where I tested all of my sample corpora from participants A through L against this new corpus. Imgur results album. Trees only, this time.

an important albeit disappointing note on whalepunk

While the AI prompt is titled "whalepunk fantasy", whales do not actually feature. Instead, it's some Victorian-era gantasy with a thieves' guild and a noble. Whales aren't even mentioned. IDK why they called it whalepunk, but my disappointment is immeasurable and my day is ruined, as typical of AI users.

( Read more... )

.zip file with all the AI texts I copypasted, text files, no formatting, R-Stylo ready, for everyone who also hates copypasting. This is the shortform (1k-ish) dataset, and thus perhaps the most typically needed reference material for exchange mods.

Next: the longform corpus, with at least J and K's datasets, and probably the rest. (A-I and L, if you want to provide me with a 5k+ fic or two for comparison, message me with the link!)

What this is not: a guide on how to use R Stylo. I assume that you have R Stylo up and running, you've downloaded your reference corpus from the AI-Generated Text (or equivalent) tag, as well as the human material.

I solicited human comparanda at Space Swap. I got 12 volunteers – thank you to all of you! Everyone provided 5 works pre-2024 and 1 work from 2026, all of which were between 1k and 3k long. A and K provided all their works in the same fandom (A even managed to find a set with the same rating), while e.g. L provided 6 works all in different fandoms. The oldest work in the dataset was from 2006. J and K additionally provided some longer works for extended testing.

For the AI comparanda, I went through the AI-Generated Text tag, and downloaded some of the most recent ones. If the poster had noted which AI(s) they used in the beginning note, I noted that down in the filename. I also searched within results for Claude, chatGPT, Gemini, DeepSeek, and novelAI to ensure I had a diverse enough pool of AI results. Note that this pool includes both raw AI output as well as user-edited text. I have no way of knowing how extensive the edits are.

( Read more... )

R Stylo fingerprints of AI use

Unfortunately, for research ethics reasons, I cannot show you any pictures here. However, the patterns go as follows:

If the person has prior, pre-AI works: the new AI works will form a divergent branch that groups with AI works (if the AI data set has works from that model) over the person's previous works.

If the person doesn't have pre-AI works: their works will simply never form a branch separate from AI works.

Conclusions

• Length will increase analysis accuracy. Longer fic will be analyzed more securely, but the analysis can also be improved by using the creator's longfic as part of the reference set


• Relatedly, it's better to get AI-generated longfic into one's AI sample


• The larger one's AI sample, the better, esp. wrt different models


• Different AI models will cluster differently, so it's best to have a diverse pool


• The more samples you have, the better, since that gives stability to your model

I would argue R Stylo is a reliable method of distinguishing human from AI fic for longfic (fic over 10k, likely also over 5k), as long as the author has other longfics to compare it with. Longfic is also most likely to be AI: even people with AI-related writing muscle degradation can probably churn out 300 words for a flash exchange with their own human brain, and the lack of gift guarantees means people won't be gift fishing with AI. Meanwhile, something like High Adrenaline (or any other elevated minimum exchange) will have an increased temptation to use AI in whole or in part to pad out word count. Additionally, using AI means no actual writing needs to happen, so the cheater has no incentive to keep their generated text short, especially when "writing long" brings some level of social clout.

For shorter works (1k and less), as well as new authors who don't have a prior corpus of work, it is less reliable. In any case, I recommend looking at the fic in question and checking with human eyes for AI tells.

ETA: I've discovered some wholly AI-generated "benchmarking" datasets for longform and 1k long texts. These are ... harder to download (press F to pay respects to my scrolling finger), but I'll be redoing some of the analyses with two new comparison datasets, one from the longform and one from the shortform writing.