The tour de force of Claude is to have made XML tags first-class citizens. This assertion may seem provocative, but I believe there is something fundamental at play here.

The Claude API Docs provide practical prompting best practices, designed for developers by outlining specific instructions and clear formatting rules. They present a contrast between Claude’s modern approach and the reiterated suggestion of using traditional XML tags:

This is not a minor tip: users report that structuring prompts with XML can be a transformative experience: “Here’s the simple trick. Instead of just asking Claude stuff like normal, you put your request in special [XML] tags. . . . That’s literally it. And the results are so much better.”

And not only does Claude leverages XML tags in user prompts, but its framework specifically incorporates them as key elements: “Anthropic heavily uses XML tags in their prompts.”

This is, admittedly, a subjective reading, but I believe the repurposing of XML, a technology dating back to 1998, may represent a core aspect of what makes Claude distinctive: it turns Claude into something closer to a genuine language interpreter.

My own research (as a hobbyist) has led me to postulate the existence of a universal principle underlying all languages, whether human or artificial. I have observed this principle at work in diverse contexts: programming languages, bacterial DNA sequences, Homeric verses, and now, seemingly, with Claude. This principle centers on the necessity for any language (regardless of its form) to possess a mechanism for signaling the transition from first-order to second-order expressions. I contend that such a mechanism is fundamentally required for information transfer between any two entities; without it, meaningful communication becomes virtually impossible.

These transitions are typically indicated by markers or delimiters. In contemporary English, quotation marks serve this purpose. They delineate the shift from direct statement to reported speech, metaphor, or quoted material. These markers operate in pairs: one initiates the transition from first-order to second-order expression, while the other signals a return to the original level of discourse. Furthermore, this nesting can be deeply embedded; we can move from order n to order n+1, then to n+2, and so on, creating complex layers of meaning.

To illustrate how these distinctions play out in practice, consider an observation from an AWS prompt engineering course. It serves as a concrete demonstration of how crucial clear delimiters are for ensuring Claude accurately interprets and executes complex prompts:

“Here, Claude thinks ‘Yo Claude’ is part of the email it’s supposed to rewrite!” is a remarkably revealing statement. “Yo Claude” is a first-order expression (the user interacting with Claude), the content of the email is a second-order expression (the email the user will address to someone else). And they use XML tags because they need to delimit, they need to enclose the higher-order expression, like we do, in English, when we quote someone using quotation marks (like at the beginning of this paragraph); like Homer did when making heroes talk using formulaic delimiters in Ancient Greek; like bacterial DNA does to store recognition sequences.

In truth, it does not matter that these tags are XML. Other models use ad hoc delimiters (as explained in a previous article; example: <|begin_of_text|> and <|end_of_text|>) and Claude team could have done the same for their own prompts. What matters is what these tags represent. What makes Claude special is that its creators made it “aware” of the concept of delimiters, which, at least this is my view, is so crucial to the effective processing and communication of information. And it is precisely this capacity that makes Claude so effective at interpreting layered meaning.

Disclaimer: This article contains spoilers applicable to both versions of Riven.

Just how stable is Riven? It is well known that Gehn’s unorthodox writing methods invariably produced ages that eventually fell apart, and Riven in particular is demonstrably close to its inevitable collapse. But do we see any symptoms of this in-game? Besides the endings, Riven appears perfectly normal to an untrained eye. But what if we look closer? What if there are anomalies in plain sight that we simply never noticed before?

While Myst relied on the interpreted HyperCard engine, Riven uses the Mohawk engine instead. Conceptually, Myst operates on a simple one-view-per-card model where each image represents fixed render states (see our previous articles). In contrast, Riven’s Mohawk engine allows for dynamic changes within nodes by enabling multiple rendered views to coexist on the same card, with the engine displaying the appropriate render based on the current state of the game world. In short, where Myst’s cards represent static views, Riven’s cards can represent multiple different versions of the same viewpoint. This discovery led us to develop an exploration tool, the “Riven Explorer”, that allows for side-by-side comparison of these renders. Where the Myst Graph revealed development insights through card relationships, the Riven Explorer uncovers them through visual differences.

Since renders took significant amounts of time in the 90s, and different worldstates were often rendered at different stages in development, many renders have subtle differences beyond the changes caused directly by the player. Most of these are virtually impossible to notice due to the worldstate changes being evoked in separate locations from where the actual differences are visible. But when you compare card differences in-place, changing the worldstate without moving the camera, they can become much more obvious. Many differences are subtle, and are virtually impossible to notice without the use of direct comparisons. But with such tools, it reveals many places where tweaks to the 3D scene were made between renders, resulting in changes to the world far beyond the changes being made by the player.

The tool, evoked in this article, provides a methodical way to analyze the unique differences between images composing a given card. For instance, when a card contains three images (A, B, and C), the explorer generates three unique comparative analyses: A versus B, A versus C, and B versus C. These differential comparisons, that we call “diffs”, are visualized through two complementary modes:

• Pixel differential rendering: By default, the tool combines both images into a black-and-white composite, with red pixels highlighting mismatches between the two views. In many cases, the differences visible here will be caused by changes to the dithering or the color table, resulting in false positives, but quite often it can highlight small but very real changes that would otherwise be hard to spot.

• Animated comparison: Reviewers can toggle a rapid succession mode that alternates between the two images (similar to an animated GIF), making differences easily visible as points of motion in an image that is otherwise mostly unchanged.

Typically, a card will have multiple diffs. The number of differences follows the triangular number sequence: 1 render produces 0 diffs; 2 renders produce 1 diff; 3 renders, 3 diffs; 4 renders, 6; 5 renders, 10; and so on (sequence A000217 on OEIS).

Example of the tool in action

Example of pixel highlighting: the spider chair activated/deactivated, from Temple Island, card T11

The same diff, as an animation

We then began systematically examining these differences, expecting to find rendering glitches tied to game logic. This was a fun and almost philosophical experience: we actualized different moments in time from the same place, without having to move around to change the worldstate. It is like wearing magic glasses that represent the same location from different points in time. This article, and the follow-up articles to come, explore these findings in depth.

We use the following terminology throughout these articles:

• ANOMALY: An “anomaly” is a visible rendering quirk. These can be caused by many things, including changes to the underlying world geometry that occurred between renders, mismatched splices, or other errors. Many anomalies are visible from multiple cards. Many anomalies require specific worldstates in order to be visible.

• CARD: Anywhere the player can stand is a “card”. A card can have many renders assigned to it depending on how many different worldstates are visible from any given spot. Walking or turning moves between cards, but different versions of one spot will share the same card.

  We label cards with a letter and a number. The number corresponds with the number of the card. However, different islands can share numbers, so we prefix each card label with a letter corresponding to the island’s label within the datafiles: B for Boiler, G for Survey (“garden”), J for Jungle, O for 233rd (“office”), P for Prison, R for Tay (“rebel”), and T for Temple. For example, card J127 (pictured at the top of this article) refers to card 127 within Jungle Island. These card numbers also correspond with the beginning of the filenames produced by the asset extraction tool Riveal by Ron Hayter, which was used heavily in the research and preparation of these articles.

  Note: some card numbers change between versions and languages. The numbers in this article are based on the 1.2 version from the English 10th Anniversary DVD. If you’re using a CD release or searching for assets dubbed into a different language, the cards may be numbered differently, but we haven’t found any differences in the actual content of the renders yet.

• WORLDSTATE: The “worldstate” refers to the state of all the player-actionable things, such as the position of a lever or an elevator. Whenever the player performs an action or the story progresses, the worldstate changes.

• PATCH: A “patch” refers to a smaller picture or video that is placed on top of the background image in real-time by the engine. Patches are often used in places where many different worldstates are visible, to prevent the need for a huge number of renders covering every possible configuration. Patches are also used for the large number of very small videos that don’t cover the entire screen, such as when a spinning dome is visible in the distance. Patches are always rectangular, and are stored as separate images/videos within the datafiles, to be overlaid in real-time within the game.

• SPLICE: A “splice” refers to when the image data of a render has been copied from another image or another render. These are sometimes used when only part of an image needs to be re-rendered, but a patch would not be appropriate or desirable for whatever reason. Splices are edits which exist within the finished render itself, and are not separate files like patches are.

• FMV: An acronym for “full motion video”, referring to the many videos used throughout the game. Almost anytime you see motion, it’s an FMV. (Rare exceptions include the water rippling effects all over the game, and the various miniscule flying bugs that appear in places like Jungle Island.)

• OCCLUSION: In computer graphics, "occlusion" refers to when one object blocks vision to another object. When one object is in the foreground and blocking another object in the background, the background object is "occluded" by the foreground object. Several anomalies involve objects blocking one another in ways that are incorrect based on where they actually exist in 3d space; these are problems with occlusion.

An example of a "Splice": the rectangular edges are easily visible in the differential comparison of J191

Part 1: Jungle Island

Jungle Island is a location very familiar to many of us, but it has a large variety of anomalies, making it suitable as our first point of examination.

Several anomalies are associated with the state of the spinning dome (this is a recurring theme- every single spinning dome in the game has anomalies.) In cards J121, J124, J133, J154, and J421, the two semicircular “hinges” that sit at the base of the dome are occluded incorrectly, making them fully or partially obstructed by other geometry depending on whether or not the dome is flipped. In J124, you can also see the hinge switch from high to low polygon count.

In card J130, the base of the dome visibly moves when the dome is flipped.

Card J154 is also the location of one of the Spyder easter egg renders. A direct comparison with a normal J154 card reveals that the camera is in a different spot. Also, the furthest of the spherical lamps in the easter egg has been raised/lit like many of the others along this walkway.

In cards J110, J119, and J127, the foliage changes (quite drastically in J127) when the dome is flipped. J119 also has some leaves incorrectly occluded by parts of the dome when closed.

At the top of the wahrk gallows, in cards J178 or J264, the doorjamb to the prison changes depending on whether the door is open or closed. In card J264 the entire jamb changes, and in J178 the differential comparison makes it obvious where the images were spliced around the jamb.

There are two patches applied when you’re standing in card J181, one for the submarine and one for submarine dock 5 (the schoolhouse).

When the submarine is raised out of the water, the submarine overlaps the railing in the foreground of J181.

Furthermore, there is a patch that shows dock 5 extended, which is correctly used to show an extended dock 5 in J181 when the prison door is open. However, the render of the closed prison door already has an extended dock 5, and there is no patch to show it retracted. As a result, if you retract dock 5 before opening the prison door, you can see the schoolhouse dock retract the moment the prison door finishes its opening FMV.

Inside the tunnels, differential comparison revealed a render that seems out of place. If you turn on exactly 3 of the lights in the tunnel, when you’re walking back towards the ocean, there is one render assigned to card J194 that’s actually a render from the perspective of J196. In the differentials, this looks like the tunnel changing shape between worldstates. In-game, this makes it seem like one of the steps towards the ocean darkens the lighting without moving forward at all. The incorrect render may be from earlier in development, as it has noticeably egregious banding on the dim/gray parts of the image, which isn’t a problem on the correct version visible from J196, or anywhere else in the tunnel. It’s likely card J194 should be completely black when 3 lights are on, as it’s barely lit (with a correct render) with 4 lights on.

Each of the 25 totems in the Tay book room can be raised or lowered individually. When you’re standing in the center of the room, individual image patches are used to allow arbitrary totems to be shown in the downwards state, and renders with Tay unlocked are made with the correct totems already lowered. However, the zoomed-in views on almost all of the 25 totems have all of the totems in the same state as the totem you’re currently zoomed in on. (Exceptions: J217 and J225)

Two anomalies are visible as a result: Firstly, and more obviously, when zoomed in on a totem, the visible edges of the 2 adjacent totems will be raised or lowered based on the state of the totem you’re currently zoomed in on. Secondly, in many of the totem views, a reflection of the room is visible on the water, which shows every other totem in the room raised or lowered in the reflection.

Also, for some reason, a few of the totem renders have bright blue pixels on the extreme right or left side:

Furthermore, there are some changes to the lighting that occur. A shadow changes visibly in card J241 based on the totem state. In cards J211 and J213, all of the shadows cast by the totems disappear when the Tay book is unlocked.

At the big wahrk head staircase, at card J375, some bright white bits in the jungle disappear when the wahrk is open.

Also, the angle of the camera in card J381 changes when the wahrk is open. Some foliage also changes, and the lighting overall becomes slightly brighter.

At the maglev leading to Temple island, at card J806, the rocks on the far left are different based on whether or not the maglev is present or missing. Also, some of the details on the concrete are slightly moved.

A few anomalies are present based on whether or not the sunners are present. In card J620, the rocks on the right change when the sunners are absent. Furthermore, in cards J620, J626, and J628, the horizon changes visibly based on the presence of the sunners. (In card J628, the mismatched horizon is only present if you enter the node while the sunners are awake/startled, probably to match the FMV).

In card J526, the shape of the background mountains change to match the FMV when you open the submarine hatch.

In card J296, if dock 4 is retracted, a red spot appears on the rocks when the submarine is turned around. It also appears some of the rocks change, but it’s unclear which of these changes are just reflections of the submarine.

In card J449, when the gallows shortcut ladder is still raised, submarine dock 2 will be completely missing when retracted (The extended dock is shown using a patch.) Furthermore, when dock 2 is visible in any worldstate, it becomes apparent that the submarine dock is on the wrong side of the walkway, a quirk also visible in J447.

In cards J468, J470, J476, and J581, the diffs reveal that the shortcut ladder isn’t actually raised, the bottom half has merely been erased. Card J334 clearly shows that the ladder should be visible when raised.

Finally, in card J459, a detached shadow probably belonging to the shortcut ladder is visible on the leftmost cliffside at all times.

Additional articles authored by Twitch are on their way, covering the large number of anomalies we have discovered across the entire game.

During my middle school studies, I was introduced to Homer’s Iliad and Odyssey. This year, I decided to revisit these epics and acquired a French edition from the Bibliothèque de la Pléiade (translated in 1955 by Victor and Jean Bérard, and Robert Flacelière). This particular edition is noteworthy for its clear demarcation of dialogue: each speech is prefixed with the speaker’s name, as illustrated by the following excerpt available on the La Pléiade website (see pages 12-13 for example).

The editorial practice of prefacing each speech with the interlocutor’s name greatly aids modern readers, giving the poems a somewhat theatrical feel. In antiquity, in an environment where the audience relied solely on auditory cues, it would have been indispensable to provide clear signals so that listeners could unambiguously follow the dialogues. Therefore, the modern convention of appending the speaker’s name appears anachronistic. It is more likely to be an editorial solution designed to help readers in navigating the complex interchanges of the poem rather than a reflection of Homeric or early performative practice. This led me to wonder how Homer originally indicated (or even encoded) these shifts in speech.

To answer, I turned to the primary text itself. Because I do not read Ancient Greek, I was primarily concerned with the structural aspects rather than lexical detail. In essence, focusing on the “architecture” of speech rather than dissecting its constituent parts. My goal was to determine if the epic contained explicit “delimiters”, or markers distinguishing Homer’s narrative passages from the direct speech of characters.

The idea of delimiters, as outlined in my previous article, posits a general principle: every interpretive language uses markers that signal transitions from a primary layer to an embedded level of expression. In contemporary English prose, quotation marks serve precisely this function; they separate the narrator’s description (e.g., It is raining) from a character’s utterance (e.g., Alice said: “It is raining”). I sought to identify analogous signals within the Homeric text that fulfil the same role.

In Homer’s era, punctuation marks such as quotation marks have not been invented yet, and even if they had existed, they would have been invisible to listeners in a purely oral performance. Therefore, my search was confined to intrinsic textual structures that could signal a shift from narrative to direct speech. It is plausible that a delimiter can be discerned through its structural properties when it shares the same alphabetic system as the surrounding text (in other words, when it is not a special character). As demonstrated in the previous article, languages can employ distinctive patterns, like palindromes, to mark boundaries. Thus, I sought regular linguistic constructs within Homer’s poems that could serve this delimiting function.

I used two resources to access the original text, hosted by Tufts University: the Perseus Digital Library (Greek text and English translation by A.T. Murray) and the Beyond Translation project (Greek text by David B. Munro and Thomas W. Allen). In this article, most of the examples come from the first book of the Iliad and the first book of the Odyssey to keep it readable.

My analysis revealed that Homer frequently employs formulaic constructions as structural cues for navigating between narrative and direct speech. These recurring phrases function as delimiters in three distinct roles:

1. Opening delimiters signal that the narrator yields the floor to a character (first‑to‑second order transition).

2. Closing delimiters indicate that a character’s speech has concluded, and the narration resumes (second‑to‑first order transition).

3. Transition delimiters are used when one speaker responds directly to another within a continuous block of dialogue (second‑to‑second order transition).

Opening Delimiters. The sentences that start a speech are highly variable. For instance: “… and he implored all the Achaeans, but most of all the two sons of Atreus, the marshallers of the people:” (Iliad 1.15-1.16) or “… yet the thing did not please the heart of Agamemnon, son of Atreus, but he sent him away harshly, and laid upon him a stern command:” (Iliad 1.24-1.25), or “… and spoke swift-footed Achilles” (Iliad 1.58). Homeric opening delimiters lack a fixed form, yet they sometimes contain the enigmatic formulaic phrase “ἔπεα πτερόεντα [winged words]”, like in “ἀγχοῦ δ᾽ ἱσταμένη ἔπεα πτερόεντα προσηύδα: [and she drew near to his side and spake to him winged words:]” (Iliad 5.123). As noted by a researcher: “All the usages of epea pteroenta / epea pteroent’—without any exceptions—introduce the direct discourse of a character in the epic.” (Françoise Letoublon. Epea Pteroenta (“Winged Words”). Oral Tradition, 1999, Oral Tradition, 14 (2), pp.321–335, p. 331). While all verses containing ἔπεα πτερόεντα are opening delimiters, all opening delimiters do not necessarily contain this expression.

The variability of opening delimiters does not appear to be surprising. When a bard sang the poem, in a first-order context (as the narrator), they just announced a speaker by preterition. For example: “[Agamemnon] laid upon him a stern command” implicitly announces that Agamemnon is about to speak. The transition to the second-order expression is obvious.

Closing Delimiters. In contrast, signaling a return to narrative from direct speech proves more problematic for an oral reciter. A speaker might employ a phrase such as “to conclude …,” but such strategies are difficult to systematize. To indicate a closure, Homer had to frequently use repetitions such as “ὣς ἔφατ᾽ [so he spoke]” (Iliad 1.33, 1.43), “ὣς φάτο [so he spoke]” (Iliad 1.188, 1.245), “ὣς ἔφαθ᾽ [so spoke]…” (Odyssey 1.43), “ὣς εἰποῦσ᾽ [so she spoke]” (Odyssey 1.95), “ὣς εἰπὼν [so saying]…” (Odyssey 1.125), or “ἤτοι ὅ γ᾽ ὣς εἰπὼν [when he has thus spoken…]” (Iliad 1.69, 1.101). They appear after the speech, much like closing quotation marks follow the enclosed text. By maintaining a uniform structure, these formulae enable listeners to recognize immediately that the current character has finished speaking and that the narrative voice resumes.

Transition Delimiters. As outlined earlier, transition delimiters demarcate the boundary between successive speakers: speaker A delivers an utterance; a delimiter signals that speaker B is now about to speak; then speaker B speaks. These delimiters typically begin with the particle τὸν δ᾽ [and him] or τὴν δ᾽ [and her] followed by the speaker and a verb indicating the act of speaking. For instance: “τὴν δ᾽ αὖ Τηλέμαχος πεπνυμένος ἀντίον ηὔδα: [Then wise Telemachus answered her:]” (Odyssey 1.230) or “τὸν δ᾽ ἀπαμειβόμενος προσέφη πόδας ὠκὺς Ἀχιλλεύς: [In answer to him spoke swift-footed Achilles:]” (Iliad 1.84). As with the closing delimiters (ὣς…), the initial word of a transition clause strongly predicts that a delimiter is present.

What I find interesting with these transition delimiters is their frequent pairing with an adjective that characterizes the speaker: the wise Telemachus, the swift-footed Achiles, etc. For instance: “τὸν δ᾽ ἠμείβετ᾽ ἔπειτα ἄναξ ἀνδρῶν Ἀγαμέμνων: [Then the king of men, Agamemnon, answered him:]” (Iliad 1.172). I therefore wonder if Homer had to use these adjectives to satisfy the metrical constraints by adding syllables necessary for dactylic hexameter. I believe that Homer fine-tuned the total syllabic count of these delimiters by using these adjectives as fillers (i.e., “to him/her, replied/answered/etc.:”). That would explain why he continuously and repeatedly uses them across both poems. This is just a hypothesis, but I would love to know what specialists think of this idea of using adjectives as fillers for delimiters.

The role of these formulaic expressions has already attracted scholar attention. A notable example is the observation of Françoise Letoublon: “In an oral epic one has a compelling need for signals of direct discourse, in principle both before and after the reported speech, so that the audience will be aware that the narrating bard is assuming the voice of his characters. And these signals must be clear, perceptible even by a less than attentive audience: they must therefore be regular enough to play the role that iconic marks of quotation (“...”) play for us in the written text. . . . Following direct discourse, the signals for closure—equivalent to closing quotation marks in the typographical tradition—at times include epos, but other formulas are used more often, most frequently with a verb of speaking in the aorist tense.” (Françoise Letoublon. Epea Pteroenta (“Winged Words”). Oral Tradition, 1999, Oral Tradition, 14 (2), pp.321–335, p. 333). In other words, the delimiters function as highly recognizable formulaic markers that cue listeners to a shift from direct to indirect discourse (i.e., first‑order to second‑order expressions) and signal a change of speaker. Letoublon illustrates this point by comparing the delimiters to quotation marks: these formulaic clauses demarcate transitions within the narrative flow.

After having applied this intuition about delimiters acting as a fundamental law of any language to LLMs, I suggest using it as a conceptual tool for analyzing ancient texts. I believe that the Iliad and the Odyssey demonstrate that delimiters can be systematically encoded using ordinary alphabetical forms in a natural language. This observation raises an intriguing possibility: if similar delimiter structures exist in ancient undeciphered scripts, they might provide a foothold for decipherment. However, successfully applying this strategy requires establishing whether a target script encodes higher-order expressions such as dialogues, which is very speculative.

The original Riven (1997), the sequel to Myst, was developed for the Mohawk engine, resulting in game files stored in a proprietary format. Myst for Mac also used a proprietary format: HyperCard. The success of the Myst Graph project relied on a readily available library (stackimport; more recently: mystextract) that simplified parsing the original game files. This can be explained by the fact that HyperCard remains relatively popular in the retro computing community. By contrast, while the Mohawk format is partially documented and supported by ScummVM, I found no easy way to navigate Riven’s files as individual cards.

To address this lack of tooling, I am currently developing an application to generate a Riven navigator (just nicknamed “Navigator” in this article) that decomposes the game world into its constituent cards. Each card will contain metadata (card name, etc.), its embedded images, the location of its “hotspots” (that is: the clickable areas within the game), and their associated scripts.

Decoding these scripts is still ongoing work. Therefore, a full analysis of relationships between cards is not yet possible. I am keeping the Navigator closed source during development but will open source it once it is complete.

But even without full script analysis capability, some initial observations are worth sharing.

First Insights

Unlike Myst, where each card typically embeds a single view (though some cards include partial images for visual effects), Riven allows a card to reference multiple images. For example, consider this card from Temple Island, embedding 20 images (If you squint, you can spot very subtle differences between these images...):

This means that in Riven, a card generally represents a location, except for special cards like the menu or those representing books. In contrast, a Myst card typically represents a single view. Furthermore, a location in Riven can have different appearances depending on gamer actions: reaching card X might show a closed door, while pulling a lever (without changing locations) could open it. Both views are handled by the same Riven card (X), whereas Myst would require two separate cards: card X for the “door-closed” view and card Y for the “door-open” view.

Based on this distinction, we can start by identifying the cards with the most images associated with them. For consistency with the Myst Graph project, I will use the same naming convention in this section: “{stack name}:{card number}”.

Cards With the Most Images

Here are the cards with the most images, listed in descending order. (To keep this article concise, I am limiting myself to the first five).

Menu:6 – 51 images

This card, belonging to the stack that also contains the menu and books (“aspit”), holds the pages of Catherine’s journal, requiring 51 images.

Temple:184 – 24 images

The entrance to the Temple, with its rotating doors, requires no fewer than 24 images to represent all unique views resulting from different combinations of actions: wall, screen door opened, lever up; wall, screen door closed, lever down; room visible (no wall), screen door opened, lever up; room visible, screen door opened, lever down; room visible, screen door closed, lever up; etc.

In this example, the image on the right (184 Tgr S1 Bdc Ld. 40) shows the screen door closed in front of the wall, while the image on the left (184 Tgr S1 Ido Ld. 40) represents it open, also in front of the wall. The middle image highlights the differences between these two views, with red pixels indicating exactly where the images differ. This diff image is generated by the navigator, that automatically compares all unique pairs of images within a single card. This technique allows for the detection of even subtle variations. For instance, while the following images may seem identical at first glance, there is a slight difference in scene lighting:

Further subtle differences are shown below.

Boiler:278 – 22 images

This card contains Gehn’s journal from his lab on Boiler Island, each page requiring an image (similarly to Catherine’s journal), for a total of 22 images.

Temple:130 and Temple:138 – 20 images each

The telescope views, shown above, result from the combination of multiple factors (bridge up/down; manhole close/open; reflection of the light on the dome; lever up/down; telescope up/down; etc.).

Cards With the Most Hotspots

As stated above, Riven cards contain hotspots. They represent areas of images with which the gamer can interact with, either to perform actions from a given location (e.g., pull a lever, push a button), or go to another card (change view, move). The Riven Navigator materializes these areas with rectangles. When clicking a hotspot rectangle, the Navigator shows the script associated with that hotspot, facilitating the analysis of each action.

Here is an example of an image with typical hotspots (usually, there are hotspots on the left and right so that the gamer can turn their head):

The spinning book domes require many hotspots, as each segment of the slider is a hotspot.

Prison:42 – 33 hotspots

The dome on Prison Island requires 33 hotspots.

Temple:392, Jungle:338 – 31 hotspots each

The domes on Temple Island and on Jungle Island require 31 hotspots.

Boiler:443, Garden:173 – 30 hotspots

The domes on Boiler Island and Garden Island require 30 hotspots.

Gehn:85 – 25 hotspots

This one is interesting. Card Gehn:85 depicts the top of Gehn’s nightstand. From here, there are only three interactive options: zoom out (hotspot 1), open his journal (hotspots 2 to 24), or examine the metal device (hotspot 25; go to new card). The unusually high number of hotspots for the journal is a consequence of the Mohawk engine’s hotspot implementation. Mohawk only supports defining non-rotated rectangular hotspots. To achieve full coverage of the slanted journal cover, the developers were forced to approximate its shape with a series of smaller, overlapping hotspots, each linked to the same action.

Card Boiler:480 employs a similar technique but works in reverse to disable interactive areas. For example, hotspot 4 takes the gamer to the boiler, but it is partially obscured by rocks in the image. To prevent accidental clicks on those non-interactive parts, hotspots 5 and 6 are used: they essentially do nothing (NOPs) and cover the area of hotspot 4 that overlaps with the rock formations. The same approach is applied to the left side of the boiler.

Diffs: Images with Minimal Variations (Close-to-Identical)

Riven’s development was remarkably meticulous, and a fascinating way to appreciate that detail is by examining pairs of near-identical images within the game files. These “diffs” (images with only slight variations) reveal how systematically the 3D scenes were rendered and updated to reflect even the smallest changes in gameplay state. Often, these differences are so subtle they are almost invisible without a direct comparison.

Here are a few examples showcasing this impressive attention to detail. (To detect them, focus on the red pixels in the middle gray image).

Boiler:477 – 9 pixels

This example perfectly illustrates Riven’s systematic rendering approach. The images depict the boiler from a distance, focusing on its front gauge (or level indicator) positioned just right of the door. One image shows the gauge empty, while the other displays it full. The change is incredibly subtle, almost imperceptible to the naked eye, with only 9 differing pixels.

Animated comparison (GIF):

(In this high-speed animated GIF above, we can see with difficulty that the gauge changes.)

Boiler:30 – 12 pixels

Looking at Gehn’s Lab from the boiler reveals another small but significant variation. This diff highlights whether the ladder hatch is open or closed. It is a logical update reflecting gamer interaction, rendered with impressive precision.

Animated comparison (GIF):

Boiler:478 – 26 pixels

Boiler Island seems to be a hotspot (ah!) for these nuanced changes. Like Boiler:477, this image focuses on the boiler’s front gauge at a distance. The game logically tracks whether it is full or empty, and this is reflected in the rendering despite the difficulty of visually discerning the difference.

Animated comparison (GIF):

Temple:447 – 46 pixels

Inside the dome, a subtle change occurs depending on whether the bridge has been set. This variation is particularly challenging to spot due to the dark environment.

Animated comparison (GIF):

Boiler:12 – 50 pixels

And back to the boiler! Once again, we see a variation focused on the front gauge.

Animated comparison (GIF):

Diffs: Images with Substantial Variations

The pairs of images with the most contrast are all related to Jungle Island.

Jungle:535, Jungle:536, Jungle:539 – 100% difference

The submarine provides high contrast in 3 locations, where its dark hull stands out sharply against the blue sky and white clouds.

Jungle:530 – 99.99%

In one location, the contrast between the submarine and the sky is slightly reduced (99.99%) because some parts of the submarine’s hull match the colors of the landscape on the right.

Jungle:154 – 99.96%

This one is particularly interesting, as it is an easter egg (the “walk in the sky”). Again, a contrast against the sky, except that the sky is below us!

Diffs: Subtle Variations

While reviewing recent diffs, I have been catching some interesting, minor variations. I have not conducted a systematic study yet but wanted to share these observations and solicit the thoughts of the readers on their potential meaning.

Lighting Details

Shadows and light cast by external elements have a minor impact on the lighting on the doors of Temple:308 and Temple:543. It is a second-order effect, but noticeable, nonetheless.

This effect is also visible in Temple:247:

Maglev

Looking at Boiler:200 and Boiler:230, the Maglev has a faint presence. In one location, it is theoretically visible through the stairs, even if not directly seen. In the other, while partially visible on the right side of the frame, there is an intriguing effect on how the stair framework supports (the metal tubes) are rendered: they appear brighter when the Maglev is present.

Animated comparisons (GIFs):

Development Artifacts?

I have also spotted what seem to be “patches” in some comparisons, like this one in Jungle:194 within the rebels’ tunnel (a rectangular shape). These patches look as though images have been layered on top of the original view rather than being fully re-rendered.

In Temple:404, I am still trying to understand the nature of that slanted line; smaller patches are also present there.

(Edit 09/20/25) On the Myst, Obduction, and Beyond Discord server, user “Twitch” (drrg) offers a credible theory: the slanted line could be a narrow beam of light filtering down from above while the elevator is in its lowered position.

You can look up the elevator shaft only when the elevator is down to see this. But when the elevator is up there's no gap.

. . . I also wonder why the button looks different between these two shots considering that the button doesn't ever permanently change states.

. . . I think the button is straight up in the wrong place in one of these renders.

Cards With the Longest Scripts

In this section, I'm sharing only the script sizes rather than their content, since I do not currently have a robust way to decode the binary data into readable scripts.

Boiler:284 – 6,390B

This location contains complex logic (complexity here correlates with data size, which holds true in this specific context). It contains 6,390 bytes of script.

Temple:268 – 5,770B

Temple:443 – 3,666B

Temple:184 – 3,450B

Jungle:327 – 3,304B

Prison:32 – 1,072B

Generally, there is a correlation between the number of images per card and script complexity. However, Prison:32 stands out as an exception; it contains just over 1KB of data but only includes one image. I do not recall ever encountering this situation in-game (that is: within the closed elevator).

Conclusion

This concludes my initial observations made possible by the Riven Navigator. I have not conducted a systematic analysis yet; these findings represent the most readily apparent patterns. My next step is to properly decode the scripts, with the goal of understanding the relationship between cards and potentially even viewing them as interconnected nodes.

Generative AI is poisoning its own well: online content is increasingly generated by AI; this data is used to train new models; those models then generate more online content, which in turn becomes training data. This creates a cycle that risks contaminating the sources AI relies on, potentially leading to diminished originality, amplified biases, and a disconnect from real-world information.

Last year, researchers already raised a critical warning about this trend: “The development of LLMs is very involved and requires large quantities of training data. Yet, although current LLMs . . . were trained on predominantly human-generated text, this may change. If the training data of most future models are also scraped from the web, then they will inevitably train on data produced by their predecessors.” (Shumailov, I., Shumaylov, Z., Zhao, Y. et al., AI models collapse when trained on recursively generated data. Nature 631, 755–759 (2024), p. 755). The authors describe this as leading to a model collapse: “Model collapse is a degenerative process affecting generations of learned generative models, in which the data they generate end up polluting the training set of the next generation. Being trained on polluted data, they then mis-perceive reality.” (Ibid.)

I believe that we may soon need “certified” human-generated data to train models. In other words, model creators, to achieve the best model performances, may benefit from a service that provides training data that is guaranteed to come from human minds. This is because human-generated data possesses qualities currently difficult for AI to replicate: nuance, creativity, common sense reasoning, and robust factual accuracy. Ensuring the source of training data will become increasingly important for them.

This idea raises multiple questions.

How can we ensure that data is generated by humans? I see one robust approach: a “human-in-a-room” experiment. To guarantee human origin, participants should have no access to AI during data generation. The ideal setup would involve having people work in a controlled environment, such as a library. While establishing such an environment presents logistical challenges (cost, participant comfort), positive incentives (financial compensation or academic credit) could encourage participation.

Who should generate the training data? Experts in their respective fields. PhDs, post-docs, doctors, mathematicians, historians, and others with deep subject matter knowledge. They would be asked to produce specific types of content (factual statements, creative writing prompts designed to test reasoning, or complex problem sets) with their knowledge, expertise, and available offline resources. This project resembles the work of encyclopedists, but instead of creating a first-order knowledge repository (an encyclopedia), they would create a higher-order one: high-quality training data for AI models.

How should this data be made available? Two approaches are possible: open access, driven by academic contributions, or a commercial market where data is sold to the highest bidder and not publicly released. Given its potential value (significantly higher quality than readily available online datasets), I anticipate the emergence of a robust market for certified human-generated training data. However, a market-driven approach raises concerns about equitable access and could favor larger companies with greater financial resources. To ensure consistent quality, a rigorous peer review process involving multiple experts would be essential.

Could AI models be trained purely on human-generated data? While humans cannot generate the sheer volume required for initial model training, a two-step approach is feasible. Models could first be pre-trained on existing large datasets and then fine-tuned using certified human data. During this second stage, we could instruct the model to prioritize learning from the human dataset by weighing its loss function more heavily. This high-quality data would also serve as an invaluable test set for evaluating model performance and identifying biases.

Model collapse is not a distant threat; it is a challenge we must confront now. By prioritizing certified human-generated data, we can unlock new levels of creativity and accuracy when creating new models. The future of AI depends not just on computational power, but on the quality of the knowledge that fuels it, a future where human ingenuity remains at the heart of artificial intelligence.

“Structurally, narrative shares the characteristics of the sentence without ever being reducible to the simple sum of its sentences: a narrative is a long sentence, just as every constative sentence is in a way the rough outline of a short narrative.” Drawing on Roland Barthes’ observation that narrative is more than the sum of its parts, the Myst Graph project explores a similar principle. It demonstrates that the relationship between game world views (the HyperCard cards, individually; the “sentences” of the game) transcends their mere aggregation: The whole (the narrative of the game) is greater than the part (the collection of views and their own puzzles). There is an emergent narrative property at play, one that can only be captured, understood, and analyzed by considering the network-level properties (clustering, re-arrangement of edges, etc.).

My goal with this project has been to reveal a new understanding of Myst by studying it structurally, treating its components as elements within a larger system. I believe that structural analysis offers a powerful tool for video game archaeology, though its applicability is limited to specific contexts—particularly games, and, more generally, interactive media, inherently structured as a graph. The Manhole, released in 1988 by the Millers, could perhaps be a natural application for this approach, mirroring Myst’s foundation in interlinked cards. This approach could be applied to any sufficiently complex application built on HyperCard—a technology created by Bill Atkinson, whose recent passing is a significant loss. Representing other games as graphs will inevitably prove more complex; the original structure of Myst facilitated a relatively natural graph creation. To apply such an approach to free-movement games, nodes must shift from static views (files) to map locations, introducing challenges in automatically identifying edges based on how game engines encode location reachability.

In this article, concluding the series of articles dedicated to the Myst Graph, let’s now turn our attention to potential improvements. As mentioned in my previous article, further development of DeMystify offers significant potential. Several improvements are possible: first, encoding the logic of the HyperTalk scripts into the graph’s edges—for example, indicating that a connection to node 1234 only occurs when button A is pressed—would add valuable detail. A recent initiative by Andrew Plotkin, inspired by this project, makes Myst’s scripts publicly available, which would greatly facilitate this effort. While displaying all these edge constraints in a static graph representation would be impractical due to label size, as the logic would be intricated for some of the edges, they could still be leveraged for other analyses (see below). Second, extracting and processing the areas of views that connect to others could potentially aid speedrunning efforts; encoding distances from a reference point (like the screen center) to clickable areas as edge weights might offer marginal benefits. For example, if the player needs to reach a level, encoding the distance to that lever as an edge weight could potentially shave off fractions of a second. However, given the established speedrunning techniques and well-known shortest paths, any practical gains would likely be minimal. Third, following a suggestion from a reader, an interactive rendering of the graph—replacing nodes with actual view images—would greatly enhance its usability. Users could select two different views and see the most optimal path, see how groups of nodes are interconnected, etc. Finally, and more simply, using the DOT-encoded graph, anyone could suggest alternative representations to the current one by tweaking the graph’s general configuration and using a different layout engine.

Of course, the Myst Graph could be combined with artificial intelligence in multiple ways. One original idea could be to “retrofy” Myst by transforming it into a text adventure game. An AI model could be used to textually describe each view, and the Graph could be leveraged to ensure that the inner logic of the game is preserved. In other words, make a 90’s game look like a 70’s game using 20’s technology. To process the natural language inputs, a LLM model could naturally be used. On this topic, it makes me wonder, outside the scope of this project, how feasible it would be to create an LLM model able to use a graph as a system prompt or even build the model upon the graph, constraining it not by a list of rules, but by the structure of a graph. That will probably constitute my next project for this website.

I am delighted to see that Myst Graph has generated interest, particularly from Cyan, the company behind Myst, and through an enthusiastic discussion on Hacker News. Initially, I envisioned a more niche audience for this project. I hope to see further creative works built upon this foundation.

Converting the Original HyperCard Files

My initial attempt to implement a parser for HyperCard files proved more challenging than expected. Creating a parser based on the unofficial file format standard (and its helpful addendum) would have been the best approach but would have required a significant effort for a single use—the Myst HyperCard files only need to be converted once, after all.

Fortunately, stackimport, a C++ library, demonstrated its ability to reliably convert HyperCard files into XML (and PBM) files. DeMystify leverages this ability and therefore requires converting the original files beforehand into XML files. The path to these files must be provided as a command-line argument. I do not provide these files in the repository as they may be copyrighted: converting the original source code is left as an exercise to the reader.

There is an opportunity for improvement here and an interesting challenge for contributors. The conversion process could be further streamlined by creating a Go-based HyperCard converter library that can generate files or even data structures directly usable by other programs.

Parsing the Converted HyperCard Files

Stacks

Demystify parser package translates these XML files into a comprehensive “proto graph” representation of HyperCard objects and their relationships. The parser logic begins by processing each stack, resulting in a slice of stack objects that contain essential information such as the stack’s name (i.e., its Age), processed HyperTalk scripts, and a mapping of card IDs to their corresponding names (if applicable). Each stack is also associated with a list of cards that are part of it. To illustrate this concept, let’s take a look at an example:

. . .
<card id="8277" file="card_8277.xml" marked="false" name="" owner="2702" />
<card id="8776" file="card_8776.xml" marked="true" name="qbow" owner="2702" />
. . .

In this example, two cards, 8277 and 8776, are part of the stack. One is marked; another is not (this information is not used for the Myst Graph). And 8776 was named “qbow” by its creator. Because the card files do not contain names, the mapping 8776 → “qbow” needs to be provided to the cards parser.

Cards

After the stacks, the cards are processed. The goal is to generate a card object that contains a card ID, a reference to the stack to which the card belongs, its original name (if any) by using—as just explained—the IDs-names map populated during the analysis of the stacks, the parsed scripts, and the backgrounds—corresponding to the name of the image featured by the card. The image name is consistently present in the <content> tag having ID 1. For instance, the asset embedded in the following card is “Caldera 4-N”:

. . .
<content>
    <layer>background</layer>
    <id>1</id>
    <text>Caldera 4-N</text>
</content>
<content>
    <layer>background</layer>
    <id>7</id>
    <text>card id 31046</text>
</content>

Furthermore, the card is internally named following the pattern “{stack}:{ID}”, where “{stack}” represents the ID of the stack containing the card, and “{ID}” is the unique identifier of the card, as explained in the first article.

The preprocessing of HyperTalk scripts is, at this stage, minimal. The scripts, in the <script></script> tags, are split by line breaks. Similarly to SQL, scripts are commented out with --. When a line is prefixed this way, the command is identified as being disabled.

References to Other Cards

Subsequently, the relationships between stacks and cards, and between cards, are inferred from the analysis of the scripts. Currently, this analysis relies on regex pattern matching to identify the links between elements. As explained in the previous articles, this analysis is not granular: it does not consider the gameplay or the scripts as a whole. It suffices for a card to point to another to identify a link between them.

HyperTalk provides a convenient mechanism for determining when a card references another card. When they belong to the same stack, the script contains a go to card id {ID} (where “to” is optional). When a card refers to another stack, the pattern is go to card id {ID} of stack “{stack name}” or go to card “{card name}” of stack “{stack name}”.

There is also a more subtle push/pop card mechanism. When a card is pushed onto the stack (push command), it is stored in memory as a reference. A pop operation is, at least the way it has been used for Myst, equivalent to a go to card id {ID}, with the ID of the card that had been stored in memory. This duality characterizes two classes of relationships.

When the first card is pushed (push card), the second card just backtracks to the first.

However, when the first card pushes another card (push card id {ID} of stack “{stack name}”), a more complex transitive relationship emerges, making the player navigate from the first card to the pushed card through an intermediate card (which “pops” to the target card).

For instance, card 100894 from Channelwood Age contains this script:

. . .
push card id 46439 of stack "Myst"
go to card id 44018 of stack "Myst"
. . .

It pushes card 46439 from a different stack, then opens card 44018 belonging to the Myst Island.

The latter embeds the following script:

on mouseUp
htlock "nobw"
htlock true
vs up
pop card
end mouseUp

It just pops itself (line 5). But instead of backtracking to Channelwood Age, it goes to the card on the top of the stack: card 44018. Therefore, the path becomes:

Channelwood:100894 (push Myst:46439; go to Myst:44018) → Myst:44018 (intermediary card: library up; pop) → Myst:46439 (bookshelf).

The script analyzer also identifies the cards that feature pages using the following patterns:

• put "{digit},A,0" into ALL_Page for blue pages (A is for Achenar)

• put "{digit},S,0" into ALL_Page for red pages (S is for Sirrus)

• put "Atrus" into ALL_Page for the white page

Last, if a card refers to a card that cannot be found, a new virtual card is created. This just means that the virtual card existed up until some point in the stack and was then deleted for the release of the game.

This initial approach may appear rudimentary but is well suited for the HyperTalk grammar and the way the Myst creators have used it. A welcome evolution would be to make the analysis more comprehensive to detect additional associations (typically capturing the navigation from page to page in a book, handled by image substitution). Also, perhaps, encode the gameplay constraints into the edges.

First Layer of Abstraction (or: Creation of the Proto Graph)

The last action of the parser is to transform stacks and cards into node objects that contain attributes (stack or card; contains colored pages; virtual) and edge objects, also containing attributes (intra-Age or cross-Ages; disabled; self-reference; etc.) and pointing to the two nodes they link. At this point, the resulting structure is not yet a proper graph but rather a preliminary representation of the data that will eventually be used to generate an actual graph.

Note that the parser ensures that the edges are unique. A card can refer several times to another card (i.e., clicking on different parts of the first card leads to the same target card). I decided, for simplification purposes, to merge edges having the same attributes and directionality. If DeMystify is updated to encode the gameplay into the edges, then it would make sense to unmerge these edges.

Also, the parser translates the properties of the cards and links into node and edge attributes. For a node, the base attribute is either stack or card. Then, if it contains a page, an additional attribute is the color of the page (can be cumulative: both blue and red). Edges can be disabled (they initially existed, but the developers disabled it by commenting it out in the script), notImplemented (the link connects to a node that refers to a card that does not exist in the stack), intraAge or crossAge (depending on whether they refer or not to a node from a different Age). An edge can have special attributes, such as a backtracking attribute (A to B, then B to A). Suppose the edge is restrictively transitive (see the figure on the right of the Transitivity section of article 1) because the transitivity relationship includes three nodes (A to B to C). In that case, the parser distinguishes between the “tail” of the transitivity and its “head”. The tail is the edge between the initial node and the intermediary node (A to B); the head is the edge between the intermediary node and the target node (B to C). This distinction is helpful for the rendering of the graph (no arrowhead on the tail section of the transitive edge).

Once this process is done, the parser returns an object containing a “bag” of nodes and edges. The next step involves transforming these components into a graph.

Second Layer of Abstraction: Creation of the Myst Graph

DeMystify leverages the Gonum library to create an instance of a graph. Due to the directionality of the edges, a directed graph is necessary. Moreover, since multiple edges can coexist between two nodes for a specific direction (even when identical edges are merged, they may differ in attributes) and self-loops should also be supported, a multigraph is required. Additionally, I had to consider backtracking edges, which should be excluded during distance calculations. To accomplish this, I utilized weighted edges with regular edges assigned a value of 1 and backtracking edges given an extremely large positive value (positive infinity).

To satisfy these constraints, after evaluating alternative solutions, I determined that multi.WeightedDirectedGraph from Gonum was the most suitable choice. This graph type effectively accommodates both directed and multigraph structures, as well as weighted edges with distinct properties.

Creating the weighted directed multigraph is straightforward, thanks to the parser’s preprocessing. Nodes are added as-is, while edges default to a weight of 1. On the other hand, backtracking edges are assigned to the math.Inf(1) value to distinguish them from regular edges. Disabled edges are skipped altogether and stored in a map for future rendering.

With the graph now created, we can analyze it.

Three types of analyses are performed: nodes, components, and paths.

• Node analysis. The node analyzer identifies key nodes based on their in-degree and out-degree. It, therefore, identifies nodes with the most incoming edges (by iterating over the nodes, counting the number of in-degree edges, then comparing with the current maximum), with the most outgoing edges, source nodes (no incoming edges and a nonzero number of outgoing edges), sink nodes (no outgoing edges and a nonzero number of incoming edges), and isolated nodes (no outgoing nor incoming edges).

• Components analysis. This analysis aims to identify the graph components (that is, subgraphs not connected to others). The general idea is to iterate over all nodes and, for each node, traverse (using BFS) the subgraph to which it belongs. By keeping a “map” (in fact, a slice, for making the outcome of the process deterministic) associating the subgraphs with the nodes that compose them, it is possible to identify the components.

• Path analysis. This analysis examines two types of paths: the shortest paths and the longest shortest paths (i.e., most separated nodes). For the computation of the shortest path, Gonum provides a valuable resource: dijkstraFrom: “DijkstraFrom returns a shortest-path tree for a shortest path from u to all nodes in the graph g. If the graph does not implement Weighted, UniformCost is used”. This function uses the Dijkstra algorithm under the hood, taking into consideration the weighted edge values in the graph. As a result, backtracking edges are de facto excluded.

When implementing DeMystify, I experimented with other conceptual tools, such as the identification of bridges. But for the purpose of the initial version of the Myst Graph, I found that focusing on just these three dimensions would provide interesting information without making it too complex. It provides a way to categorize the nodes from a graph theory perspective, identify the isolated subgraphs, and compute varieties of paths. I would be very interested in the readers’ findings in addition to these dimensions.

At this point, the Graph Myst is analyzed. It is now time to visually render it.

Rendering the Graph

At the start of this project, I wanted to use the DOT language due to its “natural” affinity for representing graphs. Initially, I used the Gonum DOT library. Then, when things became too complex for this library, I assessed other libraries. Unfortunately, none of them were flexible enough to support the requirements: I had no choice but to develop a custom encoding mechanism using string manipulation techniques. This aspect of DeMystify lacks elegance and was a source of frustration during its implementation.

From the generated DOT object, I wanted to keep things simple for the first graph version, so I decided to generate a static file. At first, I tried generating PNG images, but that quickly became impractical because of the sheer size of the graph. Creating a PDF file proved to be a good solution, also having the benefit of being searchable. At first, I did it manually, using the DOT file as input. After experimenting with different CLI tools, I found that Neato generated an acceptable output. As for generating the DOT object, I wanted to use a library to generate the PDF à la Neato. Unfortunately, I did not find any: consequently and unfortunately, DeMystify must externally call the Neato binary.

There are several areas of improvement. First and ideally, the rendering process should be refactored to use libraries. This process would be unglamorous, as the behavior would remain unnoticeable, make DeMystify more robust. Second, it would be interesting to generate a dynamic graph. From a visual perspective, I would like to view alternative renderings from contributors. On Hacker News, a commenter suggested rendering the actual images as nodes. While I (really) like this suggestion, I wonder how feasible that would be, particularly from a legal perspective (Myst not being open source).

Conclusion

As it stands, the initial prototyping phase focused on quickly demonstrating a working graph representation, though significant improvements are needed. Currently, the project lacks unit tests and relies on external dependencies (manual HyperCard file conversion and binary calls), generating a static graph instead of a dynamic one, etc.

Because this is a personal project with limited time commitment, I am actively seeking contributions from the open-source community. Your expertise and passion could dramatically improve DeMystify. I believe that with collaborative effort, DeMystify can truly reach its full potential. A fourth article suggests possible directions for the project.

Please contact me at myst.graph@glthr.com to discuss potential collaborations, ask about implementation questions, or share your own generated graph (I would reference it in the fourth article with your permission). I am excited to see how the community will shape and evolve this project: feel free to fork the repository and contribute your ideas.

Previous analyses of Myst’s source code have indicated that some assets are not seen in the game, such as Catherine’s crumpled note and the opened music box in the Mechanical Age. Also, the analysis of commented-out scripts indicates the reference to an intriguing “Rattle Ratchet Routine” (Myst:70105), as revealed during the “Reverse Engineering Myst” presentation at Mysterium Con 2024:

-- Rattle Ratchet Routine…

-- It’s no longer with us, but it may be back!!!

These findings offer a fascinating glimpse into the artistic, gameplay, and last-minute decisions made by the Myst creators. By creating the “Myst graph” (shared in the previous article), I aim to reveal new insights into the game’s creation by focusing on the relationships between cards instead of just the cards themselves.

Visual Indicators

Before diving into the findings, let’s clarify one point related to visual indicators used in this article.

A close-eye icon superimposed on a game screenshot signifies that the asset is not accessible within the game environment. This typically occurs when the asset is featured in a source node with no incoming edges, effectively making it invisible and unreachable. On the other hand, an open-eye icon indicates that the corresponding asset is visible and can be accessed while playing the game (I typically used reachable nodes to show a contrast in viewing perspective). By default, if no indicator is shown, the image is viewable in the game.

Also, for demonstration purposes (especially when the edges-density is high), edges may be highlighted with thick red lines.

Disclaimer

This project is a personal initiative to analyze and understand the classic Myst game. It is an unofficial and nonlucrative open-source effort created for educational purposes only. This project is not affiliated with Cyan Worlds, Inc. or the original creators of Myst. Any opinions, insights, or analyses presented on this blog are my personal views and do not reflect any official positions or statements of Cyan Worlds, Inc., its affiliates, or any organizations I might be affiliated with.

Before analyzing this graph, please be aware that this series of articles will contain spoilers for the game. If you are new to Myst or plan to play it soon, I recommend skipping these articles and experiencing it firsthand: Myst remains such a fantastic game!

Revisions

This article only scratches the surface of these discoveries: there are likely many hidden gems waiting to be uncovered. If readers have additional insights from the analysis of the Myst graph, I encourage them to share their work by emailing me at myst.graph@glthr.com. I will do my best to update this article to refer to their publication (if they have a blog, etc.) or, with their permission, feature their contributions by updating this article. I will also update this article with new findings and corrections. These changes are tracked on this page.

General

Shortest Theoretical Path

From the dock (Myst:8336), where the player starts the game, to Dunny:11088, where the player can return to Myst Island, the theoretical minimum distance is 24 edges. (Note that the last node can be subject to interpretation, as some people may consider that the return to the library is the actual ending).

One of the possible paths (there are alternative sub-paths in the library, after Myst:46094) is: Myst:8336 → Myst:2737 → Myst:5026 → Myst:6798 → Myst:7516 → Myst:7902 → Myst:8079 → Myst:3604 → Myst:9960 → Myst:11521 → Myst:46094 → Myst:49958 → Myst:47338 → Myst:49781 → Myst:56926 → Myst:63906 → Myst:71909 → Myst:23753 → Myst:28552 → Dunny Age:6532 → Dunny Age:2943 → Dunny Age:9757 → Dunny Age:4840 → Dunny Age:10805 → Dunny Age:11088.

As explained in the first article, this calculation assumes no gameplay constraints and provides a baseline for understanding the graph’s topology. One instance of the shortest theoretical path can be downloaded here.

Longest Shortest Theoretical Path

The most distant nodes are Selenitic Age:26154 (the tower controls) and Channelwood Age:88814 (the Achenar monitor), which are separated by 130 edges. Coincidentally and anecdotally, these nodes feature screen-based interfaces with button controls. The graph representing this path can be downloaded here.

Self-References

Three nodes refer to themselves. These self-loops are used as a mechanism to represent interactive effects instead of a navigational means.

In Myst Island, a self-referencing node represents the view of the control panel of the underground room next to the ship (90877). In Mechanical Age, 34613, representing the hologram in Achernar’s room, also self-loops. The goal of these actions seems to reset the state of the card, which contains variables.

In the Mechanical Age, the cage card (14628) redirects to itself too, but for a more practical effect: cycling quickly between cards to simulate, alongside with sound effects, the rendering of electricity arcs: cage, cage electrified, cage electrified bright, then cage again. Interestingly, it also connects to two sink nodes (i.e., nodes with no outgoing edge, leading nowhere), as they are used for this effect, not for navigation.

Isolated Cluster

As indicated in the previous article, there is an isolated cluster of three nodes unreachable from the game. The puzzle they represent has not been removed from the game; it has just been duplicated and replaced by Mechanical Age:33452, which is part of the game. The key difference is that the duplicating card plays a video named “MU Lower Layer” before going to the card Mechanical Age:35943.

Cross-Ages

Mysterious Missing Direct Link from Myst Island to Selenitic Age

Initially, the Spaceship on Myst Island (78050) was somehow directly linked to the Selenitic Age (31832, virtual). This link was removed (but still visible in the code), and the destination card was removed, too. As of the game’s release, only the exterior path (Myst:14069) is reachable when leaving the ship.

Myst Island

Centrality of the Bookshelf

With 19 incoming and 10 outgoing edges, the bookshelf (Myst:48938) is a central hub connecting various nodes throughout the graph. This position can be attributed to its role as a gateway for accessing openable books, which are treated as individual nodes that share connections with the bookshelf. This feeling is reinforced by the fact that returning from an Age places the player in front of the bookshelf (see the “transitivity” section of the previous article). This is unsurprising in a universe where books are the gateway to other worlds.

Additional Note on the Crumpled Note

It is a well-known fact that Catherine’s note is not represented in the final game as a crumpled letter.

The graph reveals that the cards featuring the crumpled note assets are still in the game, like the assets themselves, but not reachable. Looking at the North, the bidirectional edges between 8631 (crumpled note) and this latter and 8079 have been transformed into directed edges from 8631, making it a source node. A bidirectional edge was created between 25971 and 8079, bypassing 8631. Looking to the South, the mechanism is identical: 28321 was transformed into an unreachable source node, while a bidirectional edge was added between 9648 and 91471.

While it does not reveal a new finding here, the graph shows that the assets do not appear in the game because they were not used but rather because the cards that feature them were disconnected, and the gamer cannot reach them.

Retrofitted Observation Platform Edges

Initially, accessing the codes on the observation platform utilized the same navigational technique as most nodes. The player would reach code 68940, click to zoom in, navigate to the relevant card, and click again to return to the zoomed-out view. A bidirectional edge would be expected between 68940 and each of the cards representing codes.

However, this approach has since been modified to use backtracking edges instead. First, the backtracking functionality was implemented by introducing a “push card” from the origin edge and a “pop card” for the target edge (this mechanism is explained in detail in the third article). Second, the directed edge from the code view (e.g., 66100) to the observation platform (68940) was disabled (prefixed with “--”) as it was no longer necessary.

In summary, bidirectional edges have been replaced with backtracking edges for the zoom in/zoom out effect.

The Planetarium Device and the True-False Source Node

According to the graph, 42800 is a source node. But something seems off: researching the contents of the cards reveals that the asset, “Planetarium3-Up LightsOn”, representing the device on top of the dentist chair when the lights are on, does not appear to be duplicated in another node. Also, this asset is reachable from the game. Consequently, 42800 must somehow be reachable, too.

Additionally, there is an inconsistency in the vicinity of this node: 42659, which represents a partial view of the device when lights are on, supposedly has a backtracking edge with 41677, which shows the device when lights are off. This is not possible, as the player is expected to switch off the lights using the button next to the door; they cannot do it from the chair.

A review of the cards reveals that 42659 indeed points to 41677, and they have a backtracking dynamic (push/pop card) conformingly to the graph indications. However, 41677 operates an immediate substitution of images to display “Planetarium3-Up Lights On” when lights are on. So, ultimately, this image is directly used by another node than 42800: it is, indeed, a source node, and most of its logic (it would be interesting to study to which extent) is present in 41677.

Selenitic Age

Unreachable Nodes

Several views cannot be seen in the Selenitic Age.

The graph reveals two alternative viewpoints on the crater rim, each representing a nuanced shift in perspective from the views selected for the release.

Also, regarding the crater rim, while 13290 is a source node, its asset is used by 12953 (qpat3) and, therefore, is viewable in the game. Another inaccessible alternate view relates to the stone forest, offering a slight change of perspective (compare 19922 and 20027).

The two last ones show that it was initially envisaged that the player could raise their head to see the wind tunnel access from below.

Mechanical Age

Unreachable Nodes

Node 38824, featuring asset “East Is. 8-W (Down)”, is inaccessible from within the game, as it is a source node. Consequently, the corresponding asset is not used. If the image seems familiar, it is because it was replaced by a slightly different change of perspective, reachable from 16009. Also, 15738, representing the player looking down to the East, is unreachable.

Channelwood Age

The Back of the Wooden Elevator, a False Sink Node

Nodes 45661, 96885, 97270, and 98476 use the same asset: “ElevatorEntered”. The graph analyzer interprets them as sink nodes. Indeed, manual verification shows that these nodes do not have special scripts on their own: their sole purpose is to display the back of the wooden elevator.

Given that no Myst player has ever been stuck in an in-game elevator, and considering that these nodes are connected to the central cluster, it is clear that they cannot be actual sinks. There must be a way to escape them. By examining the nodes pointing to these pseudo-sink nodes, we can identify the trick: they first reach the node representing the back of the elevator, then utilize a command that, to date, I have been unable to identify in the HyperTalk reference books (“vs left” or “vs right”). Finally, they reach the actual node. This behavior is similar to the electrified cage effect from the Mechanical Age, where the card just serves as a visual transition to the actual view.

Stoneship Age

Unreachable Nodes

In the Stoneship Age, the unreachability of the nodes is subtle. First, 17724, representing an open door in the tunnel, was replaced by 17526, which uses its scripting capacity to display the door open.

Node 53791, featuring the book linking to Myst Island, is interesting because it demonstrates a debugging device used by the Myst creators: the beep. The beep is noticeable on cards Mechanical Age:33554 (commented out), Mechanical Age:34613 (commented out), Myst:42800 (beep enabled, unreachable node), Selenitic Age:54693 (the Mazerunner ship; still enabled; a card so complex in comparison with the others that it would deserve an article of its own), and Stoneship Age:53791 (beep enabled, unreachable node). 53791 was replaced by 23473, which incorporates a very different script. Anecdotally, they both represent the same image and use different image files.

Dunny (D’ni) Age

The Shared Card That Goes Nowhere

Last (at least for the scope of this article), two very similar locations—at first glance, the images may appear identical, but subtle differences suggest the rendering of different views—, 7365 and 8487, have backtracking edges to a common node, 11283. Because their architecture is the same, it makes sense for them to share the same zoomed-in perspective.

Next Steps

These findings presented here are low-hanging fruits revealed by the graph analysis, and probably more will emerge. In the third article, I go into great technical details to explain how the Myst graph generator, named DeMystify, works under the hood and open source it.

In preparation for the release of DeMystify, if you have features you would like the graph generator to support, feel free to send your suggestions to myst.graph@glthr.com.

It is now literally one. Here is Myst as a graph:

The Myst graph, a comprehensive representation of the game’s structure that maps the connections between various views and locations, is downloadable as a poster-size PDF.

Introduction

Origins of the Myst Graph

I fondly recall creating, just like many others (including more skilled cartographers), a rudimentary topological map of the infamous Myst Mazerunner puzzle when I was young. The idea recently struck me: could this approach be extended to the entire game? The HyperCard implementation’s deterministic connection of cards seemed tailor-made for an even more abstract map: a graph. After all, graphs are a powerful tool for information analysis. By having the ability to represent the original Myst as a network of interconnected nodes and edges, we could gain a deeper understanding of how the different aspects of the game are related and uncover new insights into its underlying mechanics.

However, I did not see myself creating it manually; it would have been a monumental and error-prone task. Fortunately, I soon discovered, thanks to the efforts of Uli Kusterer (Youtube channel: “Masters of the Void”) and the “Reverse Engineering Myst” presentation at Mysterium Con 2024, that the game’s source code was accessible. That changed everything: how about generating the graph programmatically?

Roadmap

(1) In this article, I introduce the concept of the Myst graph, explaining why it is an important tool for analyzing the game, and walk readers through how to interpret the graph, highlighting key concepts and insights. (2) A second article, already published, explores new findings that have emerged from analyzing the Myst graph. This article builds on the present article, providing additional context and explanations for the insights gained from the graph. (3) A third article details the technical approach and open source “DeMystify,” the program I created to generate the graph. This provides readers with hands-on experience exploring the Myst graph using code. (4) Last, a fourth article discusses how this graph can be used as a starting point for new projects and speculate on what other insights might be gained by further analysis.

Minor Terminology Clarifications

Before diving into the presentation of the graph, let’s clarify some words. “Dunny” refers to D’ni: this is how the D’ni stack was initially named. Also, I may refer to the Myst Age as the “Myst Island” for variation. When discussing graph theory, I will use the following terms: nodes will refer to the vertices, while edges represent the connections between them. For those unfamiliar with graph theory, just remember that edges are arrows pointing from one box (or view, or card) to another, the nodes, illustrating a relationship between the two.

Disclaimer

This project is a personal initiative to analyze and understand the classic Myst game. It is an unofficial and nonlucrative open-source effort created for educational purposes only. This project is not affiliated with Cyan Worlds, Inc. or the original creators of Myst. Any opinions, insights, or analyses presented on this blog are my personal views and do not reflect any official positions or statements of Cyan Worlds, Inc., its affiliates, or any organizations I might be affiliated with.

Before analyzing this graph, please be aware that this series of articles will contain spoilers for the game. If you are new to Myst or plan to play it soon, I recommend skipping these articles and experiencing it firsthand: Myst remains such a fantastic game!

Basic Properties of the Myst Graph

Nodes and Edges

The game comprises 6 HyperCard stacks, one for each Age, totaling 1,355 cards. The graph abstracts these into 1,364 nodes connected by 3,189 edges.

The number of nodes does not coincide with the number of cards and stacks because 3 nodes are virtual. They represent cards that were not shipped in the release. They only exist because they are referred to by actual cards. These virtual nodes are Mechanical Age 17673 and 20348, and Selenitic Age 31832.

In this version of the graph, edges do not encode gameplay constraints. A card is considered connected to another if it references that card; the specifics of how the player interacts with the card to reach the other card—whether by simply clicking or solving a complex puzzle—are left implicit. (This choice is discussed later.)

Paths

Paths are sequences of edges from node A to node B. Because the gameplay has not been integrated into the graph, they do not necessarily represent feasible direct paths in the game. For instance, a puzzle on the path may require performing actions outside the path. In other words, the paths evoked in these articles abstract away all the constraints of the game, except for the connections between cards.

Theoretically, the shortest path between the starting point and the (good) ending consists of 24 edges. From a pure graph distance point of view, reaching the beginning of the game and its ending only takes 24 movements or a change of direction. The most distant nodes (in other words, the longest shortest path in the graph) are separated by 130 edges. In the context of the game, these paths are detailed in the second article.

Clusters

Interestingly, while the file that encodes the graph does not contain any information about clusters (except for isolated clusters, as explained in the second article, and for the colors of the nodes, for rendering purposes), the rendering engine that transforms it into a PDF, creates an organic (albeit imperfect) formation of clusters: the Myst Island in the middle, and the Ages at the periphery. This is explainable: Myst Island refers to all Ages; each Age only refers to Myst Island.

There is one disconnected cluster outside a few isolated nodes—that, consequently, cannot be reached from within the game. This isolated cluster comprises 3 nodes, including 2 virtual ones, from the Mechanical Age: originally, 17552 had directed edges with 20348 and 17673, both absent from the Mechanical Age stack.

Legend: How to Read the Graph

Nodes

Colors

Each Age has its own color to facilitate the identification of stack-based clusters.

Nodes Labels

When a node abstracts a stack, the name of the stack (corresponding to the Age) is prefixed with “[Stack]”. For instance, “[Stack] Myst” is the stack containing the cards belonging to the Myst Island.

Nodes representing cards are labeled following this pattern: “{Stack}:{ID} [({Original name})] \ {Image name}”.

The cards are identified by an ID (see page 280 of this reference), a HyperCard-generated integer. Example: “Mechanical Age:12345” refers to card ID 12345 belonging to the Mechanical Age stack. These IDs are unique per stack. Since each stack has its own set of unique IDs, there can be collisions across stacks where two or more IDs are identical between different stacks. This is evident in the Myst graph, where some IDs appear twice across Ages (8059, 8794, 9075, 18304, 19870, 21064, 32302, 33578, 34421, 39710, 52457, 55809, and 74269). As HyperCard automatically increments these IDs within each stack, one can reasonably infer that the greater an ID value is, the closer in time it was created relative to the initial release. However, this relationship may not hold for the underlying assets, as they were rendered separately through a different process. In other words, a “recent” (relative to the release) card does not necessarily mean that the asset it represents was also rendered close to the release.

Cards can have been voluntarily named. When that occurs, the original name is specified between parentheses after the ID. For example, “Myst:81655 (AchenarLose)” was named “AchenarLose” by the Myst creators.

Last, cards represent a view or an image, so they need a reference to an asset filename. This information is displayed just below the label. So “Myst:8336 (dock) \ Dock1-E” means that card ID 8336, belonging to the Myst Island and named initially “dock” features an image called “Dock1-E”. This specific node plays an important role in the game, as it is where the player starts.

Special Nodes

Sink nodes serve as destinations, receiving connections but not providing any exit paths. They can be reached within the game, contrary to the following categories of nodes. Source nodes act as starting points, sending out connections and initiating new paths without incoming edges. They are unreachable in the game. Isolated nodes are unreachable from the rest of the graph, having no incoming or outgoing edges that could be followed.

More specific to the game, nodes with blue borders represent views containing blue pages, while those with red borders show views having red pages. The unique, purple-bordered node denotes a view encompassing blue and red pages, and the white-bordered node represents the white page.

Edges

Directions

The Myst graph is a directed graph. The quasi-totality of the edges is either directed (they connect two nodes in one direction) or bidirectional (they connect two nodes in both directions). Rarely (3 occurrences), some edges are self-loops, connecting a node to itself.

In the game, a bidirectional edge typically represents a direct change of orientation (cardinal points, looking up and down). Movements are usually represented by directed edges, as walking backward is impossible, except for some exceptions (backtracking edges; see below).

Here is a concrete demonstration. In the village of the Channelwood Age, changing between one of the staircase ascending views (17225) and its corresponding pathway view (73038) can be done in a single click as the edge is bidirectional. However, once the player takes the stairs to go up (73346), it is impossible to backtrack directly to the original staircase ascending view without first turning around to look down (73587), then reaching the original pathway view (73038) and last turning around again (17225).

To summarize, and with some exceptions, from a navigational perspective, while directed edges can be interpreted as walking, bidirectional edges can be interpreted as changing directions.

Special Edges

Edges connecting two nodes from different Ages (cross-Ages edges) are rendered with a thick line. Edges that connect exclusively two nodes are backtracking edges. They act as a device to zoom in/zoom out (one click to enter the next node, one click to go back to the previous node) and are represented by edges with inversed arrow heads and tails. Last, some of the connections are commented out in the source code of Myst: they characterize disabled edges, represented by dashed lines with a tee arrow shape.

Transitivity

In Myst, there are three forms of transitivity.

First, by default, intermediary nodes are fully transitive (edges directions notwithstanding, obviously). If node B can access A, and C and D can access B, then C and D can access A. This is the common navigational mechanism.

Second, some edges are intransitive: while B can access A and, reciprocally, A can access B, C and D can access B too, but they cannot access A. This relationship characterizes the backtracking edges.

Lastly, some edges are partially transitive: if node B can access A, C, and D can access B, only D can access A. In the realm of Myst, this is only true of the edges making the player return to Myst after completing an Age. Upon returning to Myst Island with the book, they first systematically see the ceiling of the library (Myst:44018). When they click the ceiling, they automatically face the bookshelf (Myst:46439). (The mechanism by which this is done with HyperTalk, the scripting language of HyperCard, is detailed in the third article.)

Limitations

First, as explained in the previous section, the graph does not capture the gameplay mechanics. Although this approach has its limitations, it provides a significant advantage. By separating the gameplay from the graph structure, we can already gain novel insights into Myst without dealing with the complexities of puzzle design. Of course, nothing prevents future versions of the graph from including this logic.

Second, the rendered clusters are not entirely separate from each other. They overlap in their boundaries. For example, Myst:38896 and Myst:30143 encroach on the D’ni Age. While it would be possible to constrain the graph so there is no overlapping, I made the rendering engine as Age agnostic as possible for this initial version. As “DeMystify”, the graph generator, will be released, readers can try different arrangements; I would be very interested in seeing their results.

Third, the graph only captures straightforward relationships between cards and stacks. Analyzing the scripts more granularly and identifying complex relationships, such as image substitutions, would be interesting. This lack of granularity is evident for the books from the bookshelf: the current graph does not sufficiently capture the pages. A future version may improve this aspect.

Next Steps

Now that we have had a chance to familiarize ourselves with the Myst graph, its structure, and how it relates to the game, let’s dive deeper with a second article, which explores some of the insights and discoveries made possible through this new conceptual tool. The article will take us through the hidden connections and relationships between various game elements, revealing new aspects of Myst that were previously unknown or unseen.

The graph will make one with the game.

This puzzle is straightforward. As explained on the webpage, a 6x6 chessboard has squares labeled from A to C, corresponding to positive integers. The puzzle is solved by assigning values to A, B, and C and creating two corner-to-corner trips using knight’s moves so that each trip scores 2024. The method to calculate the score is: (a) moving between two different integers multiplies the score by the value the knight is moving to, and (b) moving between two identical integers increments the score with the value the knight is moving to. Last constraint: a square can only be visited once per trip (but a square does not necessarily have to be visited).

The official webpage distinguishes between optimal and trips which, while not optimal, are qualifying (i.e., A + B + C < 50).

To solve the puzzle, we developed a solver which generates random legal knight moves for each trip. The solver also produces combinations of ABC values that satisfy the constraint A + B + C < 50. It calculates the score corresponding to each combination of values for each path. When a score equals 2024, the solver verifies whether an opposing trip exists with the same values and score. If not, the trip is retained in memory until an opposing trip is found with the same values. However, if an opposing trip does exist, the two trips are evaluated against the global best score. If the total score is the lowest recorded, the combination of possible values is updated to ensure that their sum cannot exceed the best sum during the next exploration, thereby facilitating a rapid convergence toward the optimal solution.

It appears that the best values correspond to the minimal sum 6 (A: 1, B: 3, C: 2) and the minimal trip length in terms of number of visited squares is 30, like in this example, rendered by the solver:

The solver has also identified several suboptimal solutions, as depicted below:

Introduction

This puzzle, described here, encompasses various types of constraints:

• Row clues to which numbers, formed by concatenating adjacent digits from non-shaded cells, must comply (e.g., square numbers, palindromic numbers).

• Rules governing the relationship between digits and regions: every cell within a region must contain the same digit, and orthogonally adjacent cells in different regions must contain different digits.

• Locations of shaded cells, which need to be sparse and not result in single-digit numbers.

The difficulty of this multi-constraint puzzle comes from the fact that regions are ultimately defined by where the solver more or less arbitrarily places shaded cells.

Solution

We submitted a unique solution, 88243711283 (the sum of all the numbers), discovered by the solver on May 12.

Initial Failed Attempt: Use of a SMT Solver

We initially attempted to solve the puzzle by modeling it as an SMT problem to leverage the Z3 solver.

However, this strategy proved unsuccessful for several reasons:

1. Our solver is implemented in Go, and the main Z3 library available for this language is no longer maintained, leading to frequent crashes. Additionally, this library does not handle parallel calls well, making it unsuitable for concurrent operations.

2. Some of the clues posed significant challenges when modeled under these conditions. For example, how can digits be restricted to form one or multiple prime numbers raised to a prime number (3rd-row clue)? More generally, how can the relationship between shaded cells, regions, and numbers be modeled?

3. The solving process was slower than expected for easily modelable clues (e.g., generating square numbers), making it unfit to tackle the puzzle as a whole.

Due to these outcomes—probably not because of Z3 itself, but rather how we used it and the limitations of the Go library—we took a different approach. We chose an iterative solving approach, where:

• (identification) each row satisfying the constraints is identified, then

• (augmentation) added to rows that are above it and that satisfy their constraints, and

• (propagation) alongside its own constraints (digits assigned to regions potentially connected to the next row), propagated to a new instance of the solver (that is: a goroutine).

Successful Approach: Solve and Propagate Valid Rows

This second approach (essentially brute forcing but propagating the candidates forward instead of backtracking) proved successful. On a recent laptop, the solver—implementing this approach—traverses the entire valid (in terms of constraints) search space within approximately five seconds and finds the solution in about one second.

A significant factor contributing to this efficiency is how the puzzle’s constraints are constructed, enabling the solver to converge quickly toward the solution. The following diagram illustrates this by showing the number of valid candidates at each level: The initial level (first row) generates just six eligible candidates. The subsequent level produces 816 such candidates. Subsequently, however, the number of valid candidates merely fluctuates between 1 (for four levels) and 8 (for one level).

For this reason, a monothread implementation of this solver takes less than 30 seconds on similar hardware to solve. Thus, it retrospectively appears that parallelism is not crucial to solving this puzzle efficiently.

Solving Row by Row

The initial challenge was determining whether this puzzle could be solved iteratively, row by row (from top to bottom), or if a more global approach was necessary due to interdependencies between cells from different rows.

Initially, our intuition leaned towards requiring a global approach, given that the definition of regions in a row depends on the shaded cells in the rows above and, critically, below, as demonstrated in the following illustration:

In this example, the current row (indicated by the arrow) may ultimately have three (A, B, and C) or four (A, B, C, and C’)—or even more—regions based on the presence of a shaded cell at a specific position within the row below that disjoints (or not) the C cells. This information cannot be determined when the current row is being solved: it is undecidable as it depends on the placement of shaded cells on a row that has not been solved yet.

Upon further consideration, we discovered that the originally joined regions could be temporarily considered disjoint at a row level, meaning they should not necessarily have the same digits. In other words, cells originally belonging to the same region could be assigned to different regions when (a) a top row is not connected to them and (b) they are separated by a region or a shaded cell in the current row. In this example, the original regions A, B, C, become a, b, c, and d.

That way, if they are effectively connected by the below (once it is the turn for the solver to consider the row below), any digit assignment that would violate the rule according to which regions should have the same digits would be excluded. Conversely, if they are disjointed, the solver just rejects digit assignments that give these regions the same digit simply by using the heuristic that identifies identical orthogonally adjacent digits.

This insight supported the idea that the puzzle could be solved on a row-by-row basis.

Before discussing how the solver traverses the search space, we must first clarify the generation of shaded cells and how regions are encoded.

Generating the Shaded Cells

The solver represents each configuration of shaded cells as a mask (a bitfield; a slice of booleans) where an unset bit 0 represents a digit, and a set bit 1 represents a shaded cell. For example, 00100000100 corresponds to three groups of digits and two shaded cells: one at x-position 2 and another at x-position 8.

Valid masks must comply with these rules:

1. Unset bits cannot be isolated (that is: 01---, ---101---, or ---10) as the numbers must be at least 2 digits long.

2. Set bits cannot be horizontally adjacent as shaded cells are horizontally sparse (i.e., ---11---).

3. Set bits cannot be vertically adjacent as shaded cells are also vertically sparse (e.g., top row 0010010 is incompatible with the bottom row 0010000 because of the shaded cells at x-position 2).

The masks are trivially generated by a function that sets the Cartesian product of sets of set and unset bits and filters out masks that do not satisfy the first two rules (the third one—vertical sparsity—can only be enforced, by a different function, between two rows).

This function generates 54 valid masks per row.

Encoding the Regions

The original regions are defined in a configuration file and denoted using uppercase letters, ranging from A to M. Updated regions are defined row-by-row by updating the original regions based on a given configuration of shaded cells—as explained above. These regions are symbolized by characters that have not yet been utilized to represent the original regions (a to z, etc.).

Regions are updated as follows:

• Initially, when cells in the current row are connected with those in the preceding row—by considering the original regions—, the updated symbols representing these regions in the upper row are carried down to the lower row.

  

• Subsequently, unassigned regions are assigned symbols not used so far to assign regions. Cells belonging to the same original region and adjacent to each other are allocated the same symbol. They belong to distinct regions if originally connected but horizontally separated by a region or a shaded cell. This does not preclude them from possessing identical digits if it is subsequently determined that they are connected from below.

  

Solving Process

Our solver proceeds like this:

• (a) The solving function, initiated in a goroutine, begins by examining a prospective candidate containing 0 to 11 valid rows.

• (b) If the candidate contains 11 rows, terminate the present exploration since the candidate contains a solution. (Solved).

• (c) Otherwise, iterate through all viable arrangements of shaded cells (as described above).

• (d) If no such configurations remain, exit the current process: the candidate cannot possibly yield a solution.

• (e) Compare the present configuration of shaded cells with one in the uppermost row. If at least two pairs are vertically adjacent, return to step (c), as vertical sparsity is required for shaded cells.

• (f) Instantiate the current row based on the original regions and the current mask.

• (g) Generate the regions for the current row by first propagating the regions of interconnected cells from the uppermost row and, secondly, creating new regions for the remaining cells (as described above).

• (h) Reuse the digits used in the uppermost row for connected regions. For the row’s own regions, generate the Cartesian product of the sets containing digits from 0 to 9 so that each new region is assigned a digit.

• (i) If no new Cartesian product can be generated, go back to (c) to try a new mask.

• (j) If the generated digits result in at least one number starting with zero, revert to step (h) until all numbers start with nonzero digits.

• (k) Verify that digits from adjacent regions are orthogonally different, digits from the same region are identical, and that top row regions retrospectively connected to the current row are identical (see above). If not, return to step (h) until they meet this condition.

• (l) Ensure that the numbers comply with the row clue. If not, revert to step (h).

• (m) At this stage, the row satisfies all constraints (regarding regions, digits, and clues). Consequently, the candidate can be augmented with this new row.

• (n) A new solving goroutine is subsequently spawned to try solving the augmented candidate, starting from step (a).

Conclusion

While solving this puzzle, we had come to the conclusion that:

• The current Z3 library for Go is suboptimal. We would love to see a more comprehensive implementation for Go.

• It would also perhaps have been easier to use a language like Python with more robust SMT libraries. We believe some of the people who solved the puzzle may have used this approach.

• We implemented a multithreaded solver because Go makes it so convenient to use goroutines, but retrospectively, it was unnecessary.

• Solving parts of the puzzle and propagating augmented candidates forward is a good alternative to the classic tree exploration and backtracking combination, as it simplifies the programmatic flow in some cases, especially in a multithreading context.

The “Kryptos” sculpture, dedicated in 1990 by its creator Jim Sanborn, contains four encoded messages. Cryptography enthusiasts have successfully decoded three of them. However, the elusive fourth ciphertext, K4, remains undeciphered, presenting a fascinating challenge to the global community of cryptanalysts, even 34 years after its creation.

Since 2010, Sanborn has offered clues (1, 2), hoping someone will solve them. To date, these cribs—as they are referred to within the field of cryptanalysis—are:

• The 22^nd through 34^th positions correspond to EASTNORTHEAST,

• The 64^th through 74^th positions correspond to BERLINCLOCK.

Despite these known plaintext words, K4 has yet to be deciphered. While our attempts have not yielded a solution, we have just discovered a pattern that may provide valuable insights into the encryption method employed for K4.

We first share our findings as a bottom line. Then, we detail our approach and evaluate the relevance of this pattern.

The Signs of a Segmentation and Recombination Technique

We believe that:

• The letter ‘W’ acts as a separator that splits K4 into six segments. (We define a segment as a contiguous series of characters that can be deciphered into a partial or a complete sentence, as in this fictitious example: HLMKZJZDDSABW, a ciphertext segment, corresponding to BROWNFOXJUMPS, a plaintext segment.)

• Consequently, unlike other characters, the letter ‘W’ is not mapped to plaintext characters. Additionally, despite the ciphertext’s total length of 97 characters, the plaintext can contain no more than 91 characters (97 - 6 occurrences of ‘W’).

• Each segment belongs to either of two groups (Group A and Group B).

• The groups appear alternatively in the ciphertext: A|B|A|B|A|B.

• The groups have identical letter frequency distribution shapes for different letters. In each group,

  • 1 letter appears 5 times,

  • 2 letters appear 4 times each,

  • 4 letters appear 3 times each,

  • 5 letters appear 2 times each, and

  • 9 letters appear 1 time each.

Based on these findings, we assume that:

• The passage has been enciphered by a mechanism that preserves the position of the words (substitution cipher). Indeed, a transposition cipher would not be compatible with such a segmentation: the latter requires preserving the original positions.

• The transformation of the plaintext into the ciphertext has been carried out using one of the following methods, both illustrating a “segmentation and recombination” technique:

  1. (Hypothesis 1) The plaintext has first been enciphered (a) and then split into segments divided into two groups (b), adjusted to have the same letter frequency distribution shapes (c), and recombined with intermediary letters ‘W’ (d),

  2. (Hypothesis 2) The plaintext has first been split into segments divided into two groups (a), then each segment has been enciphered using a technique specific to its group (b); additionally, the groups have been adjusted to have the same letter frequency distribution shapes (c), before being recombined with intermediary letters ‘W’ (d). (Note that, in that case, we do not exclude the possibility that the steps (b) and (c) could have been performed in one step).

Our Approach

While studying Enigma (hypothesizing it would help solve K4), we learned that separators represented punctuation marks as the machine only supported alphabetic characters. For instance, the doublet “YY” represented a dot. These separators were used for plaintext because of a technical constraint. We pondered if this technique could have been replicated in ciphertext to misdirect analysts.

To systematically explore this idea, we created K4nundrum.

This open-source tool splits K4 on different separators and analyzes the resulting segments. K4nundrum explores separators ranging from ‘A’ to ‘Z’ and uses them to split the ciphertext, subsequently permuting the resulting segments. Every permutation is distributed into different groups (from two groups for the whole ciphertext, up to one group per segment). Ultimately, K4nundrum examines each collection of groups to ascertain whether they exhibit identical letter distribution characteristics.

Using this process, K4nundrum unveiled that splitting K4 based on the letter ‘W’ produces groups of segments with identical letter frequency distribution shapes when permuted according to a specific arrangement (alternating groups). As mentioned above, these two groups are { INFBNYPVTTMZFPK, OBKRUOXOGHULBSOLIFBB, TQSJQSSEKZZ } and { ATJKLUDIA, FLRVQQPRNGKSSOT, GDKZXTJCDIGKUHUAUEKCAR }.

An Intentional Pattern?

This pattern is interesting: it is nonobvious, which would explain why it protected K4’s plaintext so well. At the same time, it would have been relatively easy for the creator of Kryptos to implement (we attempt to demonstrate it with an illustrative example below). But couldn’t it be just coincidental?

To understand how frequently such a pattern would occur in a string of text the length of K4, we have added a feature to K4nundrum that simulates “pseudo-K4s” (enabled with the --sim flag). These pseudo-K4s are random strings of 97 uppercase letters. (By definition, we cannot create a process that would generate perfect K4-like strings, as the underlying encryption method is unknown). While these strings are not generated like the original K4, this simulation gives a general indication of how frequently this phenomenon can be noticeable by an external observer.

Simulations empirically indicate, based on the generation of more than one million pseudo-K4s, that this pattern naturally emerges less than 2% of the time. This probability is low but does not necessarily make this hypothesis a reality. Fortunately, several indicators demonstrate that K4’s pattern has characteristics that make it less likely to occur just by chance.

First, the simulator tends to produce groups containing tiny segments (1 or 2 characters long). For instance, and typically:

> GKUCPFTQBJCKSEVMFWRLWKPEKCZHFFYNPLZZINWAFVKZEMJNRCAUYHUMITLKEZKQGZKBG
PIRRPUGKVSMHXASUIYPVJBZKFUKX

  Group 1:           BGPIRRPUG FU G PE QGZ UCPFTQBJC X ZEMJNRCAUYHUMITL
  Letter Freq.:      U:5 G:4 P:4 R:3 C:3 E:2 B:2 I:2 F:2 Z:2 M:2 J:2 
                     T:2 Q:2 H:1 L:1 X:1 N:1 Y:1 A:1

  Group 2:           CZHFFYNPLZZINWAFV EZ SEVMFWRLW VSMHXASUIYPVJBZ
  Letter Freq.:      Z:5 V:4 F:4 W:3 S:3 H:2 E:2 P:2 L:2 M:2 N:2 Y:2 
                     I:2 A:2 J:1 X:1 R:1 U:1 C:1 B:1

The random string produces segments such as ‘EZ’ or ‘G’ in this example.

If we assume that segments represent sequences of whole words, it is implausible that the creator of a ciphertext would use small segments. They would additionally needlessly add complexity to the encryption process. When the simulator excludes tiny segments (less than three characters), ~0.5% of the pseudo-K4 passages can be split into appropriately sized groups with identical distribution shapes.

Second, K4’s groups alternate. This additional constraint is even more rarely visible in the segments generated by the simulator: ~0.07% of the pseudo-K4s alternate. When all these constraints are considered (identical letter frequency distribution shapes, appropriately sized segments, and alternating groups), only ~0.05% of the generated strings are similar to K4. One way of putting it is that if we consider that this pattern happens merely by chance, on average, Sanborn would have had to create 2,000 Kryptos—each encoding a different message—so that one K4-like pattern would have been noticeable to an external observer.

Third, when we superpose the cribs with K4, the plaintext words are immediately adjacent to other segments. In other words, when mapped to the ciphertext, their first or last letter is directly next to a ‘W’: they start or end a segment. This observation could support the idea of segments representing sequences of whole words.

Suppose we assume that segments are to be deciphered to (partial) sentences with complete words (the second segment, consisting of 15 characters, could, for instance, mean “… east-northeast of …”). In that case, the probability that such a pattern would emerge just by chance becomes even lower. Speculatively, it may be significant that Jim Sanborn, anxious to see his work finally deciphered, revealed words belonging to two different groups (BERLINCLOCK belonging to Group A, and EASTNORTHEAST to Group B)—as a kind of meta-clue.

For these reasons, a coincidence, while not impossible, seems very unlikely. In other words, we are confident that this pattern is intentional.

A Technique to Mask Letter Frequencies

Fabricating Letter Frequencies

Why would Sanborn use such a technique?

First, as mentioned above, a segmenting and recombining technique conveniently produces nonobvious patterns. It is asymmetrical: it does not require a deep knowledge of cryptographic methods for its implementation; at the same time, it easily defeats classic cryptanalysis attacks by adding characters—the separators—that are not mapped to the plaintext and misleading people to think that the ciphertext must be analyzed as a whole. In the context of K4, this is all the truer because the passage is so brief.

Second, this technique can mask the letter frequency by combining multiple letter frequencies. Let’s take ‘K’ for example. It appears eight times in the original K4. By analyzing it under the lens of the segmentation technique, these occurrences result from combining three occurrences in Group A and five in Group B. By adjusting the letter frequency of one (or both) of the groups, it becomes possible to fabricate a new letter frequency distribution. Additionally, the letter separator pollutes the letter frequency distribution.

We do not know yet the exact mechanism employed to alter the distributions of K4’s groups. It is, however, important not to assume that identical distribution shapes necessarily mean that the deciphered segments of Group A contain the same letters as the ones of Group B (which would be very unlikely for a natural language). On one hand, this mechanism could have been applied to encrypted segments (second-order encoding). On the other hand, intentional spelling errors, similar to other passages like UNDERGRUUND (K2) or DESPARATLY (K3), could have been introduced to slightly modify the letter frequencies. With this simple technique, one letter’s occurrences are decremented while another’s are incremented. For example, IQLUSION (K1) decrements the Ls while incrementing the Qs. Similarly, superfluous letters could have been introduced in the style of XCANYOUSEEANYTHINGQ (K3).

Illustration: Let’s Encrypt Alice

We can test our hypothesis by applying this technique to a fictitious example.

Let’s say we want to encrypt this passage from Alice’s Adventures in Wonderland:

“You promised to tell me your history, you know,” said Alice.

The first step is to transform the sentence by removing the punctuation marks and the spaces and by capitalizing the letters:

Then, the sentence is split in half and segmented, with segments alternated between two groups. Last, the letter frequencies of the respective groups are computed.

The distributions are dissimilar. We want to reduce this difference. To that end, we encode the first group using standard Vigenère (obviously, we could have used a different cipher) and try several keys to improve the situation. The key SECRET quickly emerges as a good candidate (“YOUPROMISEDTO YOUR YOUKNOW” becomes “QSWGVHEMUVHMG CQLV RGYMESP”):

The letter frequency distribution shapes are now similar but have yet to be identical. We need to adjust the second group to make them match. In this case, it is just a matter of adding letters (as the second group is shorter than the first). “TELLME HISTORY SAIDALICE” becomes “YTELLME HISTORY SAIDALICEX”. The original plaintext segments remain intelligible.

We obfuscate the second group for demonstration by merely applying a trivial ROT13 cipher, which preserves the distribution shapes (“YTELLME HISTORY SAIDALICEX” becomes “LGRYYZR UVFGBEL FNVQNYVPRK”).

The shapes of the distributions are identical. The last step is to identify a letter not present in the ciphertext segments (‘X’, for instance) and recombine the segments by alternating them:

The outcome is a ciphertext that cannot be attacked with classic methods because of the fabricated letter frequency.

Importantly, this example demonstrates that the two plaintext groups are not required to contain the same letters. Also, the separator that combines the segments is one of the most frequent letters in the ciphertext, even though it is the odd one out.

Next Steps

Once the technique is known, the separator is identified, and the correct groups are determined, deciphering this kind of code is a matter of understanding how each group has been “encoded” before attacking it with classic cryptanalysis methods.

This is the path we would like to follow to crack K4.

• Part I: https://guyomel.hashnode.dev/is-artificial-intelligence-suitable-to-solve-cains-jawbone-i

• Part II: https://guyomel.hashnode.dev/is-artificial-intelligence-suitable-to-solve-cains-jawbone-part-ii

We have now reached the conclusion of our initial novice examination into the role of artificial intelligence in solving—or, at the very least, aiding in the resolution of—Cain’s Jawbone. An article summarizing this research’s findings will be available on glthr.com shortly. In the meantime, we share the latest results here.

Driven by the excellent performance of the binary classification of pages belonging to one of the narrators (Bill Hardy), we tried to extend it using multiclass classification. This attempt resulted in abysmal performance, almost undistinguishable from randomness, as demonstrated by this confusion matrix (each label corresponds to a narrator).

Subsequently, we performed a binary classification of all narrators to identify the root causes of this duality between excellent binary classification and poor multiclass classification. To that end, we trained a BERT model for each narrator, computed their confusion matrix, and represented the incorrect predictions.

Here are the results:

The models perform poorly for all narrators except Bill Hardy and, to a lesser extent, Henry the Dog. These results confirm the initial K-means analysis.

To explain Bill Hardy’s model’s performance, we suggest that Bill’s pages have a textual characteristic that makes them easily identifiable. This characteristic is grammatical: Bill Hardy is the only narrator who narrates in the present tense. All other narrators narrate in the past tense.

This hypothesis was evaluated by grammatically transforming some of the narratives from various authors into the present tense and altering a portion of Bill’s narratives to be in the past tense. Bill’s model erroneously labels these pages, categorizing non-Bill pages written in the present tense as belonging to Bill. It also categorizes Bill’s pages written in the past tense as not belonging to Bill.

This experiment demonstrates that we ultimately trained this model to identify whether pages are mainly written in the present tense. In other words, Bill’s model probably represents a function that produces primarily results based on the grammatical tense predominant on a given page.

The way Henry’s (or also John Walker’s) model operates is more puzzling. We have not identified an apparent textual characteristic that would explain its performance. Readers of this article may be interested in figuring it out—please let us know!

In conclusion, in the context of this project, we failed to create a model that autonomously assigns the pages to their respective narrators. Several factors may explain why.

The first reason is that this is our first machine-learning project: a more experienced reader would certainly have trained a more robust model.

However, we do not think a model can achieve the primary goal for the reasons evoked in the first article. A significant issue is the fact that most of the clues are obscure, extratextual, and highly subject to subjective interpretation. For instance:

• “there was the oldest brass in England, saying: SIRE:IOHAN:DAUBERNOUN:CHIVALER:GIST:ICY:DEV:DE:SA:ALME:EYT:MERCY”, referring to Stoke d’Abernon, Surrey (extratextual reference)

• “Apparently the person who slept in the lock-up at that county town on the Severn, or perhaps woke, would hear this time,” meaning 8 AM (obscure reference)

• “I sat and faced the old school colours frozen there before me? Green and white and rose, grit, wisdom and reliability, the find old Head, as we called him, had quipped it,” a reference to a frozen dessert (highly subjective interpretation)

To be a suitable candidate for solving this puzzle, a model should be able to analyze and contextualize both textual and extratextual clues.

But that would not be enough.

An ideal model should also perform second-order reasoning to characterize the interactions between the narrators. That is: going beyond the mere labelling of text tokens. For instance, while a good model would identify the existence of spoonerism in Jasmine’s monologue (e.g., “Useful, courteous little chip of a bat”, page 27), a great model would be able to identify when a different narrator refers to Jasmine’s spoonerisms: “He said he’d put that right, but he couldn’t find the silly old jossers, as Jasmine might so easily have called them” (page 99).

Although we cannot categorically rule out the use of machine learning in addressing this puzzle, we would be eager to observe how others approach this issue. It is worth noting that there are still many potential avenues for exploring the application of machine learning to Cain’s Jawbone.