atom.xml

<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">

 <title>kivikakk.ee</title>
 <link href="/atom.xml" rel="self"/>
 <link href="https://kivikakk.ee/"/>
 <updated>2025-01-04T22:46:19+11:00</updated>
 <id>https://kivikakk.ee</id>
 <author>
   <name>Asherah Connor</name>
   <email>ashe@kivikakk.ee</email>
 </author>

 
 <entry>
   <title>Nix humour</title>
   <updated>2025-01-04T00:00:00+11:00</updated>
   <id>https://kivikakk.ee/2025/01/04/nix-humour</id>
    
     <link href="https://kivikakk.ee/2025/01/04/nix-humour/"/>
     <content type="html">&lt;p&gt;Something about this really fucks.&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/post-img/expected-scenario.png&quot; alt=&quot;Screenshot of an excerpt from a NixOS wiki page: &amp;quot;Tip - The above assumes
you have root access on the remote machine, that allows you to create such a
weakly priviledged user, or that you have good relationship with your system
administrator - which is the expected scenario if they have installed Nix
system-wide.&amp;quot;&quot; /&gt;&lt;/p&gt;
</content>
    
 </entry>
 
 <entry>
   <title>Fear</title>
   <updated>2025-01-03T00:00:00+11:00</updated>
   <id>https://kivikakk.ee/2025/01/03/fear</id>
    
     <link href="https://kivikakk.ee/2025/01/03/fear/"/>
     <content type="html">&lt;p&gt;content note: physical illness, death, suicide.&lt;/p&gt;

&lt;p&gt;one thing about spending enough time fearing death, for whatever reason and
reasons that may be, is that one end to it is not removing the originator of
that fear — which depending on your psyche or conditions may not always be
possible — but learning to not fear death instead.&lt;/p&gt;

&lt;p&gt;i basically think anyone who spends enough time in panic over enough years
is liable to end up this way. the alternatives are burning to a husk, loss of
reality checking, or suicide.&lt;/p&gt;

&lt;p&gt;personally still a bit afraid — not so indifferent that i’d go hard for no
reason when exercising yesterday, for instance, i give my body what it asks
for (fluid movements, light effort, nothing anaerobic), but not so afraid that
i won’t do my best to purposefully trigger the new arrhythmia when doing the
holter monitor next week, despite risk inherent in inducing one.&lt;/p&gt;

&lt;p&gt;i find i keep having to say to myself, if this is all it would take to kill
me, then now or in two weeks accidentally say, it really makes no odds. that
i have to deploy this regularly to move forward with my days evinces a certain
something. and while the uncertainty itself is one degree of unbearable, a kind
of assertion that i cannot trust in my self-knowledge, the collective absence of
faith from others is another.&lt;/p&gt;

&lt;p&gt;if the condition finally gets worse after decades, if everyone’s been telling
you it’s false/“Just Anxiety”/psychosomatic/“functional” (given decades of panic
disorder etc.), and now it’s enough to be visible, it’s a relief and starts to
feel your only ally — the only true thing in all this time. i don’t know if it’s
true that the same death i feared and felt daily in my mid-teens until my early
20s &lt;em&gt;is&lt;/em&gt; this one, the one now manifesting so physically, or the one that stole
a few years away from me from 2016, but on reflection i’ve spent most of the
last 22 years actively fearing and feeling death.&lt;/p&gt;

&lt;p&gt;an end to this uncertainty and the loneliness within it is not exactly an
unwelcome thing in and of itself. we were never promised even one day of the
beauty of life, y’know? and i have experienced so much of it, awe-inspiring and
terrible and banal. nothing was ever ever promised.&lt;/p&gt;

</content>
    
 </entry>
 
 <entry>
   <title>2025</title>
   <updated>2025-01-02T00:00:00+11:00</updated>
   <id>https://kivikakk.ee/2025/01/02/2025</id>
    
     <link href="https://kivikakk.ee/2025/01/02/2025/"/>
     <content type="html">&lt;p&gt;Happy new year!&lt;/p&gt;

&lt;p&gt;I guess I’m taking a slightly more laid-back approach to goals this year.
&lt;a href=&quot;https://kivikakk.ee/2024/10/13/led/&quot;&gt;Health issues&lt;/a&gt; have unfortunately progressed; whatever pain relief I
was getting from the duloxetine (or otherwise), either that effect has worn
off or the pain levels have risen to break through.&lt;/p&gt;

&lt;p&gt;I’ve also encountered a new arrhythmia, so “sudden cardiac death” is back on the
menu of dread. Holter monitor next week where I’ll try to reproduce that without
dying. Rheumatologist follow-up appointment where I expect to be diagnosed with
hEDS in … 12 weeks.  So it goes.&lt;/p&gt;

&lt;p&gt;Goals, then.&lt;/p&gt;

&lt;h3 id=&quot;keep-up-duolingo&quot;&gt;Keep up Duolingo.&lt;/h3&gt;

&lt;p&gt;866 days now; the streak isn’t the important thing, but it has been a really
good lesson in proving to myself I can stay consistent with things given
the right motivation. (In this case, being able to speak my partner’s native
language.)&lt;/p&gt;

&lt;p&gt;Duolingo isn’t for everyone, and the course quality varies wildly, but the
Spanish course for English speakers is incredible, and if you spend 15-20
minutes a day on it, and let it be your fallback activity when bored, you will
steadily but surely gain an extremely real facility with the language.&lt;/p&gt;

&lt;p&gt;(This much time is also enough to reliably sit around Obsidian/Diamond league,
and sometimes it can be a nice dopamine hit to make an effort to finish at #1.)&lt;/p&gt;

&lt;p&gt;I have the benefit of being around a native speaker all the time, but the first
year or so my level wasn’t anywhere near being able to use it casually without
it getting annoying for both of us. By now, we can switch in and out pretty
comfortably around the topics I’ve covered lexicon for.&lt;/p&gt;

&lt;p&gt;I’ve also added Portuguese on the side, spending roughly half my time each day
on each, and it works pretty well! I wouldn’t start two at once (especially
closely related languages like these; too confusing), but being a few years in
with one makes it an easy way to bootstrap another habit.  Português é estranho,
mas eu gosto :)  It’s a much shorter and less developed course than the Spanish
one, so I expect I’ll finish it this year and start another in its stead.&lt;/p&gt;

&lt;h3 id=&quot;daily-meditation&quot;&gt;Daily meditation.&lt;/h3&gt;

&lt;p&gt;Planning to slot this in with my Duolingo practice.  I’ve been an on-and-off
meditator for 20 years or so now.  I need the equanimity now more than ever.&lt;/p&gt;

&lt;h3 id=&quot;home-cooked-meals&quot;&gt;Home-cooked meals.&lt;/h3&gt;

&lt;p&gt;This has been a real struggle over the last year or two, with energy levels so
low (and no microwave in the Tallinn apartment!), but since moving into my new
place back in Melbourne it’s been a key focus.  Just need to keep it up.&lt;/p&gt;

&lt;h3 id=&quot;deliver-the-ava-mvp&quot;&gt;Deliver the Ava MVP.&lt;/h3&gt;

&lt;p&gt;Charlotte’s been working non-stop on &lt;a href=&quot;https://github.com/charlottia/ava&quot;&gt;Ava BASIC&lt;/a&gt; (&lt;a href=&quot;https://kivikakk.ee/zig/2024/08/06/t%C3%BC%C3%BCpiautomaat/&quot;&gt;previously&lt;/a&gt;) since
July, and hopefully we can get that into a complete MVP this year.&lt;/p&gt;

&lt;h3 id=&quot;secondary-project&quot;&gt;Secondary project.&lt;/h3&gt;

&lt;p&gt;Having an “off” project would be nice.  This is a vaguer goal, but I get the
feeling technical writing would be a good fit for us.&lt;/p&gt;

&lt;h3 id=&quot;keep-learning&quot;&gt;Keep learning.&lt;/h3&gt;

&lt;p&gt;So far I have an unsorted list of things I’d like to learn or improve at:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Formal methods in digital design. We’ve played around with this a bit (e.g. with
&lt;a href=&quot;https://github.com/robertbaruch/amaranth-exercises&quot;&gt;Robert Baruch’s Amaranth exercises&lt;/a&gt;; solving Sudoku with
HDL and formal was a cool and unexpected twist), but not enough to usefully use it
in actual designs.&lt;/li&gt;
  &lt;li&gt;Likewise, something like &lt;a href=&quot;https://lean-lang.org/&quot;&gt;Lean&lt;/a&gt; and perhaps &lt;a href=&quot;https://github.com/informalsystems/quint&quot;&gt;Quint&lt;/a&gt;.&lt;/li&gt;
  &lt;li&gt;Stenography with &lt;a href=&quot;http://www.openstenoproject.org/&quot;&gt;Open Steno&lt;/a&gt;.&lt;/li&gt;
  &lt;li&gt;Training on the Africa Twin; I’ve been rebuilding confidence since a fall
after a tyre replacement early last year, but I’d like some controlled
conditions to push the envelope a bit.&lt;/li&gt;
&lt;/ul&gt;

</content>
    
 </entry>
 
 <entry>
   <title>StaleObjectStateException</title>
   <updated>2024-12-30T00:00:00+11:00</updated>
   <id>https://kivikakk.ee/2024/12/30/StaleObjectStateException</id>
    
     <link href="https://kivikakk.ee/2024/12/30/StaleObjectStateException/"/>
     <content type="html">&lt;p&gt;&lt;img src=&quot;/assets/post-img/StaleObjectStateException.jpg&quot; alt=&quot;Photograph of a Woolworths self-checkout machine, with an error message:
&amp;quot;Transaction concurrency error (StaleObjectStateException), Please retry&amp;quot;&quot; /&gt;&lt;/p&gt;

</content>
    
 </entry>
 
 <entry>
   <title>lottia notes continues</title>
   <updated>2024-12-24T00:00:00+11:00</updated>
   <id>https://kivikakk.ee/2024/12/24/lottia-notes-continues</id>
    
     <link href="https://kivikakk.ee/2024/12/24/lottia-notes-continues/"/>
     <content type="html">&lt;p&gt;A follow-up to &lt;a href=&quot;https://kivikakk.ee/2024/01/03/lottia-notes/&quot;&gt;lottia notes&lt;/a&gt;:&lt;/p&gt;

&lt;p&gt;Charlotte continues to write &lt;a href=&quot;https://lottia.net/notes/&quot;&gt;there&lt;/a&gt; (though about more topics now, including Nix
and Git/jj), and I’m considering her readership bootstrapped after a year of
crossposts.  &lt;a href=&quot;https://lottia.net/notes/atom.xml&quot;&gt;Her Atom feed is linked&lt;/a&gt;.&lt;/p&gt;

</content>
    
 </entry>
 
 <entry>
   <title>Comptime, flow, and exhaustiveness</title>
   <updated>2024-12-17T21:02:36+11:00</updated>
   <id>https://kivikakk.ee/2024/12/17/comptime-flow-exhaustiveness</id>
    
      <link href="https://lottia.net/notes/0015-comptime-flow-exhaustiveness.html"/>
      <content type="html">Make your own exhaustiveness rules up.</content>
    
 </entry>
 
 <entry>
   <title>1.2E+2 Volts (AC)</title>
   <updated>2024-12-15T00:00:00+11:00</updated>
   <id>https://kivikakk.ee/2024/12/15/12e2v</id>
    
     <link href="https://kivikakk.ee/2024/12/15/12e2v/"/>
     <content type="html">&lt;p&gt;&lt;img src=&quot;/assets/post-img/12e2v.png&quot; alt=&quot;Amazon entry excerpt which reads: &amp;quot;Voltage: 1.2E+2 Volts (AC)&amp;quot;&quot; /&gt;&lt;/p&gt;
</content>
    
 </entry>
 
 <entry>
   <title>Platform-specific hacks for high-DPI</title>
   <updated>2024-11-24T22:20:00+11:00</updated>
   <id>https://kivikakk.ee/2024/11/24/hidpi-platform-specific-hacks</id>
    
      <link href="https://lottia.net/notes/0014-hidpi-platform-specific-hacks.html"/>
      <content type="html">We need to avoid getting scaled by the compositor, to preserve our sharp edge.</content>
    
 </entry>
 
 <entry>
   <title>Git and jujutsu: in miniature</title>
   <updated>2024-11-09T14:30:00+11:00</updated>
   <id>https://kivikakk.ee/2024/11/09/git-jujutsu-miniature</id>
    
      <link href="https://lottia.net/notes/0013-git-jujutsu-miniature.html"/>
      <content type="html">A concrete example of Git and jujutsu usage compared.</content>
    
 </entry>
 
 <entry>
   <title>Soft skills: jujutsu early feelings</title>
   <updated>2024-11-07T17:55:00+11:00</updated>
   <id>https://kivikakk.ee/2024/11/07/soft-skills</id>
    
      <link href="https://lottia.net/notes/0012-soft-skills.html"/>
      <content type="html">Launching into jujutsu.</content>
    
 </entry>
 
 <entry>
   <title>Non-intrusive vtable</title>
   <updated>2024-10-29T19:06:00+11:00</updated>
   <id>https://kivikakk.ee/2024/10/29/non-intrusive-vtable</id>
    
      <link href="https://lottia.net/notes/0011-non-intrusive-vtable.html"/>
      <content type="html">OOP is the world's most consistently-used term in any field of any kind, ever.</content>
    
 </entry>
 
 <entry>
   <title>Reply to email, October 15</title>
   <updated>2024-10-29T00:00:00+11:00</updated>
   <id>https://kivikakk.ee/2024/10/29/reply-to-email-october-15</id>
    
     <link href="https://kivikakk.ee/2024/10/29/reply-to-email-october-15/"/>
     <content type="html">&lt;p&gt;The sender address bounced today, so I’m just publishing my reply here. Hope you
see it! :)&lt;/p&gt;

&lt;hr /&gt;

&lt;p&gt;Hiya &amp;lt;3&lt;/p&gt;

&lt;p&gt;Thank you very much for your email; it was really lovely to receive. (Markdown
ecosystem work is not very rewarding a lot of the time, lol :p)&lt;/p&gt;

&lt;p&gt;But yeah — I don’t think there’s anything in particular you could do, and
that’s quite OK! The thought by itself is enough, thank you :) Financially I’m
completely fine. So I hope you get that Clojure gig and get money to make your
own life even cooler! (Throwback to that time I wrote a
&lt;a href=&quot;https://web.archive.org/web/20130423114048/http://rouge.io/&quot;&gt;Clojure impl in Ruby!&lt;/a&gt;)&lt;/p&gt;

&lt;p&gt;Be well &amp;lt;3&lt;/p&gt;

</content>
    
 </entry>
 
 <entry>
   <title>Peak performance</title>
   <updated>2024-10-28T00:00:00+11:00</updated>
   <id>https://kivikakk.ee/2024/10/28/peak-performance</id>
    
     <link href="https://kivikakk.ee/2024/10/28/peak-performance/"/>
     <content type="html">&lt;p&gt;&lt;img alt=&quot;A small bar graph from GitHub, titled &amp;quot;Languages&amp;quot;. The legend shows 81% Nix, 18.4% Shell, 0.6% HTML.&quot; src=&quot;/assets/post-img/peakperformance.png&quot; width=&quot;336&quot; /&gt;&lt;/p&gt;

&lt;p&gt;You may not like it, but this is what peak performance looks like.&lt;/p&gt;
</content>
    
 </entry>
 
 <entry>
   <title>Led</title>
   <updated>2024-10-13T18:13:00+11:00</updated>
   <id>https://kivikakk.ee/2024/10/13/led</id>
    
     <link href="https://kivikakk.ee/2024/10/13/led/"/>
     <content type="html">&lt;p&gt;Content note: pain, bad mental health, health issues.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;/eesti/2024/04/11/early-nyonks/&quot;&gt;Six months later&lt;/a&gt;, we’re back. It’s sad — we were acclimating well,
met our neighbours, started riding all over the country, and then I started to
be in pain.&lt;/p&gt;

&lt;p&gt;Any kind of pain is bad, but the body part(s) affected really define the flavour
of the impact it has, I guess, and this has been the first time I’ve had really
significant leg pain.  That’s sort of underselling it; it’s sometimes in my
knees, usually ankles, feet, and though it seems like it shouldn’t be related,
also in my elbows, fingertips, and even the sides of my neck, and it all feels
very much related.&lt;/p&gt;

&lt;p&gt;Still, for the most part it’s in my lower legs, and it. is. constant. I went
back on nifedipine, since a Raynaud’s “diagnosis” last year had me try that
and it sort of helped during winter. My hands and feet are always really cold,
and they were worse than usual, so I was hopeful it’d help. Estonia only had
the &lt;a href=&quot;https://www.health.nsw.gov.au/sabs/Documents/2021-sn-001.pdf&quot;&gt;“medium acting”&lt;/a&gt; variant, and it felt a lot bumpier than the
modified-release ones in Australia.&lt;/p&gt;

&lt;p&gt;After that didn’t help at all, it was time to see the doctor. And see the
doctor I did. And again, and again. And a physiotherapist. And an orthopaedist.
Two X-rays, two different kinds of strong anti-inflammatory, three weeks of
gabapentin, a month and a half of physio, and three months later in total and
not even the slightest improvement. Exercising didn’t help, resting didn’t help.
Ruled out referred back pain, inflammation, neuropathy, muscular.&lt;/p&gt;

&lt;p&gt;Meanwhile, the time between appointments started to stretch out, I guess as
folks returned from summer holidays, and I was just miserable. I hadn’t been
able to do much of anything in a long time, and the &lt;a href=&quot;https://lobste.rs/s/jqfp5d/what_are_you_doing_this_week#c_hihjcw&quot;&gt;healthcare system in
Estonia&lt;/a&gt; was really making it hard. The pain on some days made me want
to beat whichever leg was hurting into a pulp, because that somehow was more
tractable and bearable.&lt;/p&gt;

&lt;p&gt;So we’re back. I’ve had some relief since, in being able to switch back to
the modified-release nifedipine, as well as get back on an antidepressant —
duloxetine, both for the general sense of despair I’ve been accumulating this
last quarter of a year (!), and for its help with ?fibromyalgia. The pain is a
little bit more distant, but it’s still there and still getting worse.&lt;/p&gt;

&lt;p&gt;Finally getting some blood tests confirmed it’s not something easy (anaemia,
B12 deficiency) or obvious (cancer, RF+ RA). My money is on COVID-related
vasculopathy, but I guess it’s not so easy to confirm, deny, or help. With some
amazing luck, I managed to score an early rheumatologist appointment (someone
had just cancelled), which then got bumped forward another week, so that’s now
tomorrow.&lt;/p&gt;

&lt;p&gt;I can’t particularly have my hopes up — we’re up to 4 months now since this
started to intrude on my life. &lt;a href=&quot;/2016/08/22/illness-1.html&quot;&gt;Last time I had a new chronic illness&lt;/a&gt;, it took more than a year for it to start subsiding regularly enough that
I could start to believe it might abate entirely. (It came back for months at a
time, years later.)&lt;/p&gt;

&lt;p&gt;And unlike that time, this one isn’t only in my head: my legs and feet are
increasingly randomly surfacing bruises that don’t clear up for &lt;em&gt;months&lt;/em&gt; at
a time. A few days ago while out to therapy, there was a particularly sharp
ache across the top of my left foot, and when I got home, I took my shoe off
and found that a vein had just turned completely black in a spot right there.
There’s a blue patch on the back of my right leg (the one that hurts less on
average!) that hasn’t gone away in the whole four months. And as my doctor here
said, it’s entirely possible that at the end of all our investigations, I’ll
still have this pain to manage, maybe indefinitely.&lt;/p&gt;

&lt;p&gt;Which, y’know? That kind of thing just happens in life.&lt;/p&gt;
</content>
    
 </entry>
 
 <entry>
   <title>RSS autodiscovery</title>
   <updated>2024-10-13T00:00:00+11:00</updated>
   <id>https://kivikakk.ee/2024/10/13/rss-autodiscovery</id>
    
     <link href="https://kivikakk.ee/2024/10/13/rss-autodiscovery/"/>
     <content type="html">&lt;p&gt;It’s come to my attention that some folks don’t know my blog has an &lt;a href=&quot;https://kivikakk.ee/atom.xml&quot;&gt;RSS feed&lt;/a&gt;!
This is kind of astonishing to me as it suggests people actually manually check
blogs for new posts sometimes?!&lt;/p&gt;

&lt;p&gt;At any rate, it does, it’s just not linked anywhere explicitly — but it is
supported via &lt;a href=&quot;https://dri.es/rss-auto-discovery&quot;&gt;RSS autodiscovery&lt;/a&gt;, which I just kind of .. assumed everyone
used, relied on, knew about? But apparently not!&lt;/p&gt;

&lt;p&gt;I guess I’ll add a link, just in case. :)&lt;/p&gt;

</content>
    
 </entry>
 
 <entry>
   <title>Tüüpiautomaat: automated testing of a compiler and target against each other</title>
   <updated>2024-08-06T00:00:00+10:00</updated>
   <id>https://kivikakk.ee/zig/2024/08/06/tüüpiautomaat</id>
    
     <link href="https://kivikakk.ee/zig/2024/08/06/t%C3%BC%C3%BCpiautomaat/"/>
     <content type="html">&lt;p&gt;I’m currently working on a bit of a &lt;a href=&quot;https://github.com/charlottia/ava&quot;&gt;long-winded project&lt;/a&gt; to recapture
&lt;a href=&quot;https://en.wikipedia.org/wiki/QuickBASIC&quot;&gt;QuickBASIC&lt;/a&gt;, the environment I learned to program in, in a bit of
a novel way, while remaining highly faithful to the original. (I actually
started a related project &lt;a href=&quot;https://github.com/kivikakk/kyuubey&quot;&gt;9 years ago&lt;/a&gt; (!), and I’ll be able to reuse
some of what I did back then!)&lt;/p&gt;

&lt;p&gt;The new project involves compiling the BASIC down into a bytecode suitable for
a stack machine, and an implementation of such a stack machine. Importantly,
there’ll soon be a second implementation which runs on a different architecture
entirely: namely, it’ll be an implementation &lt;em&gt;of&lt;/em&gt; this architecture on an FPGA.
So the machine needs to be reasonably easy to implement in hardware — if it
simplifies the gateware design, I’ll happily take tradeoffs that involve making the
compiler more complicated, or the ISA more verbose.&lt;/p&gt;

&lt;p&gt;The ISA started out with opcodes like:&lt;/p&gt;

&lt;div class=&quot;language-zig highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;k&quot;&gt;pub&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;const&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;Opcode&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;enum&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;kt&quot;&gt;u8&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;PUSH_IMM_INTEGER&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;0x01&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;PUSH_IMM_LONG&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;0x02&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;PUSH_IMM_SINGLE&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;0x03&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;PUSH_IMM_DOUBLE&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;0x04&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;PUSH_IMM_STRING&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;0x05&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;PUSH_VARIABLE&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;0x0a&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
    &lt;span class=&quot;c&quot;&gt;// [...]&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;OPERATOR_ADD&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;0xd0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;OPERATOR_MULTIPLY&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;0xd1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;OPERATOR_NEGATE&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;0xd2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
    &lt;span class=&quot;c&quot;&gt;// [...]&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;};&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;For the binary operations, the stack machine would pop off two elements, and then
look at the types of those elements to determine how to add them. This is easy to
do in Zig, but this is giving our core a lot of work to do — especially when we
consider how extensive the rules involved are:&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/post-img/typeconversions.png&quot; alt=&quot;Screenshot of QuickBASIC 4.5 open to the &amp;quot;Type Conversions&amp;quot; help page.
It&apos;s scrolled some way down and yet the screen is full of text describing how
operands are converted to the same degree of precision, and further how result
types may affect that.&quot; /&gt;&lt;/p&gt;

&lt;p&gt;Ideally, the executing machine doesn’t &lt;em&gt;ever&lt;/em&gt; need to check the type of a value
on the stack before doing something with it — the bytecode itself can describe
whatever needs to happen instead.&lt;/p&gt;

&lt;p&gt;In other words, when compiling this&lt;sup id=&quot;fnref:highlighting&quot; role=&quot;doc-noteref&quot;&gt;&lt;a href=&quot;#fn:highlighting&quot; class=&quot;footnote&quot; rel=&quot;footnote&quot;&gt;1&lt;/a&gt;&lt;/sup&gt;:&lt;/p&gt;

&lt;div class=&quot;language-zig highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;n&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;%&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;42&lt;/span&gt;       &lt;span class=&quot;err&quot;&gt;&apos;&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;This&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;is&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;an&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;INTEGER&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;-&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;32768&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;..&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;32767&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;).&lt;/span&gt;
&lt;span class=&quot;py&quot;&gt;b&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;&amp;amp;&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;32800&lt;/span&gt;    &lt;span class=&quot;err&quot;&gt;&apos;&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;This&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;is&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;a&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;LONG&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;-&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;2147483648&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;..&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;2147483647&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;).&lt;/span&gt;
&lt;span class=&quot;py&quot;&gt;c&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;%&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;b&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;&amp;amp;&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;%&lt;/span&gt;  &lt;span class=&quot;err&quot;&gt;&apos;&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;This&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;should&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;result&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;in&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;c&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;%&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;32758&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;  &lt;span class=&quot;py&quot;&gt;A&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;smaller&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;%&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;would&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;result&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;in&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;overflow&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;… instead of having an op stream like this:&lt;/p&gt;

&lt;div class=&quot;language-zig highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;n&quot;&gt;PUSH_IMM_INTEGER&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;42&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;LET&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;PUSH_IMM_LONG&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;32800&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;LET&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;PUSH_VARIABLE&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;PUSH_VARIABLE&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;OPERATOR_SUBTRACT&lt;/span&gt;     &lt;span class=&quot;c&quot;&gt;// runtime determines it must promote RHS and do LONG subtraction&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;CAST_INTEGER&lt;/span&gt;          &lt;span class=&quot;c&quot;&gt;// runtime determines it has a LONG to demote&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;LET&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;— and so leaving the runtime environment to decide when to promote, or demote,
or coerce, and when to worry about under/overflow — we instead have this:&lt;/p&gt;

&lt;div class=&quot;language-zig highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;n&quot;&gt;PUSH_IMM_INTEGER&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;42&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;LET&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;PUSH_IMM_LONG&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;32800&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;LET&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;PUSH_VARIABLE&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;PUSH_VARIABLE&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;PROMOTE_INTEGER_LONG&lt;/span&gt;     &lt;span class=&quot;c&quot;&gt;// compiler knows we&apos;re about to do LONG subtraction&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;OPERATOR_SUBTRACT_LONG&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;COERCE_LONG_INTEGER&lt;/span&gt;      &lt;span class=&quot;c&quot;&gt;// compiler knows we&apos;re assigning to an INTEGER&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;LET&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;A proliferation of opcodes results, but crucially it means most operations
execute unconditionally, resulting in a far simpler design when it comes to
putting this on a chip.&lt;/p&gt;

&lt;p&gt;The upshot of this design, however, is that the compiler needs a far greater degree
of awareness and ability:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;
    &lt;p&gt;It needs to be aware of what types the arguments to the binary operation are.&lt;/p&gt;

    &lt;p&gt;In the example above, we have a LONG and an INTEGER. This appears trivial, but
we need to keep in mind that the arguments can be arbitrary expressions.&lt;/p&gt;

    &lt;p&gt;We need this in order to determine the opcode that gets emitted for the
operation, and to know if any operand reconciliation is necessary.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;It needs to be able to reconcile different types of operands, where necessary.&lt;/p&gt;

    &lt;p&gt;BASIC’s rules here are simple: we coerce the lesser-precision
value to the greater precision. In effect the rule is
INTEGER &amp;lt; LONG &amp;lt; SINGLE &amp;lt; DOUBLE.&lt;/p&gt;

    &lt;p&gt;STRING is not compatible with any other type — there’s no &lt;nobr&gt;&lt;code&gt;&quot;x&quot; * 3 = &quot;xxx&quot;&lt;/code&gt;&lt;/nobr&gt; here.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;It needs to be aware of what type a binary operation’s result will be in.&lt;/p&gt;

    &lt;p&gt;I started out with the simple rule that the result will be of the same type
of the (reconciled) operands. This worked fine when I just had addition,
subtraction and multiplication; above, we add a LONG and an INTEGER, the INTEGER
gets promoted to a LONG, and the result is a LONG.&lt;/p&gt;

    &lt;p&gt;Division broke this assumption, and resulted in the motivation for this
write-up.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;It needs to be able to pass that information up the compilation stack.&lt;/p&gt;

    &lt;p&gt;Having calculated the result is a LONG, we need to return that information to
the procedure that compiled this expression, as it may make decisions based
on what’s left on the stack after evaluating it — such as in the first
dot-point here, or when assigning to a variable (which may be of any type, and
so require a specific coercion).&lt;/p&gt;
  &lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This all kind of Just Worked, right up until I implemented division. I
discovered QuickBASIC actually has floating &lt;em&gt;and&lt;/em&gt; integer division! Young me
was not aware of the backslash “&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;\&lt;/code&gt;” operator (or any need for it, I suppose).
Floating division always returns a floating-point number, even when the inputs
are integral. Integer division always returns an integral, even when the inputs
are floating. The precision of the types returned in turn depends on that of the
input types.&lt;/p&gt;

&lt;table&gt;
  &lt;thead&gt;
    &lt;tr&gt;
      &lt;th style=&quot;text-align: right&quot;&gt;operands&lt;/th&gt;
      &lt;th style=&quot;text-align: right&quot;&gt;fdiv “&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;/&lt;/code&gt;”&lt;/th&gt;
      &lt;th style=&quot;text-align: right&quot;&gt;idiv “&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;\&lt;/code&gt;”&lt;/th&gt;
    &lt;/tr&gt;
  &lt;/thead&gt;
  &lt;tbody&gt;
    &lt;tr&gt;
      &lt;td style=&quot;text-align: right&quot;&gt;INTEGER&lt;/td&gt;
      &lt;td style=&quot;text-align: right&quot;&gt;SINGLE&lt;/td&gt;
      &lt;td style=&quot;text-align: right&quot;&gt;INTEGER&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td style=&quot;text-align: right&quot;&gt;LONG&lt;/td&gt;
      &lt;td style=&quot;text-align: right&quot;&gt;DOUBLE&lt;/td&gt;
      &lt;td style=&quot;text-align: right&quot;&gt;LONG&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td style=&quot;text-align: right&quot;&gt;SINGLE&lt;/td&gt;
      &lt;td style=&quot;text-align: right&quot;&gt;SINGLE&lt;/td&gt;
      &lt;td style=&quot;text-align: right&quot;&gt;INTEGER&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td style=&quot;text-align: right&quot;&gt;DOUBLE&lt;/td&gt;
      &lt;td style=&quot;text-align: right&quot;&gt;DOUBLE&lt;/td&gt;
      &lt;td style=&quot;text-align: right&quot;&gt;LONG&lt;/td&gt;
    &lt;/tr&gt;
  &lt;/tbody&gt;
&lt;/table&gt;

&lt;p&gt;&lt;center&gt;&lt;em&gt;Output types of each division operation given (reconciled) input type.&lt;/em&gt;&lt;/center&gt;&lt;/p&gt;

&lt;p&gt;This presented a problem for the existing compiler, which assumed the result
would be the same type of the operands: having divided two INTEGERs, for
example, the bytecode emitted &lt;em&gt;assumed&lt;/em&gt; there would be an INTEGER left on the
stack, and so further emitted code would carry that assumption.&lt;/p&gt;

&lt;p&gt;Upon realising how division was supposed to work, the first place I made the
change was in the actual stack machine: correct the behaviour by performing
floating division when floating division was asked for, regardless of the
operand type. Thus dividing two INTEGERs pushed a SINGLE. The (Zig) machine then
promptly asserted upon hitting any operation on that value at all, expecting an
INTEGER there instead. (The future gateware implementation would probably not
assert, and instead produce confusing garbage.)&lt;/p&gt;

&lt;p&gt;And so opened up a new kind of bug to look out for: a mismatch between (a)
the assumptions made by the compiler about the effects on the stack of the
operations it was emitting, and (b) the actual effects on the stack produced by
those operations running on the stack machine.&lt;/p&gt;

&lt;p&gt;Rather than just some isolated tests (though short of creating some whole
contract interface between compiler and machine — maybe later), why not be
thorough while using the best witness for behaviour there is? Namely, the
compiler and stack machine themselves:&lt;/p&gt;

&lt;div class=&quot;language-zig highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;k&quot;&gt;test&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;&quot;compiler and stack machine agree on binop expression types&quot;&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;for&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;std&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;py&quot;&gt;meta&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;tags&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Expr&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;py&quot;&gt;Op&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;))&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;|&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;op&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;|&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
        &lt;span class=&quot;k&quot;&gt;for&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;std&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;py&quot;&gt;meta&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;tags&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;ty&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;py&quot;&gt;Type&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;))&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;|&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;tyLhs&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;|&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
            &lt;span class=&quot;k&quot;&gt;for&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;std&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;py&quot;&gt;meta&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;tags&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;ty&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;py&quot;&gt;Type&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;))&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;|&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;tyRhs&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;|&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
                &lt;span class=&quot;k&quot;&gt;var&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;c&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;try&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;Compiler&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;init&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;testing&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;py&quot;&gt;allocator&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;kc&quot;&gt;null&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;);&lt;/span&gt;
                &lt;span class=&quot;k&quot;&gt;defer&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;c&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;deinit&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;();&lt;/span&gt;

                &lt;span class=&quot;k&quot;&gt;const&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;compilerTy&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;c&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;compileExpr&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;{&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;py&quot;&gt;binop&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
                    &lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;py&quot;&gt;lhs&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;&amp;amp;&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Expr&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;init&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Expr&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;py&quot;&gt;Payload&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;oneImm&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;tyLhs&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;),&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;{}),&lt;/span&gt;
                    &lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;py&quot;&gt;op&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;loc&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;WithRange&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Expr&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;py&quot;&gt;Op&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;).&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;init&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;op&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;{}),&lt;/span&gt;
                    &lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;py&quot;&gt;rhs&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;&amp;amp;&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Expr&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;init&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Expr&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;py&quot;&gt;Payload&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;oneImm&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;tyRhs&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;),&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;{}),&lt;/span&gt;
                &lt;span class=&quot;p&quot;&gt;}&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;})&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;catch&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;|&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;err&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;|&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;switch&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;err&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
                    &lt;span class=&quot;n&quot;&gt;Error&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;py&quot;&gt;TypeMismatch&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&amp;gt;&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;continue&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;c&quot;&gt;// keelatud eine&lt;/span&gt;
                    &lt;span class=&quot;k&quot;&gt;else&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&amp;gt;&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;return&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;err&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
                &lt;span class=&quot;p&quot;&gt;};&lt;/span&gt;

                &lt;span class=&quot;k&quot;&gt;var&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;m&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;stack&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;Machine&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;stack&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;py&quot;&gt;TestEffects&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;).&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;init&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;
                    &lt;span class=&quot;n&quot;&gt;testing&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;py&quot;&gt;allocator&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
                    &lt;span class=&quot;k&quot;&gt;try&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;stack&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;py&quot;&gt;TestEffects&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;init&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(),&lt;/span&gt;
                    &lt;span class=&quot;kc&quot;&gt;null&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
                &lt;span class=&quot;p&quot;&gt;);&lt;/span&gt;
                &lt;span class=&quot;k&quot;&gt;defer&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;m&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;deinit&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;();&lt;/span&gt;

                &lt;span class=&quot;k&quot;&gt;const&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;code&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;try&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;c&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;py&quot;&gt;buf&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;toOwnedSlice&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;testing&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;py&quot;&gt;allocator&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;);&lt;/span&gt;
                &lt;span class=&quot;k&quot;&gt;defer&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;testing&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;py&quot;&gt;allocator&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;free&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;code&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;);&lt;/span&gt;

                &lt;span class=&quot;k&quot;&gt;try&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;m&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;run&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;code&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;);&lt;/span&gt;

                &lt;span class=&quot;k&quot;&gt;try&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;testing&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;expectEqual&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;m&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;py&quot;&gt;stack&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;py&quot;&gt;items&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;py&quot;&gt;len&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;);&lt;/span&gt;
                &lt;span class=&quot;k&quot;&gt;try&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;testing&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;expectEqual&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;m&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;py&quot;&gt;stack&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;py&quot;&gt;items&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;].&lt;/span&gt;&lt;span class=&quot;k&quot;&gt;type&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(),&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;compilerTy&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;);&lt;/span&gt;
            &lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;
        &lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;
    &lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;We go as follows:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;
    &lt;p&gt;For all binary operations, enumerate all permutations of left- and right-hand
side types.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;For each such triple, try compiling the binary operation with the “one”-value&lt;sup id=&quot;fnref:one&quot; role=&quot;doc-noteref&quot;&gt;&lt;a href=&quot;#fn:one&quot; class=&quot;footnote&quot; rel=&quot;footnote&quot;&gt;2&lt;/a&gt;&lt;/sup&gt;
of each type as its operands. (For integrals, it’s literally the number one.)&lt;/p&gt;

    &lt;ul&gt;
      &lt;li&gt;
        &lt;p&gt;If we get a type error from this attempt, skip it — we don’t care that we can’t
divide a DOUBLE by a STRING, or whatever.&lt;/p&gt;
      &lt;/li&gt;
      &lt;li&gt;
        &lt;p&gt;If not, the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;compileExpr&lt;/code&gt; method returns to us the type of what it believes
the result to be. This is the same information used elsewhere in the
compiler to guide opcode and coercion decisions.&lt;/p&gt;
      &lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Create a stack machine, and run the compiled code.&lt;/p&gt;

    &lt;ul&gt;
      &lt;li&gt;Seeing as we didn’t actually compile a full statement, we expect the result
of the operation to be left sitting on the stack. (Normally a statement
would ultimately consume what’s been pushed.)&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Assert that the type of the runtime value left on the stack is the same as
what the compiler expects!&lt;/p&gt;
  &lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;I love how much this solution takes care of itself. While it lacks the
self-descriptive power of a more coupled approach to designing the compiler and
runtime, it lets the implementations remain quite clear and straightforward to
read. It also lets them stand alone, which is handy when a second implementation
of the runtime part is forthcoming, and is (by necessity) in a completely
different language environment.&lt;/p&gt;

&lt;p&gt;There was greater value than just from floats, of course: relational operators
like equal “&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;=&lt;/code&gt;”, not equal “&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;&amp;lt;&amp;gt;&lt;/code&gt;”, etc. all return INTEGERs 0 or -1, but
can operate with any combination of numeric types, or with pairs of STRINGs.
Bitwise operators like &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;AND&lt;/code&gt;, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;OR&lt;/code&gt;, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;XOR&lt;/code&gt;, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;IMP&lt;/code&gt; (!), etc. can operate with any
combination of numeric types, but coerce any floats to integrals before doing
so. I bet there’ll be more to uncover, too.&lt;/p&gt;

&lt;p&gt;Hope this was fun to read!&lt;/p&gt;

&lt;div class=&quot;footnotes&quot; role=&quot;doc-endnotes&quot;&gt;
  &lt;ol&gt;
    &lt;li id=&quot;fn:highlighting&quot; role=&quot;doc-endnote&quot;&gt;
      &lt;p&gt;I will get some real BASIC highlighting up here by the next
             post on this! &lt;a href=&quot;#fnref:highlighting&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
    &lt;li id=&quot;fn:one&quot; role=&quot;doc-endnote&quot;&gt;
      &lt;p&gt;I first used each type’s zero value, but then we get division by zero errors while trying
    to execute the code! &lt;a href=&quot;#fnref:one&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
  &lt;/ol&gt;
&lt;/div&gt;
</content>
    
 </entry>
 
 <entry>
   <title>Kalakarp</title>
   <updated>2024-07-27T00:00:00+10:00</updated>
   <id>https://kivikakk.ee/2024/07/27/kalakarp</id>
    
     <link href="https://kivikakk.ee/2024/07/27/kalakarp/"/>
     <content type="html">&lt;p&gt;&lt;img src=&quot;/assets/post-img/kalakarp.png&quot; alt=&quot;&quot; /&gt;&lt;/p&gt;

&lt;p&gt;Feeling very seen by fish today.&lt;/p&gt;
</content>
    
 </entry>
 
 <entry>
   <title>Ravimid</title>
   <updated>2024-07-26T00:00:00+10:00</updated>
   <id>https://kivikakk.ee/2024/07/26/ravimid</id>
    
     <link href="https://kivikakk.ee/2024/07/26/ravimid/"/>
     <content type="html">&lt;p&gt;Reviewing all the medications I’ve ever been on. This will be intensely specific to my experiences;
YMMV! Content note: mentions of mental illness, suicidal feelings, assault, sex.&lt;/p&gt;

&lt;h2 id=&quot;mental-health-ish&quot;&gt;mental health(-ish)&lt;/h2&gt;

&lt;ul&gt;
  &lt;li&gt;escitalopram
    &lt;ul&gt;
      &lt;li&gt;A day or two after starting it I felt like the world had colour in it again for the first time.&lt;/li&gt;
      &lt;li&gt;Not very effective for anything other than “light depression” (?) imo.&lt;/li&gt;
      &lt;li&gt;Prescribed initially for depression/I’m 23 And Life Is Generally Hard, was on it for two years.&lt;/li&gt;
      &lt;li&gt;Later prescribed in an attempt to counter chronic idiopathic nausea, to no effect.&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;quetiapine
    &lt;ul&gt;
      &lt;li&gt;Wow! I didn’t know what “suicidal compulsion” actually felt like until I took my first dose.
That was also my last dose.&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;desvenlafaxine
    &lt;ul&gt;
      &lt;li&gt;Going up from 100mg to 200mg, I’d just stop knowing how to form sentences halfway through them.&lt;/li&gt;
      &lt;li&gt;Effective on moderate GAD-type symptoms.&lt;/li&gt;
      &lt;li&gt;Coming off it was, contrary to all expectations, pretty uneventful. Except that for about three
weeks, all I could smell was blood, with no apparent source.&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;brexpiprazole
    &lt;ul&gt;
      &lt;li&gt;If you close your eyes, you will fall asleep. At any time.&lt;/li&gt;
      &lt;li&gt;Definitely calm.&lt;/li&gt;
      &lt;li&gt;Much too sleepy to continue, but it definitely brought me down after increasingly intensifying
distress post-sexual assault.&lt;/li&gt;
      &lt;li&gt;As with any antipsychotic/dopamine antagonist, &lt;a href=&quot;https://en.wikipedia.org/wiki/Tardive_dyskinesia&quot;&gt;not particularly inclined&lt;/a&gt; to remain on one
for any longer period of time.&lt;/li&gt;
      &lt;li&gt;
        &lt;p&gt;At the time I was put on it, the &lt;em&gt;only&lt;/em&gt; article in the literature about it had the (actually!)
reassuring title: &lt;a href=&quot;https://pubmed.ncbi.nlm.nih.gov/26913177/&quot;&gt;“Brexpiprazole: so far so good”&lt;/a&gt;.&lt;/p&gt;

      &lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;lamotrigine
    &lt;ul&gt;
      &lt;li&gt;Unclear whether I felt much of anything on this, even after three years at 200mg.&lt;/li&gt;
      &lt;li&gt;Coming off it: ANGER. Wow. (Remember to titrate; SJS can be triggered both on increase and
decrease, apparently.)&lt;/li&gt;
      &lt;li&gt;After coming off it: oh my goodness I suddenly feel like being creative again.&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;mirtazapine
    &lt;ul&gt;
      &lt;li&gt;eeeeeeeepy and hunnnnggggy.&lt;/li&gt;
      &lt;li&gt;Extremely and rapidly effective with chronic idiopathic nausea. &lt;em&gt;Nothing&lt;/em&gt; else worked.&lt;/li&gt;
      &lt;li&gt;Weight gain pretty woof. That said, the weight I gained on it, I managed to drop while still on
it.&lt;/li&gt;
      &lt;li&gt;I don’t remember if I maxed out at 30mg or 45mg — at higher doses the dissociation made
memory very wonky.&lt;/li&gt;
      &lt;li&gt;Discontinuation meant a lot of rebound nausea and anxiety. It took me about 4 attempts, over
several years, with various titration schedules; the successful attempt was just dropping from
a sustained 15mg to 0. Trying to step it down at 7.5mg only prolonged the unpleasantness and
resulted in aborting the attempt — subtherapeutic but side-effects remain.&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;duloxetine
    &lt;ul&gt;
      &lt;li&gt;Really pleasant and noticeable, even at 30mg. Everything feels a bit lighter, and mild
neuropathy totally neutralised.&lt;/li&gt;
      &lt;li&gt;Night sweats :( Became intolerable not long after increasing to 60mg, and didn’t subside after
reducing again.&lt;/li&gt;
      &lt;li&gt;Discontinuation was reasonably uneventful; (manageably) shorterer temper and very mild nausea.&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;gabapentin
    &lt;ul&gt;
      &lt;li&gt;I just started this! Early effects for neuropathy are promising. Slightly dissociative/floaty.&lt;/li&gt;
      &lt;li&gt;Was always curious about gabapentin/pregabalin since I first started having noticeable anxiety,
so I’m glad I’ve gotten to try it. Definitely somewhat calming.&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;diazepam
    &lt;ul&gt;
      &lt;li&gt;It is what it is.&lt;/li&gt;
      &lt;li&gt;Avoid taking on an empty stomach if you’re having issues with nausea — it can get so much
worse.&lt;/li&gt;
      &lt;li&gt;I treat it with extreme caution. I’ve read so many personal horror stories about benzodiazepine
dependence and it takes a really bad time for me to consider taking even 2.5mg. I’ll go up to
5mg if need be, but in general would refuse to take it if I’ve had any separate dose within the
last week.&lt;/li&gt;
      &lt;li&gt;For the above reason, despite having been in some very bad places (where doctors were
prescribing it in larger quantities at a time), I’ve maybe taken 100mg total over the last
decade. Just not worth the risk.&lt;/li&gt;
      &lt;li&gt;Even less inclined to if I’ve taken stimulants on the same day, but I’ve only ever once
(namely, in a life-threatening situation) had to even consider diazepam since getting medicated
for ADHD 2 years ago. Funny how that works!&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;temazepam
    &lt;ul&gt;
      &lt;li&gt;Not a fan; taken a handful of times ever, mostly at music festivals.&lt;/li&gt;
      &lt;li&gt;I have chronic insomnia and this is just not any part of an answer to it.&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;adhd&quot;&gt;ADHD&lt;/h2&gt;

&lt;ul&gt;
  &lt;li&gt;dextroamphetamine
    &lt;ul&gt;
      &lt;li&gt;Current weapon of choice. Calm, alert, focussed.&lt;/li&gt;
      &lt;li&gt;Latent anxiety is reduced a lot, probably because it lets me have some agency in my own head.&lt;/li&gt;
      &lt;li&gt;My doctor gradually increased my dose to 10mg, 4 times a day. This is absolutely fucking nuts.
Don’t do that.&lt;/li&gt;
      &lt;li&gt;After 1.5 years I’ve settled on 2.5mg, twice a day at a 4 hour interval, and it’s completely
adequate.&lt;/li&gt;
      &lt;li&gt;If I’m having a very long day, a third dose is fine too. 2.5mg doses are very light.&lt;/li&gt;
      &lt;li&gt;For a long while there I was finding myself increasing my dose by 2.5mg or 5mg (per day) every
week or so since the tolerance builds so rapidly. You will deplete every last monoamine you
have like this. I managed to tolerate 40mg/day (after a buildup) for about two days before I 
crashed. Do Not.&lt;/li&gt;
      &lt;li&gt;Sometimes 2.5mg doses can feel a little bit &lt;em&gt;too&lt;/em&gt; light, but my health right now is such that
increasing even just one of the doses leaves me feeling empty the next day. I’m glad to have
something that works.&lt;/li&gt;
      &lt;li&gt;Small doses is also good because I’ve yet to see a psychiatrist in Estonia to get a
prescription here, so I’m working through my stock from Australia very slowly. As far as I can
tell, currently authorised dextroamphetamine products &lt;a href=&quot;https://www.ravimiregister.ee/publichomepage.aspx?pv=PublicMedDetail&amp;amp;vid=988182bb-38b5-4201-bc1f-8ec0eeb820fb&quot;&gt;haven’t been imported to Estonia&lt;/a&gt;,
so it’s unclear how easy it’d be to access it.&lt;/li&gt;
      &lt;li&gt;There’s &lt;a href=&quot;https://www.ravimiregister.ee/publichomepage.aspx?pv=PublicMedDetail&amp;amp;vid=7e33f799-6b5e-402f-b654-cda2d0e5fe66&quot;&gt;this one&lt;/a&gt;, but it’s only available in 10mg? And searching for it online shows up
something with the same name, but it’s &lt;a href=&quot;https://www.google.com/search?hl=en&amp;amp;q=attentin+10mg&amp;amp;udm=2&quot;&gt;atomoxetine&lt;/a&gt; instead? Very suss on that.&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;lisdexamfetamine
    &lt;ul&gt;
      &lt;li&gt;Hrrrrrrr. Not a fan.&lt;/li&gt;
      &lt;li&gt;Something about this leaves me feeling depleted at any effective dose. I think I like the
little humps I get between dex doses — lisdex is just exhausting, and an absolute pain if you
wake up late.&lt;/li&gt;
      &lt;li&gt;30mg? Not enough to feel anything — too spread out.&lt;/li&gt;
      &lt;li&gt;40mg? I kinda feel something. It’s a bit eh.&lt;/li&gt;
      &lt;li&gt;50mg? Yeah, okay, maybe this starts to feel like I’m having an OK dose of dex — I can think,
focus, actually listen to people when they talk to me. I feel like I am spread very thin.&lt;/li&gt;
      &lt;li&gt;60–70mg? Effective — wired — but also I start sleeping longer and longer because of how
exhausted it leaves me, making my doses later and later, in turn fucking up my sleep schedule;
not to mention the pervasive sense of Bad that starts to accumulate. No ty.&lt;/li&gt;
      &lt;li&gt;My god. I tried combining ~low dose lisdex with a small dex booster in the afternoon. Quick way
to know what it feels like to fry your brain. You can do all the stimulant equivalence
arithmetic you want, it just doesn’t add up that way.&lt;/li&gt;
      &lt;li&gt;If I was offered lisdex or nothing, it’s not clear to me that I would take it.&lt;/li&gt;
      &lt;li&gt;In general, I’m not sold on long-acting stimulants. I get the abuse angle (it’s what I was put
on first, for this reason), and I do read stories of folks who find it legitimately helpful —
like most others with ADHD, I regularly forget to take my stimulants — but there was no way
to make it work for me. It’s too inflexible, and there’s some (possibly fear-mongering) news
out there that suggests long-acting stimulants carry a greater risk of psychosis than
short-acting ones?&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;methylphenidate
    &lt;ul&gt;
      &lt;li&gt;Very conflicted on this. (Annoying that it’s so much more readily available, especially in
Europe.)&lt;/li&gt;
      &lt;li&gt;
        &lt;p&gt;&lt;a href=&quot;https://elifesciences.org/articles/68165&quot;&gt;Possibly neuroprotective vs. an active COVID infection!&lt;/a&gt; I was on MPH when I finally
got COVID, and was wondering if I should keep taking stimulants through a moderately bad case
of it or not. This article swayed me in favour of maintaining it:&lt;/p&gt;

        &lt;blockquote&gt;
          &lt;p&gt;Remarkably, several drugs acting on receptors to neurotransmitters also appear to decrease
hospitalization risk: rizatriptan (OR=0.118, CI 0.003 to 0.750), bupropion (OR=0.399, CI
0.136 to 0.976), and methylphenidate (OR=0.444, CI 0.178 to 0.972).&lt;/p&gt;
        &lt;/blockquote&gt;

        &lt;p&gt;I’ll take an OR of 0.444ish. I had to argue with a doctor from the government(!) to avoid
hospitalisation, but avoid it I did.&lt;/p&gt;

      &lt;/li&gt;
      &lt;li&gt;I haven’t gotten a sense for how shared this is, but I get noticeable euphoria when starting on
10mg, and it makes one very inclined to up the dose when you stop getting that response after a
few days.&lt;/li&gt;
      &lt;li&gt;Dragon-chasing aside, MPH consistently produces a sort of global “bad” sensation that I just
cannot with. Even with the initial euphoria, it’s still noticeable — there’s just this
unpleasant but persistent background qualia, almost a certain flavour of experience. Unclear if
shared or just me.&lt;/li&gt;
      &lt;li&gt;All that notwithstanding, from 10mg it feels 70–80% as effective as dex at ~equivalent dose
(5mg dex ≈ 10mg MPH).&lt;/li&gt;
      &lt;li&gt;5mg didn’t feel like it had any impact at all, but that was a while ago, and I’d revisit it
if I had to.&lt;/li&gt;
      &lt;li&gt;If I was offered MPH or nothing, I would probably take it.&lt;/li&gt;
      &lt;li&gt;If I was offered MPH or lisdex, I would probably take MPH.&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;vascular&quot;&gt;vascular&lt;/h2&gt;

&lt;ul&gt;
  &lt;li&gt;propranolol
    &lt;ul&gt;
      &lt;li&gt;Very chill. Noticeable sensation in the chest.&lt;/li&gt;
      &lt;li&gt;I once had a kind of shock reaction to a piercing which resulted in a really high heart rate,
intense and sudden nausea, cold shivers all over, and me on the floor. (I find lying on a cold
hard floor fixes most things.) The next time I went for a piercing I took 10mg propranolol
before and it was easy.&lt;/li&gt;
      &lt;li&gt;Initially prescribed to try to combat panic disorder/idiopathic nausea. It at least brought my
heart rate down and made it possible to eat again.&lt;/li&gt;
      &lt;li&gt;Later took it for arrythmia on demand. Reasonable at this — would often get painful(!)
arrythmia at night which would make sleep impossible, and leave me feeling sore (and exhausted)
the whole next day. 10mg propranolol would usually convert it within half an hour.&lt;/li&gt;
      &lt;li&gt;100–200mg magnesium seems to work for that too.&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;nifedipine
    &lt;ul&gt;
      &lt;li&gt;Certainly feel that one on starting. Not like I didn’t already have some orthostatic
hypotension …&lt;/li&gt;
      &lt;li&gt;Take for Raynaud’s in the Australian winter/unspecified (micro?)vascular issues. Recently my
legs have been feeling Not OK and so I’ve restarted it. Unfortunately, we don’t seem to have
the modified-release ones here, so I have to dose twice a day.&lt;/li&gt;
      &lt;li&gt;Doesn’t impair exercise.&lt;/li&gt;
      &lt;li&gt;Not to be combined with propranolol (or magnesium, possibly).&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;gastric&quot;&gt;gastric&lt;/h2&gt;

&lt;ul&gt;
  &lt;li&gt;esomeprazole/omeprazole
    &lt;ul&gt;
      &lt;li&gt;Unremarkable. Not super effective for (probably stress-induced) heartburn/reflux.&lt;/li&gt;
      &lt;li&gt;Was also prescribed it at one point as an attempt to help with idiopathic nausea — no
response.&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;ondansetron
    &lt;ul&gt;
      &lt;li&gt;This is meant to be the Big Guns when it comes to nausea, and it did nothing for mine.&lt;/li&gt;
      &lt;li&gt;I was scrambling to find something that would work — it’d slowly subsided on desvenlafaxine
in the first instance, and I’d tapered off that after about a year and was all good. A
particularly bad anxiety attack provoked it into a full reoccurrence, which was terrifying. I
wasn’t inclined to retry stepping up on desvenlafaxine and waiting it out to see if it’d work
again — I felt like it might’ve also just been a matter of time, then, as it remitted only 
after more than a year of (essentially constant) suffering! — so I looked for other things.&lt;/li&gt;
      &lt;li&gt;My research lead me to mirtazapine; &lt;a href=&quot;https://en.wikipedia.org/wiki/Mirtazapine#Pharmacodynamics&quot;&gt;there’s a number of different ways in which it’s
antiemetic&lt;/a&gt;. My GP at the time was happy to prescribe it for me based on my conviction,
but wanted to try ondansetron first to see if it’d work as a circuit breaker (and thus avoid
necessitating something taken over a longer time). Alas.&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;cannabidiol&quot;&gt;cannabidiol&lt;/h2&gt;

&lt;p&gt;Usual effects of cannabis products apply.&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;THC oil
    &lt;ul&gt;
      &lt;li&gt;First trials found it to be extremely intense and disorienting.&lt;/li&gt;
      &lt;li&gt;Retrying years later (once I was using flower more medicinally and regularly), found it to be a
nice way to dose THC in a more controlled manner, without the usual associations that come with
vaping bud (i.e. “Time To Relax, And Perhaps Eat Much Šokolaadi.”)&lt;/li&gt;
      &lt;li&gt;Slower onset can be quite pleasant.&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;CBD oil
    &lt;ul&gt;
      &lt;li&gt;First trials found it to make me feel generally unwell!&lt;/li&gt;
      &lt;li&gt;Later trials together with THC oil found it to be a nice way to dampen some aspects of the THC
high, if needed.&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;cannabis flower
    &lt;ul&gt;
      &lt;li&gt;Nothing much to report here. Australian medicinal cannabis was decent, actually decent value
(per volume), and there was a reasonable variety of strains to choose from, which improved over
the years. Doctors were not inclined to gatekeep (&lt;a href=&quot;https://www.abc.net.au/news/2024-07-21/medicinal-cannabis-psychosis-harm-risk-prescription-marijuana/104116952&quot;&gt;perhaps concerningly so&lt;/a&gt;) or judge.&lt;/li&gt;
      &lt;li&gt;Driving was a bit nerve-wracking; regular use builds up a level in the blood which can be
detected by roadside random breath tests even if you’ve not used any that day, so I was pretty
much always on alert/police-avoidance mode. It was a pretty strong deterrent, but I lived in a
public transport deadzone so there wasn’t much choice.&lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;https://www.abc.net.au/news/2024-05-21/victoria-medicinal-cannabis-driving-cbd-thc-explainer/103872650&quot;&gt;This may be changing soon in Victoria!&lt;/a&gt;&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;misc&quot;&gt;misc&lt;/h2&gt;

&lt;ul&gt;
  &lt;li&gt;naproxen
    &lt;ul&gt;
      &lt;li&gt;Prescribed for leg pain which was suspected referred back pain.&lt;/li&gt;
      &lt;li&gt;Came in a combination form with esomeprazole because &lt;a href=&quot;https://en.wikipedia.org/wiki/Nonsteroidal_anti-inflammatory_drug#Gastrointestinal&quot;&gt;COX inhibitors be wild&lt;/a&gt;.&lt;/li&gt;
      &lt;li&gt;No effect.&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;tadalafil/sildenafil
    &lt;ul&gt;
      &lt;li&gt;Mildly effective for antiandrogen-related difficulties with penetrative sex.&lt;/li&gt;
      &lt;li&gt;Not a fan of the way these make my heart feel; wouldn’t opt for tadalafil again since it’s just
the same but for days instead of hours.&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;emtricitabine+tenofovir
    &lt;ul&gt;
      &lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Pre-exposure_prophylaxis_for_HIV_prevention&quot;&gt;PrEP&lt;/a&gt;. This is important, good stuff!&lt;/li&gt;
      &lt;li&gt;Mild nausea during the first two weeks of starting it, otherwise side-effect-free.&lt;/li&gt;
      &lt;li&gt;Been on and off it a half-dozen times over the last 8 years as my risk factors have increased
or decreased. I have nothing but good things to say about PrEP.&lt;/li&gt;
      &lt;li&gt;Caution while travelling; less progressive governments may infer any number of things about you
based on the medication you carry.&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;hrt&quot;&gt;HRT&lt;/h2&gt;

&lt;ul&gt;
  &lt;li&gt;oestradiol valerate
    &lt;ul&gt;
      &lt;li&gt;Standard first-line transfem primary HRT.&lt;/li&gt;
      &lt;li&gt;In the first few years I seemed to manage decent levels while increasing to 8mg/day, but over
time I seemed to stop being able to absorb oral oestrogen and had to switch off.&lt;/li&gt;
      &lt;li&gt;Not particularly sad about that: other forms are less work for my liver.&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;oestradiol gel
    &lt;ul&gt;
      &lt;li&gt;Absorbs well. A little bit annoying to make your skin sticky twice a day but wycd ¯\_(ツ)_/¯&lt;/li&gt;
      &lt;li&gt;Seems to be widely available; the last few years have had many HRT shortages, but I’ve not had
issues with this.&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;micronised progesterone
    &lt;ul&gt;
      &lt;li&gt;Preferred over synthetics (e.g. levonorgestrel) due to much reduced blood clotting risk, fewer
negative mood side-effects.&lt;/li&gt;
      &lt;li&gt;Helpful for sleep — take at night.&lt;/li&gt;
      &lt;li&gt;First time I started it, within days I was sobbing (+) while watching Frozen. Really unlocks
some emotions.&lt;/li&gt;
      &lt;li&gt;200-300mg is an actual recreational high (N.B.! loss of consciousness/respiratory depression
may result at extreme doses).&lt;/li&gt;
      &lt;li&gt;Also an antiandrogen!&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;spironolactone
    &lt;ul&gt;
      &lt;li&gt;At least in Australia, first-line antiandrogen.&lt;/li&gt;
      &lt;li&gt;Diuretic and generally not fun. At the doses required for me, also precipitated pretty intense
low mood which abated upon discontinuation.&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;cyproterone
    &lt;ul&gt;
      &lt;li&gt;Antiandrogen. Extremely effective in even tiny doses.&lt;/li&gt;
      &lt;li&gt;My doctor started me on 50mg a day. I don’t have a good analogy for how nuclear this is.&lt;/li&gt;
      &lt;li&gt;May need more initially to get levels down, but (depending on prevailing E levels) you may be
fine with 12.5 mg &lt;strong&gt;once a week&lt;/strong&gt;, i.e. 3.6% of the dose I was started on.&lt;/li&gt;
      &lt;li&gt;Some folks report low mood on cypro; it hasn’t been an issue for me.&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
&lt;/ul&gt;

</content>
    
 </entry>
 
 <entry>
   <title>VyxOS</title>
   <updated>2024-07-19T00:00:00+10:00</updated>
   <id>https://kivikakk.ee/2024/07/19/vyxos</id>
    
      <link href="https://lottia.net/notes/0010-vyxos.html"/>
      <content type="html">Nix configuration revealed!</content>
    
 </entry>
 
 <entry>
   <title>Sada päeva</title>
   <updated>2024-07-18T00:00:00+10:00</updated>
   <id>https://kivikakk.ee/2024/07/18/sada-päeva</id>
    
     <link href="https://kivikakk.ee/2024/07/18/sada-p%C3%A4eva/"/>
     <content type="html">&lt;p&gt;Yesterday was our 100th day in Estonia. Taking a little bit of stock of what we’ve managed:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Visited the 4 largest cities in the country.&lt;/li&gt;
  &lt;li&gt;Rented an apartment in the biggest one!&lt;/li&gt;
  &lt;li&gt;Furnished what the apartment didn’t come with.&lt;/li&gt;
  &lt;li&gt;Shipped our things from Australia. (Maybe a month off those arriving.)&lt;/li&gt;
  &lt;li&gt;Got our motorcycles; put 900km on each.&lt;/li&gt;
  &lt;li&gt;Got medium-term visa for A, and long-term one applied for.&lt;/li&gt;
  &lt;li&gt;Got our medications prescribed locally.&lt;/li&gt;
  &lt;li&gt;Financial/bureaucratic overhead.&lt;/li&gt;
  &lt;li&gt;Changed my name and got new ID.&lt;/li&gt;
  &lt;li&gt;Got onto a good family doctor’s list.&lt;/li&gt;
  &lt;li&gt;Kept in touch with families and psychologists.&lt;/li&gt;
  &lt;li&gt;Vaccinations.&lt;/li&gt;
  &lt;li&gt;Saw the border.&lt;/li&gt;
  &lt;li&gt;Went to a cat café.&lt;/li&gt;
  &lt;li&gt;Went to sauna.&lt;/li&gt;
  &lt;li&gt;Concluded jaanipäev with clothes smelling deeply of bonfire.&lt;/li&gt;
  &lt;li&gt;Went to a live show (Estonian).&lt;/li&gt;
  &lt;li&gt;Went to a live show (non-Estonian).&lt;/li&gt;
  &lt;li&gt;Walked about 500km.&lt;/li&gt;
  &lt;li&gt;Taken a lot of public transport.&lt;/li&gt;
  &lt;li&gt;Met a range of people.&lt;/li&gt;
  &lt;li&gt;Spoken quite a bit of Estonian.&lt;/li&gt;
  &lt;li&gt;Kept up with projects.&lt;/li&gt;
  &lt;li&gt;Finished a diary!&lt;/li&gt;
  &lt;li&gt;Grew a lot.&lt;/li&gt;
&lt;/ul&gt;
</content>
    
 </entry>
 
 <entry>
   <title>Activity period</title>
   <updated>2024-07-09T00:00:00+10:00</updated>
   <id>https://kivikakk.ee/2024/07/09/activity-period</id>
    
     <link href="https://kivikakk.ee/2024/07/09/activity-period/"/>
     <content type="html">&lt;p&gt;&lt;img src=&quot;/assets/post-img/eesti/activityperiod.png&quot; alt=&quot;Screenshot of a website. The title is &amp;quot;Activity period&amp;quot;. Text underneath reads: &amp;quot;The sole proprietor may notify the registrar in advance of the suspension of activities, temporary activities or set periods of seasonal activities. Below that is a timeline with &amp;quot;2024&amp;quot; underneath it, and 12 points marked along the timeline. All 12 points have the legend &amp;quot;Invalid Date&amp;quot;.&quot; /&gt;&lt;/p&gt;
</content>
    
 </entry>
 
 <entry>
   <title>Time travel</title>
   <updated>2024-07-06T00:00:00+10:00</updated>
   <id>https://kivikakk.ee/2024/07/06/time-travel</id>
    
      <link href="https://lottia.net/notes/0008-time-travel.html"/>
      <content type="html">The typical hypothetical “who are you coding for” example meant to shock you into writing better code is “yourself in six months”, but it turns out four is completely adequate to get lost.</content>
    
 </entry>
 
 <entry>
   <title>Python still surprises</title>
   <updated>2024-06-27T00:00:00+10:00</updated>
   <id>https://kivikakk.ee/2024/06/27/python-still-surprises</id>
    
     <link href="https://kivikakk.ee/2024/06/27/python-still-surprises/"/>
     <content type="html">&lt;p&gt;After the better part of 20 years working with Python, it still managed to
surprise me today.&lt;/p&gt;

&lt;p&gt;I’m so used to languages treating &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;x += y&lt;/code&gt; et al. as pure sugar for &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;x = x + y&lt;/code&gt;
that it skipped my mind that some don’t.&lt;/p&gt;

&lt;p&gt;I’m not surprised that you &lt;em&gt;can&lt;/em&gt; override them separately in some languages (e.g.
I simply assume this to be the case in C++, and on checking it turns out to
be true — but that seems fair enough given the scope of the language), but I
really am so accustomed to them being only sugar in Ruby that I assumed the same
would hold, at least in effect, in Python.&lt;/p&gt;

&lt;p&gt;Thus my surprise on &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;some_list += x&lt;/code&gt; modifying &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;some_list&lt;/code&gt; in place (unlike
&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;some_list = some_list + x&lt;/code&gt;), but once observed, I realised there’d be a
separately-overridden operator function — namely &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;__iadd__&lt;/code&gt; — and so I
figured it “had” to be that way.&lt;/p&gt;

&lt;p&gt;Or did it? I then found myself assuming it’s because these operators can’t
actually reassign the receiver, but in fact they can and do: the return value is
what’s assigned to the LHS. So it’s just a matter of convention.&lt;/p&gt;

</content>
    
 </entry>
 
 <entry>
   <title>sint</title>
   <updated>2024-06-22T00:00:00+10:00</updated>
   <id>https://kivikakk.ee/2024/06/22/sint</id>
    
     <link href="https://kivikakk.ee/2024/06/22/sint/"/>
     <content type="html">&lt;p&gt;Notes.app, kell 08:03:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;my new theory is that satan Crept into this world through signed integers&lt;/p&gt;
&lt;/blockquote&gt;
</content>
    
 </entry>
 
 <entry>
   <title>zxxrtl</title>
   <updated>2024-06-17T00:00:00+10:00</updated>
   <id>https://kivikakk.ee/digital/2024/06/17/zxxrtl</id>
    
     <link href="https://kivikakk.ee/digital/2024/06/17/zxxrtl/"/>
     <content type="html">&lt;p&gt;I’ve been getting back into using &lt;a href=&quot;https://yosyshq.readthedocs.io/projects/yosys/en/latest/cmd/write_cxxrtl.html&quot;&gt;CXXRTL&lt;/a&gt; and &lt;a href=&quot;https://ziglang.org&quot;&gt;Zig&lt;/a&gt; together, so I’ve extracted
and rendered somewhat reusable the bindings I made to use them together!&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://github.com/kivikakk/zxxrtl&quot;&gt;zxxrtl&lt;/a&gt; uses CXXRTL’s C API to provide a somewhat idiomatic way to access,
manipulate, and respond to events happening in the design. Its README covers the
setup — it’s a bit involved as it’s necessarily something of a build system,
but once you’re done it’s good to go and flexible enough to be instrumented from
a higher build system.&lt;/p&gt;

&lt;p&gt;I’m going to paste the example usage here; this doesn’t use the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Sample&lt;/code&gt; API for
edge detection, and just drives the simulation while optionally recording VCD:&lt;/p&gt;

&lt;div class=&quot;language-zig highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;k&quot;&gt;const&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;Cxxrtl&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;@import&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;zxxrtl&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;);&lt;/span&gt;

&lt;span class=&quot;c&quot;&gt;// Initialise the design.&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;const&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;cxxrtl&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;Cxxrtl&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;init&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;();&lt;/span&gt;

&lt;span class=&quot;c&quot;&gt;// Optionally start recording VCD. Assume `vcd_out` is `?[]const u8` representing an&lt;/span&gt;
&lt;span class=&quot;c&quot;&gt;// optional output filename.&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;var&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;vcd&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;?&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Cxxrtl&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;py&quot;&gt;Vcd&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;kc&quot;&gt;null&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;if&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;vcd_out&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;!=&lt;/span&gt; &lt;span class=&quot;kc&quot;&gt;null&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;vcd&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;Cxxrtl&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;py&quot;&gt;Vcd&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;init&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;cxxrtl&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;);&lt;/span&gt;

&lt;span class=&quot;k&quot;&gt;defer&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;if&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;vcd&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;|&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;*&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;vcdh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;|&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;vcdh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;deinit&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;();&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;cxxrtl&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;deinit&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;();&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;

&lt;span class=&quot;c&quot;&gt;// Get handles to the clock and reset lines.&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;const&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;clk&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;cxxrtl&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;get&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;k&quot;&gt;bool&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;&quot;clk&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;);&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;const&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;rst&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;cxxrtl&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;get&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;k&quot;&gt;bool&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;&quot;rst&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;);&lt;/span&gt;  &lt;span class=&quot;c&quot;&gt;// These are of type `Cxxrtl.Object(bool)`.&lt;/span&gt;

&lt;span class=&quot;c&quot;&gt;// Reset for a tick.&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;rst&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;next&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;kc&quot;&gt;true&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;);&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;clk&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;next&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;kc&quot;&gt;false&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;);&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;cxxrtl&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;step&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;();&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;if&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;vcd&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;|&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;*&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;vcdh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;|&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;vcdh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;sample&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;();&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;clk&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;next&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;kc&quot;&gt;true&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;);&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;cxxrtl&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;step&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;();&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;if&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;vcd&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;|&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;*&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;vcdh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;|&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;vcdh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;sample&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;();&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;rst&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;next&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;kc&quot;&gt;false&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;);&lt;/span&gt;

&lt;span class=&quot;c&quot;&gt;// Play out 10 cycles.&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;for&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;..&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;10&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;|&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;_&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;|&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;clk&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;next&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;kc&quot;&gt;false&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;);&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;cxxrtl&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;step&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;();&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;if&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;vcd&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;|&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;*&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;vcdh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;|&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;vcdh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;sample&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;();&lt;/span&gt;

    &lt;span class=&quot;n&quot;&gt;clk&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;next&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;kc&quot;&gt;true&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;);&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;cxxrtl&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;step&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;();&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;if&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;vcd&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;|&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;*&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;vcdh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;|&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;vcdh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;sample&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;();&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;

&lt;span class=&quot;k&quot;&gt;if&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;vcd&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;|&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;*&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;vcdh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;|&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
    &lt;span class=&quot;c&quot;&gt;// Assume `alloc` exists.&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;const&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;buffer&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;try&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;vcdh&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;read&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;alloc&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;);&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;defer&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;alloc&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;free&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;buffer&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;);&lt;/span&gt;

    &lt;span class=&quot;k&quot;&gt;var&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;file&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;try&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;std&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;py&quot;&gt;fs&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;cwd&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;().&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;createFile&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;vcd_out&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.?&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;{});&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;defer&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;file&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;close&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;();&lt;/span&gt;

    &lt;span class=&quot;k&quot;&gt;try&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;file&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;writeAll&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;buffer&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;);&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Hopefully this is useful to someone else!&lt;/p&gt;

</content>
    
 </entry>
 
 <entry>
   <title>fren</title>
   <updated>2024-06-13T00:00:00+10:00</updated>
   <id>https://kivikakk.ee/eesti/2024/06/13/fren</id>
    
     <link href="https://kivikakk.ee/eesti/2024/06/13/fren/"/>
     <content type="html">&lt;p&gt;&lt;img src=&quot;/assets/post-img/eesti/fren.jpg&quot; alt=&quot;low-exposure photograph of a cat walking along the ground&quot; /&gt;&lt;/p&gt;
</content>
    
 </entry>
 
 <entry>
   <title>Täna</title>
   <updated>2024-06-06T00:00:00+10:00</updated>
   <id>https://kivikakk.ee/eesti/2024/06/06/täna</id>
    
     <link href="https://kivikakk.ee/eesti/2024/06/06/t%C3%A4na/"/>
     <content type="html">&lt;p&gt;Täna oli saunapäev. ^_^&lt;/p&gt;
</content>
    
 </entry>
 
 <entry>
   <title>Chisel and C++, recorded at last</title>
   <updated>2024-05-29T00:00:00+10:00</updated>
   <id>https://kivikakk.ee/digital/2024/05/29/chisel-and-cxx-recording</id>
    
     <link href="https://kivikakk.ee/digital/2024/05/29/chisel-and-cxx-recording/"/>
     <content type="html">&lt;p&gt;Yay! &lt;a href=&quot;https://www.youtube.com/watch?v=_-oqnf9gYuE&quot;&gt;Here’s the recording&lt;/a&gt; for &lt;a href=&quot;https://kivikakk.ee/digital/2024/05/28/chisel-and-cxx/&quot;&gt;Chisel and C++, together at last&lt;/a&gt;.&lt;/p&gt;

</content>
    
 </entry>
 
 <entry>
   <title>Chisel and C++, together at last</title>
   <updated>2024-05-28T00:00:00+10:00</updated>
   <id>https://kivikakk.ee/digital/2024/05/28/chisel-and-cxx</id>
    
     <link href="https://kivikakk.ee/digital/2024/05/28/chisel-and-cxx/"/>
     <content type="html">&lt;p&gt;I gave a lightning talk at last night’s &lt;a href=&quot;https://blog.yosyshq.com/p/yosys-users-group/&quot;&gt;Yosys Users Group&lt;/a&gt; about combining
Chisel and &lt;nobr&gt;C++&lt;/nobr&gt; with Yosys/CXXRTL. &lt;del&gt;I think there’ll be a recording
of them that goes up on YouTube eventually?&lt;/del&gt;&lt;/p&gt;

&lt;iframe width=&quot;560&quot; height=&quot;315&quot; src=&quot;https://www.youtube.com/embed/_-oqnf9gYuE?si=sOh9tujGCab9fHcZ&quot; title=&quot;YouTube video player&quot; frameborder=&quot;0&quot; allow=&quot;accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share&quot; referrerpolicy=&quot;strict-origin-when-cross-origin&quot; allowfullscreen=&quot;&quot;&gt;&lt;/iframe&gt;

&lt;p&gt;Here’s my &lt;a href=&quot;https://f.hrzn.ee/chiselcxx.pdf&quot;&gt;slides&lt;/a&gt;; the transcript follows.&lt;/p&gt;

&lt;p&gt;&lt;a id=&quot;more&quot;&gt;&lt;/a&gt;&lt;/p&gt;

&lt;!--more--&gt;

&lt;p&gt;&lt;img src=&quot;/assets/post-img/chiselcxx/slides.001.jpeg&quot; alt=&quot;Chisel and C++, together at last. Yosys% speedrun. 2024.05.27 — @kivikakk&quot; /&gt;&lt;/p&gt;

&lt;p&gt;Hi folks! I’m kivikakk, and I’m here to talk about connecting Chisel and C++,
leaning on Yosys for all the hard work.&lt;/p&gt;

&lt;p&gt;&lt;br /&gt;&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/post-img/chiselcxx/slides.002.jpeg&quot; alt=&quot;@kivikakk — no verilog pls. Senior systems engineer; no EE/DD background. Australian startups, agencies, GitHub. Started playing with FPGAs early 2023, OSS toolchains only. ~40 commits in Yosys, mostly in support of CXXRTL and alternative frontends. To the right of the slide is a snapshot of my GitHub profile showing recent work, and an excerpt of my commits in Yosys.&quot; /&gt;&lt;/p&gt;

&lt;p&gt;In the workplace I’m a “systems engineer”, which usually means weaving together
low- and high- level languages in dark ways; think writing Erlang C nodes,
combining Ruby, Go and C++, that kind of thing.&lt;/p&gt;

&lt;p&gt;In open source, I’m regrettably best-known for my work with Markdown. I have
zero electrical or digital background — or formal education — but after
microcontrollers failed to capture my interest, FPGAs succeeded, and I started
exploring in earnest last year.&lt;/p&gt;

&lt;p&gt;Now, this is something I do for fun, which meant Verilog and VHDL were
completely capable of turning me off this path forever. I’m really into
programming language theory and design, and uh, well, Verilog sure could’ve used
some of either. I found Amaranth (formerly nMigen) pretty quickly, and so I
started hacking on Yosys too. I’ve particularly enjoyed working on CXXRTL, which
is the focus of this talk. I spent about 9 months learning with Amaranth, but—&lt;/p&gt;

&lt;p&gt;&lt;br /&gt;&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/post-img/chiselcxx/slides.003.jpeg&quot; alt=&quot;A labrador in a science lab wearing safety goggles, pouring a beaker into a mug, with the text superimposed: &amp;quot;I have no idea what I&apos;m doing.&amp;quot;&quot; /&gt;&lt;/p&gt;

&lt;p&gt;I’m still this dog, and there are more perspectives out there.&lt;/p&gt;

&lt;p&gt;&lt;br /&gt;&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/post-img/chiselcxx/slides.004.jpeg&quot; alt=&quot;Screenshot of the Chisel homepage, an excerpt of Chisel code, and the SystemVerilog generated from that code.&quot; /&gt;&lt;/p&gt;

&lt;p&gt;I decided to learn Chisel, which is an HDL in Scala like Amaranth is an HDL in
Python. These aren’t high-level synthesis tools, you still describe hardware in
them, just using DSLs embedded in a regular programming language.&lt;/p&gt;

&lt;p&gt;You write code which generates hardware, in a metaprogramming kind of way,
except the metaprogramming is regular programming and the programming is circuit
definition instead. You run your code,&lt;/p&gt;

&lt;p&gt;and out pops something that can go into your toolchain’s frontend. Chisel
outputs SystemVerilog, and is easily configured to avoid constructs Yosys
doesn’t like.&lt;/p&gt;

&lt;p&gt;&lt;br /&gt;&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/post-img/chiselcxx/slides.005.jpeg&quot; alt=&quot;Screenshot of SystemVerilog code with an arrow pointing to the Yosys cat logo. From the Yosys logo arrows point to a Lattice iCE40 chip with the associated text &amp;quot;Project IceStorm, nextpnr-ice40&amp;quot;, a Lattice ECP5 chip with the text &amp;quot;Project Trellis, nextpnr-ecp5&amp;quot;, and a cute C++ logo.&quot; /&gt;&lt;/p&gt;

&lt;p&gt;So we have our Verilog, and we feed it into Yosys.&lt;/p&gt;

&lt;p&gt;Using the rest of the suite, we can synthesise for iCE40, ECP5 and more, but we can also target C++!&lt;/p&gt;

&lt;p&gt;Yosys has its own C++ backend, CXXRTL. It’s similar to Verilator, but has some
unique advantages. For starters, if you’re using Yosys anyway, we can avoid
adding another tool. Moreover, the C++ comes directly from Yosys’ internal RTL
model — you can perform transforms and optimisations and then generate the
simulation without a Verilog roundtrip. It also supports runtime introspection
of the design, as well as exposing its API to C. This makes it feasible to use
the generated simulations from any language with C FFI, like Rust or Zig.&lt;/p&gt;

&lt;p&gt;&lt;br /&gt;&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/post-img/chiselcxx/slides.006.jpeg&quot; alt=&quot;Screenshot of Chisel code describing an instruction set for a stack machine and the core logic of the stack machine, as well as some code that shows a memory read port generated from a static ROM.&quot; /&gt;&lt;/p&gt;

&lt;p&gt;One of the most fun parts, though, is the ability to instantiate blackboxes
anywhere in your hierarchy, which you implement in C++. I’m going to show you
real fast what that can look like.&lt;/p&gt;

&lt;p&gt;Here’s a tiny stack machine. It knows how to read and write bytes on UART, some
trivial stack manipulation, and how to jump back to zero. The implementation
itself isn’t very challenging, but the important part is that it gets its
instructions from a synchronous memory.&lt;/p&gt;

&lt;p&gt;For unit tests  in Chisel, I instantiate a vector like a ROM, and implement the
other side side of the read port, making sure to return data in the right cycle.
So far so good.&lt;/p&gt;

&lt;p&gt;&lt;br /&gt;&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/post-img/chiselcxx/slides.007.jpeg&quot; alt=&quot;C++ logo in the middle with a small flash chip above it. Three dot points are listed: A. Emulate in gateware. B. Emulate in C++ by monitoring the top-level IO pins. C. Emulate in C++ with a &amp;quot;whitebox&amp;quot; implementation that responds to your module&apos;s IOs.&quot; /&gt;&lt;/p&gt;

&lt;p&gt;Let’s initialise our instruction memory from SPI flash. The iCEBreaker I’m using
as a dev board puts its bitstream on one, and there’s plenty of room left for
user data. So I flash my little “ROM” into the upper half, and on reset the
gateware’s SPI reader module populates the memory from it before starting the
stack machine.&lt;/p&gt;

&lt;p&gt;What about our C++ simulation? We have a few options that are more interesting
than “ignore the flash reader”:&lt;/p&gt;

&lt;p&gt;A, we can do like we did with the static memory and emulate the SPI flash in
gateware, and put that into the design when elaborating for CXXRTL.&lt;/p&gt;

&lt;p&gt;This approach is fine for simple external interfaces, but for more complex ones,
such an implementation may not be feasible, and writing gateware for sim means
writing testbenches for your sim gateware. It’s also going to run as slow as any
other logic.&lt;/p&gt;

&lt;p&gt;B, we can emulate the SPI flash in C++ by watching the top-level output pins and
toggling input pins as necessary. This is straight-forward, though it means you
have to co-simulate your peripheral at the same time as stepping the design.&lt;/p&gt;

&lt;p&gt;C, we can drop a blackbox into the design, and hook up the SPI reader module to
the blackbox instead of external IO. Then, we implement the blackbox internals
in C++.&lt;/p&gt;

&lt;p&gt;&lt;br /&gt;&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/post-img/chiselcxx/slides.008.jpeg&quot; alt=&quot;Chisel code sample that describes a whitebox, pointing to a C++ class definition that matches it with no implementation details, pointing in turn to a subclass of that class which overrides and fills out the logic.&quot; /&gt;&lt;/p&gt;

&lt;p&gt;This is where CXXRTL’s blackbox support comes in: you give it a module interface
definition, and it generates a C++ class for it the same way it would for any
other non-flattened module in your design. Then you subclass it, implementing
logic internal to the blackbox in C++, reacting to events at the simulation step
level, without having to rewrite your whole simulation driver into an event
loop.&lt;/p&gt;

&lt;p&gt;This is super powerful, and it’s a lot easier to implement a peripheral in
full-blown C++ than it is in gateware.&lt;/p&gt;

&lt;p&gt;&lt;br /&gt;&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/post-img/chiselcxx/slides.009.jpeg&quot; alt=&quot;C++ logo in the middle with the flash chip again. Now a fourth point is added: D. Simulate the module itself with a blackbox implementation that produces the right IOs.&quot; /&gt;&lt;/p&gt;

&lt;p&gt;Now, I tend to call this approach a “whitebox” implementation, to contrast with—&lt;/p&gt;

&lt;p&gt;D, take the SPI reader out of the design, and drop in a blackbox which emulates
the reader’s interface instead.&lt;/p&gt;

&lt;p&gt;&lt;br /&gt;&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/post-img/chiselcxx/slides.010.jpeg&quot; alt=&quot;Chisel code sample that dsecribes a blackbox, and the C++ logic in its subclass.&quot; /&gt;&lt;/p&gt;

&lt;p&gt;So whereas the whitebox watches chip-select and data-in and toggles data-out
accordingly, painstakingly pretending to be a real flash module, this blackbox
goes one level higher, and monitors the read strobe from your design and
responds to it directly. This can significantly speed up your simulation,
particularly if your design clock rate is high but the peripheral’s is much
lower, like in I²C.&lt;/p&gt;

&lt;p&gt;As with the other non-gateware options, you can source the data from a file on
disk, a buffer compiled in, or from the network or whatever you like, it’s your
C++ code.&lt;/p&gt;

&lt;p&gt;&lt;br /&gt;&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/post-img/chiselcxx/slides.011.jpeg&quot; alt=&quot;Chisel code sample showing target specific wiring for three platforms: IceBreaker, CXXRTL whitebox, CXXRTL blackbox.&quot; /&gt;&lt;/p&gt;

&lt;p&gt;This is the target-specific gateware for this example, all in the top-level
module. Most of this depends on my little framework for Chisel, but it’s all
just ergonomics and instrumenting Yosys really.&lt;/p&gt;

&lt;p&gt;For iCEBreaker, we instantiate a real UART driver and wire it up to the IO pins;
its control interface is connected to the stack machine. The CXXRTL targets skip
the UART and just expose the control interface at the top level. Those are what
the C++ sim driver interacts with.&lt;/p&gt;

&lt;p&gt;Similarly, for iCEBreaker we instantiate the flash reader, connect its pins and
hook its control interface to a wire bundle. The whitebox also instantiates the
flash reader and wires the control interface, but connects its pins to the C++
whitebox module instead. The blackbox skips the reader entirely, and instead
connects the control interface wire bundle to the C++ blackbox module.&lt;/p&gt;

&lt;p&gt;&lt;br /&gt;&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/post-img/chiselcxx/slides.012.jpeg&quot; alt=&quot;Two README screenshots both demonstrating hardware and matching software simulations. On the left is a photograph of a 4-digit 7-segment display spelling out the word &amp;quot;pong&amp;quot;. Underneath is a screenshot of software showing the same display and the same output. On the right is a screenshot of some software demonstrating a 128x128 OLED with some text and ASCII drawing characters on it. There&apos;s a photograph of an OLED display wired up to an IceBreaker showing the same output.&quot; /&gt;&lt;/p&gt;

&lt;p&gt;So, I really enjoy this approach! It’s a lot of fun, and being able to stub out
my design at various levels turns out to be really handy as my logic gets more
involved. CXXRTL’s simulation isn’t as fast as Verilator’s, but it’s within the
same order of magnitude, and it lets me make these changes essentially
hot-swappable, because the blackboxes are instantiated if and where they occur
in the design.&lt;/p&gt;

&lt;p&gt;&lt;br /&gt;&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/post-img/chiselcxx/slides.013.jpeg&quot; alt=&quot;A list of links and acknowledgements, included below the text that follows.&quot; /&gt;&lt;/p&gt;

&lt;p&gt;And that’s it! The main takeaway really is that you can do this kind of thing
with Yosys with any HDL — none of this is Chisel specific, it’s just what I
happened to pick. Thanks all.&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;&lt;a href=&quot;https://github.com/kivikakk/spifrbb&quot;&gt;SPI flash reader example&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;https://github.com/chryse-hdl/chryse&quot;&gt;Chryse, experimental framework for Chisel/Yosys&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;https://tomverbeure.github.io/2020/08/08/CXXRTL-the-New-Yosys-Simulation-Backend.html&quot;&gt;CXXRTL primer (a little out of date now)&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;https://x.com/sawaratsuki1004&quot;&gt;C++ logo by Sawaratsuki&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
</content>
    
 </entry>
 
 <entry>
   <title>Digital design bash.org</title>
   <updated>2024-05-10T00:00:00+10:00</updated>
   <id>https://kivikakk.ee/2024/05/10/digital-design-bash-org</id>
    
     <link href="https://kivikakk.ee/2024/05/10/digital-design-bash-org/"/>
     <content type="html">&lt;p&gt;Identities changed to protect the innocent.&lt;/p&gt;

&lt;p&gt;&lt;code style=&quot;font-family: monospace;&quot;&gt;
&lt;span style=&quot;opacity: 0.6&quot;&gt;&amp;lt;&lt;/span&gt;&lt;span style=&quot;color: green&quot;&gt;pestopasta&lt;/span&gt;&lt;span style=&quot;opacity: 0.6&quot;&gt;&amp;gt;&lt;/span&gt; How do you do 128bit memory buses and stuff like that&lt;br /&gt;
&lt;span style=&quot;opacity: 0.6&quot;&gt;&amp;lt;&lt;/span&gt;&lt;span style=&quot;color: green&quot;&gt;pestopasta&lt;/span&gt;&lt;span style=&quot;opacity: 0.6&quot;&gt;&amp;gt;&lt;/span&gt; Like what is going on in those 128 bits&lt;br /&gt;
&lt;span style=&quot;opacity: 0.6&quot;&gt;&amp;lt;&lt;/span&gt;&lt;span style=&quot;color: orange&quot;&gt;Rice&lt;/span&gt;&lt;span style=&quot;opacity: 0.6&quot;&gt;&amp;gt;&lt;/span&gt; uh, data that&apos;s being read from or written to memory?&lt;br /&gt;
&lt;span style=&quot;opacity: 0.6&quot;&gt;&amp;lt;&lt;/span&gt;&lt;span style=&quot;color: orange&quot;&gt;Rice&lt;/span&gt;&lt;span style=&quot;opacity: 0.6&quot;&gt;&amp;gt;&lt;/span&gt; What is the issue you&apos;re not understanding&lt;br /&gt;
&lt;span style=&quot;opacity: 0.6&quot;&gt;&amp;lt;&lt;/span&gt;&lt;span style=&quot;color: green&quot;&gt;pestopasta&lt;/span&gt;&lt;span style=&quot;opacity: 0.6&quot;&gt;&amp;gt;&lt;/span&gt; &lt;span style=&quot;opacity: 0.6&quot;&gt;@&lt;/span&gt;&lt;span style=&quot;color: orange&quot;&gt;Rice&lt;/span&gt; What is transferred over it&lt;br /&gt;
&lt;span style=&quot;opacity: 0.6&quot;&gt;&amp;lt;&lt;/span&gt;&lt;span style=&quot;color: pink&quot;&gt;HamSandwich&lt;/span&gt;&lt;span style=&quot;opacity: 0.6&quot;&gt;&amp;gt;&lt;/span&gt; data that&apos;s being read from or written to memory 👀&lt;br /&gt;
&lt;span style=&quot;opacity: 0.6&quot;&gt;&amp;lt;&lt;/span&gt;&lt;span style=&quot;color: green&quot;&gt;pestopasta&lt;/span&gt;&lt;span style=&quot;opacity: 0.6&quot;&gt;&amp;gt;&lt;/span&gt; &lt;span style=&quot;opacity: 0.6&quot;&gt;@&lt;/span&gt;&lt;span style=&quot;color: pink&quot;&gt;HamSandwich&lt;/span&gt; Yea. How do you manage 128 bits though. That&apos;s a lot&lt;br /&gt;
&lt;span style=&quot;opacity: 0.6&quot;&gt;&amp;lt;&lt;/span&gt;&lt;span style=&quot;color: pink&quot;&gt;HamSandwich&lt;/span&gt;&lt;span style=&quot;opacity: 0.6&quot;&gt;&amp;gt;&lt;/span&gt; They are written to and from caches via the cache controller, not the core. The core has a maximum of 32-bit access.&lt;br /&gt;
&lt;span style=&quot;opacity: 0.6&quot;&gt;&amp;lt;&lt;/span&gt;&lt;span style=&quot;color: orange&quot;&gt;Rice&lt;/span&gt;&lt;span style=&quot;opacity: 0.6&quot;&gt;&amp;gt;&lt;/span&gt; &lt;span style=&quot;opacity: 0.6&quot;&gt;@&lt;/span&gt;&lt;span style=&quot;color: green&quot;&gt;pestopasta&lt;/span&gt; ...the same way you handle 32 or 64 bits of data, just double or quadruple?&lt;br /&gt;
&lt;span style=&quot;opacity: 0.6&quot;&gt;&amp;lt;&lt;/span&gt;&lt;span style=&quot;color: green&quot;&gt;pestopasta&lt;/span&gt;&lt;span style=&quot;opacity: 0.6&quot;&gt;&amp;gt;&lt;/span&gt; I don&apos;t know what you mean. Is there a video explaining this?
&lt;/code&gt;&lt;/p&gt;
</content>
    
 </entry>
 
 <entry>
   <title>Amaranth to Chisel</title>
   <updated>2024-05-09T00:00:00+10:00</updated>
   <id>https://kivikakk.ee/2024/05/09/amaranth-to-chisel</id>
    
      <link href="https://lottia.net/notes/0007-amaranth-to-chisel.html"/>
      <content type="html">Learning your second HDL is kinda like learning your second programming language. Or just learning your second language.</content>
    
 </entry>
 
 <entry>
   <title>Post-vaccination advice by country</title>
   <updated>2024-05-03T00:00:00+10:00</updated>
   <id>https://kivikakk.ee/eesti/2024/05/03/post-vaccination-advice-by-country</id>
    
     <link href="https://kivikakk.ee/eesti/2024/05/03/post-vaccination-advice-by-country/"/>
     <content type="html">&lt;p&gt;As dispensed by the vaccinating staff:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Australia: Now, no heavy lifting for a few hours!&lt;/li&gt;
  &lt;li&gt;Estonia: Mm.. probably better not to go in sauna today.&lt;/li&gt;
&lt;/ul&gt;
</content>
    
 </entry>
 
 <entry>
   <title>Volbripäev</title>
   <updated>2024-05-01T00:00:00+10:00</updated>
   <id>https://kivikakk.ee/eesti/2024/05/01/volbripäev</id>
    
     <link href="https://kivikakk.ee/eesti/2024/05/01/volbrip%C3%A4ev/"/>
     <content type="html">&lt;p&gt;&lt;a href=&quot;https://kivikakk.ee/eesti/2024/04/29/naming/&quot;&gt;Never mind that&lt;/a&gt;, lol. I have to update my Estonian documents to match
my Australian ones for my partner’s immigration, so back to Ashe it is!&lt;/p&gt;

</content>
    
 </entry>
 
 <entry>
   <title>Naming</title>
   <updated>2024-04-29T00:00:00+10:00</updated>
   <id>https://kivikakk.ee/eesti/2024/04/29/naming</id>
    
     <link href="https://kivikakk.ee/eesti/2024/04/29/naming/"/>
     <content type="html">&lt;p&gt;Noting that, having moved (“returned!”) to Estonia, I’m going by Amelia (or Amy)
again, since it’s the name on all my documentation here.
&lt;a href=&quot;/assets/post-img/eesti/head-isu.jpg&quot; title=&quot;Photo of a power distribution box, with graffiti art of a very long, splotchy, odd looking dog, and the text &apos;Head isu! #14&apos; above it.&quot;&gt;Head isu!&lt;/a&gt;&lt;/p&gt;
</content>
    
 </entry>
 
 <entry>
   <title>Continued nyõnks</title>
   <updated>2024-04-18T00:00:00+10:00</updated>
   <id>https://kivikakk.ee/eesti/2024/04/18/continued-nyonks</id>
    
     <link href="https://kivikakk.ee/eesti/2024/04/18/continued-nyonks/"/>
     <content type="html">&lt;p&gt;It’s been a week since I last wrote. Some more little bits and pieces.&lt;/p&gt;

&lt;p&gt;The DPDR/dissociation has become even bit worse; somehow still keeping it
together. Watching The Princess Bride with Annie last night helped.&lt;/p&gt;

&lt;p&gt;Miscellany:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;
    &lt;p&gt;Presented to the Police and Border Guard Board office to renew my ID card. Sat
down in front of the worker once my number was called and asked, “kas te
räägite inglise keelt?” (do you speak English?). With almost a tinge of pride,
she replied, “ei!” (no!). It took everything in me not to &lt;a href=&quot;https://youtu.be/CgjMfaQ94Co?si=Qig3i8dlKwjSIOC3&amp;amp;t=117&quot;&gt;burst out
laughing&lt;/a&gt;.&lt;/p&gt;

    &lt;ul&gt;
      &lt;li&gt;There’s an oddity in that I can pronounce Estonian very well, despite not
really knowing much of it (i.e. reading out text, like from a menu or Google
Translate or whatever, I sound semi-native, but actually producing that text
myself I’m nowhere near yet). This makes for fun situations.&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;We have a new home :) An r/eesti comment mentioned someone once found an
apartment in Tallinn on day 5 and was moved in by day 7. That was encouraging,
if a little unrealistic-seeming. But nope: we inspected ours on day 4 (!), and
moved in on day 8. Now we can ship our boxes from Australia.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Food’s so nice here.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Sure it’s 2° outside, that’s no problem, but then you walk into any building
and it’s 22° and you’re shedding layers as fast as you can. Other people just
seem not to?! I don’t get it.&lt;/p&gt;

    &lt;ul&gt;
      &lt;li&gt;
        &lt;p&gt;The worst part for me is that, even with all our apartment floor-heating
turned “off” (or as off as you can, which is to say, the set point is set
lower than the actual point), we’re on the top of the building, so it’s over
20° regardless. This is troublesome at night, and the street noise might be
a bit much to open the windows, but using the aircon when it’s below zero
outside feels a bit …&lt;/p&gt;
      &lt;/li&gt;
      &lt;li&gt;
        &lt;p&gt;Also, sharing a bedroom for the first time in so many years is a strange
experience, like I’m cosplaying being a different kind of adult.&lt;/p&gt;
      &lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Public transport’s so nice here — it’s already happened that we’ve taken
train+bus+tram all in the one day just out of convenience.&lt;/p&gt;

    &lt;ul&gt;
      &lt;li&gt;
        &lt;p&gt;Not that I’ve worked out how to get the free transit for locally-registered
residents working on my Ühiskaart yet.&lt;/p&gt;

        &lt;ul&gt;
          &lt;li&gt;
            &lt;p&gt;Is it only on TLT-operated busses? Is that it?? The information’s all very
vague.&lt;/p&gt;

            &lt;ul&gt;
              &lt;li&gt;DISREGARD THAT I S—eem to have gotten it on a further attempt with the
card reader.&lt;/li&gt;
            &lt;/ul&gt;
          &lt;/li&gt;
        &lt;/ul&gt;
      &lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;It just started snowing!?!!?&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;&lt;a href=&quot;/assets/post-img/eesti/juustuga.png&quot; title=&quot;Screenshot of a food delivery app, automatically translated from Estonian.
The section header is &apos;With justice&apos;, and the options are &apos;Without cheese&apos; and
&apos;With justice&apos;.&quot;&gt;Without cheese or with justice?&lt;/a&gt; (&lt;a href=&quot;https://en.wiktionary.org/wiki/juust#Estonian&quot;&gt;juust&lt;/a&gt;)&lt;/p&gt;
  &lt;/li&gt;
&lt;/ul&gt;
</content>
    
 </entry>
 
 <entry>
   <title>Early nyõnks</title>
   <updated>2024-04-11T00:00:00+10:00</updated>
   <id>https://kivikakk.ee/eesti/2024/04/11/early-nyonks</id>
    
     <link href="https://kivikakk.ee/eesti/2024/04/11/early-nyonks/"/>
     <content type="html">&lt;p&gt;We’ve been in the country a little over two days now. It’s been one of the most
serene times of my life, and even the flights from Australia seemed to pass in
an instant.&lt;/p&gt;

&lt;p&gt;This is in some part due to intense derealisation, but as uncanny as life has
seemed for months now, I can’t say I hate it — I’ve been able to do so many
things without crushing anxiety killing it, whether due to COVID or whatever
else. (Not that the skin on my face is recovered yet from the 30 hours of
wearing an elastomeric respirator.)&lt;/p&gt;

&lt;p&gt;Some fun moments and things I don’t want to forget:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;
    &lt;p&gt;We had 6 hours to kill between getting out of Tallinn airport and our
temporary accommodation becoming available to us. We walked around the Old
Town after storing our bags, still wearing the clothes from our flights. I had
on a plain blue dress and black knee-high compression socks, and didn’t
realise quite how I looked until we spotted two men, bottles in hand and
visibly drunk, who spotted us in turn and called out to me, “Гермааааания!
Айайайайяаа!”, and all four of us were laughing.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;&lt;a href=&quot;/assets/post-img/eesti/thisguy.jpg&quot; title=&quot;A photograph of an ape (?)
sculpture (??) painted bright red, sitting in the front garden of some house.
A large EU flag is visible hanging on an adjacent property.&quot;&gt;This guy&lt;/a&gt;.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;&lt;a href=&quot;/assets/post-img/eesti/thisgirl.jpg&quot; title=&quot;A photograph of Annie
standing in front of a wall covered in various graffiti, herself right next to
a pretty flower in red and white. Also visible is &apos;follow sheep, not the
state&apos; next to an anarchist circle-A symbol. In the background are trees
(without leaves yet) and tourists.&quot;&gt;This girl&lt;/a&gt;.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Beer and cider at 3pm in an upstairs nook of a café with no-one in it but us
(still waiting for our accommodation to become available). The owner came up
and gave us some cake that was left-over from a birthday party.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Walking to the Rimi express on my own first thing in the morning to get cereal
and &lt;del&gt;milk&lt;/del&gt; oat drink. I feel so calm here.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Bike riding to Rocca al Mare :) We took panniers and had probably the nicest
and most “successful” shopping experience of my life. (warm clothes for the
new climate!)&lt;/p&gt;

    &lt;ul&gt;
      &lt;li&gt;NEXT TIME HIRING A MOUNTAIN BIKE, PLEASE, “CITY BIKES” ARE A GRIFT&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Vapiano vegan chicken pastaaaaaaaaa, next time I will take much more bread&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Going from straight-up pre-emptive “Inglise keel?” at the beginning of
exchanges with service staff to full Estonian in the course of a day, and just
hoping I was guessing the questions right when I didn’t know the operative
words. (usually assuming it’s “do you have a membership with us?”, since “ei
ole” seems to get me through …)&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;We were having a sit in the square after exploring the new town today,
people-watching, when we noticed a group of younger Russian girls having fun,
chasing pigeons, etc. After a while, I began to notice the most adventurous
one tracing out a circuitous path from their spot under the town hall over to
us, inverted phone in hand, ready to record an exchange.&lt;/p&gt;

    &lt;p&gt;Without much ado, she finally came over and sat down right next to me, holding
her phone up and asking something in somewhat-shy-and-therefore-mumbled
Russian. I just said “Что?”, and even more embarrassed she repeated her
question. I got the gist of it and first said “Я не знаю русский.. English?”
She haltingly replied, “m-maybe?”, and so I finally answered her question and
said “Half-half”, using a gesture to reinforce the meaning.&lt;/p&gt;

    &lt;p&gt;The question was if I was a boy or a girl, which is a pretty common one when
I’ve been overseas. She then apologised and said she didn’t mean to be rude,
and that she thought it was a totally fine thing to be different and not at
all a problem here, and I made sure she knew I didn’t mind the question at
all! It was pretty daring of her to just come over and ask; I think she and
her friends must’ve been observing us two as much as we had been watching
them. It was a really nice exchange across cultures and generations.&lt;/p&gt;

    &lt;p&gt;(I only realised once home that I was wearing a &lt;a href=&quot;https://transjustice.org.au/&quot;&gt;Trans Justice
Project&lt;/a&gt; T-shirt underneath my coat! Missed
opportunity.)&lt;/p&gt;
  &lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Tomorrow we take a train trip out to Narva so I can renew my ID card, explore
there for a day, and then get back in time for an evening inspection of what
might be our future home. Fingers crossed :)&lt;/p&gt;
</content>
    
 </entry>
 
 <entry>
   <title>Literate pickle</title>
   <updated>2024-04-03T00:00:00+11:00</updated>
   <id>https://kivikakk.ee/2024/04/03/literate-pickle</id>
    
     <link href="https://kivikakk.ee/2024/04/03/literate-pickle/"/>
     <content type="html">&lt;p&gt;I’ve been deleting hundreds of accounts and cleaning up many more while
preparing for the move, and I found I have an npm account with one (1)
package, last published May 2012 (!): &lt;a href=&quot;https://www.npmjs.com/package/allium&quot;&gt;allium&lt;/a&gt;,
a parser for the &lt;a href=&quot;https://en.wikipedia.org/wiki/Cucumber_(software)#Gherkin_language&quot;&gt;“Gherkin” syntax&lt;/a&gt;
used to define Cucumber BDD tests, back when that was all the rage.&lt;/p&gt;

&lt;p&gt;The README captured by npm refers to the literate source code as once published
on GitHub Pages. That was long ago, but I figured it might be fun to look at
again. And it is! Have look: &lt;a href=&quot;https://f.hrzn.ee/allium/&quot;&gt;https://f.hrzn.ee/allium/&lt;/a&gt;&lt;/p&gt;
</content>
    
 </entry>
 
 <entry>
   <title>Eesti keele abilised</title>
   <updated>2024-03-25T00:00:00+11:00</updated>
   <id>https://kivikakk.ee/2024/03/25/eesti-keele-abilised</id>
    
     <link href="https://kivikakk.ee/2024/03/25/eesti-keele-abilised/"/>
     <content type="html">&lt;p&gt;Here’s a little tool I’ve made to help my partner and I while learning Estonian.
Very much nothing special, but I’ll hopefully find more resources to add to it
over time!&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://eka.kivikakk.ee&quot;&gt;Eesti keele abilised&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;(Previously on this topic: &lt;a href=&quot;https://kivikakk.ee/estonian/2016/04/18/estonian-morphology-guide-mirror.html&quot;&gt;Estonian Morphology Guide
(mirror)&lt;/a&gt;.
I no longer collect analytics on my site, but when I did it was consistently one
of the most-visited parts of it, presumably because the link broke at some
point. I’ve just found its new canonical home, though, so I’ve updated the
link!)&lt;/p&gt;
</content>
    
 </entry>
 
 <entry>
   <title>lottia notes</title>
   <updated>2024-01-03T00:00:00+11:00</updated>
   <id>https://kivikakk.ee/2024/01/03/lottia-notes</id>
    
     <link href="https://kivikakk.ee/2024/01/03/lottia-notes/"/>
     <content type="html">&lt;p&gt;I’ve been writing notes on FPGA/digital design and Nix over at
&lt;a href=&quot;https://lottia.net/notes&quot;&gt;&lt;em&gt;lottia notes&lt;/em&gt;&lt;/a&gt;.&lt;/p&gt;
</content>
    
 </entry>
 
 <entry>
   <title>Comrak on Akkoma</title>
   <updated>2024-01-02T00:00:00+11:00</updated>
   <id>https://kivikakk.ee/2024/01/02/comrak-on-akkoma</id>
    
      <link href="https://lottia.net/notes/0006-comrak-on-akkoma.html"/>
      <content type="html">I've barely touched Elixir before. How hard could shoving a Rust dependency into it be?</content>
    
 </entry>
 
 <entry>
   <title>Jambalam</title>
   <updated>2023-09-18T00:00:00+10:00</updated>
   <id>https://kivikakk.ee/2023/09/18/jambalam</id>
    
      <link href="https://lottia.net/notes/0005-jambalam.html"/>
      <content type="html">Have it your way.</content>
    
 </entry>
 
 <entry>
   <title>Happy birthday!</title>
   <updated>2023-08-23T00:00:00+10:00</updated>
   <id>https://kivikakk.ee/2023/08/23/happy-birthday</id>
    
      <link href="https://lottia.net/notes/0004-happy-birthday.html"/>
      <content type="html">It's always your birthday!</content>
    
 </entry>
 
 <entry>
   <title>Nix revisited</title>
   <updated>2023-07-15T00:00:00+10:00</updated>
   <id>https://kivikakk.ee/2023/07/15/nix-revisited</id>
    
      <link href="https://lottia.net/notes/0003-nix-revisited.html"/>
      <content type="html">An unsystematic collection of thoughts while adopting Nix.</content>
    
 </entry>
 
 <entry>
   <title>Untangling cycles</title>
   <updated>2023-06-29T00:00:00+10:00</updated>
   <id>https://kivikakk.ee/2023/06/29/untangling-cycles</id>
    
      <link href="https://lottia.net/notes/0002-untangling-cycles.html"/>
      <content type="html">Wherein the author perceives time in multiple lights, simultaneously, and logic does too, but, uh, differently.</content>
    
 </entry>
 
 <entry>
   <title>Installing an HDL toolchain from source</title>
   <updated>2023-06-27T00:00:00+10:00</updated>
   <id>https://kivikakk.ee/2023/06/27/hdl-toolchain-source</id>
    
      <link href="https://lottia.net/notes/0001-hdl-toolchain-source.html"/>
      <content type="html">A fairly detailed guide on building and installing a gateware toolchain in a self-contained and repeatable way.</content>
    
 </entry>
 
 <entry>
   <title>DDR the second!!</title>
   <updated>2021-08-16T00:00:00+10:00</updated>
   <id>https://kivikakk.ee/2021/08/16/ddr-the-second</id>
    
     <link href="https://kivikakk.ee/2021/08/16/ddr-the-second/"/>
     <content type="html">&lt;p&gt;Two weeks gets incremental improvements.&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/post-img/moreonenight2.jpg&quot; alt=&quot;More One Night score (AAA 97.13%)&quot; /&gt;
&lt;img src=&quot;/assets/post-img/badapple2.jpg&quot; alt=&quot;Bad Apple!! score (AAA 97.57%)&quot; /&gt;
&lt;img src=&quot;/assets/post-img/danceallnight2.jpg&quot; alt=&quot;DANCE ALL NIGHT score (AAA 98.55%)&quot; /&gt;&lt;/p&gt;
</content>
    
 </entry>
 
 <entry>
   <title>DDR</title>
   <updated>2021-08-02T00:00:00+10:00</updated>
   <id>https://kivikakk.ee/2021/08/02/ddr</id>
    
     <link href="https://kivikakk.ee/2021/08/02/ddr/"/>
     <content type="html">&lt;p&gt;Not much to report lately.  DDR continues to be fun, though I think given these
timings I’m past due for replacing the control board in the pad so I can get
better than 8ms resolution on steps.&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/post-img/moreonenight.jpg&quot; alt=&quot;More One Night score (AA 97.13%)&quot; /&gt;
&lt;img src=&quot;/assets/post-img/badapple.jpg&quot; alt=&quot;Bad Apple!! score (AAA 96.65%)&quot; /&gt;
&lt;img src=&quot;/assets/post-img/danceallnight.jpg&quot; alt=&quot;DANCE ALL NIGHT score (AAA 98.34%)&quot; /&gt;&lt;/p&gt;
</content>
    
 </entry>
 
 <entry>
   <title>You Got Me</title>
   <updated>2021-04-27T00:00:00+10:00</updated>
   <id>https://kivikakk.ee/2021/04/27/you-got-me</id>
    
     <link href="https://kivikakk.ee/2021/04/27/you-got-me/"/>
     <content type="html">&lt;center&gt;&lt;iframe style=&quot;border: 0; width: 350px; height: 442px;&quot; src=&quot;https://bandcamp.com/EmbeddedPlayer/track=3439825641/size=large/bgcol=333333/linkcol=fe7eaf/tracklist=false/transparent=true/&quot; seamless=&quot;&quot;&gt;&lt;a href=&quot;https://tanoc.bandcamp.com/track/you-got-me&quot;&gt;You Got Me by USAO &amp;amp; Shandy Kubota&lt;/a&gt;&lt;/iframe&gt;&lt;/center&gt;
</content>
    
 </entry>
 
 <entry>
   <title>MAX 300</title>
   <updated>2021-04-24T00:00:00+10:00</updated>
   <id>https://kivikakk.ee/2021/04/24/max-300</id>
    
     <link href="https://kivikakk.ee/2021/04/24/max-300/"/>
     <content type="html">&lt;p&gt;&lt;a href=&quot;https://remywiki.com/MAX_300&quot;&gt;MAX 300&lt;/a&gt; was the first&lt;sup id=&quot;fnref:sakura&quot; role=&quot;doc-noteref&quot;&gt;&lt;a href=&quot;#fn:sakura&quot; class=&quot;footnote&quot; rel=&quot;footnote&quot;&gt;1&lt;/a&gt;&lt;/sup&gt; level 10
song I cleared at the arcade.  I learned to play DDR with my older brother; I
distinctly recall the sensation of finally being better than him at a
game—whether it was Descent, StarCraft, anything, he always had such a lead.&lt;/p&gt;

&lt;p&gt;I started DDR a bit later than him and had some catching up to do, but
eventually crossed the level 9 mark before he did.  It turned out rhythm games
would be a good place for me to excel.&lt;/p&gt;

&lt;p&gt;He lives overseas now, and at some point got an &lt;a href=&quot;https://www.maty-taneczne.pl&quot;&gt;L-TEK DDR
pad&lt;/a&gt;.  I was a bit envious, but felt the shipping
expense—far worse to Australia than the US—was too hard to justify.
Pandemic closing the gym made it muuuuuch more palatable, and plus it’d mean
I’d get to play DDR with my brother again in a way.&lt;/p&gt;

&lt;p&gt;It’s been a very good way to get fit again, and hitting old milestones again is
a lot of fun.  I’ve done some other harder ones already, but today was the day
for clearing MAX&lt;nobr&gt; &lt;/nobr&gt;300 again.  My scores are &lt;em&gt;much&lt;/em&gt; better than my
13-year-old self’s, even though my endurance isn’t.&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/post-img/max-300.jpg&quot; alt=&quot;MAX 300 score (87.23%)&quot; /&gt;&lt;/p&gt;

&lt;div class=&quot;footnotes&quot; role=&quot;doc-endnotes&quot;&gt;
  &lt;ol&gt;
    &lt;li id=&quot;fn:sakura&quot; role=&quot;doc-endnote&quot;&gt;
      &lt;p&gt;Not counting 桜 or bag here. &lt;a href=&quot;#fnref:sakura&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
  &lt;/ol&gt;
&lt;/div&gt;
</content>
    
 </entry>
 
 <entry>
   <title>As best we can tell, the pub is shut</title>
   <updated>2021-03-29T00:00:00+11:00</updated>
   <id>https://kivikakk.ee/2021/03/29/as-best-we-can-tell-the-pub-is-shut</id>
    
     <link href="https://kivikakk.ee/2021/03/29/as-best-we-can-tell-the-pub-is-shut/"/>
     <content type="html">&lt;p&gt;I was alerted by a commenter that it’s been more than a year, now, since this
video dropped:&lt;/p&gt;

&lt;center&gt;&lt;iframe width=&quot;560&quot; height=&quot;315&quot; src=&quot;https://www.youtube.com/embed/83cEEyTuAMM&quot; title=&quot;YouTube video player&quot; frameborder=&quot;0&quot; allow=&quot;accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture&quot; allowfullscreen=&quot;&quot;&gt;&lt;/iframe&gt;&lt;/center&gt;

&lt;p&gt;COVID measures had already begun to be implemented; national borders shut, most
schools already closed.  Watching this press conference, &lt;a href=&quot;https://www.youtube.com/watch?v=SqDP8SnPVA0&quot;&gt;a scene from The
Simpsons&lt;/a&gt; played in my mind.  I’d
been getting a little comfy with a video editing program to record &lt;a href=&quot;https://www.youtube.com/watch?v=8tzZYDXHHG4&quot;&gt;IIDX
plays&lt;/a&gt;, so I gave it a crack.&lt;/p&gt;

&lt;p&gt;I don’t really have networks to tap, but Niki liked it so much she diligently
dropped it into comments on Facebook and Twitter replies wherever it seemed
appropriate.  Before I knew it, I had a moderately popular YouTube video.  It
entered the popular discourse when it was &lt;a href=&quot;https://www.youtube.com/watch?v=7hOK5JF5XGA&quot;&gt;further
remixed&lt;/a&gt;, but if you ask me, the
Trump oversamples are just kinda gross.&lt;/p&gt;

&lt;p&gt;One thing that’s been interesting to see has been how the popularity of the
video corresponded with (literally) viral events:&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/post-img/views.png&quot; alt=&quot;A graph showing the views for the video&quot; /&gt;&lt;/p&gt;

&lt;p&gt;The three major events were:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Late March, video released, Dan Andrews said “get on the beers”.&lt;/li&gt;
  &lt;li&gt;Mid-May, first lockdown restrictions eased.&lt;/li&gt;
  &lt;li&gt;&lt;a id=&quot;more&quot;&gt;&lt;/a&gt;October 26, Victoria recorded zero new cases/deaths for the
first time since June.  Dan reported that he “might go a little higher up the
shelf” than beers.&lt;/li&gt;
&lt;/ul&gt;

&lt;!--more--&gt;

&lt;hr /&gt;

&lt;p&gt;Daniel Andrews, Victoria’s Premier, is widely liked, and he’s largely been
credited with producing the results we’ve had locally, the likes of which have
only really been seen in Taiwan, New Zealand and Singapore.  We’ve had our
share of anti-lockdown protests, too, but overall the sentiment has been that
Victorians have been willing to accept discomfort up front to mitigate a
disaster later.&lt;/p&gt;

&lt;p&gt;Sharing this video online more recently has had some interesting reactions;
namely, Americans being like “lol government interference!!!”  Melbourne’s
second lockdown was 112 days long (July 9 until October 28), and it was
challenging in its own ways, and for some very difficult.  I don’t seek to deny
that, but the practical upshot has been a few dozen cases in the last six
months.&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/post-img/community.png&quot; alt=&quot;Graph from covid19data.com.au showing community spread of COVID in Victoria&quot; /&gt;&lt;/p&gt;

&lt;p&gt;It’s a fucking pandemic.  You don’t just ask everyone nicely to please do the
right thing, because that’s not human beings work at the population level.  If
you &lt;em&gt;actually&lt;/em&gt; want to beat it, you need to be realistic about what works.
Ideals fall flat in the face of an airborne pathogen.  If there are any doubts
about that, please consult &lt;a href=&quot;https://coronavirus.jhu.edu/map.html&quot;&gt;the big red number at the top
left&lt;/a&gt;.  Empirically, government
interference gets it done&lt;sup id=&quot;fnref:hif&quot; role=&quot;doc-noteref&quot;&gt;&lt;a href=&quot;#fn:hif&quot; class=&quot;footnote&quot; rel=&quot;footnote&quot;&gt;1&lt;/a&gt;&lt;/sup&gt;.&lt;/p&gt;

&lt;p&gt;Someone in chat a couple weeks back lamented how they wish they could go out
again like normal, and I didn’t have the heart to tell them that I had been for
nearly half a year now, safely; I just said, “in some places in the world you
can.  soon!”  Y’know, vaccinations proceeding apace and all that.&lt;/p&gt;

&lt;p&gt;Another person agreed, saying Japan was reopening for dine-in in a couple
weeks, and the first said how much they wished the borders were open so they
could visit.  Japan reported 2,080 new cases this weekend, a steady increase
from the most recent low 7-day average of ~900 in early March.&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/post-img/japan.png&quot; alt=&quot;Japan&apos;s graph&quot; /&gt;&lt;/p&gt;

&lt;p&gt;That’s not what I meant.&lt;/p&gt;

&lt;div class=&quot;footnotes&quot; role=&quot;doc-endnotes&quot;&gt;
  &lt;ol&gt;
    &lt;li id=&quot;fn:hif&quot; role=&quot;doc-endnote&quot;&gt;
      &lt;p&gt;Never mind the fact that, say, Australia and New Zealand both rank higher in the &lt;a href=&quot;https://www.cato.org/human-freedom-index/2020&quot;&gt;Human Freedom Index&lt;/a&gt; than the US—what matters is how you feel when you say it! &lt;a href=&quot;#fnref:hif&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
  &lt;/ol&gt;
&lt;/div&gt;
</content>
    
 </entry>
 
 <entry>
   <title>Knowing when to look past your code</title>
   <updated>2021-02-28T00:00:00+11:00</updated>
   <id>https://kivikakk.ee/2021/02/28/loader</id>
    
     <link href="https://kivikakk.ee/2021/02/28/loader/"/>
     <content type="html">&lt;p&gt;There’s a weird tension in programming — on the one hand, as you learn the
ropes, you (hopefully) learn very quickly that the problem is almost &lt;em&gt;always&lt;/em&gt;
in your code, and not, say, the compiler, stdlib, kernel, etc.  This is usually
very correct; the people who’ve worked on those things have many times the
experience you did when you decided that there must be a bug in &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;printf&lt;/code&gt; or
something.&lt;/p&gt;

&lt;p&gt;You’ll later realise you tried to print something through a pointer to a
stack-allocated variable that’s long since gone.  These accusations tend to
wane as you gain familiarity with your subject matter, and wax as you step out
into lands populated with ever more footguns, exposing more of the architecture
than you ever suspected was there.  (See also: the emails from me to the libev
mailing list in 2011.)&lt;/p&gt;

&lt;p&gt;At some point, though, your journies will take you to places where things
aren’t so clear cut, and you’ll start to gain a sixth sense; a kind of visceral
experience that &lt;em&gt;things are not as they have been promised to be&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;&lt;a id=&quot;more&quot;&gt;&lt;/a&gt;A few weeks ago, that sixth sense whispered in my ear: “what
if, instead of your cruddy bootloader written in a pre-1.0 systems language for
a platform you don’t fully understand, it’s the 20 year-old project with 80,000
commits that’s wrong?”  And it was right.&lt;/p&gt;

&lt;!--more--&gt;

&lt;hr /&gt;

&lt;p&gt;&lt;a href=&quot;https://github.com/kivikakk/daintree&quot;&gt;Daintree&lt;/a&gt;’s bootloader, dainboot, worked
great on QEMU, but would fail hard and fast on hardware with synchronous
aborts.  It’s a UEFI application, which means we get a lot for free – see how
easy it is to &lt;a href=&quot;https://github.com/kivikakk/daintree/blob/c8ecf0131aba16fb6ef1991393b7088037526f77/dainboot/src/dainboot.zig#L98-L160&quot;&gt;search connected FAT filesystems for
binaries&lt;/a&gt;.
I don’t particularly want to spend much effort on a bootloader, so this makes
sense to me.&lt;/p&gt;

&lt;p&gt;QEMU comes bundled with a build of &lt;a href=&quot;https://www.tianocore.org&quot;&gt;TianoCore EDK2&lt;/a&gt;,
which makes it really easy to get started.  On my ROCKPro64 I have a build of
&lt;a href=&quot;https://www.denx.de/wiki/U-Boot&quot;&gt;U-Boot&lt;/a&gt; installed to the eMMC, an extremely
versatile bootloader found on all kinds of devices&lt;sup id=&quot;fnref:why-no-edk2-on-rk&quot; role=&quot;doc-noteref&quot;&gt;&lt;a href=&quot;#fn:why-no-edk2-on-rk&quot; class=&quot;footnote&quot; rel=&quot;footnote&quot;&gt;1&lt;/a&gt;&lt;/sup&gt;.  It
makes a TFTP-based development cycle remarkably pleasant.&lt;/p&gt;

&lt;p&gt;But different bootloaders mean very different execution environments.  For one,
EDK2 seems to execute the UEFI application in EL1, whereas U-Boot gives over
control in EL2.  There are many, many differences in the state of the various
system control registers.  And in this case, right in the beginning, we were
getting an exception in U-Boot before we’d done barely any work:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;Booting /efi\boot\BOOTAA64.efi
AC&quot;Synchronous Abort&quot; handler, esr 0x96000010
elr: fffffffffd1c4d98 lr : fffffffffd1c4d5c (reloc)
elr: 0000000078f07d98 lr : 0000000078f07d5c
x0 : 0000000000000000 x1 : 0000000000000000
x2 : 000000007bfdf450 x3 : 000000000000004c
x4 : 0000000000002800 x5 : 000000007bfdf480
x6 : 000000007bfdcb50 x7 : 0000000079f71680
x8 : 0000000078f0ab78 x9 : 0000000000003b5c
x10: 0000000000000000 x11: 0000000000000020
x12: 000000000000ed83 x13: 000000000000ed9c
x14: 0000000079f29d28 x15: 0000000008100000
x16: 0000000000000010 x17: 0000000000000000
x18: 0000000000000000 x19: 000000007bf43b78
x20: 0000000079f29fa0 x21: 0000000078f10040
x22: 0000000000005800 x23: 0000000079f542e0
x24: 000000007bff4eac x25: 0000000000000000
x26: 0000000000000000 x27: 0000000000000000
x28: 0000000079f4ba00 x29: 0000000079f29a30

Code: f9400fe8 f9400109 f94007ea 8b0a0129 (3940012b)
UEFI image [0x0000000078f07000:0x0000000078f0c35f] pc=0xd98
           &apos;/efi\boot\BOOTAA64.efi&apos;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;The disassembly showed that at &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;pc=0xd98&lt;/code&gt; in the image we were attempting to
load a byte from the address pointed to by register &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;x9&lt;/code&gt;:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;d98: 2b 01 40 39                   ldrb    w11, [x9]
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;In the register dump, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;x9&lt;/code&gt; has the value &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0000000000003b5c&lt;/code&gt;, whereas we are
clearly relocated much higher in memory (note the UEFI image offset; the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;pc&lt;/code&gt;
is relative to that).  Should &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;x9&lt;/code&gt; actually have a higher computed address?  I
hacked some things together to get a register dump from a similar place in QEMU
(on EDK2, where this all worked):&lt;/p&gt;

&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;(gdb) info registers
x0             0x0                 0
x1             0x2                 2
x2             0x1                 1
x3             0x5f46d944          1598478660
x4             0x43                67
x5             0x0                 0
x6             0x70616d6d          1885433197
x7             0x0                 0
x8             0x5c1f5b78          1545558904
&lt;strong&gt;x9             0x5c1f5b5c          1545558876&lt;/strong&gt;
x10            0x0                 0
x11            0x64                100
x12            0x0                 0
x13            0x8                 8
x14            0x0                 0
x15            0x0                 0
x16            0x5f6b4ab0          1600866992
x17            0xffffa6ac          4294944428
x18            0x0                 0
x19            0x0                 0
x20            0x5f37d000          1597493248
x21            0x5f37f000          1597501440
x22            0x0                 0
x23            0x5f37f000          1597501440
x24            0x0                 0
x25            0x1                 1
x26            0x0                 0
x27            0x5f37f000          1597501440
x28            0x5f37d55d          1597494621
x29            0x5f6b45b0          1600865712
x30            0x5c1f2d5c          1545547100
sp             0x0                 0x0
pc             0x5c1f2da0          0x5c1f2da0&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;I’ve highlighted &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;x9&lt;/code&gt; – it has a value that’s clearly much more
valid-pointer-looking, and it even ends in &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;...b5c&lt;/code&gt;, which makes me think it’s
a corrected version of the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;3b5c&lt;/code&gt; value we saw on the ROCKPro64.&lt;/p&gt;

&lt;p&gt;What could cause a register to have a correct-looking address on one platform
but not on the other?  Let’s look at the code up and until the point where
things fall apart.  Unfortunately, UEFI code objects are all
&lt;a href=&quot;https://en.wikipedia.org/wiki/COFF&quot;&gt;COFF&lt;/a&gt;s.  I’m super inexperienced with
these, and so too it turns out is the tooling in the area; I think it must be a
bit of a hack that Zig or LLVM knows how to produce them, because it also
produces a &lt;a href=&quot;https://en.wikipedia.org/wiki/Program_database&quot;&gt;PDB&lt;/a&gt; alongside that
presumably contains the debugging/line info, but then &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;llvm-objdump&lt;/code&gt; refuses to
use the same thing, helpfully declaring:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;llvm-objdump: warning: &apos;dainboot/zig-cache/bin/BOOTAA64.
rockpro64.efi&apos;: failed to parse debug information for
dainboot/zig-cache/bin/BOOTAA64.rockpro64.efi
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;&lt;img src=&quot;/assets/post-img/disas.jpg&quot; alt=&quot;Hand-written disassembly.&quot; /&gt;&lt;/p&gt;

&lt;p&gt;(The right-hand page has the disassembly in reverse, since we only have the
state of registers at the point in time of the last instruction and have to
trace data dependencies backwards.  The precise values are different because
they shifted every time I added some breaks or debugging helpers anywhere.)&lt;/p&gt;

&lt;p&gt;I have no line numbers – it’s up to disassembly and guesswork.  Here’s the
former:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;d8c: 09 01 40 f9                   ldr     x9, [x8]
d90: ea 07 40 f9                   ldr     x10, [sp, #8]
d94: 29 01 0a 8b                   add     x9, x9, x10
d98: 2b 01 40 39                   ldrb    w11, [x9]
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;I’ve grown very fond of aarch64 (dis)assembly.  Look at those four-byte
instructions.  This experience was a crash course in learning it.&lt;/p&gt;

&lt;p&gt;So what have we here?  Translated into pseudo-C:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;The faulting instruction tries to load a byte from the address stored in
&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;x9&lt;/code&gt;, which we know to be &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;00003b5c&lt;/code&gt;, i.e. &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;w11 = *(u8 *)x9&lt;/code&gt;.&lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;x9&lt;/code&gt; was calculated as &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;x9 = x9 + x10&lt;/code&gt;.&lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;x10&lt;/code&gt; came from &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;x10 = *(sp + 8)&lt;/code&gt;.&lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;x9&lt;/code&gt; came from &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;x9 = *x8&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;At first I thought that &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;x10&lt;/code&gt; must’ve been some relocation base, but it’s zero
on both QEMU and hardware.  The pointer is whatever we get from &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;x8&lt;/code&gt;, which is
&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0000000078f0ab78&lt;/code&gt; on hardware and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x5c1f5b78&lt;/code&gt; on QEMU.  The last few digits
line up nicely, again, so it looks like whatever’s at that address is a
relocated address on QEMU/EDK2 but just &lt;em&gt;not&lt;/em&gt; on ROCKPro64/U-Boot.  What even
is at that address?  Let’s look at the symbol table.&lt;/p&gt;

&lt;p&gt;U-Boot told us in the exception dump that the UEFI image was loaded at
&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x78f07000&lt;/code&gt;, so &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x78f0ab78&lt;/code&gt; is at offset &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x3b78&lt;/code&gt; from the start of the
image.  What’s that?  &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;objdump&lt;/code&gt; to the rescue.  It’s in our &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;.data&lt;/code&gt; section:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;3b70 287b737d 290d0a00 5c3b0000 00000000  ({s})...........
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;I really had to squint at this for a moment before realising that &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x3b78&lt;/code&gt;
actually contains a 64-bit pointer value, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x00000000003b5c&lt;/code&gt; — in other
words, the exact value of &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;x8&lt;/code&gt; we saw!  So this was pulled directly out of the
loaded image.  Two questions arose: what &lt;em&gt;is&lt;/em&gt; it?  And how is it that QEMU/EDK2
got something different here?&lt;/p&gt;

&lt;p&gt;It felt off that it should point to a value directly before itself in memory.
Here’s context:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;3b50 02200000 00000000 00000000 6461696e  ............dain
3b60 74726565 20626f6f 746c6f61 64657220  tree bootloader 
3b70 287b737d 290d0a00 5c3b0000 00000000  ({s})...........
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;A string!  &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x3b78&lt;/code&gt; points to &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x3b5c&lt;/code&gt;, which is the greeting string printed
from the bootloader.  Why this indirection in the binary itself?&lt;/p&gt;

&lt;p&gt;There are three sections in the PE/COFF files generated by the build process:
&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;.text&lt;/code&gt;, which contains the executable code, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;.data&lt;/code&gt;, which contains strings
and other bits, and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;.reloc&lt;/code&gt;.  It still felt like this was a relocation issue,
so I read Microsoft’s &lt;a href=&quot;https://docs.microsoft.com/en-us/windows/win32/debug/pe-format&quot;&gt;PE
Format&lt;/a&gt;
documentation carefully.  &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;objdump&lt;/code&gt; was doing a lot of the heavy lifting for
us, but to really understand it, I wanted to &lt;a href=&quot;https://github.com/kivikakk/daintree/blob/master/tools/pe-parser.rb&quot;&gt;pull apart the format
myself&lt;/a&gt;.
This approach would turn out to be invaluable.&lt;/p&gt;

&lt;p&gt;The PE relocation section is comprised of a number of base relocation blocks,
which defines a 32-bit base address for the block, each with any number of
entries that specify the type of relocation and the 12-bit offset in that block
to apply it at.  Here’s a decoded &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;.reloc&lt;/code&gt; we got:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;Page RVA: 0xa000
28 relocations:
  0xaca8 0xacb8 0xacc8 0xacd8 0xace8 0xacf8 0xad08 0xad18
  0xad28 0xad38 0xad48 0xad58 0xad68 0xad78 0xadc8 0xadd8
  0xae10 0xae40 0xae90 0xaeb0 0xaef8 0xaf00 0xaf08 0xaf10
  0xaf48 0xaf88 0xafb8 0xafe0
Page RVA: 0xb000
53 relocations:
  0xb010 0xb040 0xb070 0xb0c0 0xb100 0xb160 0xb188 0xb1a0
  0xb210 0xb240 0xb270 0xb2a0 0xb2d0 0xb300 0xb328 0xb370
  0xb3b0 0xb3f8 0xb450 0xb4a0 0xb4b0 0xb4c8 0xb528 0xb578
  0xb5c0 0xb610 0xb680 0xb6d0 0xb720 0xb778 0xb7c8 0xb7e0
  0xb848 0xb898 0xb8e8 0xb938 0xb9c8 0xba18 0xba68 0xbab8
  0xbb08 0xbb58 0xbba8 0xbbd8 0xbc08 0xbc40 0xbcd0 0xbd20
  0xbd70 0xbdc0 0xbe18 0xbe68 0xbea8
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;objdump&lt;/code&gt; couldn’t give us this!  (The addresses don’t line up with the above
because I continued to hack on and modify things, shifting everything in the
binary around.  This was as frustrating as it might seem.  These lined up
exactly with the indirection seen above.)&lt;/p&gt;

&lt;p&gt;For my purposes, I ignore all entries except of the type
&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;IMAGE_REL_BASED_DIR64&lt;/code&gt;: “The base relocation applies the difference to the
64-bit field at offset.”  It turns out Zig/LLVM only generates those, and they
work as simply as they sound: at the address specified by the relocation entry,
treat it as a 64-bit field and add the relocation offset to it.&lt;/p&gt;

&lt;p&gt;So, a PE loader should, after loading all data, visit all those addresses and
add the relocation offset to them.  It seemed like that was happening correctly
on QEMU/EDK2 — when we loaded the addresses, they had been shifted for us.
But why not on U-Boot?&lt;/p&gt;

&lt;p&gt;To answer this, I looked at the U-Boot source code.  And when looking wasn’t
enough, it was time to &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;printf&lt;/code&gt; debug, which meant getting a U-Boot build that
actually ran on my hardware.  There’s a 4+ hour gap between the two blocks of
chat here:&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/post-img/fuck.png&quot; alt=&quot;&amp;quot;finally got my own uboot build running!! fuck&amp;quot;&quot; /&gt;&lt;/p&gt;

&lt;p&gt;(I nuked the boot environment quite a few times getting to this point.)&lt;/p&gt;

&lt;p&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;printf&lt;/code&gt; debugging to the rescue: U-Boot wasn’t doing any of the relocations,
at all.  It thought there weren’t any:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;doing relocations (rel 0x0000000078efb2e0 rel_size 0x68
                   efi_reloc 0x0000000078ef6000
                   image_base 0x0)
DAINDBG: delta: 0x78ef6000
rel: 0x0000000078efb2e0, end: 0x0000000078efb348,
     rel-&amp;gt;SizeOfBlock: 0x0
DAINDBG: done
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;rel-&amp;gt;SizeOfBlock&lt;/code&gt; here corresponds to the &lt;a href=&quot;https://docs.microsoft.com/en-us/windows/win32/debug/pe-format#base-relocation-block&quot;&gt;Block
Size&lt;/a&gt;
field in the base relocation block.  My tools were clearly showing that block
had a very non-zero size, so how was U-Boot getting a different idea?&lt;/p&gt;

&lt;p&gt;I tried dumping the table raw, 32 bytes as hex to see what we got, and it was
all zero.  Something was still up.  I had to keep going back.&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/post-img/suspect.png&quot; alt=&quot;&amp;quot;so now i need to suspect the code immediately
before&amp;quot;&quot; /&gt;&lt;/p&gt;

&lt;p&gt;I riddled the loader with &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;printf&lt;/code&gt;s:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;DAINDBG: loading section[2]: VA 0x0000000078efb2e0 --
    setting 0x68 bytes to zero, then loading 0x200 bytes
    from 0x0000000002085600
    (efi 0x0000000002080000 + PointerToRawData 0x5600)
DAINDBG: loading section[1]: VA 0x0000000078ef9860 --
    setting 0x1a75 bytes to zero, then loading 0x1c00 bytes
    from 0x0000000002083a00
    (efi 0x0000000002080000 + PointerToRawData 0x3a00)
DAINDBG: loading section[0]: VA 0x0000000078ef6200 --
    setting 0x364c bytes to zero, then loading 0x3800 bytes
    from 0x0000000002080200
    (efi 0x0000000002080000 + PointerToRawData 0x200)
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Here’s the code that corresponds to the above, with the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;printf&lt;/code&gt;
re-approximated post-hoc for your reading pleasure:&lt;/p&gt;

&lt;div class=&quot;language-c highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;cm&quot;&gt;/* Load sections into RAM */&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;for&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;i&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;num_sections&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;i&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;&amp;gt;=&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;i&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;--&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
	&lt;span class=&quot;cm&quot;&gt;/* XXX */&lt;/span&gt;
	&lt;span class=&quot;n&quot;&gt;printf&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;DAINDBG: loading section[%d]: VA %p -- &quot;&lt;/span&gt;
	       &lt;span class=&quot;s&quot;&gt;&quot;setting 0x%x bytes to zero, then loading &quot;&lt;/span&gt;
	       &lt;span class=&quot;s&quot;&gt;&quot;0x%x bytes from %p (efi %p + &quot;&lt;/span&gt;
	       &lt;span class=&quot;s&quot;&gt;&quot;PointerToRawData %x)&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\n&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; 
	       &lt;span class=&quot;n&quot;&gt;i&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
	       &lt;span class=&quot;n&quot;&gt;efi_reloc&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;sec&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;-&amp;gt;&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;VirtualAddress&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
	       &lt;span class=&quot;n&quot;&gt;sec&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;-&amp;gt;&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Misc&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;VirtualSize&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
	       &lt;span class=&quot;n&quot;&gt;sec&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;-&amp;gt;&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;SizeOfRawData&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
	       &lt;span class=&quot;n&quot;&gt;efi&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;sec&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;-&amp;gt;&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;PointerToRawData&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
	       &lt;span class=&quot;n&quot;&gt;efi&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
	       &lt;span class=&quot;n&quot;&gt;sec&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;-&amp;gt;&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;PointerToRawData&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;);&lt;/span&gt;
	&lt;span class=&quot;cm&quot;&gt;/* XXX */&lt;/span&gt;
	&lt;span class=&quot;n&quot;&gt;IMAGE_SECTION_HEADER&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;*&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;sec&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;&amp;amp;&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;sections&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;i&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;];&lt;/span&gt;
	&lt;span class=&quot;n&quot;&gt;memset&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;efi_reloc&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;sec&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;-&amp;gt;&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;VirtualAddress&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
	       &lt;span class=&quot;n&quot;&gt;sec&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;-&amp;gt;&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Misc&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;VirtualSize&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;);&lt;/span&gt;
	&lt;span class=&quot;n&quot;&gt;memcpy&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;efi_reloc&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;sec&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;-&amp;gt;&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;VirtualAddress&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
	       &lt;span class=&quot;n&quot;&gt;efi&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;sec&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;-&amp;gt;&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;PointerToRawData&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
	       &lt;span class=&quot;n&quot;&gt;sec&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;-&amp;gt;&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;SizeOfRawData&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;);&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Firstly, we load them in reverse.  Weird, but okay.  Secondly, it became clear
that &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;section[1]&lt;/code&gt; (&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;.data&lt;/code&gt;) overlaps &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;section[2]&lt;/code&gt; (&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;.reloc&lt;/code&gt;) by 384 bytes (!!):&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;.reloc&lt;/code&gt;: load at &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x78efb2e0&lt;/code&gt;, zero &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x68&lt;/code&gt; bytes at start, then load
&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x200&lt;/code&gt; bytes at start, until &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x78efb4e0&lt;/code&gt; non-incl.&lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;.data&lt;/code&gt;: load at &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x78ef9860&lt;/code&gt;, zero &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x1a75&lt;/code&gt; bytes at start, then load
&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x1c00&lt;/code&gt; bytes at start, until &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x78efb460&lt;/code&gt; non-incl. (&lt;strong&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x180&lt;/code&gt; bytes into
&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;.reloc&lt;/code&gt; target !!&lt;/strong&gt;)&lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;.text&lt;/code&gt;: load at &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x78ef6200&lt;/code&gt;, zero &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x364c&lt;/code&gt; bytes at start, then load
&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x3800&lt;/code&gt; bytes at start, until &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x78ef9a00&lt;/code&gt; non-incl. (&lt;strong&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x1a0&lt;/code&gt; bytes into
&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;.data&lt;/code&gt; target !!&lt;/strong&gt;)&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Oh boy.  I didn’t even notice until now that the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;.text&lt;/code&gt; overlapped &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;.data&lt;/code&gt; too.
Who &lt;em&gt;knows&lt;/em&gt; what would’ve happened if the sections aligned such that the
relocations worked.  It would’ve been much harder to diagnose.&lt;/p&gt;

&lt;p&gt;Why would this happen?  It seems like &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;VirtualAddress + SizeOfRawData&lt;/code&gt; overlaps
with other &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;VirtualAddress&lt;/code&gt;es.  What does the &lt;a href=&quot;https://docs.microsoft.com/en-us/windows/win32/debug/pe-format#section-table-section-headers&quot;&gt;PE
format&lt;/a&gt;
say about &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;VirtualAddress&lt;/code&gt;?&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;For executable images, the address of the first byte of the section relative
to the image base when the section is loaded into memory. For object files,
this field is the address of the first byte before relocation is applied; for
simplicity, compilers should set this to zero. Otherwise, it is an arbitrary
value that is subtracted from offsets during relocation.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;SizeOfRawData&lt;/code&gt; is below it and caught my eye.  Emphasis mine:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;The size of the section (for object files) or the size of the initialized
data on disk (for image files). For executable images, this must be a
multiple of FileAlignment from the optional header. If this is less than
VirtualSize, the remainder of the section is zero-filled. &lt;strong&gt;Because the
SizeOfRawData field is rounded but the VirtualSize field is not, it is
possible for SizeOfRawData to be greater than VirtualSize as well.&lt;/strong&gt; When a
section contains only uninitialized data, this field should be zero.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;SizeOfRawData&lt;/code&gt; is rounded to a multiple of &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;FileAlignment&lt;/code&gt;, which the docs
tell us is typically 512 bytes.  (Notice they all divide by &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x200&lt;/code&gt;.)  But
that’s presumably just because of the way they align in the file!  What &lt;em&gt;is&lt;/em&gt;
&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;VirtualSize&lt;/code&gt;, which until now we’re only using to determine how much to zero?&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;&lt;strong&gt;The total size of the section when loaded into memory.&lt;/strong&gt; If this value is
greater than SizeOfRawData, the section is zero-padded. This field is valid
only for executable images and should be set to zero for object files.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The &lt;strong&gt;total size&lt;/strong&gt;.  It seems pretty clear that we should never load more than
this many bytes, even if &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;SizeOfRawData&lt;/code&gt; happens to be bigger.  The size of the
section can’t be bigger than this.  If we were to constrain our &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;memcpy&lt;/code&gt;s to
&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;min(VirtualSize, SizeOfRawData)&lt;/code&gt;, we get this instead:&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;.reloc&lt;/code&gt;: load at &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x78efb2e0&lt;/code&gt;, zero &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x68&lt;/code&gt; bytes at start, then load
&lt;del&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x200&lt;/code&gt;&lt;/del&gt; &lt;strong&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x68&lt;/code&gt;&lt;/strong&gt; bytes at start, until &lt;strong&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x78efb348&lt;/code&gt;&lt;/strong&gt; non-incl.&lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;.data&lt;/code&gt;: load at &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x78ef9860&lt;/code&gt;, zero &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x1a75&lt;/code&gt; bytes at start, then load
&lt;del&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x1c00&lt;/code&gt;&lt;/del&gt; &lt;strong&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x1a75&lt;/code&gt;&lt;/strong&gt; bytes at start, until &lt;strong&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x78efb2d5&lt;/code&gt;&lt;/strong&gt; non-incl.
(&lt;strong&gt;before &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;.reloc&lt;/code&gt; begins&lt;/strong&gt;)&lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;.text&lt;/code&gt;: load at &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x78ef6200&lt;/code&gt;, zero &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x364c&lt;/code&gt; bytes at start, then load
&lt;del&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x3800&lt;/code&gt;&lt;/del&gt; &lt;strong&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x364c&lt;/code&gt;&lt;/strong&gt; bytes at start, until &lt;strong&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x78ef984c&lt;/code&gt;&lt;/strong&gt; non-incl.
(&lt;strong&gt;before &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;.data&lt;/code&gt; begins&lt;/strong&gt;)&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;It looked like a bug.  If U-Boot loaded sections forwards, this wouldn’t have
been exposed, but either way it appeared to be an error to do this at all.  The
section shouldn’t be loaded beyond its VirtualSize.&lt;/p&gt;

&lt;p&gt;A quick trip to the &lt;a href=&quot;https://github.com/tianocore/edk2/blob/c640186ec8aae6164123ee38de6409aed69eab12/BaseTools/Source/C/Common/BasePeCoff.c#L1028-L1031&quot;&gt;EDK2 PE
loader&lt;/a&gt;
shows they load at most &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;min(VirtualSize, SizeOfRawData)&lt;/code&gt; bytes into memory,
and then pad up to &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;VirtualSize&lt;/code&gt; with zeroes if needed.  (The zeroing behaviour
is for BSS-style initialisation.)  They never touch memory past &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;VirtualSize&lt;/code&gt;
bytes.&lt;/p&gt;

&lt;div class=&quot;language-diff highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt; memcpy(efi_reloc + sec-&amp;gt;VirtualAddress,
        efi + sec-&amp;gt;PointerToRawData,
&lt;span class=&quot;gd&quot;&gt;-       sec-&amp;gt;SizeOfRawData);
&lt;/span&gt;&lt;span class=&quot;gi&quot;&gt;+       min(sec-&amp;gt;Misc.VirtualSize, sec-&amp;gt;SizeOfRawData));
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;&lt;a href=&quot;http://u-boot.10912.n7.nabble.com/Re-PATCH-efi-loader-don-t-load-beyond-VirtualSize-td440790.html#a440833&quot;&gt;One short conversation
later&lt;/a&gt;,
and &lt;a href=&quot;https://source.denx.de/u-boot/u-boot/-/commit/9d30a941cce5ed055da18398f4deba18830d00d6&quot;&gt;the bug was
fixed&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/post-img/dainboot.png&quot; alt=&quot;Screenshot of it working.&quot; /&gt;&lt;/p&gt;

&lt;p&gt;(This is what success looks like.)&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Sometimes&lt;/em&gt;, the problem is not in your code.&lt;/p&gt;

&lt;div class=&quot;footnotes&quot; role=&quot;doc-endnotes&quot;&gt;
  &lt;ol&gt;
    &lt;li id=&quot;fn:why-no-edk2-on-rk&quot; role=&quot;doc-endnote&quot;&gt;

      &lt;p&gt;Maybe it’d be nice if I could slap EDK2 on the ROCKPro64 –
&lt;a href=&quot;https://github.com/edk2-porting/edk2-rk3399&quot;&gt;edk2-porting/edk2-rk3399&lt;/a&gt;
appears to be the strongest effort in this area so far, and it doesn’t look
great.  U-Boot is mature on many embedded platforms. &lt;a href=&quot;#fnref:why-no-edk2-on-rk&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
  &lt;/ol&gt;
&lt;/div&gt;
</content>
    
 </entry>
 
 <entry>
   <title>Inkplate done quick</title>
   <updated>2021-02-19T00:00:00+11:00</updated>
   <id>https://kivikakk.ee/2021/02/19/inkplate-done-quick</id>
    
     <link href="https://kivikakk.ee/2021/02/19/inkplate-done-quick/"/>
     <content type="html">&lt;p&gt;I recently received an &lt;a href=&quot;https://inkplate.io&quot;&gt;Inkplate&lt;/a&gt;, and while I’m in the
middle of a few interesting projects already, I couldn’t let it sit there
unused.  Until I get a longer chunk of time to turn it into something really
nifty — maybe an embedded debugging helper of some kind — it can at least
mean I no longer need to have Mail.app open.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://github.com/kivikakk/kmlyink&quot;&gt;kmlyink&lt;/a&gt;’s README explains:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;This repo has two parts:&lt;/p&gt;

  &lt;ul&gt;
    &lt;li&gt;
      &lt;p&gt;a Dockerised IMAP proxy written in Ruby.&lt;/p&gt;

      &lt;p&gt;It speaks plain HTTP, like an ESP can manage. It just fetches your Inbox
listing and puts it in JSON.&lt;/p&gt;
    &lt;/li&gt;
    &lt;li&gt;
      &lt;p&gt;a MicroPython module that connects to your wifi, speaks to the IMAP proxy,
and formats it into the display.&lt;/p&gt;
    &lt;/li&gt;
  &lt;/ul&gt;
&lt;/blockquote&gt;

&lt;p&gt;It took just a few hours to get it all up and running.  Delightful!&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/post-img/kmlyink.jpg&quot; alt=&quot;A photo of kmlyink in action. There&apos;s some emails listed on an e-ink
display.&quot; /&gt;&lt;/p&gt;

</content>
    
 </entry>
 
 <entry>
   <title>DTB parser implementing notes</title>
   <updated>2021-02-13T00:00:00+11:00</updated>
   <id>https://kivikakk.ee/2021/02/13/dtb-parser-implementing-notes</id>
    
     <link href="https://kivikakk.ee/2021/02/13/dtb-parser-implementing-notes/"/>
     <content type="html">&lt;p&gt;Ever find yourself needing to implement a &lt;a href=&quot;https://devicetree-specification.readthedocs.io/en/latest/chapter5-flattened-format.html&quot;&gt;device tree
blob&lt;/a&gt;
(aka FDT, flattened device tree) parser and want to save yourself some time?
Learn from my mistakes!&lt;/p&gt;

&lt;h2 id=&quot;if-you-try-to-do-it-in-one-pass-you-will-hurt-yourself&quot;&gt;If you try to do it in one pass, you will hurt yourself&lt;/h2&gt;

&lt;p&gt;&lt;a id=&quot;more&quot;&gt;&lt;/a&gt;I charged headlong into writing
&lt;a href=&quot;https://github.com/kivikakk/dtb.zig&quot;&gt;dtb.zig&lt;/a&gt;
by starting at the top of the Devicetree Specification page on the “Flattened
Devicetree (DTB)” Format” and reading down. It looked delightfully simple. Keep
in mind, I still didn’t know what I yet needed out of it, just that I probably
needed to reference the DTB to get it.  (I &lt;a href=&quot;https://github.com/kivikakk/daintree/commit/1a65076a36301f0fb33748b8da644010a178b58e#diff-5e1ca02318cf3679c3aa9a422be7adfefe1fefdd76d297d676770edeacdb5e67R329-R349&quot;&gt;kind of know better now&lt;/a&gt;.)&lt;/p&gt;

&lt;!--more--&gt;

&lt;hr /&gt;

&lt;p&gt;The tree was taking shape, and then I had to parse the contents of one field by
the contents of a prop in its parent (&lt;a href=&quot;https://devicetree-specification.readthedocs.io/en/latest/chapter2-devicetree-basics.html#address-cells-and-size-cells&quot;&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;#address-cells&lt;/code&gt; and
&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;#size-cells&lt;/code&gt;&lt;/a&gt;).
Add some contexts and derive them from their parent, allowing overriding for
children. Easy.&lt;/p&gt;

&lt;p&gt;Then I needed to parse
&lt;a href=&quot;https://devicetree-specification.readthedocs.io/en/latest/chapter2-devicetree-basics.html#interrupts-and-interrupt-mapping&quot;&gt;interrupts&lt;/a&gt;.
It turns out the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;interrupts&lt;/code&gt; property of a node has its format defined by the
&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;#interrupt-cells&lt;/code&gt; of the “binding of the interrupt domain root”.  It turns out
your &lt;a href=&quot;https://devicetree-specification.readthedocs.io/en/latest/chapter2-devicetree-basics.html#interrupt-parent&quot;&gt;“interrupt
parent”&lt;/a&gt;
might be defined &lt;em&gt;forward&lt;/em&gt; in the file, as referenced by its phandle.&lt;/p&gt;

&lt;p&gt;You find out the same thing about clocks, though the documentation is &lt;a href=&quot;https://android.googlesource.com/kernel/msm.git/+/android-msm-shamu-3.10-lollipop-mr1/Documentation/devicetree/bindings/clock/clock-bindings.txt&quot;&gt;harder
to
find&lt;/a&gt;.
A clock provider specifies &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;#clock-cells&lt;/code&gt;, which is usually 0 or 1. When
another node refers to a clock on that node, it addresses the phandle of the
clock provider, followed by &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;#clock-cells&lt;/code&gt; worth of cells to index which clock
on that provider.&lt;/p&gt;

&lt;p&gt;In other words, a &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;clocks&lt;/code&gt; like this:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;00000000: 00000085 0000001c 0000002e
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;could refer to either:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;one clock specified by phandle &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x85&lt;/code&gt;, with a &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;#clock-cells&lt;/code&gt; of 2, the index
being &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x1c 0x2e&lt;/code&gt;,&lt;/li&gt;
  &lt;li&gt;two clocks;
    &lt;ul&gt;
      &lt;li&gt;either a clock at phandle &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x85&lt;/code&gt; with a &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;#clock-cells&lt;/code&gt; of 1 indexed by
&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x1c&lt;/code&gt;, and a clock at &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x2e&lt;/code&gt; with no index, or,&lt;/li&gt;
      &lt;li&gt;a clock at &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x85&lt;/code&gt; with no index, and a clock at &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x1c&lt;/code&gt; indexed by &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x2e&lt;/code&gt;;
or,&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;three clocks, all with no index; &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x85&lt;/code&gt;, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x1c&lt;/code&gt;, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0x2e&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;You need to be able to look up the clocks and obtain their properties to
interpret this, so you &lt;strong&gt;need&lt;/strong&gt; a second pass, or delayed/on-time resolution of
fields, or whatever.  There end up being &lt;a href=&quot;https://github.com/kivikakk/dtb.zig/blob/9bc32d41ae83586a422dd5f10c943021592613cd/src/parser.zig#L133-L142&quot;&gt;quite a few
props&lt;/a&gt;
that need a second pass.&lt;/p&gt;

&lt;h2 id=&quot;how-big&quot;&gt;How big?&lt;/h2&gt;

&lt;p&gt;It’s worth noting all numbers and indexes in DTB are in big-endian, unsigned
32-bit integer cells. That makes hexdumps easier, since you can read them
byte-by-byte or in groups of 4 and don’t need to rearrange them in your head.&lt;/p&gt;

&lt;p&gt;You’ll see &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;#address-cells&lt;/code&gt; of 2 and similar for most 64-bit devices. I saw an
&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;#address-cells&lt;/code&gt; of 3 once in a PCIe node and it scared me.&lt;/p&gt;

&lt;h3 id=&quot;strings-are-nul-terminated-and-nul-padded&quot;&gt;Strings are NUL-terminated, and NUL padded&lt;/h3&gt;

&lt;p&gt;This tripped me up.  Strings are NUL-terminated, and then the field will be
padded with NULs (if needed) to align on a &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;u32&lt;/code&gt; (i.e. offset divisible by 4).
This is helpful, because a &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;u32&lt;/code&gt; is literally what will always follow, and Arm
devices (which DTBs are often used on) don’t like unaligned reads.&lt;/p&gt;

&lt;p&gt;So, when you need to read a NUL-terminated string, don’t do what I did first:&lt;/p&gt;

&lt;figure class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-zig&quot; data-lang=&quot;zig&quot;&gt;&lt;span class=&quot;k&quot;&gt;var&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;name_end&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;kt&quot;&gt;usize&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;while&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;index&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;name_end&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;!=&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;name_end&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+=&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;{}&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;const&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;name&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;index&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;..&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;name_end&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;];&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;index&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;index&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;name_end&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;3&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;&amp;amp;&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;~&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;@as&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;kt&quot;&gt;usize&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;3&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;..&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;];&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;

&lt;p&gt;It seems reasonable at first blush: count the NULs (&lt;a href=&quot;https://github.com/ziglang/zig/blob/9270aae071a4ee840193afe1162b24945cbd6d9e/lib/std/mem.zig#L680-L711&quot;&gt;there’s a much better
way&lt;/a&gt;),
then advance the index past the name, plus align to advance past padding.
(Hack for aligning to a power of two, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;n&lt;/code&gt;: add &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;n-1&lt;/code&gt;, then logical AND with
&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;n-1&lt;/code&gt;.)&lt;/p&gt;

&lt;p&gt;The problem is that I never advanced past the NUL terminator, which is still
part of the string.  Here are some example NUL-terminated strings:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;00000000: 6100                                 a.
00000000: 616200                               ab.
00000000: 61626300                             abc.
00000000: 61626364 00                          abcd.
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Here are the same strings padded with NULs to align on &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;u32&lt;/code&gt;:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;00000000: 61000000                             a...
00000000: 61620000                             ab..
00000000: 61626300                             abc.
00000000: 61626364 00000000                    abcd....
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Here’s the corrected code:&lt;/p&gt;

&lt;figure class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-zig&quot; data-lang=&quot;zig&quot;&gt;&lt;span class=&quot;k&quot;&gt;var&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;name_end&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;kt&quot;&gt;usize&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;while&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;index&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;name_end&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;!=&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;name_end&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+=&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;{}&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;const&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;name&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;index&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;..&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;name_end&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;];&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;name_end&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+=&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt; &lt;span class=&quot;c&quot;&gt;// advance past NUL byte&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;index&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;index&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;name_end&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;3&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;&amp;amp;&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;~&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;@as&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;kt&quot;&gt;usize&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;3&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;..&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;];&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;

&lt;h2 id=&quot;thats-all-for-now&quot;&gt;That’s all for now&lt;/h2&gt;

&lt;p&gt;I ended up separating dtb.zig into two parts, given it’s used in boot-time code where allocating memory can mess around with things:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;allocator-free
&lt;a href=&quot;https://github.com/kivikakk/dtb.zig/blob/9bc32d41ae83586a422dd5f10c943021592613cd/src/traverser.zig&quot;&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Traverser&lt;/code&gt;&lt;/a&gt;,
which emits events SAX style. I tried using Zig’s &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;suspend&lt;/code&gt;/&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;resume&lt;/code&gt; here,
and it works pretty well.&lt;/li&gt;
  &lt;li&gt;allocating
&lt;a href=&quot;https://github.com/kivikakk/dtb.zig/blob/9bc32d41ae83586a422dd5f10c943021592613cd/src/parser.zig&quot;&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Parser&lt;/code&gt;&lt;/a&gt;
which uses the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Traverser&lt;/code&gt; and creates an AST, parsing props into an
immediately usable AST in two passes.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Traverser&lt;/code&gt; is used in &lt;a href=&quot;https://github.com/kivikakk/daintree&quot;&gt;daintree&lt;/a&gt;’s
bootloader,
&lt;a href=&quot;https://github.com/kivikakk/daintree/blob/209782a93de9088cba20808644c89cd9898ddada/dainboot/src/dtb.zig&quot;&gt;dainboot&lt;/a&gt;,
to find a probable serial UART device. I’ll use the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Parser&lt;/code&gt; later in the OS
proper to bring up more devices.&lt;/p&gt;
</content>
    
 </entry>
 
 <entry>
   <title>Breaking homegrown crypto</title>
   <updated>2016-02-20T11:56:00+11:00</updated>
   <id>https://kivikakk.ee/cryptography/2016/02/20/breaking-homegrown-crypto</id>
    
     <link href="https://kivikakk.ee/cryptography/2016/02/20/breaking-homegrown-crypto/"/>
     <content type="html">&lt;p&gt;Note: this is a pretty long article which does a deep dive into breaking some amateur crypto.  I go on for quite a bit.  Make a cup of tea before reading, and get ready to read some code!&lt;/p&gt;

&lt;h3 id=&quot;introduction&quot;&gt;introduction&lt;/h3&gt;

&lt;p&gt;Everyone knows it.  Rolling your own cryptography is a terrible idea.  Here’s &lt;a href=&quot;https://www.schneier.com/essays/archives/1999/03/cryptography_the_imp.html&quot;&gt;Bruce Schneier writing about it in &lt;strong&gt;1999&lt;/strong&gt;&lt;/a&gt;.  Here’s &lt;a href=&quot;http://security.stackexchange.com/a/18198/34825&quot;&gt;an excellent answer on the Infosec Stack Exchange&lt;/a&gt; about why you shouldn’t do it.  Here’s &lt;a href=&quot;https://www.schneier.com/blog/archives/2015/05/amateurs_produc.html&quot;&gt;another Scheiner post&lt;/a&gt; with an excellent opening sentence.&lt;/p&gt;

&lt;p&gt;This, then, is a post about a broken homegrown cryptosystem; namely, that used in &lt;a href=&quot;https://www.codeigniter.com&quot;&gt;CodeIgniter&lt;/a&gt;, pre-2.2.  This version was current until the release of CodeIgniter 2.2, on &lt;a href=&quot;https://ellislab.com/blog/entry/codeigniter-2.2.0-released&quot;&gt;the 5th of June, 2014&lt;/a&gt;, and you can still find sites on it today.&lt;/p&gt;

&lt;p&gt;The attack described in the post depends on a lot of things to go right (or wrong, if you will); it’s not just that they used a bad cipher, but also the fact that they rolled their own session storage, and implemented a fallback, and a dozen other things.  This is probably typical for most bugs of this class; a bunch of bad decisions which aren’t thought through find their logical conclusion in complete insecurity.&lt;/p&gt;

&lt;p&gt;&lt;a id=&quot;more&quot;&gt;&lt;/a&gt;Let’s get into it!&lt;/p&gt;

&lt;!--more--&gt;

&lt;hr /&gt;

&lt;h3 id=&quot;what-are-sessions-and-why-do-you-want-them&quot;&gt;what are sessions and why do you want them?&lt;/h3&gt;

&lt;p&gt;When you visit a website, you might want to log in, and have the website remember that you’re logged in.&lt;/p&gt;

&lt;p&gt;When I was 13, I wrote a website with a “login” feature; I didn’t know about cookies, so instead the logged in part of the website just passed around your credentials in URL parameters.  To obscure them from the user, the login page was actually a POST form which rendered a frameset (!); this way you’d never see your password in the address bar.&lt;/p&gt;

&lt;p&gt;It was a great idea!  But, all it took was someone right-clicking a link and selecting “copy” and along went their credentials too.  So, an imperfect idea.&lt;/p&gt;

&lt;p&gt;I learned about cookies.&lt;/p&gt;

&lt;p&gt;When I verified the user’s credentials against the backend (and to be honest, it was probably SQLi-filled), I put a &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;user_id&lt;/code&gt; cookie on their machine.  When they come back, if they have the cookie, they’re in!&lt;/p&gt;

&lt;p&gt;Modifying cookies didn’t seem like the easiest or most obvious thing, but eventually I tried it, and found I could become whomever I wanted to be.&lt;/p&gt;

&lt;p&gt;I learned about sessions.&lt;/p&gt;

&lt;p&gt;PHP’s implementation ran this way: we’ll throw a cookie on the user’s machine, maybe called &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;PHPSESSID&lt;/code&gt;, which is just an opaque identifier.  Session variables accumulated throughout the script execution will then get written to storage keyed by that ID; often just files in &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;/tmp&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;This has a few advantages:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;The user can’t modify their own session data.  No more setting &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;user_id&lt;/code&gt; or &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;isAdmin&lt;/code&gt; for you!&lt;/li&gt;
  &lt;li&gt;The user can’t &lt;em&gt;see&lt;/em&gt; their own session data.  This one might be less obviously bad, but in general the less data you (needlessly) expose the better.&lt;/li&gt;
  &lt;li&gt;You can perform other tricks to verify the session owner.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For instance, you could — completely hypothetically — store the user’s IP address or user agent in the session.  Then, when they use the session, you confirm the session data against their IP/UA.  This prevents an attack where someone sniffs or steals the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;PHPSESSID&lt;/code&gt; of another user and attempts to use it themselves.&lt;/p&gt;

&lt;h3 id=&quot;what-happens-when-you-dont-have-a-good-place-to-store-session-data&quot;&gt;what happens when you don’t have a good place to store session data?&lt;/h3&gt;

&lt;p&gt;Say you’re on a weird shared host and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;/tmp&lt;/code&gt; is unwritable, or shared, or filled with piranhas.  Your database has a limit of one write per minute.  Where do you put your sessions?&lt;/p&gt;

&lt;p&gt;“Maybe,” you think, “maybe I put the sessions in the cookie I give to the user!?”&lt;/p&gt;

&lt;p&gt;This is not a bad idea.  This is essentially “store everything in the cookie” per above, although it presumes a level of structure given by the session storage mechanism.  The key realisation is that you still want those three things above:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;The user can’t modify their own session data.&lt;/li&gt;
  &lt;li&gt;The user can’t see their own session data.&lt;/li&gt;
  &lt;li&gt;You can perform other tricks to verify the session owner.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;How do we stop them modifying their own session data?  You &lt;a href=&quot;https://en.wikipedia.org/wiki/Hash-based_message_authentication_code&quot;&gt;HMAC&lt;/a&gt; it.
Of course, most people don’t &lt;em&gt;actually&lt;/em&gt; use HMAC and instead just use
&lt;strong&gt;H&lt;/strong&gt;(&lt;em&gt;k&lt;/em&gt; || &lt;em&gt;m&lt;/em&gt;)
or
&lt;strong&gt;H&lt;/strong&gt;(&lt;em&gt;k&lt;/em&gt; || &lt;em&gt;m&lt;/em&gt; || &lt;em&gt;k&lt;/em&gt;)
or whatever their “instincts” told them to do; the latter getting the job done while admitting a few attacks that a Sufficiently Capable (or Cashed Up) Adversary can follow through on; the former practically negligent (see &lt;a href=&quot;https://en.wikipedia.org/wiki/Length_extension_attack&quot;&gt;length extension attack on Wikipedia&lt;/a&gt;; thanks to &lt;a href=&quot;https://www.reddit.com/r/PHP/comments/46pv94/breaking_codeigniters_homegrown_crypto/d06ze0h&quot;&gt;nikic for the correction&lt;/a&gt;).&lt;/p&gt;

&lt;p&gt;So, we transmit the MAC — maybe you just append or prepend it to the session, or put it in a separate cookie, whatever — and then when we get a session back we authenticate it.  If authentication fails, we don’t touch it, we throw it away; certainly we don’t try to e.g. unserialise it or anything.  We didn’t produce it, so it’s a live wire.&lt;/p&gt;

&lt;p&gt;That done, we now have “the user can’t modify their own session”, and this is a pretty good start.  We have a secure storage mechanism, albeit one where the user can see their own session data.&lt;/p&gt;

&lt;p&gt;We can tackle the “verify the session owner” point by storing IP and UA in the session as before; they can’t modify these values themselves, so an attacker can’t either.  That said, they can see these values and realise they’re probably used in session authentication, which makes impersonating the user that much easier.&lt;/p&gt;

&lt;p&gt;To finally achieve a desirable level of security, we might want to stop them from seeing their session data too.  Thus we symmetrically encrypt the session data.&lt;/p&gt;

&lt;p&gt;Done!&lt;/p&gt;

&lt;h3 id=&quot;whats-one-good-way-to-screw-this-up&quot;&gt;what’s one good way to screw this up?&lt;/h3&gt;

&lt;p&gt;&lt;a href=&quot;http://security.stackexchange.com/a/2210/34825&quot;&gt;Encryption without authentication&lt;/a&gt;.  In other words, the user can’t see their session, but &lt;em&gt;can&lt;/em&gt; modify it.&lt;/p&gt;

&lt;p&gt;At first blush, this doesn’t sound so bad: they can’t know what they’re changing the data to, so changes are essentially random and astronomically unlikely to produce a “working” result.&lt;/p&gt;

&lt;p&gt;In reality, it’s &lt;em&gt;quite&lt;/em&gt; bad: if there’s any pattern to the encrypted data (i.e. it’s not indistinguishable from random noise), then it can be exploited; for example, to repeat or remove certain sections of the data. With enough analysis, you could even start crafting arbitrary results.&lt;/p&gt;

&lt;p&gt;If the encryption algorithm used is reasonable, this shouldn’t be possible; any change should cause the decryption to fail or produce garbage results.  This could still be a vector for an effective DoS, though.&lt;/p&gt;

&lt;h3 id=&quot;what-if-the-encryption-algorithm-used-isnt-reasonable&quot;&gt;what if the encryption algorithm used isn’t reasonable?&lt;/h3&gt;

&lt;p&gt;Let’s finally turn our attention to CodeIgniter.  As a reminder, we’re looking at the pre-2.2 code, the latest release then being 2.1.4.&lt;/p&gt;

&lt;p&gt;First, let’s look at their &lt;a href=&quot;https://github.com/bcit-ci/CodeIgniter/blob/2.1.4/system/libraries/Session.php&quot;&gt;session storage mechanism, system/libraries/Session.php&lt;/a&gt;.  It’s highly configurable (probably a bad thing); note these options and their meanings:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;$sess_encrypt_cookie&lt;/code&gt; — do we encrypt the cookie?  Defaults to false, but I bet we want this turned on.&lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;$sess_use_database&lt;/code&gt; — do we stick the session data in the database?  Defaults to false.  Maybe it’s fine to leave this.&lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;$encryption_key&lt;/code&gt; — sounds very important.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If we read the constructor, we see the encryption key is indeed required, regardless of whether the cookie itself is encrypted.  Why would that be?&lt;/p&gt;

&lt;p&gt;If we scroll down to &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;_set_cookie&lt;/code&gt;, we can see &lt;a href=&quot;https://github.com/bcit-ci/CodeIgniter/blob/2.1.4/system/libraries/Session.php#L655-L663&quot;&gt;something strange&lt;/a&gt;:&lt;/p&gt;

&lt;figure class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-php&quot; data-lang=&quot;php&quot;&gt;&lt;span class=&quot;k&quot;&gt;if&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;$this&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;-&amp;gt;&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;sess_encrypt_cookie&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;==&lt;/span&gt; &lt;span class=&quot;kc&quot;&gt;TRUE&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
    &lt;span class=&quot;nv&quot;&gt;$cookie_data&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;$this&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;-&amp;gt;&lt;/span&gt;&lt;span class=&quot;no&quot;&gt;CI&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;-&amp;gt;&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;encrypt&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;-&amp;gt;&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;encode&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;$cookie_data&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;);&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;else&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
    &lt;span class=&quot;c1&quot;&gt;// if encryption is not used, we provide an md5 hash to prevent userside tampering&lt;/span&gt;
    &lt;span class=&quot;nv&quot;&gt;$cookie_data&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;$cookie_data&lt;/span&gt;&lt;span class=&quot;mf&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;md5&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;$cookie_data&lt;/span&gt;&lt;span class=&quot;mf&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;$this&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;-&amp;gt;&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;encryption_key&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;);&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;

&lt;p&gt;Pay close attention to the comment: they prevent userside tampering &lt;strong&gt;if&lt;/strong&gt; encryption is not used.  And if it is?  Let’s have a look at &lt;a href=&quot;https://github.com/bcit-ci/CodeIgniter/blob/2.1.4/system/libraries/Encrypt.php&quot;&gt;Encrypt.php&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;A brief scan of &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;encode&lt;/code&gt; and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;decode&lt;/code&gt; strongly suggests that no authentication is done; this means we can change the data on the client-side with ease; there’s no MAC protecting it.  The natural follow-up question is, can we make anything of this leeway?&lt;/p&gt;

&lt;h3 id=&quot;say-no-to-fallback&quot;&gt;say no to fallback&lt;/h3&gt;

&lt;p&gt;The &lt;a href=&quot;https://github.com/bcit-ci/CodeIgniter/blob/2.1.4/system/libraries/Encrypt.php#L103-L109&quot;&gt;header of &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;encode&lt;/code&gt;&lt;/a&gt; has this to say about itself:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;Encodes the message string using bitwise XOR encoding. The key is combined
with a random hash, and then it too gets converted using XOR. The whole thing
is then run through mcrypt (if supported) using the randomized key. The end
result is a double-encrypted message string that is randomized with each call
to this function, even if the supplied message and key are the same.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Do you see what I see?&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;Encodes the message string using bitwise XOR encoding. The key is combined
with a random hash, and then it too gets converted using XOR. The whole thing
is then run through mcrypt &lt;strong&gt;(if supported)&lt;/strong&gt; using the randomized key. The end
result is a double-encrypted message string that is randomized with each call
to this function, even if the supplied message and key are the same.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;mcrypt is old, and &lt;a href=&quot;https://paragonie.com/blog/2015/05/if-you-re-typing-word-mcrypt-into-your-code-you-re-doing-it-wrong&quot;&gt;gets bad press for very valid
reasons&lt;/a&gt;,
but it provides some primitives that can work.&lt;/p&gt;

&lt;p&gt;So if it’s not supported, we’re left with the rest of the trash in that
paragraph.  “The key is combined with a random hash, and then it too gets
converted using XOR”.  Converted?? what does that mean???&lt;/p&gt;

&lt;p&gt;It’s worth calling out &lt;a href=&quot;https://cdn.rawgit.com/bcit-ci/CodeIgniter/2.1.4/user_guide/libraries/encryption.html&quot;&gt;CodeIgniter’s own documentation&lt;/a&gt; here:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;If Mcrypt is not available on your server the encoded message will still
provide a reasonable degree of security for encrypted sessions or other such
“light” purposes.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;What the hell is a “light” purpose of encryption?  Those inverted commas in the
quoted portion are verbatim, I should add.  Even they aren’t convinced that a
“light” purpose of encryption exists, yet they claim encrypted sessions are
such a case.  Just the foundation of your site or app’s security, nbd.&lt;/p&gt;

&lt;p&gt;Let’s find out just how bad an idea it is to have “fallback crypto” that you
cooked up yourself.  And here’s the thing: &lt;strong&gt;it’s going to get called&lt;/strong&gt;.  You
don’t add fallback code without someone using it.  This stuff is criminally bad; you can’t say “oh well, it’s a fallback, no-one should use it”.  If that’s the case, remove it; fail to work without the dependency.&lt;/p&gt;

&lt;p&gt;This is exactly what they did in 2.2, but there was a good &lt;em&gt;eight years&lt;/em&gt; while
this stuff was in &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;HEAD&lt;/code&gt;.  The fallback code got called.&lt;/p&gt;

&lt;h3 id=&quot;how-do-you-use-key-material-not-like-this-not-like-this&quot;&gt;how do you use key material? not like this… not like this.&lt;/h3&gt;

&lt;p&gt;Let’s start with &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;encode&lt;/code&gt;:&lt;/p&gt;

&lt;figure class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-php&quot; data-lang=&quot;php&quot;&gt;&lt;span class=&quot;k&quot;&gt;function&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;encode&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;$string&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;$key&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;&apos;&apos;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
    &lt;span class=&quot;nv&quot;&gt;$key&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;$this&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;-&amp;gt;&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;get_key&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;$key&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;);&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;if&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;$this&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;-&amp;gt;&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;_mcrypt_exists&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;===&lt;/span&gt; &lt;span class=&quot;kc&quot;&gt;TRUE&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
    &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
        &lt;span class=&quot;nv&quot;&gt;$enc&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;$this&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;-&amp;gt;&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;mcrypt_encode&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;$string&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;$key&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;);&lt;/span&gt;
    &lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;else&lt;/span&gt;
    &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
        &lt;span class=&quot;nv&quot;&gt;$enc&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;$this&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;-&amp;gt;&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;_xor_encode&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;$string&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;$key&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;);&lt;/span&gt;
    &lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;return&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;base64_encode&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;$enc&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;);&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;

&lt;p&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;$key&lt;/code&gt; wasn’t passed from the session library, so &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;get_key&lt;/code&gt; is called with an empty string; that, in turn, does the following:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;returns &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;$this-&amp;gt;encryption_key&lt;/code&gt; directly if it’s set.  You can grep (or &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;ag&lt;/code&gt;) the codebase quickly to find the only setter is &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;set_key&lt;/code&gt; in the class, which isn’t called by CI itself.&lt;/li&gt;
  &lt;li&gt;fetches &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;&apos;encryption_key&apos;&lt;/code&gt; from the CI config.  This is the same one that the Session class mandates is set, though Session doesn’t use it itself in encrypted-cookie mode.&lt;/li&gt;
  &lt;li&gt;surprisingly: returns the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;md5()&lt;/code&gt; of &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;$key&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;It’s surprising because the comment says: “Returns it as MD5 in order to have an exact-length 128 bit key”.  But PHP’s &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;md5&lt;/code&gt;, by default, returns the digest as a hexstring, not as raw data, meaning each byte will have &lt;em&gt;four&lt;/em&gt; bits of entropy and not eight.  It also means the mcrypt codepath might well be ignoring half the key.&lt;/p&gt;

&lt;p&gt;There is a noteworthy remark on the &lt;a href=&quot;http://php.net/manual/en/function.mcrypt-encrypt.php#refsect1-function.mcrypt-encrypt-changelog&quot;&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;mcrypt_encrypt&lt;/code&gt; changelog&lt;/a&gt;:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;Changed in: 5.6.0&lt;/p&gt;

  &lt;p&gt;Invalid key and iv sizes are no longer accepted. &lt;strong&gt;mcrypt_encrypt()&lt;/strong&gt; will now throw a warning and return &lt;strong&gt;FALSE&lt;/strong&gt; if the inputs are invalid. Previously keys and IVs were padded with ‘\0’ bytes to the next valid size.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;&lt;a href=&quot;https://soundcloud.com/kivikakk/gems&quot;&gt;Outrageous&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Let’s move back to &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;encode&lt;/code&gt;.  &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;$key&lt;/code&gt; now has the hexed MD5 of our actual encryption key (i.e. it’s a 32 byte string of hex digits).  We throw the plaintext string and that MD5 into &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;_xor_encode&lt;/code&gt;, base64 the result, and that’s our session cookie.&lt;/p&gt;

&lt;p&gt;Let’s look at &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;_xor_encode&lt;/code&gt;.&lt;/p&gt;

&lt;h3 id=&quot;how-not-to-use-randomness&quot;&gt;how not to use randomness&lt;/h3&gt;

&lt;p&gt;Reminder: &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;$string&lt;/code&gt; is plaintext, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;$key&lt;/code&gt; is a 32-byte hexstring.&lt;/p&gt;

&lt;figure class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-php&quot; data-lang=&quot;php&quot;&gt;&lt;span class=&quot;k&quot;&gt;function&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;_xor_encode&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;$string&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;$key&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
    &lt;span class=&quot;nv&quot;&gt;$rand&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;&apos;&apos;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;while&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;strlen&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;$rand&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;&amp;lt;&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;32&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
    &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
        &lt;span class=&quot;nv&quot;&gt;$rand&lt;/span&gt; &lt;span class=&quot;mf&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;mt_rand&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;mt_getrandmax&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;());&lt;/span&gt;
    &lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;
    &lt;span class=&quot;nv&quot;&gt;$rand&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;$this&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;-&amp;gt;&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;hash&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;$rand&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;);&lt;/span&gt;
    &lt;span class=&quot;nv&quot;&gt;$enc&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;&apos;&apos;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;for&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;$i&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;$i&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;&amp;lt;&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;strlen&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;$string&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;);&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;$i&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;++&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
    &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
        &lt;span class=&quot;nv&quot;&gt;$enc&lt;/span&gt; &lt;span class=&quot;mf&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;substr&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;$rand&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;$i&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;%&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;strlen&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;$rand&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)),&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
              &lt;span class=&quot;mf&quot;&gt;.&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;substr&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;$rand&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;$i&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;%&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;strlen&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;$rand&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)),&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;^&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;substr&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;$string&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;$i&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;));&lt;/span&gt;
    &lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;return&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;$this&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;-&amp;gt;&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;_xor_merge&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;$enc&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;$key&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;);&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;

&lt;p&gt;Let’s break it down:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;We generate random numbers and &lt;em&gt;concatenate&lt;/em&gt; them until we have more than 32 digits.  That’s just bizarre.&lt;/li&gt;
  &lt;li&gt;We then call &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;$this-&amp;gt;hash&lt;/code&gt; on the random number string, which by default will just &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;sha1()&lt;/code&gt; it, &lt;em&gt;again&lt;/em&gt; returning a hexstring, this time 40 characters long.&lt;/li&gt;
  &lt;li&gt;We now iterate over each character of the plaintext and the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;$rand&lt;/code&gt; string, cycled.
    &lt;ul&gt;
      &lt;li&gt;We append to &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;$enc&lt;/code&gt; the byte from the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;$rand&lt;/code&gt; string.&lt;/li&gt;
      &lt;li&gt;We then append to &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;$enc&lt;/code&gt; the same byte XOR’d with the corresponding plaintext byte.&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;We then &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;_xor_merge&lt;/code&gt; the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;$enc&lt;/code&gt; with &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;$key&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;In other words, if:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;$rand&lt;/code&gt; looks like &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;&quot;RRRRRRR...&quot;&lt;/code&gt;, and&lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;$string&lt;/code&gt; looks like &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;&quot;xxxx&quot;&lt;/code&gt;, then&lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;$enc&lt;/code&gt; will look like &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;&quot;R*R*R*R*&quot;&lt;/code&gt;,&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;where &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;&apos;R&apos;&lt;/code&gt; ⊕ &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;&apos;x&apos;&lt;/code&gt; = &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;&apos;*&apos;&lt;/code&gt;.  We can XOR the pairs together to recover the plaintext.&lt;/p&gt;

&lt;p&gt;Of course, we haven’t even used the key material yet.  That’s entirely the domain of &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;_xor_merge&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;_xor_merge&lt;/code&gt; is defined as a “key + string” combiner by the header doc.  Let’s see what that means.&lt;/p&gt;

&lt;figure class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-php&quot; data-lang=&quot;php&quot;&gt;&lt;span class=&quot;k&quot;&gt;function&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;_xor_merge&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;$string&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;$key&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
    &lt;span class=&quot;nv&quot;&gt;$hash&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;$this&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;-&amp;gt;&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;hash&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;$key&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;);&lt;/span&gt;
    &lt;span class=&quot;nv&quot;&gt;$str&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;&apos;&apos;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;for&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;$i&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;$i&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;&amp;lt;&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;strlen&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;$string&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;);&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;$i&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;++&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
    &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
        &lt;span class=&quot;nv&quot;&gt;$str&lt;/span&gt; &lt;span class=&quot;mf&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;substr&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;$string&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;$i&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;^&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;substr&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;$hash&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;$i&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;%&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;strlen&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;$hash&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)),&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;);&lt;/span&gt;
    &lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;return&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;$str&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;

&lt;p&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;$key&lt;/code&gt; as passed into the function is the hexstring of the MD5 of &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;encryption_key&lt;/code&gt;.  We then produce the hexed SHA1 of &lt;em&gt;that&lt;/em&gt;, and perform a 1-to-1 XOR of &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;$string[$i]&lt;/code&gt; with &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;$hash[$i]&lt;/code&gt; for all &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;$i&lt;/code&gt;; the hash being a cycled 40 characters.  Remember that &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;$string&lt;/code&gt; here is &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;$enc&lt;/code&gt; above (&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;&quot;R*R*R*R*&quot;&lt;/code&gt;).&lt;/p&gt;

&lt;p&gt;It’s important to note that these 40 hexadecimal characters are all that’s left of the key material, and they’re applied in &lt;a href=&quot;https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Electronic_Codebook_.28ECB.29&quot;&gt;ECB mode&lt;/a&gt; (like Snapchat, &lt;a href=&quot;https://kivikakk.ee/2013/05/10/snapchat.html&quot;&gt;which I’ve written about&lt;/a&gt;).  The main implication of this is that later blocks aren’t affected by earlier blocks; if we correctly decrypt byte &lt;em&gt;k&lt;/em&gt; of the output, then we’ll also get every byte (&lt;em&gt;k&lt;/em&gt; + &lt;strong&gt;W&lt;/strong&gt;&lt;em&gt;n&lt;/em&gt;), where &lt;strong&gt;W&lt;/strong&gt; is the width of the block.&lt;/p&gt;

&lt;p&gt;Usually — using a strong block cipher mode — getting any part of a block wrong would ensure every subsequent block would be corrupted, as the plaintext output of the earlier blocks are fed into later ones.  Not so with ECB, which they’ve unwittingly used. (Though “used” seems to be a bit of a stretch here.)&lt;/p&gt;

&lt;p&gt;Call the hexed SHA1 key material &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;&quot;kkkkkkk...&quot;&lt;/code&gt;.  We XOR this with the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;&quot;R*R*R*R*&quot;&lt;/code&gt; string, producing &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;&quot;9A9A9A9A&quot;&lt;/code&gt;.  This is the final output.&lt;/p&gt;

&lt;p&gt;Let’s review the entire process key material and plaintext takes to be encrypted, with some simplifications to make it readable in English:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;A random SHA1 is produced (&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;&quot;RRRRRRR...&quot;&lt;/code&gt;).&lt;/li&gt;
  &lt;li&gt;We interleave characters of the plaintext (&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;&quot;xxxx&quot;&lt;/code&gt;) with the random data, XORing each plaintext byte against the respective random byte (&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;&quot;R*R*R*R*&quot;&lt;/code&gt;).&lt;/li&gt;
  &lt;li&gt;We take the SHA1 of the MD5 of the key (&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;&quot;kkkkkkk...&quot;&lt;/code&gt;).&lt;/li&gt;
  &lt;li&gt;We bytewise XOR this SHA1 against each byte of the plaintext–random pairs (&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;&quot;9A9A9A9A&quot;&lt;/code&gt;).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;To recover the first byte of plaintext, then, looks like this:&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;Take the first two bytes of the ciphertext (&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;&quot;9A&quot;&lt;/code&gt;).&lt;/li&gt;
  &lt;li&gt;Take the SHA1 of the MD5 of the key (&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;&quot;kkkkkkk...&quot;)&lt;/code&gt;).&lt;/li&gt;
  &lt;li&gt;XOR the ciphertext bytewise with the SHA1 (&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;&quot;R*&quot;&lt;/code&gt;).&lt;/li&gt;
  &lt;li&gt;XOR the resulting pair of bytes (&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;&quot;x&quot;&lt;/code&gt;).&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Note that the key material is only ever used in SHA1(MD5(&lt;em&gt;key&lt;/em&gt;)) form.  For our purposes, then, we only need to know the resulting SHA1; the source key doesn’t matter.  To put it more clearly, we only need to recover the SHA1 output itself to break the effective key.&lt;/p&gt;

&lt;p&gt;Looking at the above list of steps, we note that the SHA1 is only involved in the 3rd step; and moreover, this being a hexed SHA1, there’s only 8 bits of entropy across the two bytes, each byte taking 16 values.&lt;/p&gt;

&lt;p&gt;(If you do the math, there’s actually only 6 bits; in the ASCII values of “0” through “9” and “a” through “f” bits 6 (always on) and 8 (always off) don’t change.)&lt;/p&gt;

&lt;p&gt;XORs commute and associate, so we can be lazy and define the plaintext byte &lt;em&gt;i&lt;/em&gt; as:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;&lt;em&gt;p&lt;/em&gt;[&lt;em&gt;i&lt;/em&gt;] = &lt;em&gt;c&lt;/em&gt;[2&lt;em&gt;i&lt;/em&gt;] ^ &lt;em&gt;c&lt;/em&gt;[2&lt;em&gt;i&lt;/em&gt;+1] ^ &lt;em&gt;k&lt;/em&gt;[2&lt;em&gt;i&lt;/em&gt; mod 40] ^ &lt;em&gt;k&lt;/em&gt;[2&lt;em&gt;i&lt;/em&gt;+1 mod 40]&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;This is very straight forward.  Simplicity is often a nice thing in crypto, but this is the wrong kind of simple.&lt;/p&gt;

&lt;p&gt;Let’s say we &lt;em&gt;already know&lt;/em&gt; &lt;em&gt;p&lt;/em&gt;[&lt;em&gt;i&lt;/em&gt;] for some &lt;em&gt;i&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;Knowing all of &lt;em&gt;c&lt;/em&gt;, this tells us something about &lt;em&gt;k&lt;/em&gt;[2&lt;em&gt;i&lt;/em&gt; mod 40] and &lt;em&gt;k&lt;/em&gt;[2&lt;em&gt;i&lt;/em&gt;+1 mod 40].  Because there are some serious value restrictions on elements of &lt;em&gt;k&lt;/em&gt; (being that there are only 16 possible values for a given element and not 256 like there should be), this lets us piece together &lt;em&gt;k&lt;/em&gt; quite neatly, and thus all the elements of &lt;em&gt;p&lt;/em&gt; we don’t know.&lt;/p&gt;

&lt;h3 id=&quot;do-we-know-the-plaintext&quot;&gt;do we know the plaintext?&lt;/h3&gt;

&lt;p&gt;Do we know any &lt;em&gt;p&lt;/em&gt;[&lt;em&gt;i&lt;/em&gt;]?  Well, what’s the plaintext?  &lt;a href=&quot;https://github.com/bcit-ci/CodeIgniter/blob/2.1.4/system/libraries/Session.php#L645-L676&quot;&gt;Session.php reveals all&lt;/a&gt;:&lt;/p&gt;

&lt;figure class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-php&quot; data-lang=&quot;php&quot;&gt;&lt;span class=&quot;nv&quot;&gt;$cookie_data&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;$this&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;-&amp;gt;&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;_serialize&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;$cookie_data&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;);&lt;/span&gt;
&lt;span class=&quot;err&quot;&gt;…&lt;/span&gt;
&lt;span class=&quot;nv&quot;&gt;$cookie_data&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;$this&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;-&amp;gt;&lt;/span&gt;&lt;span class=&quot;no&quot;&gt;CI&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;-&amp;gt;&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;encrypt&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;-&amp;gt;&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;encode&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;$cookie_data&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;);&lt;/span&gt;
&lt;span class=&quot;err&quot;&gt;…&lt;/span&gt;
&lt;span class=&quot;nb&quot;&gt;setcookie&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;
    &lt;span class=&quot;nv&quot;&gt;$this&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;-&amp;gt;&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;sess_cookie_name&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
    &lt;span class=&quot;nv&quot;&gt;$cookie_data&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
    &lt;span class=&quot;nv&quot;&gt;$expire&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
    &lt;span class=&quot;nv&quot;&gt;$this&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;-&amp;gt;&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;cookie_path&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
    &lt;span class=&quot;nv&quot;&gt;$this&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;-&amp;gt;&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;cookie_domain&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
    &lt;span class=&quot;nv&quot;&gt;$this&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;-&amp;gt;&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;cookie_secure&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;);&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;

&lt;p&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;_serialize&lt;/code&gt; massages the data in a way we can ignore for our purposes before passing it through to PHP’s own &lt;a href=&quot;http://php.net/serialize&quot;&gt;serialize&lt;/a&gt;.  So the session data is pure PHP serialised data.  What does this look like?&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://github.com/bcit-ci/CodeIgniter/blob/2.1.4/system/libraries/Session.php#L264-L273&quot;&gt;Go up to &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;sess_write&lt;/code&gt;&lt;/a&gt;; you’ll see &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;$cookie_userdata&lt;/code&gt; is an array, which has keys set in a particular order: &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;&apos;session_id&apos;&lt;/code&gt;, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;&apos;ip_address&apos;&lt;/code&gt;, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;&apos;user_agent&apos;&lt;/code&gt;, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;&apos;last_activity&apos;&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;It’s important to know that PHP arrays are both associative &lt;em&gt;and&lt;/em&gt; ordered!  The same datatype is used for both key–value dictionaries as well as integer-indexed arrays.  This means order is present &lt;em&gt;and&lt;/em&gt; preserved in &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;$cookie_userdata&lt;/code&gt;, and this remains true for the serialised form.&lt;/p&gt;

&lt;p&gt;If we fake our own &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;$cookie_userdata&lt;/code&gt; in PHP and then serialise it, what would it look like?  Here’s an example:&lt;/p&gt;

&lt;figure class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-php&quot; data-lang=&quot;php&quot;&gt;&lt;span class=&quot;k&quot;&gt;echo&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;serialize&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;k&quot;&gt;array&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;
    &lt;span class=&quot;c1&quot;&gt;// Session.php:317 shows that session IDs are hexed MD5, so always 32 chars&lt;/span&gt;
    &lt;span class=&quot;s1&quot;&gt;&apos;session_id&apos;&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&amp;gt;&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;&apos;1234567890abcdef1234567890abcdef&apos;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;  
    &lt;span class=&quot;s1&quot;&gt;&apos;ip_address&apos;&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&amp;gt;&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;&apos;0.0.0.0&apos;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
    &lt;span class=&quot;s1&quot;&gt;&apos;user_agent&apos;&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&amp;gt;&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;&apos;welp/1.0&apos;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
    &lt;span class=&quot;s1&quot;&gt;&apos;last_activity&apos;&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&amp;gt;&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;1455953571&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;));&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;

&lt;p&gt;Here’s the output:&lt;/p&gt;

&lt;figure class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-php&quot; data-lang=&quot;php&quot;&gt;&lt;span class=&quot;n&quot;&gt;a&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;4&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;s&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;10&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&quot;session_id&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;s&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;32&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&quot;1234567890abcdef1234567890abcdef&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;s&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;10&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&quot;ip_address&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;s&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;7&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&quot;0.0.0.0&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;s&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;10&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&quot;user_agent&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;s&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;8&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&quot;welp/1.0&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;s&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;13&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&quot;last_activity&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;i&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1455953571&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;}&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;

&lt;p&gt;The format is pretty simple to glean from this. Excuse my pseudo-not-even-BNF:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;&lt;em&gt;object&lt;/em&gt; = &lt;em&gt;array&lt;/em&gt; | &lt;em&gt;string&lt;/em&gt; | &lt;em&gt;integer&lt;/em&gt;&lt;/li&gt;
  &lt;li&gt;&lt;em&gt;array&lt;/em&gt; = &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;a&lt;/code&gt; &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;:&lt;/code&gt; number &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;:&lt;/code&gt; &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;{&lt;/code&gt; (&lt;em&gt;object&lt;/em&gt; &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;;&lt;/code&gt; &lt;em&gt;object&lt;/em&gt; &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;;&lt;/code&gt;)* &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;}&lt;/code&gt;&lt;/li&gt;
  &lt;li&gt;&lt;em&gt;string&lt;/em&gt; = &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;s&lt;/code&gt; &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;:&lt;/code&gt; number &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;:&lt;/code&gt; &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;&quot;&lt;/code&gt; (any-char)* &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;&quot;&lt;/code&gt;&lt;/li&gt;
  &lt;li&gt;&lt;em&gt;integer&lt;/em&gt; = &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;i&lt;/code&gt; &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;:&lt;/code&gt; number&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So what do we have after all this? The first two bytes of our known plaintext: &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;a:&lt;/code&gt;.&lt;/p&gt;

&lt;h3 id=&quot;lets-put-this-together&quot;&gt;let’s put this together&lt;/h3&gt;

&lt;p&gt;Here’s what we do: iterate over all 256 possible values of &lt;em&gt;k&lt;/em&gt;[0] and &lt;em&gt;k&lt;/em&gt;[1]; these correspond to &lt;em&gt;p&lt;/em&gt;[0].  The rest of &lt;em&gt;k&lt;/em&gt; doesn’t matter.&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;For each &lt;em&gt;k&lt;/em&gt;, we calculate &lt;em&gt;p&lt;/em&gt;′ = decrypt(&lt;em&gt;k&lt;/em&gt;, &lt;em&gt;c&lt;/em&gt;).&lt;/li&gt;
  &lt;li&gt;Check &lt;em&gt;p&lt;/em&gt;′[0] == &lt;em&gt;p&lt;/em&gt;[0].&lt;/li&gt;
  &lt;li&gt;If valid, we found &lt;em&gt;k&lt;/em&gt;[0] and &lt;em&gt;k&lt;/em&gt;[1]!  In an average of 128 decrypt operations (which are just a bunch of XORs)!  Continue on to &lt;em&gt;p&lt;/em&gt;[1] and &lt;em&gt;k&lt;/em&gt;[2] and &lt;em&gt;k&lt;/em&gt;[3].&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This can be done very, very fast.&lt;/p&gt;

&lt;p&gt;This is actually slightly off; because of the weird keyspace and the fact that we’re XORing two parts of the key for one byte, (&lt;em&gt;k&lt;/em&gt;[&lt;em&gt;n&lt;/em&gt;], &lt;em&gt;k&lt;/em&gt;[&lt;em&gt;n&lt;/em&gt;+1]) = (&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;&apos;2&apos;&lt;/code&gt;, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;&apos;1&apos;&lt;/code&gt;) works the same as it would if it were equal to (&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;&apos;3&apos;&lt;/code&gt;, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;&apos;0&apos;&lt;/code&gt;).  You don’t necessarily get the exact SHA1 of the hex MD5 of the key, but you get one that works identically.&lt;/p&gt;

&lt;p&gt;(In other words, there are even fewer distinct keys than there appear! Wow.)&lt;/p&gt;

&lt;p&gt;We hit a slight hiccup because we don’t necessarily know the size of the session array, so the third byte of plaintext is unknown.  But it doesn’t take too long to realise that &lt;em&gt;k&lt;/em&gt;[0] and &lt;em&gt;k&lt;/em&gt;[1] correspond not only to &lt;em&gt;p&lt;/em&gt;[0], but also &lt;em&gt;p&lt;/em&gt;[20], &lt;em&gt;p&lt;/em&gt;[40], etc., because the key material is cycled (see above re: ECB).&lt;/p&gt;

&lt;p&gt;Let’s align the known plaintext data in rows of 20 bytes:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;&lt;tt&gt;&lt;strong&gt;a:&lt;/strong&gt;&lt;span style=&quot;color: #777&quot;&gt;4&lt;/span&gt;&lt;strong&gt;:{s:10:”session_i&lt;/strong&gt;&lt;/tt&gt;&lt;br /&gt;
&lt;tt&gt;&lt;strong&gt;d”;s:32:”&lt;/strong&gt;&lt;span style=&quot;color: #777&quot;&gt;1234567890a&lt;/span&gt;&lt;/tt&gt;&lt;br /&gt;
&lt;tt&gt;&lt;span style=&quot;color: #777&quot;&gt;bcdef1234567890abcde&lt;/span&gt;&lt;/tt&gt;&lt;br /&gt;
&lt;tt&gt;&lt;span style=&quot;color: #777&quot;&gt;f&lt;/span&gt;&lt;strong&gt;“;s:10:”ip_address”&lt;/strong&gt;&lt;/tt&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;In bold is everything we’re sure about.  You can see that, in the first two blocks (rows) alone, we have enough known-plaintext to gather all 40 bytes of key data; we’re missing the 3rd byte in the first block, but we have the 3rd byte in the second block.&lt;/p&gt;

&lt;p&gt;There is the concern that the session array may have more than 9 keys, making the unknown part of the first block one byte greater, in which case it looks like this:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;&lt;tt&gt;&lt;strong&gt;a:&lt;/strong&gt;&lt;span style=&quot;color: #777&quot;&gt;12&lt;/span&gt;&lt;strong&gt;:{s:10:”session_&lt;/strong&gt;&lt;/tt&gt;&lt;br /&gt;
&lt;tt&gt;&lt;strong&gt;id”;s:32:”&lt;/strong&gt;&lt;span style=&quot;color: #777&quot;&gt;1234567890&lt;/span&gt;&lt;/tt&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;This is fine; we get both bytes 3 and 4 from the second block.  The same goes for many digits more, by which time we leave the realm of probability and cookie size limits.&lt;/p&gt;

&lt;p&gt;And that’s it.  We’ve done it.  You can write an automated cracker based on this alone.&lt;/p&gt;

&lt;p&gt;Here’s a PoC in action, artifically slowed down so you can watch its progress:&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/post-img/codeigniter.gif&quot; /&gt;&lt;/p&gt;

&lt;p&gt;Too simple!  Now that you have the key (or rather, the SHA1 of the MD5 hexstring of the key — good enough), you can feel free to change the data in the session, do what you like, and become whom you like.  Maybe even realise that PHP serialisation admits arbitrary objects, which you might be able to do something special with.&lt;/p&gt;

&lt;p&gt;If you liked the article, please share!  I also welcome your feedback/thoughts on &lt;a href=&quot;https://twitter.com/kivikakk&quot;&gt;Twitter&lt;/a&gt;.  Special thanks to &lt;a href=&quot;https://twitter.com/ibutsu&quot;&gt;Kairi&lt;/a&gt; for editing this article.&lt;/p&gt;

&lt;p&gt;The PoC code follows.  Enjoy!&lt;/p&gt;

&lt;script src=&quot;https://gist.github.com/kivikakk/c510b82a6ad828c3c5d3.js&quot;&gt;&lt;/script&gt;

</content>
    
 </entry>
 
 <entry>
   <title>Snapchat: not for state secrets</title>
   <updated>2013-05-10T20:44:00+10:00</updated>
   <id>https://kivikakk.ee/2013/05/10/snapchat</id>
    
     <link href="https://kivikakk.ee/2013/05/10/snapchat/"/>
     <content type="html">&lt;p&gt;I use &lt;a href=&quot;http://www.snapchat.com/&quot;&gt;Snapchat&lt;/a&gt;.  It’s an app where you can take a
photo or short (&amp;lt; 10 second) video and send it to your friends who use the
service; they’ll then be able to see it, once, before it disappears forever.&lt;/p&gt;

&lt;p&gt;Ostensibly, the app is for sexting, because there’s no fear that your photo
will get spread around (no forwarding/etc.) or retained for longer than you’d
like, but it &lt;a href=&quot;http://survata.com/blog/is-snapchat-only-used-for-sexting-we-asked-5000-people-to-find-out/&quot;&gt;seems like it’s not as much a sexter’s hangout as the media might
want you to
think&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;My circle of friends use it basically as an extension of weird Twitter – most
snaps I send and receive are strange angles of weird objects; the completely
mundane but somehow therapeutic (7 seconds of the camera pointed outside the
window of a tram, pointed at the ground moving below); or just closeups of
&lt;a href=&quot;https://www.google.com.au/search?q=curtis+stone&quot;&gt;Curtis Stone’s face&lt;/a&gt;,
wherever we see him.&lt;/p&gt;

&lt;p&gt;&lt;img style=&quot;border: 1px solid #000; margin: 0px auto; display: block;&quot; title=&quot;Curtis Stone. Ugh.&quot; src=&quot;/assets/post-img/stone.jpg&quot; /&gt;&lt;/p&gt;

&lt;p&gt;&lt;a id=&quot;more&quot;&gt;&lt;/a&gt;Of course, the promise that they won’t get retained is just that: a promise.
Since your phone receives this image and shows it to you at some point, it must
be downloaded by your phone.  If it can be downladed by the phone, it can be
downloaded by something else.  We decided to find out how.&lt;/p&gt;

&lt;!--more--&gt;

&lt;hr /&gt;

&lt;p&gt;My first thought was to use
&lt;a href=&quot;http://en.wikipedia.org/wiki/Cain_and_Abel_%28software%29&quot;&gt;Cain&lt;/a&gt; to re-route
the phone’s traffic to Snapchat via a computer with ARP poisoning, then
Wireshark to packet-sniff.  For whatever reason, we weren’t able to make this
work; while we did see some of the traffic (the non-HTTPS stuff), HTTPS
wouldn’t seem to pass through my friend’s computer.&lt;/p&gt;

&lt;p&gt;Another got to work using a different set of tools on a Linux machine to do ARP
stuff, and I took a more direct route.&lt;/p&gt;

&lt;p&gt;First, I set my phone’s proxy on WiFi to point to my machine.  Then I just
listened with netcat.  After receiving lots of apparently unrelated requests
(and the aforementioned HTTP requests&lt;sup id=&quot;fnref:flurry&quot; role=&quot;doc-noteref&quot;&gt;&lt;a href=&quot;#fn:flurry&quot; class=&quot;footnote&quot; rel=&quot;footnote&quot;&gt;1&lt;/a&gt;&lt;/sup&gt;), I found that Snapchat was
requesting an SSL forward:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;$ nc -l -p 5588
CONNECT feelinsonice.appspot.com:443 HTTP/1.1
Host: feelinsonice.appspot.com
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;“feelinsonice”.  Snapchat are hosted on GAE!&lt;/p&gt;

&lt;p&gt;Having confirmed confirmed that this is a worthwhile approach, I wrote a little
Ruby to receive requests and start coaxing data from Snapchat.  The first
iteration was something like this:&lt;/p&gt;

&lt;figure class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-ruby&quot; data-lang=&quot;ruby&quot;&gt;&lt;span class=&quot;c1&quot;&gt;#!/usr/bin/env ruby&lt;/span&gt;

&lt;span class=&quot;nb&quot;&gt;require&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;&apos;openssl&apos;&lt;/span&gt;
&lt;span class=&quot;nb&quot;&gt;require&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;&apos;socket&apos;&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;l&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;no&quot;&gt;TCPServer&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;new&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;5588&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;l&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;listen&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;10&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

&lt;span class=&quot;k&quot;&gt;while&lt;/span&gt; &lt;span class=&quot;kp&quot;&gt;true&lt;/span&gt;
  &lt;span class=&quot;n&quot;&gt;s&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;l&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;accept&lt;/span&gt;

  &lt;span class=&quot;n&quot;&gt;d&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;s&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;readpartial&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;8192&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
  &lt;span class=&quot;k&quot;&gt;if&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;d&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;!~&lt;/span&gt; &lt;span class=&quot;sr&quot;&gt;/CONNECT feelinsonice.appspot.com:443/&lt;/span&gt;
    &lt;span class=&quot;no&quot;&gt;STDERR&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;puts&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;rejecting unwanted client &lt;/span&gt;&lt;span class=&quot;si&quot;&gt;#{&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;d&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;inspect&lt;/span&gt;&lt;span class=&quot;si&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&quot;&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;s&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;close&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;next&lt;/span&gt;
  &lt;span class=&quot;k&quot;&gt;end&lt;/span&gt;

  &lt;span class=&quot;no&quot;&gt;STDERR&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;puts&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;probably good client &lt;/span&gt;&lt;span class=&quot;si&quot;&gt;#{&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;d&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;inspect&lt;/span&gt;&lt;span class=&quot;si&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&quot;&lt;/span&gt;

  &lt;span class=&quot;n&quot;&gt;ctx&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;no&quot;&gt;OpenSSL&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;::&lt;/span&gt;&lt;span class=&quot;no&quot;&gt;SSL&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;::&lt;/span&gt;&lt;span class=&quot;no&quot;&gt;SSLContext&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;new&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&quot;SSLv23_server&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
  &lt;span class=&quot;n&quot;&gt;ctx&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;cert&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;no&quot;&gt;OpenSSL&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;::&lt;/span&gt;&lt;span class=&quot;no&quot;&gt;X509&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;::&lt;/span&gt;&lt;span class=&quot;no&quot;&gt;Certificate&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;new&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;no&quot;&gt;File&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;read&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&quot;server.crt&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;))&lt;/span&gt;
  &lt;span class=&quot;n&quot;&gt;ctx&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;key&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;no&quot;&gt;OpenSSL&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;::&lt;/span&gt;&lt;span class=&quot;no&quot;&gt;PKey&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;::&lt;/span&gt;&lt;span class=&quot;no&quot;&gt;RSA&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;new&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;no&quot;&gt;File&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;read&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&quot;server.key&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;))&lt;/span&gt;
  &lt;span class=&quot;n&quot;&gt;ctx&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;verify_mode&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;no&quot;&gt;OpenSSL&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;::&lt;/span&gt;&lt;span class=&quot;no&quot;&gt;SSL&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;::&lt;/span&gt;&lt;span class=&quot;no&quot;&gt;VERIFY_NONE&lt;/span&gt;

  &lt;span class=&quot;n&quot;&gt;s&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;write&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;HTTP/1.1 200 OK&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\r\n\r\n&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&quot;&lt;/span&gt;
  &lt;span class=&quot;n&quot;&gt;s&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;flush&lt;/span&gt;

  &lt;span class=&quot;n&quot;&gt;ss&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;no&quot;&gt;OpenSSL&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;::&lt;/span&gt;&lt;span class=&quot;no&quot;&gt;SSL&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;::&lt;/span&gt;&lt;span class=&quot;no&quot;&gt;SSLSocket&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;new&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;s&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;ctx&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
  &lt;span class=&quot;n&quot;&gt;ss&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;accept&lt;/span&gt;

  &lt;span class=&quot;no&quot;&gt;STDERR&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;puts&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;I THINK WE&apos;RE IN, JOHN.&quot;&lt;/span&gt;

  &lt;span class=&quot;n&quot;&gt;d&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;ss&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;readpartial&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;8192&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
  &lt;span class=&quot;no&quot;&gt;STDERR&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;puts&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;got data: &lt;/span&gt;&lt;span class=&quot;si&quot;&gt;#{&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;d&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;inspect&lt;/span&gt;&lt;span class=&quot;si&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&quot;&lt;/span&gt;

  &lt;span class=&quot;n&quot;&gt;ss&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;close&lt;/span&gt;
  &lt;span class=&quot;n&quot;&gt;s&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;close&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;end&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;

&lt;p&gt;To go with, I generated &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;server.crt&lt;/code&gt; and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;server.key&lt;/code&gt; with CN &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;*.appspot.com&lt;/code&gt;,
wondering if Snapchat are checking for a valid cert or not.&lt;/p&gt;

&lt;p&gt;Turns out they are: I was getting &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;EOFError&lt;/code&gt; thrown at the last &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;readpartial&lt;/code&gt;
call, so presumably that was Snapchat not liking my identity.&lt;/p&gt;

&lt;p&gt;Thankfully, the workaround wasn’t hard: make a local CA, install its
certificate on the phone, re-generate the SSL certificate with that
CA&lt;sup id=&quot;fnref:ssl-ca&quot; role=&quot;doc-noteref&quot;&gt;&lt;a href=&quot;#fn:ssl-ca&quot; class=&quot;footnote&quot; rel=&quot;footnote&quot;&gt;2&lt;/a&gt;&lt;/sup&gt;, and away we go!&lt;/p&gt;

&lt;p&gt;Next, capture the data from the phone, and establish the connection to Snapchat
ourselves to complete this man-in-the-middle. We put this after reading the
first block of data from the client above:&lt;/p&gt;

&lt;figure class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-ruby&quot; data-lang=&quot;ruby&quot;&gt;&lt;span class=&quot;c1&quot;&gt;# make the real connection&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;begin&lt;/span&gt;
  &lt;span class=&quot;n&quot;&gt;up&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;no&quot;&gt;TCPSocket&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;new&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s1&quot;&gt;&apos;feelinsonice.appspot.com&apos;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;443&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;rescue&lt;/span&gt; &lt;span class=&quot;no&quot;&gt;Errno&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;::&lt;/span&gt;&lt;span class=&quot;no&quot;&gt;ECONNREFUSED&lt;/span&gt;
  &lt;span class=&quot;no&quot;&gt;STDERR&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;puts&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;snapchat getting weary?&quot;&lt;/span&gt;
  &lt;span class=&quot;n&quot;&gt;ss&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;close&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;rescue&lt;/span&gt; &lt;span class=&quot;kp&quot;&gt;false&lt;/span&gt;
  &lt;span class=&quot;n&quot;&gt;s&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;close&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;rescue&lt;/span&gt; &lt;span class=&quot;kp&quot;&gt;false&lt;/span&gt;
  &lt;span class=&quot;k&quot;&gt;next&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;end&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;ups&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;no&quot;&gt;OpenSSL&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;::&lt;/span&gt;&lt;span class=&quot;no&quot;&gt;SSL&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;::&lt;/span&gt;&lt;span class=&quot;no&quot;&gt;SSLSocket&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;new&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;up&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;to_io&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;ups&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;connect&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;ups&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;write&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;d&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;no&quot;&gt;STDERR&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;puts&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;forwarded request from phone&quot;&lt;/span&gt;

&lt;span class=&quot;k&quot;&gt;while&lt;/span&gt; &lt;span class=&quot;kp&quot;&gt;true&lt;/span&gt;
  &lt;span class=&quot;k&quot;&gt;begin&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;r&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;ups&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;readpartial&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1024&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
  &lt;span class=&quot;k&quot;&gt;rescue&lt;/span&gt; &lt;span class=&quot;no&quot;&gt;EOFError&lt;/span&gt;
    &lt;span class=&quot;no&quot;&gt;STDERR&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;puts&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;no more&quot;&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;ss&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;close&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;s&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;close&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;break&lt;/span&gt;
  &lt;span class=&quot;k&quot;&gt;end&lt;/span&gt;
  &lt;span class=&quot;no&quot;&gt;STDERR&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;puts&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;they say: &lt;/span&gt;&lt;span class=&quot;si&quot;&gt;#{&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;r&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;inspect&lt;/span&gt;&lt;span class=&quot;si&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&quot;&lt;/span&gt;

  &lt;span class=&quot;c1&quot;&gt;# send back to phone&lt;/span&gt;
  &lt;span class=&quot;n&quot;&gt;ss&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;write&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;r&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
  
  &lt;span class=&quot;no&quot;&gt;STDERR&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;puts&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;written back&quot;&lt;/span&gt;

  &lt;span class=&quot;k&quot;&gt;break&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;if&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;r&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;length&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;zero?&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;end&lt;/span&gt;

&lt;span class=&quot;no&quot;&gt;STDERR&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;puts&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;quiet&quot;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;

&lt;p&gt;We open a regular SSL socket to Snapchat’s GAE server, forward the request from
the phone, and read back what Snapchat said.&lt;/p&gt;

&lt;p&gt;It turns out this is enough to start getting sensitive data in a form where we
can attack it offline&lt;sup id=&quot;fnref:http&quot; role=&quot;doc-noteref&quot;&gt;&lt;a href=&quot;#fn:http&quot; class=&quot;footnote&quot; rel=&quot;footnote&quot;&gt;3&lt;/a&gt;&lt;/sup&gt;.&lt;/p&gt;

&lt;p&gt;Add some logging of responses to file; you’ll see a request to &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;/ph/sync&lt;/code&gt;,
followed by a bulk of data indicating who our friends are, and information
regarding new snaps.  Then, the phone will try to fetch those snaps: you’ll see
requests to &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;/ph/blob&lt;/code&gt;, like:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;/ph/blob?id=...&amp;amp;username=hellomoto&amp;amp;timestamp=1368171438418&amp;amp;req_token=...
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;It turns out all the data required is in the URI; no funny header business.
You can paste the requested URL directly into a browser and fetch the blob.&lt;/p&gt;

&lt;p&gt;What you get is identified by the BSD file tool as &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;data&lt;/code&gt; – not obviously an
image, video, or whatever, nor any headers indicating encryption or
compression.&lt;/p&gt;

&lt;p&gt;First, I had a look at the size: 19,712 bytes, which is divisible by 256.  This
feels like too much of a coincidence; I’d be surprised by divisibility by
anything above 4 or 8.  My going assumption is that images are transferred in
JPEG, and ~half the JPEGs I looked at on my disk have odd numbers of bytes, so
I’m guessing there’s nothing frame-y about JPEG that would cause the plaintext
to be in a regular block of bytes – so conclusion, it’s probably a block
cipher.&lt;/p&gt;

&lt;p&gt;Next, it was time to see if there were any obvious cryptographic errors.  A
repeated block in the ciphertext might give us a hint about the encryption
mode.&lt;/p&gt;

&lt;p&gt;Sure enough, a repeated 16-byte block:&lt;/p&gt;

&lt;figure class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-ruby&quot; data-lang=&quot;ruby&quot;&gt;&lt;span class=&quot;o&quot;&gt;&amp;gt;&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;data&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;no&quot;&gt;File&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;open&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s1&quot;&gt;&apos;x&apos;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;&apos;r:ASCII-8BIT&apos;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;).&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;read&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt; &lt;span class=&quot;kp&quot;&gt;nil&lt;/span&gt;
&lt;span class=&quot;o&quot;&gt;=&amp;gt;&lt;/span&gt; &lt;span class=&quot;kp&quot;&gt;nil&lt;/span&gt;
&lt;span class=&quot;o&quot;&gt;&amp;gt;&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;data&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;bytes&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;each_slice&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;16&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;).&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;to_a&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;length&lt;/span&gt;
&lt;span class=&quot;o&quot;&gt;=&amp;gt;&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;1232&lt;/span&gt;
&lt;span class=&quot;o&quot;&gt;&amp;gt;&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;data&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;bytes&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;each_slice&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;8&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;).&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;to_a&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;length&lt;/span&gt;
&lt;span class=&quot;o&quot;&gt;=&amp;gt;&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;2464&lt;/span&gt;
&lt;span class=&quot;o&quot;&gt;&amp;gt;&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;data&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;bytes&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;each_slice&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;8&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;).&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;to_a&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;uniq&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;length&lt;/span&gt;
&lt;span class=&quot;o&quot;&gt;=&amp;gt;&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;2462&lt;/span&gt;
&lt;span class=&quot;o&quot;&gt;&amp;gt;&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;data&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;bytes&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;each_slice&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;16&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;).&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;to_a&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;length&lt;/span&gt;
&lt;span class=&quot;o&quot;&gt;=&amp;gt;&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;1232&lt;/span&gt;
&lt;span class=&quot;o&quot;&gt;&amp;gt;&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;data&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;bytes&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;each_slice&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;16&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;).&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;to_a&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;uniq&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;length&lt;/span&gt;
&lt;span class=&quot;o&quot;&gt;=&amp;gt;&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;1231&lt;/span&gt;
&lt;span class=&quot;o&quot;&gt;&amp;gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;

&lt;p&gt;So apparently a 16-byte (128-bit) block cipher, in &lt;a href=&quot;http://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Electronic_codebook_.28ECB.29&quot;&gt;ECB
mode&lt;/a&gt;
at that.  (Not a good thing.)   Seeing as there wasn’t a demonstration of a lot
of intelligence this far, I started to wonder if it wasn’t just XORed, as it’d
look the same.&lt;/p&gt;

&lt;p&gt;A &lt;a href=&quot;https://twitter.com/kyhwana/status/332770066150068227&quot;&gt;friend on Twitter
noted&lt;/a&gt; that the repeated
block was at the same location as JPEG files have a string of repeated bytes.&lt;/p&gt;

&lt;p&gt;Here’s a JPEG surrounding the repeated bytes:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;00000a0: 090c 0b0c 180d 0d18 3221 1c21 3232 3232  ........2!.!2222
00000b0: 3232 3232 3232 3232 3232 3232 3232 3232  2222222222222222
00000c0: 3232 3232 3232 3232 3232 3232 3232 3232  2222222222222222
00000d0: 3232 3232 3232 3232 3232 3232 3232 ffc0  22222222222222..
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Here’s the ciphertext around the repeated 16-byte blocks:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;0000060: 61e0 3cb3 ca5d 4ebe 4cd9 2212 3e9a 40ba  a.&amp;lt;..]N.L.&quot;.&amp;gt;.@.
0000070: 39e4 8dc2 ac39 8f10 59d8 fc08 9d19 b239  9....9..Y......9
0000080: 39e4 8dc2 ac39 8f10 59d8 fc08 9d19 b239  9....9..Y......9
0000090: 4614 4d69 7c61 5d11 cabb 5310 1697 5b4f  F.Mi|a]...S...[O
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Note that the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;32&lt;/code&gt; bytes are more than 2x 16-byte blocks in length: they extend
well before and after the 16-byte alignment.  But the ciphertext doesn’t show
that at all: this rules out a plain repeating XOR, as we’d otherwise expect to
see something more like this:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;0000060: 61e0 3cb3 ca5d 4ebe 4cd9 2212 9d19 b239  a.&amp;lt;..]N.L.&quot;....9
0000070: 39e4 8dc2 ac39 8f10 59d8 fc08 9d19 b239  9....9..Y......9
0000080: 39e4 8dc2 ac39 8f10 59d8 fc08 9d19 b239  9....9..Y......9
0000090: 39e4 8dc2 ac39 8f10 59d8 fc08 9d19 5b4f  9....9..Y.....[O
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;… assuming the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;32&lt;/code&gt; bytes above are exactly the number you’d expect to find
anywhere, which isn’t the case; but you get the point: there’d be some, but
there are none.  The key and data are totally mixed, which suggests a real
block cipher.&lt;/p&gt;

&lt;p&gt;Since there aren’t really good ways to attack this directly (at least, not for
me, an utter novice), it seemed much faster just look for the cipher/key/etc.
in the source.&lt;/p&gt;

&lt;p&gt;I was thinking I’d have to do some MitM of Google Play or root my Android, but
it turns out Googling ‘snapchat apk download’ is enough. Hah.&lt;/p&gt;

&lt;p&gt;The first tool I found for getting the contents and decompiling the APK was
&lt;a href=&quot;http://code.google.com/p/android-apktool/&quot;&gt;android-apktool&lt;/a&gt;; there are surely
better tools (this gives you smali output, not Java or Java-ish), but it was
easy enough to peruse, given I just wanted to know what the key was and what
primitives were being used.&lt;/p&gt;

&lt;p&gt;The code was totally unobfuscated, so it wasn’t hard to find the
&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;com.snapchat.android.api.SnapchatServer&lt;/code&gt;: the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;.smali&lt;/code&gt; file is a bit weird to
read, but sure enough there’s:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;.field private static final BASE_URL:Ljava/lang/String; =
        &quot;https://feelinsonice.appspot.com&quot;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;and:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;.line 247
.local v2, image:[B
sget-object v6, Lcom/snapchat/android/util/AESEncrypt;-&amp;gt;ENCRYPT_KEY_2:Ljava/lang/String;

invoke-static {v2, v6},
        Lcom/snapchat/android/util/AESEncrypt;-&amp;gt;encrypt([BLjava/lang/String;)[B

move-result-object v0

.line 248
.local v0, encryptedImage:[B
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;I guess that’d be like:&lt;/p&gt;

&lt;figure class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-java&quot; data-lang=&quot;java&quot;&gt;&lt;span class=&quot;c1&quot;&gt;// byte[] image;&lt;/span&gt;
&lt;span class=&quot;kt&quot;&gt;byte&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;[]&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;encryptedImage&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;nc&quot;&gt;AESEncrypt&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;na&quot;&gt;encrypt&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;image&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;nc&quot;&gt;AESEncrypt&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;na&quot;&gt;ENCRYPT_KEY_2&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;);&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;

&lt;p&gt;or something.  I don’t actually do Java, so maybe that’s all backwards, but the
point seems pretty clear.  It occurs to me I’m reading the encryption code, but
while there are two keys, only ENCRYPT_KEY_2 is ever used.&lt;/p&gt;

&lt;p&gt;So, what encryption is going on?  &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;AESEncrypt.smali&lt;/code&gt; reads:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;.line 8
const-string v0, &quot;1234567891123456&quot;

sput-object v0, Lcom/snapchat/android/util/AESEncrypt;-&amp;gt;ENCRYPT_KEY:Ljava/lang/String;

.line 9
const-string v0, &quot;M02cnQ51Ji97vwT4&quot;

sput-object v0, Lcom/snapchat/android/util/AESEncrypt;-&amp;gt;ENCRYPT_KEY_2:Ljava/lang/S
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Here are the keys!  Is it really as simple as 128-bit AES in ECB mode?&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;.line 21
const-string v3, &quot;AES/ECB/PKCS5Padding&quot;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Looks like it.  Note the padding scheme; seems weird to use PKCS#5 which has
apparently “only been defined for block ciphers that use 64 bit (8 byte) block
size”, when the size here is 128-bit.  Let’s give it a go.&lt;/p&gt;

&lt;figure class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-ruby&quot; data-lang=&quot;ruby&quot;&gt;&lt;span class=&quot;o&quot;&gt;&amp;gt;&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;data&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;no&quot;&gt;File&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;open&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s1&quot;&gt;&apos;x&apos;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;&apos;r:ASCII-8BIT&apos;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;).&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;read&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt; &lt;span class=&quot;kp&quot;&gt;nil&lt;/span&gt;
&lt;span class=&quot;o&quot;&gt;=&amp;gt;&lt;/span&gt; &lt;span class=&quot;kp&quot;&gt;nil&lt;/span&gt;
&lt;span class=&quot;o&quot;&gt;&amp;gt;&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;c&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;no&quot;&gt;OpenSSL&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;::&lt;/span&gt;&lt;span class=&quot;no&quot;&gt;Cipher&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;new&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s1&quot;&gt;&apos;AES-128-ECB&apos;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;o&quot;&gt;=&amp;gt;&lt;/span&gt; &lt;span class=&quot;c1&quot;&gt;#&amp;lt;OpenSSL::Cipher:0x007f8182658618&amp;gt;&lt;/span&gt;
&lt;span class=&quot;o&quot;&gt;&amp;gt;&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;c&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;decrypt&lt;/span&gt;
&lt;span class=&quot;o&quot;&gt;=&amp;gt;&lt;/span&gt; &lt;span class=&quot;c1&quot;&gt;#&amp;lt;OpenSSL::Cipher:0x007f8182658618&amp;gt;&lt;/span&gt;
&lt;span class=&quot;o&quot;&gt;&amp;gt;&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;c&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;key&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;&apos;M02cnQ51Ji97vwT4&apos;&lt;/span&gt;
&lt;span class=&quot;o&quot;&gt;=&amp;gt;&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;M02cnQ51Ji97vwT4&quot;&lt;/span&gt;
&lt;span class=&quot;o&quot;&gt;&amp;gt;&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;o&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;&apos;&apos;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;force_encoding&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s1&quot;&gt;&apos;ASCII-8BIT&apos;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;o&quot;&gt;=&amp;gt;&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;&quot;&lt;/span&gt;
&lt;span class=&quot;o&quot;&gt;&amp;gt;&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;data&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;bytes&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;each_slice&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;16&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;|&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;s&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;|&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;o&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;c&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;update&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;s&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;map&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;&amp;amp;&lt;/span&gt;&lt;span class=&quot;ss&quot;&gt;:chr&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;).&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;join&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)}&lt;/span&gt;
&lt;span class=&quot;o&quot;&gt;=&amp;gt;&lt;/span&gt; &lt;span class=&quot;kp&quot;&gt;nil&lt;/span&gt;
&lt;span class=&quot;o&quot;&gt;&amp;gt;&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;o&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;c&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;final&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt; &lt;span class=&quot;kp&quot;&gt;nil&lt;/span&gt;
&lt;span class=&quot;o&quot;&gt;=&amp;gt;&lt;/span&gt; &lt;span class=&quot;kp&quot;&gt;nil&lt;/span&gt;
&lt;span class=&quot;o&quot;&gt;&amp;gt;&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;o&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;...&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;60&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;
&lt;span class=&quot;o&quot;&gt;=&amp;gt;&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\xFF\xD8\xFF\xE0\0\x10&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;JFIF&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\0\x01\x01\0\0\x01\0\x01\0\0\xFF\xDB\0&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;C&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\0\x14\x0E&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;\
x0F&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\x12\x0F\r\x14\x12\x10\x12\x17\x15\x14\x18\x1E&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;2!&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\x1E\x1C\x1C\x1E&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;=,.$2I@LKG@FE
PZ&quot;&lt;/span&gt;
&lt;span class=&quot;o&quot;&gt;&amp;gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;

&lt;p&gt;JFIF! Hello! We got our man.&lt;/p&gt;

&lt;p&gt;&lt;img style=&quot;border: 1px solid #000; margin: 0px auto; display: block;&quot; title=&quot;Thanks, @lapscallion.&quot; src=&quot;/assets/post-img/snapchat.jpg&quot; /&gt;&lt;/p&gt;

&lt;p&gt;That’s as far as I got; it was at this stage that I thought of Googling the
encryption key, to see if anyone else had tried this.
&lt;a href=&quot;http://adamcaudill.com/2012/12/31/revisiting-snapchat-api-and-security/&quot;&gt;Seems&lt;/a&gt;
&lt;a href=&quot;https://github.com/tlack/snaphax&quot;&gt;they&lt;/a&gt;
&lt;a href=&quot;https://gist.github.com/NeilHanlon/4686779&quot;&gt;had&lt;/a&gt;.  Still, I’m glad it wasn’t
until here that I searched.&lt;/p&gt;

&lt;p&gt;Many thanks to the &lt;a href=&quot;http://www.matasano.com/articles/crypto-challenges/&quot;&gt;Matasano crypto
challenges&lt;/a&gt; – some of the
above came from knowledge picked up doing them.&lt;/p&gt;

&lt;p&gt;The conclusion is that it’s easy to intercept and decrypt the data Snapchat on
your phone receives; it’d be one, maybe two hours of work to turn the above
code into something I could just switch on and forget about, while it happily
archives every Snapchat I ever receive.&lt;/p&gt;

&lt;p&gt;Truth be told, I can’t be bothered – it’s fun, and I don’t want to ruin the
unique feeling it has by virtue of being an ephemeral medium – but don’t think
it’s hard for someone who cared enough to.&lt;/p&gt;

&lt;div class=&quot;footnotes&quot; role=&quot;doc-endnotes&quot;&gt;
  &lt;ol&gt;
    &lt;li id=&quot;fn:flurry&quot; role=&quot;doc-endnote&quot;&gt;
      &lt;p&gt;To &lt;a href=&quot;http://www.flurry.com/&quot;&gt;Flurry&lt;/a&gt;, a mobile analytics company, containing data like my phone model, screen res, probably some unique ID … &lt;a href=&quot;#fnref:flurry&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
    &lt;li id=&quot;fn:ssl-ca&quot; role=&quot;doc-endnote&quot;&gt;
      &lt;p&gt;I used &lt;a href=&quot;http://www.freebsdmadeeasy.com/tutorials/freebsd/create-a-ca-with-openssl.php&quot;&gt;this guide&lt;/a&gt;, but there are better ones out there which are probably more explanatory. &lt;a href=&quot;#fnref:ssl-ca&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
    &lt;li id=&quot;fn:http&quot; role=&quot;doc-endnote&quot;&gt;
      &lt;p&gt;Note that this client doesn’t speak HTTP; accordingly, it doesn’t know when to close the connection (GAE’s HTTP server sends the responses with &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Content-Length&lt;/code&gt;, so we should parse that and know when we’ve received all the data, but it’s easier just to restart the server over and over at this stage). &lt;a href=&quot;#fnref:http&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
  &lt;/ol&gt;
&lt;/div&gt;
</content>
    
 </entry>
 
 <entry>
   <title>Escapology: how, when and why to encode and escape</title>
   <updated>2012-04-18T21:10:00+10:00</updated>
   <id>https://kivikakk.ee/2012/04/18/escapology</id>
    
     <link href="https://kivikakk.ee/2012/04/18/escapology/"/>
     <content type="html">&lt;p&gt;As programmers, we spend a lot of time just carting data from one place to
another.  Sometimes that’s the entire purpose of a program or library (data
conversion whatevers), but more often it’s just something that needs to happen
in the course of getting a certain task done.  When we’re sending a request,
using a library, executing templates or whatever, it’s important to be 100%
clear on the format of the data, which is a fancy way of saying how the data is
encoded.&lt;/p&gt;

&lt;p&gt;Let’s do the tacky dictionary thing:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;&lt;strong&gt;&lt;a href=&quot;http://en.wiktionary.org/wiki/encoding&quot;&gt;encoding&lt;/a&gt;&lt;/strong&gt; (&lt;em&gt;plural&lt;/em&gt; encodings)&lt;/p&gt;

  &lt;ol&gt;
    &lt;li&gt;
      &lt;p&gt;(computing) The way in which symbols are mapped onto bytes, e.g. in the
rendering of a particular font, or in the mapping from keyboard input into
visual text.&lt;/p&gt;
    &lt;/li&gt;
    &lt;li&gt;
      &lt;p&gt;A conversion of plain text into a code or cypher form (for decoding by the
recipient).&lt;/p&gt;
    &lt;/li&gt;
  &lt;/ol&gt;
&lt;/blockquote&gt;

&lt;p&gt;&lt;a id=&quot;more&quot;&gt;&lt;/a&gt;I think these senses are a bit too specific—if your data is in a computer in
any form, then it’s already encoded.  The keyboard doesn’t even have to come
into it.&lt;/p&gt;

&lt;!--more--&gt;

&lt;hr /&gt;

&lt;p&gt;If you’re like me and you come from an English-speaking country, there’s a good
chance that this might seem farfetched, or totally obvious but lacking in
depth.  The letter &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;A&lt;/code&gt; is represented in ASCII by the integer 65, or hex &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;41&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;From hereon, if I refer to a number with regular formatting, it’s decimal
unless specified otherwise; likewise with &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;code&lt;/code&gt; formatting, it’s hexadecimal.&lt;/p&gt;

&lt;p&gt;You are also probably aware that non-Latin characters like &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;恋&lt;/code&gt; do not have any
mapping in ASCII, that people all tried to make their own ways to get around
this—none of which interoperated particularly well—and that at some stage,
a bunch of smart people decided to create Unicode, which assigns a unique
integer codepoint to every character of every language (and then some), such
that the character just mentioned is &lt;a id=&quot;604b&quot;&gt;&lt;/a&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;U+604b&lt;/code&gt;, and that there are character
encodings, like UTF-8, which are used to represent the codepoints in a
bytestream, such that &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;恋&lt;/code&gt; becomes &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;e6 81 8b&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;This is all well and good.  But what do you do with this stuff in your program?&lt;/p&gt;

&lt;p&gt;Firstly, we need to straighten out what your environment does, or doesn’t do,
with character encodings.  I’m going to use PHP, Erlang and HTML as my
examples, because they’re things I work with at work, and they each have
slightly different ways of dealing with encoding &lt;sup id=&quot;fnref:php-silly&quot; role=&quot;doc-noteref&quot;&gt;&lt;a href=&quot;#fn:php-silly&quot; class=&quot;footnote&quot; rel=&quot;footnote&quot;&gt;1&lt;/a&gt;&lt;/sup&gt; owing to their
internal representation of strings.&lt;/p&gt;

&lt;p&gt;Secondly, I’m going to expand this beyond character encodings to &lt;strong&gt;any
encoding&lt;/strong&gt;—which is ultimately what I want to talk about here.  We’re not
just encoding the textual content for decoding into codepoints; we’re also
often encoding data to put it within other data in a demarcated way.  In this
case, we tend to refer to &lt;strong&gt;escaping&lt;/strong&gt;, but escaping and encoding are different
ways of talking about the same &lt;em&gt;process&lt;/em&gt;.&lt;/p&gt;

&lt;h2 id=&quot;php&quot;&gt;PHP&lt;/h2&gt;

&lt;p&gt;The best way to describe PHP’s character encoding is with the words “not at
all”.  Strings do not have metadata associated with encoding; to all string
manipulation functions, a string might as well be an array of bytes,
representing the raw bytes from the disk that occured between two ASCII &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;&quot;&lt;/code&gt;
(&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;22&lt;/code&gt;) characters.&lt;/p&gt;

&lt;p&gt;In other words, PHP treats the input PHP file as byte soup—nominally ASCII.&lt;/p&gt;

&lt;p&gt;Say I want to store &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;恋は戦争&lt;/code&gt; &lt;sup id=&quot;fnref:love-is-war&quot; role=&quot;doc-noteref&quot;&gt;&lt;a href=&quot;#fn:love-is-war&quot; class=&quot;footnote&quot; rel=&quot;footnote&quot;&gt;2&lt;/a&gt;&lt;/sup&gt; in a string in PHP.  I boot up my
editor, open a new file called &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;koi.php&lt;/code&gt; and enter:&lt;/p&gt;

&lt;figure class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-php&quot; data-lang=&quot;php&quot;&gt;&lt;span class=&quot;cp&quot;&gt;&amp;lt;?&lt;/span&gt;

&lt;span class=&quot;nv&quot;&gt;$koi&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;恋は戦争&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt;

&lt;span class=&quot;k&quot;&gt;echo&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;$koi&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt;

&lt;span class=&quot;cp&quot;&gt;?&amp;gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;

&lt;p&gt;What do I get?&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/post-img/koihasensou-utf8.png&quot; alt=&quot;The text &amp;quot;恋は戦争&amp;quot; displays correctly in my
browser.&quot; /&gt;&lt;/p&gt;

&lt;p&gt;Hey, the text displays correctly!  PHP must be super-smart and it’s doing
everything right!  Right?&lt;/p&gt;

&lt;p&gt;Maybe.  My text-editor decided to save the file in UTF-8 by default.  If I was
in Japan and I dealt mostly with Japanese, it could be that I tended to save
files in some popular non-Unicode encodings, like
&lt;a href=&quot;http://en.wikipedia.org/wiki/Shift_JIS&quot;&gt;Shift JIS&lt;/a&gt; or
&lt;a href=&quot;http://en.wikipedia.org/wiki/ISO-2022-JP&quot;&gt;ISO-2022-JP&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;What happens if I resave the file in Shift JIS?&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/post-img/koihasensou-sjis.png&quot; alt=&quot;Non-descript characters appear instead.&quot; /&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;&lt;a href=&quot;http://en.wikipedia.org/wiki/Mojibake&quot;&gt;Mojibake!&lt;/a&gt;&lt;/em&gt;  Character encoding issues
are so common, Japanese has a word for it.&lt;/p&gt;

&lt;p&gt;What happened?  PHP actually has no idea about what the text is, encoding,
whatever.  When it looks at the file, it sees this:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;0000000: 3c3f 0a0a 246b 6f69 203d 2022 97f6 82cd  &amp;lt;?..$koi = &quot;....
0000010: 90ed 9188 223b 0a0a 6563 686f 2024 6b6f  ....&quot;;..echo $ko
0000020: 693b 0a0a 3f3e 0a                        i;..?&amp;gt;.
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Note that we have a &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;22&lt;/code&gt; at byte &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;0b&lt;/code&gt;, indicating the start of a string, and
then another &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;22&lt;/code&gt; at byte &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;14&lt;/code&gt;.  I’m contending that the bytes between—i.e.
&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;97 f6 82 cd 90 ed 91 88&lt;/code&gt;—are what gets stored in the string, without any
further knowledge.&lt;/p&gt;

&lt;p&gt;If this were the case, then &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;strlen($koi)&lt;/code&gt; should be equal to &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;8&lt;/code&gt; (despite
there being &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;4&lt;/code&gt; Japanese characters).  I modify &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;koi.php&lt;/code&gt;, still in Shift JIS:&lt;/p&gt;

&lt;figure class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-php&quot; data-lang=&quot;php&quot;&gt;&lt;span class=&quot;cp&quot;&gt;&amp;lt;?&lt;/span&gt;

&lt;span class=&quot;nv&quot;&gt;$koi&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;恋は戦争&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt;

&lt;span class=&quot;k&quot;&gt;echo&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;strlen&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;$koi&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;mf&quot;&gt;.&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;&amp;lt;br&amp;gt;&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\n&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;echo&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;$koi&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt;

&lt;span class=&quot;cp&quot;&gt;?&amp;gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;

&lt;p&gt;And now?&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/post-img/koihasensou-sjislen.png&quot; alt=&quot;The number 8 prefaces the mojibake.&quot; /&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;&lt;a href=&quot;http://patrickmccoy.typepad.com/lost_in_translation/2006/02/yappari_thats_r_1.html&quot;&gt;Yappari&lt;/a&gt;&lt;/em&gt;.
PHP has &lt;em&gt;no clue&lt;/em&gt; what encoding the string is in—it’s just saving those
bytes, counting them, and throwing them back out.  So why does the UTF-8 one
look alright in the browser and the Shift JIS one doesn’t?&lt;/p&gt;

&lt;p&gt;The first tripping point is the webserver; the Apache on the machine I’m
testing has this directive in a conffile:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;AddDefaultCharset UTF-8
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Sure enough, when we take a look at the headers sent on the wire:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;HTTP/1.1 200 OK
Date: Wed, 18 Apr 2012 04:17:51 GMT
Server: Apache/2.2.11 (Fedora)
X-Powered-By: PHP/5.2.9
Content-Length: 14
Connection: close
Content-Type: text/html; charset=UTF-8
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;So the browser is trying to read the Shift JIS as UTF-8, hence mojibake.  We
can force that by adding an appropriate &lt;a href=&quot;http://php.net/header&quot;&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;header()&lt;/code&gt;&lt;/a&gt;
call, but it goes to show that PHP isn’t cognisant of what’s going on here.&lt;/p&gt;

&lt;p&gt;Another grand example is using other built-in string functions.
&lt;a href=&quot;http://php.net/str_replace&quot;&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;str_replace()&lt;/code&gt;&lt;/a&gt; and
&lt;a href=&quot;http://php.net/substr&quot;&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;substr()&lt;/code&gt;&lt;/a&gt; are fraught with difficulty, no matter what
encoding you use.&lt;/p&gt;

&lt;p&gt;We haven’t even hit the fun stuff yet.  What happens if we use another popular
Japanese encoding, ISO-2022-JP?  ISO-2022, also known as ECMA-35, is a standard
for mechanisms for encoding foreign language text, and is used for Japanese in
ISO-2022-JP, Chinese in ISO-2022-CN, and more, including extensions of the
same.&lt;/p&gt;

&lt;p&gt;Being a more complicated system, it fails to make some guarantees about the
encoded data which other encodings do make; Shift JIS, for instance, &lt;a href=&quot;http://en.wikipedia.org/wiki/Shift_JIS#Shift_JIS_byte_map&quot;&gt;does not use common ASCII special characters in its second byte&lt;/a&gt;,
such as &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;$&lt;/code&gt;.  This means a &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;$&lt;/code&gt; symbol in Shift JIS-encoded text always means a
&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;$&lt;/code&gt;, whereas the letter &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;A&lt;/code&gt; could occur as either a literal &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;A&lt;/code&gt;, or as the
second byte in a double-byte character &lt;sup id=&quot;fnref:mb-encoding&quot; role=&quot;doc-noteref&quot;&gt;&lt;a href=&quot;#fn:mb-encoding&quot; class=&quot;footnote&quot; rel=&quot;footnote&quot;&gt;3&lt;/a&gt;&lt;/sup&gt;.&lt;/p&gt;

&lt;p&gt;Keeping in mind that ISO-2022-JP doesn’t exercise such care, let’s see what we
get …&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/post-img/koihasensou-iso2022jp.png&quot; alt=&quot;We get a few &amp;quot;undefined variable&amp;quot; PHP
warnings.&quot; /&gt;&lt;/p&gt;

&lt;p&gt;Uhhhh.  What’s stored on the disk?&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;0000000: 3c3f 0a0a 246b 6f69 203d 2022 1b24 424e  &amp;lt;?..$koi = &quot;.$BN
0000010: 7824 4f40 6f41 681b 2842 223b 0a0a 6563  x$O@oAh.(B&quot;;..ec
0000020: 686f 2073 7472 6c65 6e28 246b 6f69 2920  ho strlen($koi) 
0000030: 2e20 223c 6272 3e5c 6e22 3b0a 6563 686f  . &quot;&amp;lt;br&amp;gt;\n&quot;;.echo
0000040: 2024 6b6f 693b 0a0a 3f3e 0a               $koi;..?&amp;gt;.
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;The ISO-2022-JP encoding of &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;恋は戦争&lt;/code&gt; contains the byte-sequence &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;24 42 4e 78
24 4f&lt;/code&gt;, which is &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;$BNx$O&lt;/code&gt; in ASCII—so PHP tries to interpolate variables so
named.&lt;/p&gt;

&lt;p&gt;Of course, this means even scarier things are possible.  I replaced &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;恋は戦争&lt;/code&gt;
with &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;あ&lt;/code&gt;—the Japanese letter “a”, essentially—and we get:&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/post-img/a-iso2022jp.png&quot; alt=&quot;A warning and a parse error.&quot; /&gt;&lt;/p&gt;

&lt;p&gt;Whoops!  On the disk:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;0000000: 3c3f 0a0a 246b 6f69 203d 2022 1b24 4224  &amp;lt;?..$koi = &quot;.$B$
0000010: 221b 2842 223b 0a0a 6563 686f 2073 7472  &quot;.(B&quot;;..echo str
0000020: 6c65 6e28 246b 6f69 2920 2e20 223c 6272  len($koi) . &quot;&amp;lt;br
0000030: 3e5c 6e22 3b0a 6563 686f 2024 6b6f 693b  &amp;gt;\n&quot;;.echo $koi;
0000040: 0a0a 3f3e 0a                             ..?&amp;gt;.
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;あ&lt;/code&gt; encodes as &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;1b 24 42 24 22 1b 28 42&lt;/code&gt;.  Those playing at home will notice a
&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;22&lt;/code&gt;—i.e. &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;&quot;&lt;/code&gt;—is stuck in the middle, so PHP thinks you’ve prematurely
terminated the string.&lt;/p&gt;

&lt;p&gt;The solution is to encode your source files as UTF-8, because UTF-8 guarantees
that all ASCII characters—that is, values &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;00&lt;/code&gt; through &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;7f&lt;/code&gt;—are both mapped
to the same byte in UTF-8, &lt;em&gt;and&lt;/em&gt; that those bytes will not occur in a UTF-8
stream as part of any other character.  UTF-8 is &lt;a href=&quot;http://en.wikipedia.org/wiki/UTF-8#Description&quot;&gt;marvelously
well-designed&lt;/a&gt; (actually).&lt;/p&gt;

&lt;p&gt;This means that PHP won’t do anything funny with your strings, though it still
treats it like a bag of bytes.  Next, use only the &lt;a href=&quot;http://php.net/manual/en/book.mbstring.php&quot;&gt;multibyte string extension&lt;/a&gt;
to do string operations.  For instance, let’s revert back to the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;恋は戦争&lt;/code&gt;
example in UTF-8, and use &lt;a href=&quot;http://php.net/mb_strlen&quot;&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;mb_strlen()&lt;/code&gt;&lt;/a&gt; instead of
the plain variety:&lt;/p&gt;

&lt;figure class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-php&quot; data-lang=&quot;php&quot;&gt;&lt;span class=&quot;cp&quot;&gt;&amp;lt;?&lt;/span&gt;

&lt;span class=&quot;nv&quot;&gt;$koi&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;恋は戦争&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt;

&lt;span class=&quot;k&quot;&gt;echo&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;mb_strlen&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;$koi&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;mf&quot;&gt;.&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;&amp;lt;br&amp;gt;&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\n&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;echo&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;$koi&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt;

&lt;span class=&quot;cp&quot;&gt;?&amp;gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;

&lt;p&gt;And we see:&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/post-img/koihasensou-utf8len.png&quot; alt=&quot;12!?&quot; /&gt;&lt;/p&gt;

&lt;p&gt;Whoops.  That’s raw bytes again!  The multibyte module has no idea what
encoding to use, so if we don’t tell it, it behaves as usefully as
&lt;a href=&quot;http://php.net/strlen&quot;&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;strlen()&lt;/code&gt;&lt;/a&gt;. We have to tell it, and in a very PHP-like
manner, we set a &lt;strong&gt;global&lt;/strong&gt; state for the interpreter.  Great.&lt;/p&gt;

&lt;figure class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-php&quot; data-lang=&quot;php&quot;&gt;&lt;span class=&quot;cp&quot;&gt;&amp;lt;?&lt;/span&gt;

&lt;span class=&quot;nb&quot;&gt;mb_internal_encoding&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&quot;UTF-8&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;);&lt;/span&gt;

&lt;span class=&quot;nv&quot;&gt;$koi&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;恋は戦争&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt;

&lt;span class=&quot;k&quot;&gt;echo&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;mb_strlen&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;$koi&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;mf&quot;&gt;.&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;&amp;lt;br&amp;gt;&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\n&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;echo&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;$koi&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt;

&lt;span class=&quot;cp&quot;&gt;?&amp;gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;

&lt;p&gt;And finally:&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/post-img/koihasensou-utf8lenmb.png&quot; alt=&quot;The number 4. Hallelujah.&quot; /&gt;&lt;/p&gt;

&lt;p&gt;Onto saner pastures.&lt;/p&gt;

&lt;h2 id=&quot;erlang&quot;&gt;Erlang&lt;/h2&gt;

&lt;p&gt;Erlang is a funny language.  &lt;a href=&quot;http://learnyousomeerlang.com/starting-out-for-real#lists&quot;&gt;It doesn’t even &lt;em&gt;have&lt;/em&gt;
strings&lt;/a&gt;&lt;sup id=&quot;fnref:inconvenient-truth&quot; role=&quot;doc-noteref&quot;&gt;&lt;a href=&quot;#fn:inconvenient-truth&quot; class=&quot;footnote&quot; rel=&quot;footnote&quot;&gt;4&lt;/a&gt;&lt;/sup&gt;.
Instead, a string is a list of numbers; the REPL guesses that a list should be
pretty-printed as a string if they all look like printable ASCII:&lt;/p&gt;

&lt;figure class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-erlang&quot; data-lang=&quot;erlang&quot;&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;&amp;gt;&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;&quot;Hello.&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;
&lt;span class=&quot;s&quot;&gt;&quot;Hello.&quot;&lt;/span&gt;
&lt;span class=&quot;mi&quot;&gt;2&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;&amp;gt;&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;72&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;101&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;108&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;108&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;111&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;46&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;].&lt;/span&gt;
&lt;span class=&quot;s&quot;&gt;&quot;Hello.&quot;&lt;/span&gt;
&lt;span class=&quot;mi&quot;&gt;3&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;&amp;gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;

&lt;p&gt;lol!  Gnarly!  But seriously, this accidentally becomes great when you start
Unicoding it up like you’re part of the UN:&lt;/p&gt;

&lt;figure class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-erlang&quot; data-lang=&quot;erlang&quot;&gt;&lt;span class=&quot;mi&quot;&gt;3&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;&amp;gt;&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;S&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;&quot;恋は戦争&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;24651&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;12399&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;25126&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;20105&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;
&lt;span class=&quot;mi&quot;&gt;4&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;&amp;gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;

&lt;p&gt;Huh!?  Well, they’re not printable ASCII, so what &lt;em&gt;are&lt;/em&gt; they?  Let’s translate
those pesky decimals into something humans can read:&lt;/p&gt;

&lt;figure class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-erlang&quot; data-lang=&quot;erlang&quot;&gt;&lt;span class=&quot;mi&quot;&gt;4&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;&amp;gt;&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;nn&quot;&gt;io&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;format&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;si&quot;&gt;~4.16.0b&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;, &quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;N&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;])&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;||&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;N&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;&amp;lt;-&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;S&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;],&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;ok&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;
&lt;span class=&quot;mi&quot;&gt;604&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;b&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;306&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;f&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;6226&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;4&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;e89&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;ok&lt;/span&gt;
&lt;span class=&quot;mi&quot;&gt;5&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;&amp;gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;

&lt;p&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;604b&lt;/code&gt;?  &lt;a href=&quot;#604b&quot;&gt;Doesn’t that sound familiar?&lt;/a&gt;.  It’s actually interpreting
each Unicode &lt;em&gt;character&lt;/em&gt; (not each &lt;em&gt;byte&lt;/em&gt;) as its integer codepoint.  That
means all lovely things we want to assume hold true, like length calculation
and substrings:&lt;/p&gt;

&lt;figure class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-erlang&quot; data-lang=&quot;erlang&quot;&gt;&lt;span class=&quot;mi&quot;&gt;5&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;&amp;gt;&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;length&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;S&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;).&lt;/span&gt;
&lt;span class=&quot;mi&quot;&gt;4&lt;/span&gt;
&lt;span class=&quot;mi&quot;&gt;6&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;&amp;gt;&lt;/span&gt; &lt;span class=&quot;nn&quot;&gt;io&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;format&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;~ts&lt;/span&gt;&lt;span class=&quot;si&quot;&gt;~n&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;nn&quot;&gt;lists&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;sublist&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;S&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;3&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)]).&lt;/span&gt;
&lt;span class=&quot;err&quot;&gt;戦争&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;ok&lt;/span&gt;
&lt;span class=&quot;mi&quot;&gt;7&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;&amp;gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;

&lt;p&gt;So Erlang provides a pretty good native data-type for storing Unicode
characters; pre-Unicode, we were just storing the numbers of ASCII characters
in a list, so now we just store the numbers of Unicode codepoints instead.&lt;/p&gt;

&lt;p&gt;Unfortunately, the simplicity of &lt;em&gt;entry&lt;/em&gt; does not extend to source files.
D’oh!  The Erlang manual specifies that source must be entered in
&lt;a href=&quot;http://en.wikipedia.org/wiki/ISO/IEC_8859-1&quot;&gt;ISO-8859-1&lt;/a&gt;, also known as
“Latin-1” encoding.  Only the REPL is smart enough to do Unicode.  This is
detailed in &lt;a href=&quot;http://www.erlang.org/doc/apps/stdlib/unicode_usage.html#id62505&quot;&gt;the Erlang manual&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;So you could consider that it’s difficult to get Unicode data &lt;em&gt;into&lt;/em&gt;
Erlang—in that you can’t enter it directly into the source—but once you
have it, it’s much more straightforward to manange than with, say, PHP.  The
reality is, most Unicode data in your program will be coming from &lt;em&gt;without&lt;/em&gt;
your program, not within—i.e. user input, API call results, etc.—so this
isn’t as bad as it sounds.&lt;/p&gt;

&lt;p&gt;Erlang doesn’t store metadata about the encoding; it avoids the problem
entirely by letting strings represent Unicode natively.  Once you start sending
or receiving them on the wire, you’ll usually want convert them to or from
binary strings with functions from the
&lt;a href=&quot;http://www.erlang.org/doc/man/unicode.html&quot;&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;unicode&lt;/code&gt;&lt;/a&gt; module, which provides
helpers for various UTF encodings, and Latin-1.  Once so-converted, the data is
unambiguously opaque … compared with PHP’s “I don’t have a clue &lt;em&gt;what&lt;/em&gt; it is”.&lt;/p&gt;

&lt;h2 id=&quot;html&quot;&gt;HTML&lt;/h2&gt;

&lt;p&gt;This is a huge kludge.&lt;/p&gt;

&lt;p&gt;HTML itself contains a way to declare its own encoding, using a &lt;a href=&quot;http://en.wikipedia.org/wiki/Meta_element&quot;&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;&amp;lt;meta&amp;gt;&lt;/code&gt; tag&lt;/a&gt;
to declare the &lt;a href=&quot;http://en.wikipedia.org/wiki/MIME#Content-Type&quot;&gt;Content-Type&lt;/a&gt;
of the document.  The issue, if you weren’t paying attention, is that reading
the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;&amp;lt;meta&amp;gt;&lt;/code&gt; tag implies you are able to make any sense of the document
whatsoever.  Valid HTML requires the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;&amp;lt;meta&amp;gt;&lt;/code&gt; to appear within the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;&amp;lt;head&amp;gt;&lt;/code&gt;,
i.e. for HTML 5:&lt;/p&gt;

&lt;figure class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-html&quot; data-lang=&quot;html&quot;&gt;&lt;span class=&quot;cp&quot;&gt;&amp;lt;!doctype html&amp;gt;&lt;/span&gt;
&lt;span class=&quot;nt&quot;&gt;&amp;lt;html&amp;gt;&lt;/span&gt;
&lt;span class=&quot;nt&quot;&gt;&amp;lt;head&amp;gt;&lt;/span&gt;
  &lt;span class=&quot;nt&quot;&gt;&amp;lt;meta&lt;/span&gt; &lt;span class=&quot;na&quot;&gt;charset=&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;utf-8&quot;&lt;/span&gt;&lt;span class=&quot;nt&quot;&gt;&amp;gt;&lt;/span&gt;
  ...&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;

&lt;p&gt;This isn’t a problem with sane encodings, because they tend to map ASCII
through; but things like UTF-16 and above require the server to declare the
content-type in the HTTP headers; interpreting
&lt;a href=&quot;http://en.wikipedia.org/wiki/UTF-32&quot;&gt;UTF-32&lt;/a&gt; as ASCII leads to madness.&lt;/p&gt;

&lt;p&gt;For comparison’s sake, the layout on disk of the above in UTF-8 (identical to
ASCII here):&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;00000000: 3c21 646f 6374 7970 6520 6874 6d6c 3e0a  &amp;lt;!doctype html&amp;gt;.
00000010: 3c68 746d 6c3e 0a3c 6865 6164 3e0a 2020  &amp;lt;html&amp;gt;.&amp;lt;head&amp;gt;.  
00000020: 3c6d 6574 6120 6368 6172 7365 743d 2275  &amp;lt;meta charset=&quot;u
00000030: 7466 2d38 223e 0a                        tf-8&quot;&amp;gt;.  ....
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;And in UTF-32:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;00000000: 0000 003c 0000 0021 0000 0064 0000 006f  ...&amp;lt;...!...d...o
00000010: 0000 0063 0000 0074 0000 0079 0000 0070  ...c...t...y...p
00000020: 0000 0065 0000 0020 0000 0068 0000 0074  ...e... ...h...t
00000030: 0000 006d 0000 006c 0000 003e 0000 000a  ...m...l...&amp;gt;....
00000040: 0000 003c 0000 0068 0000 0074 0000 006d  ...&amp;lt;...h...t...m
00000050: 0000 006c 0000 003e 0000 000a 0000 003c  ...l...&amp;gt;.......&amp;lt;
00000060: 0000 0068 0000 0065 0000 0061 0000 0064  ...h...e...a...d
00000070: 0000 003e 0000 000a 0000 0020 0000 0020  ...&amp;gt;....... ... 
00000080: 0000 003c 0000 006d 0000 0065 0000 0074  ...&amp;lt;...m...e...t
...
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;(it goes on like this)&lt;/p&gt;

&lt;p&gt;Triples of &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;NUL&lt;/code&gt;-bytes separating everything!  This is UTF-32’s grand plan to
support everything without variable-width encoding, meaning operations like
string length, slicing and substring matching could be done fairly
cheaply&lt;sup id=&quot;fnref:utf-32-clusterfuck&quot; role=&quot;doc-noteref&quot;&gt;&lt;a href=&quot;#fn:utf-32-clusterfuck&quot; class=&quot;footnote&quot; rel=&quot;footnote&quot;&gt;5&lt;/a&gt;&lt;/sup&gt;.&lt;/p&gt;

&lt;p&gt;In these cases you should be ensuring the server sends the correct
&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Content-Type&lt;/code&gt;, implying the server has a clue—and if you’re lucky, enough
users’ browsers will guess.&lt;/p&gt;

&lt;p&gt;This could be considered a non-solution.&lt;/p&gt;

&lt;p&gt;The other thing HTML brings to the table, via SGML, is character entity
references, giving you the ability to &lt;a href=&quot;http://en.wikipedia.org/wiki/List_of_XML_and_HTML_character_entity_references&quot;&gt;refer to any Unicode character by
codepoint&lt;/a&gt;.
This doesn’t make for happy editing, but it does mean that HTML can &lt;em&gt;represent&lt;/em&gt;
arbitrary Unicode characters even when the HTML itself is ASCII, for instance.&lt;/p&gt;

&lt;h2 id=&quot;representing-unicode-characters&quot;&gt;Representing Unicode characters&lt;/h2&gt;

&lt;p&gt;This is an important concept.  In HTML, one can enter:&lt;/p&gt;

&lt;figure class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-html&quot; data-lang=&quot;html&quot;&gt;&lt;span class=&quot;ni&quot;&gt;&amp;amp;#x604b;&amp;amp;#x306f;&amp;amp;#x6226;&amp;amp;#x4e89;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;

&lt;p&gt;And get this in their browser:&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/post-img/koihasensou-utf8.png&quot; alt=&quot;The text &amp;quot;恋は戦争&amp;quot;.&quot; /&gt;&lt;!-- Yes, I&apos;m reusing this
image. --&gt;&lt;/p&gt;

&lt;p&gt;This is without any other markup or declaration of encoding; we’re telling the
browser to render the characters by Unicode codepoint directly.  This is
equivalent to entering them into a list in Erlang source directly:&lt;/p&gt;

&lt;figure class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-erlang&quot; data-lang=&quot;erlang&quot;&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;16#604b&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;16#306f&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;16#6226&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;16#4e89&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;

&lt;p&gt;It’s important to make clear a distinction here, however: HTML is &lt;em&gt;markup&lt;/em&gt;, and
Erlang is a &lt;em&gt;programming language&lt;/em&gt;.  HTML gives you an escape route to render a
given codepoint, which is good, but what we’re talking about when we talk about
Erlang is actually an &lt;strong&gt;internal representation&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;This is what I’m more interested in, so I’ll put HTML to the side for now.&lt;/p&gt;

&lt;p&gt;When data enters your Erlang program, it’s most likely going to be encoded in
some form; whichever service receives that data is responsible for making sense
of it.  Imagine you’re writing a webserver: people might submit forms in UTF-8,
UTF-16, Shift JIS, Latin-1, whatever.  No matter what you’re doing with that
data—you might be spitting it right back in the response; hacking it up into
pieces; storing in a database, maybe for later hacking—you need to normalise
the format of the data &lt;em&gt;while you still know what format it’s in&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;If you’re coding in PHP and you receive a string full of bits, if you throw
that into a database without noting the encoding
&lt;sup id=&quot;fnref:whose-encoding-is-it-anyway&quot; role=&quot;doc-noteref&quot;&gt;&lt;a href=&quot;#fn:whose-encoding-is-it-anyway&quot; class=&quot;footnote&quot; rel=&quot;footnote&quot;&gt;6&lt;/a&gt;&lt;/sup&gt;, you’ve permanently lost the ability to say for
sure what the data actually is.&lt;/p&gt;

&lt;p&gt;The solution, then, is to normalise the data at the point of entry, once, and
to normalise it into an &lt;em&gt;internal format&lt;/em&gt; that makes sense.  In Erlang, you
might store a string as a codepoint list.  In PHP, you’ve little option but to
normalise it to another encoding like UTF-8, and to decide that UTF-8 &lt;em&gt;is&lt;/em&gt; the
internal format for textual data.&lt;/p&gt;

&lt;h2 id=&quot;check-your-blindspots&quot;&gt;Check your blindspots&lt;/h2&gt;

&lt;p&gt;Do &lt;strong&gt;not&lt;/strong&gt; fall into the trap of saying “well, I know my users will only enter
Latin-1 data, which happens to be the default, so I’ll just save that and print
that.”  Guess what?  That’s what most of Japan said when they decided to use
Shift JIS.  Except for those who used ISO-2022-JP.  Or EUC-JP.  Good luck to
most of these people when people from other countries start submitting data.&lt;/p&gt;

&lt;p&gt;If you won’t listen to me, listen to the experience of the top 25 Japanese
websites according to Alexa.  &lt;a href=&quot;http://www.webmasterworld.com/printerfriendlyv5.cgi?forum=32&amp;amp;discussion=3373673&amp;amp;serial=3376486&amp;amp;user=&quot;&gt;A forum post from
mid-2007&lt;/a&gt;
detailed 10 of them using UTF-8.  When I check the other 15, 8 have moved to
UTF-8, leaving 7 not using UTF-8.  3 of those are on Shift JIS because they’re
either tailored for the Japanese mobile market—which endemically tends to
only support Shift JIS (Japanese mobile phones are not like your mobile phones)
—or because they make heavy use of &lt;a href=&quot;http://en.wikipedia.org/wiki/Shift_JIS_art&quot;&gt;Shift_JIS
art&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Storing data authentically is something that, as programmers, we need to get
used to, and clobbering data that doesn’t conform to your expectation makes no
sense.&lt;/p&gt;

&lt;p&gt;When data comes in, store it in a normal form.  If you’re building a webservice
in a sane language or framework, it’s probable that the environment has done
this work for you.&lt;/p&gt;

&lt;h2 id=&quot;dont-escape-data&quot;&gt;Don’t escape data&lt;/h2&gt;

&lt;p&gt;The key is to strongly mark the boundaries of keeping data in one format and
another.  Strongly-typed languages can distinguish this at compile time and
stop mistakes, but that’s a rant for another day.&lt;/p&gt;

&lt;p&gt;I’m going to start talking about another encoding process, often referred to as
escaping.&lt;/p&gt;

&lt;p&gt;Pop-quiz: what’s wrong with this PHP?&lt;/p&gt;

&lt;figure class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-php&quot; data-lang=&quot;php&quot;&gt;&lt;span class=&quot;cp&quot;&gt;&amp;lt;?&lt;/span&gt;

&lt;span class=&quot;c1&quot;&gt;// Omitted: init.&lt;/span&gt;

&lt;span class=&quot;nv&quot;&gt;$username&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;$_POST&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&quot;username&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;];&lt;/span&gt;
&lt;span class=&quot;nv&quot;&gt;$password&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;$_POST&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&quot;password&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;];&lt;/span&gt;
&lt;span class=&quot;nb&quot;&gt;mysql_query&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&quot;INSERT INTO tblUsers VALUES (&apos;&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;$username&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&apos;, &apos;&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;$password&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&apos;)&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;);&lt;/span&gt;

&lt;span class=&quot;cp&quot;&gt;?&amp;gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;

&lt;p&gt;A related question is “what’s &lt;em&gt;not&lt;/em&gt; wrong with this PHP?”.  I can think of nine
separate issues with it &lt;sup id=&quot;fnref:php-issue-register&quot; role=&quot;doc-noteref&quot;&gt;&lt;a href=&quot;#fn:php-issue-register&quot; class=&quot;footnote&quot; rel=&quot;footnote&quot;&gt;7&lt;/a&gt;&lt;/sup&gt;, but let’s look at the obvious
one: &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;$username&lt;/code&gt; and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;$password&lt;/code&gt; are inserted into the query unescaped.  Maybe.&lt;/p&gt;

&lt;p&gt;Unless you live under a rock, you’ll recognise the &lt;a href=&quot;http://en.wikipedia.org/wiki/SQL_injection&quot;&gt;SQL injection attack&lt;/a&gt;,
also known as &lt;strong&gt;SQLi&lt;/strong&gt;.  The issue is because we’re substituting
&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;$username&lt;/code&gt;—which we should think of as user data—directly into the stream
of a MySQL command.  By doing so, we’re essentially saying that the user data
&lt;em&gt;is&lt;/em&gt; part of a MySQL command—because it is.  A user could enter &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;&quot;&apos;); DROP TABLE tblUsers; --&lt;/code&gt;,
and because we failed to encode user data as user data (which really means
escaping here), it happens.&lt;/p&gt;

&lt;p&gt;So how do we “encode” data?  The correct way is to let the layer of abstraction
handle that for you.  If you use &lt;a href=&quot;http://php.net/manual/en/book.pdo.php&quot;&gt;PDO&lt;/a&gt;,
and put the laundry list of issues with the above code aside, it looks like
this:&lt;/p&gt;

&lt;figure class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-php&quot; data-lang=&quot;php&quot;&gt;&lt;span class=&quot;cp&quot;&gt;&amp;lt;?&lt;/span&gt;

&lt;span class=&quot;c1&quot;&gt;// Omitted: init, database connection in $db.&lt;/span&gt;

&lt;span class=&quot;nv&quot;&gt;$stmt&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;$db&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;-&amp;gt;&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;prepare&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s1&quot;&gt;&apos;INSERT INTO tblUsers VALUES (:username, :password)&apos;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;);&lt;/span&gt;

&lt;span class=&quot;nv&quot;&gt;$stmt&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;-&amp;gt;&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;execute&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;k&quot;&gt;array&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;
	&lt;span class=&quot;s1&quot;&gt;&apos;:username&apos;&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&amp;gt;&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;$_POST&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&quot;username&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;],&lt;/span&gt;
	&lt;span class=&quot;s1&quot;&gt;&apos;:password&apos;&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&amp;gt;&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;$_POST&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&quot;password&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;],&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;));&lt;/span&gt;

&lt;span class=&quot;cp&quot;&gt;?&amp;gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;

&lt;p&gt;We let PDO deal with constructing the query—being the most-informed part of
the system to deal with query parameter encoding.  If we do it ourselves, we’ll
probably make a mistake.&lt;/p&gt;

&lt;p&gt;Note that &lt;strong&gt;we are &lt;em&gt;not&lt;/em&gt; escaping the data&lt;/strong&gt;—we’re deciding that another part
of the system which knows how to do it should do it instead.  Knowing when
&lt;strong&gt;not&lt;/strong&gt; to escape is often just as important as knowing when &lt;strong&gt;to&lt;/strong&gt; escape.&lt;/p&gt;

&lt;h3 id=&quot;caveat&quot;&gt;Caveat&lt;/h3&gt;

&lt;p&gt;Q. When will the above go terribly wrong?&lt;/p&gt;

&lt;p&gt;A. &lt;a href=&quot;http://web.archive.org/web/20171002204939/http://php.net/manual/en/security.magicquotes.php&quot;&gt;When a deficient language tries to automate security&lt;/a&gt;,
and finds that’s actually not possible according to the definition of security.&lt;/p&gt;

&lt;p&gt;Hello, your database is now full of backslashes.  Unfortunately, magic quotes
is enabled at the level of the webserver, so if your host has it enabled, you
have to try to turn it off.  &lt;a href=&quot;http://web.archive.org/web/20210101040748/https://www.php.net/manual/en/security.magicquotes.disabling.php&quot;&gt;Look at that Example #2.  Mmmmmm.&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Do &lt;strong&gt;not&lt;/strong&gt; rely on broken auto-escaping.  Your code will become unportable (and
insecure) if you end up hosting the same stuff elsewhere where this process
does not take place.  It’s tantamount to assuming all data will come in encoded
in UTF-16 and being surprised when your application breaks at inopportune moments.&lt;/p&gt;

&lt;p&gt;(Edit 2021-02-18: I’ve needed to put in archive links here, as this has been
thoroughly sweeped under the rug; deprecated in PHP 5.3 and removed in PHP 5.4.
Keep in mind you can still find PHP 4 webhosting if you search for it.)&lt;/p&gt;

&lt;h2 id=&quot;the-golden-rule&quot;&gt;The golden rule&lt;/h2&gt;

&lt;p&gt;So, what’s the guiding principle?&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Data in your application should be &lt;em&gt;semantically pure&lt;/em&gt;&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;If this doesn’t make any sense to you, read it once more, and I’ll explain.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Data in your application should be &lt;em&gt;semantically pure&lt;/em&gt;&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Let’s say you take a username as input in a web application.  When the client
makes the request and sends us this username, we receive it into a variable.
What does the variable contain, right from the start?  Is it the exact text
they entered?  Does it have quotes escaped?  Are characters like &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;&amp;lt;&lt;/code&gt; and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;&amp;gt;&lt;/code&gt;
converted to HTML entities so we can output it back into the page if we need
to?&lt;/p&gt;

&lt;p&gt;No.  The variable contains the username.  &lt;em&gt;Nothing&lt;/em&gt; else.&lt;/p&gt;

&lt;p&gt;It is not escaped.  It has no special encoding &lt;strong&gt;other&lt;/strong&gt; than the normalisation
about textual data encoding we’ve already discussed.  It is in &lt;em&gt;no way&lt;/em&gt; prepped
for output, because it is the &lt;em&gt;pure&lt;/em&gt; username.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;When&lt;/em&gt; we want to output it, we pass it to the presentation layer &lt;em&gt;pure&lt;/em&gt;, and
let the presentation layer apply as many post-processing layers of encoding as
is required for the context.  It will probably convert special characters to
entities if they could cause issues.  If it’s being put in a link URI it will
be URI encoded.  If it’s ending up in some server-side generated JavaScript it
may need to be escaped appropriately.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;When&lt;/em&gt; we want to do a match against the database, we pass it to the database
layer &lt;em&gt;pure&lt;/em&gt;, and let the database layer perform the correct escaping for the
context.&lt;/p&gt;

&lt;p&gt;Let’s say you take a user’s age as input in a web application.  You may have a
mind for UX, so you check that the field appears to contain a number.  Now,
before you start passing that variable around everywhere, what do you do?
&lt;strong&gt;You represent it internally as a number&lt;/strong&gt;.  You call
&lt;a href=&quot;http://php.net/intval&quot;&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;intval&lt;/code&gt;&lt;/a&gt; or &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;list_to_integer/1&lt;/code&gt; and you &lt;strong&gt;let the
presentation layer decide what to do with a number&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Otherwise you’re going to leave yourself open for all sorts of trouble.&lt;/p&gt;

&lt;p&gt;The key is that the value is &lt;em&gt;semantically&lt;/em&gt; pure—it carries the meaning of
the value in the programming language as it does in your mind.  No extra
backslashes in your mind?  Make sure the application sees that too.  This is
why we add structs, classes, new types and so forth; to better model the
semantics of the values in the programming language.&lt;/p&gt;

&lt;h2 id=&quot;mvc&quot;&gt;MVC&lt;/h2&gt;

&lt;p&gt;The model is the purest part of the application.  You ask it for some value,
and it gives it to you, completely unadorned.  It’s the controller’s
responsibility to request data of the model, and to hand all requisite data so
gathered to the view; it’s ultimately the view’s responsibility to perform
encoding of data according to context.&lt;/p&gt;

&lt;p&gt;Similarly, when the controller receives request data, it should parse it into
semantically meaningful values, which are then passed back to the view and into
models as appropriate.&lt;/p&gt;

&lt;p&gt;If the view then, say, uses some of this data in a query string value, it URI
encodes it.  If it’s including the data on the page directly, it converts HTML
entities to avoid &lt;a href=&quot;http://en.wikipedia.org/wiki/Cross-site_scripting&quot;&gt;XSS&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;And so on.&lt;/p&gt;

&lt;h2 id=&quot;escape-data-unescape-data&quot;&gt;Escape data, unescape data&lt;/h2&gt;

&lt;p&gt;I hope this entry might have shown you some of the parallels between
character-set encoding and escaping; they are both forms of processing data
into different formats, and it’s nearly always a mistake to not know if this
form of processing has occurred yet on a given piece of data, anywhere in an
application.&lt;/p&gt;

&lt;p&gt;Repeating the process (double-escaping or double-encoding) &lt;em&gt;without intending
to&lt;/em&gt; means you’re actually talking about those bytes that represent the encoded
data &lt;em&gt;in&lt;/em&gt; the encoded data.  Without being sure of the state of your inputs,
this can happen easily, and the next time you type &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;恋は戦争&lt;/code&gt;, you end up
seeing &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;æ&amp;lt;81&amp;gt;&amp;lt;8b&amp;gt;ã&amp;lt;81&amp;gt;¯æ&amp;lt;88&amp;gt;¦äº&amp;lt;89&amp;gt;&lt;/code&gt;—which is what happens if you interpret
the UTF-8 bytes of the string as a Latin-1 sequence&lt;sup id=&quot;fnref:encodeception&quot; role=&quot;doc-noteref&quot;&gt;&lt;a href=&quot;#fn:encodeception&quot; class=&quot;footnote&quot; rel=&quot;footnote&quot;&gt;8&lt;/a&gt;&lt;/sup&gt;.&lt;/p&gt;

&lt;p&gt;Here are some other actions analogous to encoding.&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Shell escaping.&lt;/li&gt;
  &lt;li&gt;Wrapping in the jQuery object&lt;sup id=&quot;fnref:jquery-idempotency&quot; role=&quot;doc-noteref&quot;&gt;&lt;a href=&quot;#fn:jquery-idempotency&quot; class=&quot;footnote&quot; rel=&quot;footnote&quot;&gt;9&lt;/a&gt;&lt;/sup&gt;.&lt;/li&gt;
  &lt;li&gt;Quoting people in conversation.&lt;/li&gt;
  &lt;li&gt;Editor’s comments in a quote in a newspaper.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Ultimately, it’s a matter of being certain about the &lt;em&gt;type&lt;/em&gt; of data you’re
handling, whereby type I mean anything relevant to parsing its semantic
content.  Both dynamically- and statically-typed languages are amenable to
annotating objects with metadata concerning the operations that have been
carried out with them.&lt;/p&gt;

&lt;p&gt;I’m also trying to point out that this is not restricted to programming
languages—it’s whenever you have different categories of things being spoken
of, or different levels of abstraction.&lt;/p&gt;

&lt;p&gt;Golang’s &lt;a href=&quot;http://golang.org/pkg/html/template/&quot;&gt;html/template package&lt;/a&gt;,
effectively performing the role of the view in MVC, does automatic encoding of
data that has come from the controller, depending on the context.  This is a
nifty feature, as it allows you to forget about escaping—so long as you are
passing it semantically pure data, of course.&lt;/p&gt;

&lt;p&gt;If you have the template &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;&amp;lt;p&amp;gt;{{.}}&amp;lt;/p&amp;gt;&lt;/code&gt;, then data to be substituted at
dot will have HTML entities inserted automatically, preventing XSS attacks.
Similarly, with the template fragment 
&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;&amp;lt;a href=&quot;/?action={{.}}&quot;&amp;gt;{{.}}&amp;lt;/a&amp;gt;&lt;/code&gt; dot’s content will be URI
encoded in the first instance and have HTML entities inserted in the second.&lt;/p&gt;

&lt;p&gt;As the package documentation explains:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;This package assumes that template authors are trusted, that Execute’s data
parameter is not, and seeks to preserve the properties below in the face of
untrusted data:&lt;/p&gt;

  &lt;p&gt;Structure Preservation Property: “… when a template author writes an HTML
tag in a safe templating language, the browser will interpret the
corresponding portion of the output as a tag regardless of the values of
untrusted data, and similarly for other structures such as attribute
boundaries and JS and CSS string boundaries.”&lt;/p&gt;

  &lt;p&gt;Code Effect Property: “… only code specified by the template author should
run as a result of injecting the template output into a page and all code
specified by the template author should run as a result of the same.”&lt;/p&gt;

  &lt;p&gt;Least Surprise Property: “A developer (or code reviewer) familiar with HTML,
CSS, and JavaScript, who knows that contextual autoescaping happens should be
able to look at a {{.}} and correctly infer what sanitization happens.”&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;This provides a lot of reassurances that we want—but then if we &lt;em&gt;do&lt;/em&gt; have
some data in the controller that we &lt;em&gt;really&lt;/em&gt; want to be substituted in as HTML,
what’s a cute programmer such as yourself to do?&lt;/p&gt;

&lt;p&gt;The same package provides the
&lt;a href=&quot;http://golang.org/pkg/html/template/#HTML&quot;&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;HTML&lt;/code&gt; type&lt;/a&gt;
—actually, a synonym for &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;string&lt;/code&gt;.  In Golang, type synonyms are different
types with respect to method sets—we can’t use a method of one on another
(methods are not inherited)—and there is no implicit conversion between them.
Objects of different types (even where the types are synonyms) are completely
different, except that we can
&lt;a href=&quot;http://golang.org/ref/spec#Conversions&quot;&gt;convert between them&lt;/a&gt; as they have the
same &lt;a href=&quot;http://golang.org/ref/spec#Types&quot;&gt;underlying type&lt;/a&gt;.  This means that we
can take a &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;string&lt;/code&gt; and turn it into a &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;HTML&lt;/code&gt; with a simple conversion:&lt;/p&gt;

&lt;figure class=&quot;highlight&quot;&gt;&lt;pre&gt;&lt;code class=&quot;language-go&quot; data-lang=&quot;go&quot;&gt;&lt;span class=&quot;c&quot;&gt;// x is of type string&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;x&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;:=&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;&quot;&amp;lt;p&amp;gt;This is delicious!&amp;lt;/p&amp;gt;&quot;&lt;/span&gt;

&lt;span class=&quot;c&quot;&gt;// y is of type HTML&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;y&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;:=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;HTML&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;x&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;

&lt;p&gt;Passing an object of type HTML to &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;html/template&lt;/code&gt; tells it that the escaping
required to sanitise HTML has already been done, and so in a context where such
escaping is necessary, the content will be included verbatim.  Of course, this
also implies you don’t let user input find its way into a &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;HTML&lt;/code&gt;-typed object
without doing necessary encoding/normalisation yourself.  I refrain from using the word “sanitisation”, as the term has plenty of bad connotations which I’m about to talk about.&lt;/p&gt;

&lt;p&gt;There are a range of other types for indicating that you’ve taken on the
responsibility of validating the variable contents are safe for use in given
contexts, i.e. giving the developer an escape for getting some data in which
you know you don’t want re-encoded.  The important thing is that you’re
categorically stating that it’s so—no part of the system will assume it
otherwise.  It’s safe by default, but in a way that doesn’t compare to “magic
quotes”, because it’s happening at the view, right before output, not at data
entry, contaminating your entire application.&lt;/p&gt;

&lt;h2 id=&quot;for-the-love-of-all-that-is-good&quot;&gt;For the love of all that is good&lt;/h2&gt;

&lt;p&gt;&lt;a href=&quot;http://coding.smashingmagazine.com/2011/01/11/keeping-web-users-safe-by-sanitizing-input-data/&quot;&gt;Don’t do this&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;This is an anti-pattern.  This is &lt;strong&gt;the&lt;/strong&gt; anti-pattern.  The author’s very
first suggestion is to tell you to set the server-wide configuration for PHP to
automatically do certain escaping on some variables.  The idea is that you can
now pretend that all your variables are ready to become a part of HTML
somewhere!&lt;/p&gt;

&lt;p&gt;Of course, if you insert these (safely) into the database, you’ve just inserted
HTML-sanitised data into the database.  Your database is perfectly capable of
storing the text &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;kivi &quot;owl&quot; kakk&lt;/code&gt;, but now you’ve got a row with the value
&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;kivi &amp;amp;quot;owl&amp;amp;quot; kakk&lt;/code&gt;.  If you still think this is a good idea, you
should leave the classroom right now.&lt;/p&gt;

&lt;p&gt;You read this out of the database.  Maybe you want to know how many characters
long it is.  &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;strlen()&lt;/code&gt;?  26.  Never mind that the actual displayed string is
16 when shown in HTML.  Maybe you want to use this value in an API call to some
other webservice.  You wrap it in a few objects, maybe encode it to JSON.  Now
you have to remember to &lt;em&gt;decode&lt;/em&gt; the entities before you use it in non-HTML
contexts.  Though if you want to put it into HTML to be part of a URI, you have
to remember to decode it, then re-encode it with URI (component) encoding.  You
lose.&lt;/p&gt;

&lt;p&gt;If you treat data semantically to begin with, and only encode it as appropriate
for the output at the &lt;em&gt;time&lt;/em&gt; of output, we don’t have an issue.  Input
filtering is just a way to make sure you’ll never really know what’s in a
variable.&lt;/p&gt;

&lt;h2 id=&quot;conclusion&quot;&gt;Conclusion&lt;/h2&gt;

&lt;p&gt;Data in your application should &lt;em&gt;mean what it means&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;When data comes in, interpret its &lt;em&gt;meaning&lt;/em&gt; once, according to context.&lt;/p&gt;

&lt;p&gt;When data goes out, encode it &lt;em&gt;meaningfully&lt;/em&gt; according to context.&lt;/p&gt;

&lt;p&gt;This applies to charsets, escaping and more.&lt;/p&gt;

&lt;h2 id=&quot;postscript-php-strings&quot;&gt;Postscript: PHP strings&lt;/h2&gt;

&lt;p&gt;It’s interesting to note here that there is neither a PHP type which represents
a string with a given encoding—
&lt;a href=&quot;http://www.php.net/manual/en/language.types.string.php#language.types.string.details&quot;&gt;the PHP string is a byte-buffer&lt;/a&gt;
and no more—nor a suitable sequence-like container for arbitrary codepoints;
you could follow the same approach that Erlang takes and store codepoints as
integers within arrays, but due to PHP’s impressive array
type&lt;sup id=&quot;fnref:array-what-array&quot; role=&quot;doc-noteref&quot;&gt;&lt;a href=&quot;#fn:array-what-array&quot; class=&quot;footnote&quot; rel=&quot;footnote&quot;&gt;10&lt;/a&gt;&lt;/sup&gt;, it would be incredibly inefficient.&lt;/p&gt;

&lt;p&gt;I’m also not finding any way to take an arbitrary list of codepoints (or a
single codepoint) and return it in a given encoding; the data is always assumed
to have come in as a string in some encoding.  Of course, you could take your
codepoints and convert them to UTF-8 yourself, and &lt;em&gt;then&lt;/em&gt; treat that with the
multibyte string module.  &lt;a href=&quot;http://stackoverflow.com/questions/1805802/php-convert-unicode-codepoint-to-utf-8&quot;&gt;This appears to not be
atypical&lt;/a&gt;.
Oh, PHP.&lt;/p&gt;

&lt;div class=&quot;footnotes&quot; role=&quot;doc-endnotes&quot;&gt;
  &lt;ol&gt;
    &lt;li id=&quot;fn:php-silly&quot; role=&quot;doc-endnote&quot;&gt;

      &lt;p&gt;Hint: the way PHP does it is silly.  See the part on “Text” in &lt;a href=&quot;http://me.veekun.com/blog/2012/04/09/php-a-fractal-of-bad-design/&quot;&gt;Eevee’s
blog post about
PHP&lt;/a&gt;. &lt;a href=&quot;#fnref:php-silly&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
    &lt;li id=&quot;fn:love-is-war&quot; role=&quot;doc-endnote&quot;&gt;

      &lt;p&gt;Japanese &lt;em&gt;koi wa sensou&lt;/em&gt;, meaning &lt;a href=&quot;http://supercell.sc/koisen/&quot;&gt;“love is war”&lt;/a&gt;. &lt;a href=&quot;#fnref:love-is-war&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
    &lt;li id=&quot;fn:mb-encoding&quot; role=&quot;doc-endnote&quot;&gt;

      &lt;p&gt;All encodings mentioned so far, except ASCII, are examples of
&lt;a href=&quot;http://en.wikipedia.org/wiki/Variable-width_encoding&quot;&gt;variable-width
encodings&lt;/a&gt;, or
&lt;em&gt;multibyte encodings&lt;/em&gt;. &lt;a href=&quot;#fnref:mb-encoding&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
    &lt;li id=&quot;fn:inconvenient-truth&quot; role=&quot;doc-endnote&quot;&gt;

      &lt;p&gt;Well, there are binaries, but &lt;strong&gt;sshhhh&lt;/strong&gt;, I’m in the middle of pointmaking. &lt;a href=&quot;#fnref:inconvenient-truth&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
    &lt;li id=&quot;fn:utf-32-clusterfuck&quot; role=&quot;doc-endnote&quot;&gt;

      &lt;p&gt;Of course, they shot themselves in all thirty-two of their feet when
English-language text in UTF-32 resembles UTF-16 so much that browsers
trying to auto-detect encoding when the server doesn’t declare it &lt;a href=&quot;http://en.wikipedia.org/wiki/UTF-32#Non-use_in_HTML5&quot;&gt;detect
it as UTF-16&lt;/a&gt;, not to
mention endianness problems also found in UTF-16 and &lt;a href=&quot;http://en.wikipedia.org/wiki/Byte_order_mark&quot;&gt;the terrible kludges
to work around
them&lt;/a&gt;.&lt;/p&gt;

      &lt;p&gt;Given UTF-16 doesn’t solve any problem (well), and creates many, I find
myself wondering why they bothered. &lt;a href=&quot;#fnref:utf-32-clusterfuck&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
    &lt;li id=&quot;fn:whose-encoding-is-it-anyway&quot; role=&quot;doc-endnote&quot;&gt;

      &lt;p&gt;And how do you know the encoding?  You don’t.  The browser encodes it in
the &lt;em&gt;same encoding as it determined the page to be&lt;/em&gt;.  Think about what that
means for a bit. &lt;a href=&quot;#fnref:whose-encoding-is-it-anyway&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
    &lt;li id=&quot;fn:php-issue-register&quot; role=&quot;doc-endnote&quot;&gt;

      &lt;ol&gt;
        &lt;li&gt;may need to check &lt;a href=&quot;http://php.net/get_magic_quotes_gpc&quot;&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;get_magic_quotes_gpc()&lt;/code&gt;&lt;/a&gt; to know if data is encoded.&lt;/li&gt;
        &lt;li&gt;password stored unhashed.&lt;/li&gt;
        &lt;li&gt;we don’t specify a connection object.&lt;/li&gt;
        &lt;li&gt;we’re using an antique MySQL library.&lt;/li&gt;
        &lt;li&gt;inputs are unescaped (maybe—see point 1).&lt;/li&gt;
        &lt;li&gt;escape inputs when you could use placeholders? Doing so successfully means ensuring data is &lt;em&gt;not&lt;/em&gt; escaped by now; see point 1.&lt;/li&gt;
        &lt;li&gt;specifying columns; inserted columns &lt;em&gt;may&lt;/em&gt; cause failure, or just data going in the wrong places.&lt;/li&gt;
        &lt;li&gt;use placeholders when you can use an ORM?&lt;/li&gt;
        &lt;li&gt;checking return value; because who cares about data integrity?&lt;/li&gt;
      &lt;/ol&gt;

      &lt;p&gt;This list itself is an example of poorly encoded data.  Note how it’s
inconsistent as to whether the items mentioned are the issues, or the fixes. &lt;a href=&quot;#fnref:php-issue-register&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
    &lt;li id=&quot;fn:encodeception&quot; role=&quot;doc-endnote&quot;&gt;

      &lt;p&gt;Of course, note that these characters are of course embedded in &lt;em&gt;this&lt;/em&gt; document with UTF-8. &lt;a href=&quot;#fnref:encodeception&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
    &lt;li id=&quot;fn:jquery-idempotency&quot; role=&quot;doc-endnote&quot;&gt;

      &lt;p&gt;jQuery is special in that it’s never a mistake to rewrap in jQuery; the
jQuery wrapper function is idempotent.  This is because you can
unambiguously distinguish between an wrapped object and a not-wrapped
object; note that the same does not hold of arbitrary strings without
metadata attached to the string describing what encoding it should be.
Ruby’s strings, for instance, have no issues, as they carry knowledge of
their encoding.&lt;/p&gt;

      &lt;p&gt;A common practice is to prefix a variable name with a &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;$&lt;/code&gt; if it’s known to
be wrapped in jQuery—because you’ve guaranteed it at the point of naming
the variable, typically.  This is a way of making the assertion about the
encoding obvious. Because you can’t accidentally overwrap an object in
JavaScript, usually an absence of &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;$&lt;/code&gt; prefix implies no guarantee—it may
be wrapped or unwrapped, and the only way to guarantee e.g. a DOM object,
is by wrapping and then using &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;get&lt;/code&gt;.  The point is that you can never
mistakenly assume something to be jQuery wrapped when it isn’t; you’ll
always know. &lt;a href=&quot;#fnref:jquery-idempotency&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
    &lt;li id=&quot;fn:array-what-array&quot; role=&quot;doc-endnote&quot;&gt;

      &lt;p&gt;Hint: there is none.  I know where you can get a hash-table-y kinda thing,
though. &lt;a href=&quot;#fnref:array-what-array&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
  &lt;/ol&gt;
&lt;/div&gt;
</content>
    
 </entry>
 

</feed>