This month marks the pub­li­ca­tion of “The Actual Star,” a new, wildly am­bi­tious, wildly suc­cess­ful novel by Monica Byrne, span­ning 3,000 years of his­tory from the an­cient Maya to a dis­tant fu­ture.

The tale braids three mo­ments: the col­lapse of the Mayan royal dy­nasty in 1012, 2012, when a Maya girl raised in Minnesota re­turns to her fa­ther’s birth­place in Belize; and 3012, when the 8m hu­mans who sur­vived the cli­mate emer­gency in­habit a high-tech, no­madic civ­i­liza­tion.

All three are a mix of mys­ti­cism, blood, car­nal­ity, and a deep love of place and the Earth. They all tell the tale of schisms, when dif­fer­ent views of how to live a right life in har­mony with na­ture are con­tested with rage and vi­o­lence, and they all con­nect to the Maya faith.

Byrne’s deep his­tor­i­cal re­search into the Maya of 1012 and her imag­in­ing of a rad­i­cal new wan­der­ing so­ci­ety of 3012 are both pro­foundly, gor­geously for­eign, rem­i­nis­cent of Ada Palmer’s books (one of the few sf writ­ers who can con­jure a truly dif­fer­ent set of so­cial norms).

The Actual Star chal­lenges the lin­ear, ef­fi­ciency-dri­ven “rationality” that has brought our civ­i­liza­tion to the brink of col­lapse and asks us to imag­ine ut­terly new ways of un­der­stand­ing the world, even as it probes the flaws in­her­ent in any sys­tem of know­ing and be­ing.

And for all that, it’s an sf novel. It’s got a plot (three, ac­tu­ally), that’s in­tense and grip­ping, as mys­ti­cal sym­bols like an­cient haunted caves and god­like jaguars aren’t just sym­bols — they’re phys­i­cal things that char­ac­ters we care about have to cope with.

It’s a book about sac­ri­fice, about the long view and deep time, about the uni­ver­sal­ity of hu­man ex­pe­ri­ence and the par­tic­u­lar­ity of any given mo­ment. It’s a first-rate work of sf, and a hope­ful and fear­ful book about the cli­mate. It’s just great.

“Breaking In,” is my lat­est col­umn for Locus Magazine; it’s both the story of how I broke into sci­ence fic­tion, and an ex­pla­na­tion of why there’s so lit­tle to learn from that story.

When I was try­ing to sell my first sto­ries, I ob­ses­sively sought ca­reer ad­vice and mem­oirs from es­tab­lished writ­ers. I sat in on count­less sf con­ven­tion pan­els in which best­selling writ­ers ex­plained how they’d but­ter up long-dead ed­i­tors to sell to long-de­funct pub­li­ca­tions.

None of them ever men­tioned that as in­ter­est­ing as this stuff might be as an his­tor­i­cal ar­ti­fact, it had zero ap­plic­a­bil­ity to the mar­ket I was try­ing to break into.

Not only did these writ­ers en­ter a fun­da­men­tally dif­fer­ent — and long-ex­tinct pub­lish­ing world than the cur­rent one, but their re­la­tion­ship to the cur­rent mar­ket was fun­da­men­tally dif­fer­ent from my own.

Editors so­licited work from them, not the other way around. When they wrote some­thing on spec, they could di­rectly con­tact ed­i­tors with whom they’d had long and fruit­ful pro­fes­sional as­so­ci­a­tions — by­pass­ing the who “slush reader” ap­pa­ra­tus.

I don’t know if these es­tab­lished writ­ers failed to men­tion that none of this ap­plied to the would-be writ­ers in the au­di­ence be­cause they thought it was ob­vi­ous or be­cause it never oc­curred to them, but ei­ther way, it did­n’t do me a lick of good.

What worked for me? Well, that’s the point, is­n’t it? What worked for me won’t work for you. Not only was my path into the field pretty idio­syn­cratic — any gen­er­ally ap­plic­a­ble prin­ci­ple to be de­rived from it has been ob­so­lete for decades.

But some things don’t change. I ben­e­fited im­mensely from the kind­ness — some­times pro­tracted, some­times mo­men­tary — of writ­ers who spoke to youth groups, served as writ­ers-in-res­i­dence, guest-lec­tured to my sum­mer D&D camp.

Above all, I ben­e­fited from Judith Merril, a tow­er­ing writer, critic and ed­i­tor who went into vol­un­tary ex­ile in Toronto af­ter the Chicago po­lice ri­ots of 1968, and opened the Spaced Out Library, now the Merril Collection, the largest pub­lic sf ref­er­ence li­brary in the world.

Judy did­n’t just serve as writer-in-res­i­dence, read­ing my man­u­scripts when I took the sub­way down­town to give them to her. She also did writer-in-the-schools pro­grams, found­ing se­ri­ous writ­ers’ work­shops that en­dured for decades.

My high-school work­shop was one such; I kept at­tend­ing it for years af­ter I grad­u­ated (I was­n’t alone). Judy also steered the writ­ers she cri­tiqued into peer groups, like the still-thriv­ing Cecil Street Irregulars, which I joined in the early 1990s.

Other writ­ers were like­wise kind and gen­er­ous with their time. Tanya Huff worked be­hind the counter at Bakka book­store; she sold me the first sf novel I ever bought with my own money (H Beam Piper’s Little Fuzzy).

Tanya was im­mensely pa­tient with me, and even read man­u­scripts I shyly brought down to the store, giv­ing me en­cour­ag­ing — but un­flinch­ing — feed­back. When Tayna quit to write full time, I got her job in the store.

Ed Llewellyn and Ed Greenwood were guest speak­ers at the D&D sum­mer camp I at­tended. Both were in­cred­i­bly en­cour­ag­ing when I ap­proached them af­ter their talks to tell them I wanted to write.

Parke Godwin was guest of honor at the first con I ever vol­un­teered at; when I brought him his cof­fee, he pa­tiently lis­tened to me as I told him I wanted to write and took me se­ri­ously, telling me about the im­por­tance of good habits.

These writ­ers did­n’t have any ca­reer ad­vice for me per se, but I would­n’t have had a ca­reer with­out them — with­out them tak­ing me se­ri­ously, even at a very young age. I try to pay them for­ward, by en­cour­ag­ing the young writ­ers in my own path:

As to com­mer­cial ad­vice, there’s very lit­tle I can of­fer, I’m afraid. I like Heinlein’s ad­vice (“1. Write. 2. Finish. 3. Submit. 4. Revise to ed­i­to­r­ial spec.“).

I have a gen­eral method (“Find pub­li­ca­tions that fea­ture work like yours, re­search their sub­mis­sion process, send your story to the high­est-pay­ing ones first”).

As for spe­cific mar­ket ad­vice, that’s some­thing that you should get from peers, not the peo­ple who came be­fore you. When I was start­ing out, other would-be writ­ers and I ob­ses­sively shared notes on new mar­kets, ed­i­to­r­ial tastes, and other nuts-and-bolts.

Writers who are at the same place in their de­vel­op­ment as you have ad­vice that is far more likely to be ap­plic­a­ble to your sit­u­a­tion. What’s more, they’re also the kinds of writ­ers you should be seek­ing out to join in a cri­tiquing group — your peers.

The re­al­ity is that “breaking in” is a grind. It took me a decade from my first sub­mis­sion to my first pro­fes­sional pub­li­ca­tion; 19 years be­fore my first novel hit the shelves.

Perseverance is the great­est pre­dic­tor of suc­cess here, and sup­port from your peers is the best source of strength and re­siliency over that long road.

The Framework lap­top is the first lap­top to ever score a 10/10 from Ifixit for re­pairabil­ity. But it’s no thick-as-a-brick throw­back the size of a 2005 Thinkpad — it’s ap­prox­i­mately the same di­men­sions as a MacBook.

Mine was de­liv­ered at the end of Aug. I got it set up by the first of September and have been us­ing it ever since. Yesterday, I put my 2019 Thinkpad on my pile of “laptops to re­fur­bish and do­nate.” I’ve bought a new Thinkpad al­most every year since 2006. I think that’s over.

I switched to Thinkpads as part of my switch to Ubuntu, a fla­vor of GNU/Linux that was de­signed to be easy to use for laypeo­ple. My Unix sys­tems ad­min­is­tra­tion days were more than a decade be­hind me when I made the switch.

I loved Thinkpads…at first. Not only were they rugged as hell, but they had an in­cred­i­ble war­ranty. For about $150/year, IBM guar­an­teed that a ser­vice tech would come to your home or ho­tel room, any­where in the world, within 24 hours, and fix your ma­chine.

Prior to my Thinkpad switch, I’d been a Powerbook user and a pris­oner to Applecare. I made a prac­tice of buy­ing two Powerbooks at a time and keep­ing them in synch so that when one in­evitably broke down, I could leave it for weeks or months with Apple and use the other one.

I was a heavy trav­eller then (I was EFF’s European Director, on the road 27+ days/​month — I even stopped plug­ging in my fridge be­cause it was cost­ing me $10/month to keep my ice-cubes frozen), and a dead lap­top meant that I was beached, un­able to do any work.

I loved Macos, but the Powerbooks were re­ally shitty ma­chines, with in­cred­i­bly poor build qual­ity and a cap­tive re­pair chain that was run in a way that made it clear that its man­agers un­der­stood that its cus­tomers had no al­ter­na­tive.

Switching to Ubuntu was dis­ori­ent­ing…at first. It was a lot like the time we ren­o­vated our kitchen and moved every­thing around, and I spent a month reach­ing for a cut­lery drawer that was­n’t there. But then, one day, I just ac­cli­mated and never no­ticed it again.

So it was with OSes. If you’re notic­ing your OS, some­thing’s wrong. With Ubuntu, I got a GUI that was sim­i­lar enough to Macos that I could re­train my­self, and when things went wrong, I had ac­cess to an (admittedly es­o­teric but) in­cred­i­bly pow­er­ful suite of com­mand-line tools.

This turned out to be an ideal com­bi­na­tion. When every­thing worked, the UX was ef­fec­tively iden­ti­cal to my Macos days. When things went wrong with my hard­ware, I never had more than 24h down­time — even when some of my RAM went bad while I was in Mumbai!

And when soft­ware got wonky — some­thing that hap­pened with the same ap­prox­i­mate fre­quency as I ex­pe­ri­enced with Macos and when I was a CIO ad­min­is­ter­ing large het­ero­ge­neous net­works of Mac/Win sys­tems — the re­cov­ery tools were far su­pe­rior.

But it was­n’t to last. IBM sold its Thinkpad di­vi­sion to Lenovo and every­thing started to go to shit. The ac­tual sys­tems ac­quired lay­ers and lay­ers of pro­pri­etary crap — se­cre­tive Nvidia graph­ics cards, strange BIOS rub­bish — that made in­stalling Ubuntu pro­gres­sively harder.

The hard­ware got worse, too. When I lived in the UK, my Thinkpads al­ways shipped with a UK key­board. I’d or­der a US key­board for

By 2015, Thinkpads re­quired a full dis­as­sem­bly with mul­ti­ple spe­cial­ized tools and tape-re­moval to fix the key­boards. Also, the key­boards got worse — I had to have three key­board re­place­ments in 2015, and I could­n’t per­form any of them,

Things re­ally came to a head in 2019. That was the year I bought and re­turned two Thinkpads be­cause I could­n’t sta­bi­lize Ubuntu on them. The third, a gi­ant, heavy Carbon X1, took three months and sev­eral bug-fixes by Lenovo’s dri­ver team be­fore it worked.

Still, I was ready to buy an­other Thinkpad by last spring. What else was I go­ing to buy? I wanted some­thing main­tain­able, and I loved the hard­ware mouse-but­tons and the Trackpoint. But Lenovo was es­ti­mat­ing 4-5 months to ful­fill or­ders, so I closed the win­dow and bailed.

Then I saw Ifixit’s tear­down of a Framework lap­top. They de­scribed a com­puter whose hard­ware was fully user-main­tain­able/​up­grade­able. The sys­tem opens with six “captive” screws (they stay in the case) and then every com­po­nent can be eas­ily ac­cessed.

There’s no tape. There’s no glue. Every part has a QR code that you can shoot with your phone to go to a ser­vice man­ual that has sim­ple-to-fol­low in­struc­tions for in­stalling, re­mov­ing and re­plac­ing it. Every part is la­beled in English, too!

The screen is re­place­able. The key­board is re­place­able. The touch­pad is re­place­able. Removing the bat­tery and re­plac­ing it takes less than five min­utes. The com­puter ac­tu­ally ships with a screw­driver.

All this, with­out sac­ri­fic­ing size or power — it’s so sim­i­lar to a Macbook that a friend who came over for din­ner (and who knows about my feel­ings about pro­pri­etary Apple hard­ware) ex­pressed shock that I’d switched to a Macbook!

The com­puter per­forms as well or bet­ter than my 2019 Thinkpad, but it does­n’t need the Thinkpad’s pro­pri­etary, ~$200 dock — a cheap, $60 de­vice lets me eas­ily con­nect it to all my pe­riph­er­als and my desk­top mon­i­tor, over USB-C. No dri­vers or con­fig­u­ra­tion needed!

Installing Ubuntu was (nearly) pain­less. I had been loathe to up­grade the ver­sion of Ubuntu I was run­ning on the Thinkpad, lest I kick off an­other cas­cade of bru­tal, tier-2 bug-hunt­ing in the sys­tem’s pro­pri­etary dri­vers. As a re­sult, I ran the 2018 “Long Term Support” OS.

When I in­stalled Ubuntu on the Framework, I used the lat­est ver­sion — the Framework ships with a very up-to-date wifi card that the older ver­sion of Ubuntu could­n’t rec­og­nize. Then I sim­ply dumped all my files over from a backup drive.

Jumping three years’ worth of OSes in one go, mov­ing over my pref­er­ences and con­fig­u­ra­tion files from a Thinkpad, did not work per­fectly. A sin­gle track­pad con­fig file did­n’t play nice and I had to hunt it down and delete it, and then every­thing else was lit­er­ally flaw­less.

The hard­ware is also nearly flaw­less, though I do have a few mi­nor caveats. The com­puter ships dis­as­sem­bled: you have to open it and in­stall your RAM, SSD, and wifi card. The first two were easy — the third was a ma­jor pain in the ass.

The stan­dard wifi card an­tenna ca­bles are ab­surdly fid­dly, and the Framework doc­u­men­ta­tion was­n’t clear enough to see me through. However, when I tweeted to the com­pany about it, they re­sponded swiftly with a video that de­mys­ti­fied it.

Another caveat. I re­ally miss my Thinkpad Trackpoint (the lit­tle nub in the mid­dle of the key­board) and the three hard­ware mouse but­tons on the track­pad. I’m find­ing it re­ally hard to re­li­ably hit the right re­gion on my track­pad to get the left-, cen­ter- and mid­dle-but­tons.

I’ve drawn lit­tle hints on in sharpie, and I’m work­ing with Canonical, who make Ubuntu, on remap­ping the but­ton ar­eas. But judg­ing from the Framework fo­rums, I’m not the only Thinkpad ex­pat who’d like to swap the key­board and track­pad.

But the good news is that if any­one wants to make that key­board and track­pad, I can swap them in my­self, in min­utes, with one tool.

That tool — a small screw­driver — is also suf­fi­cient to up­grade the CPU or re­place the screen, speak­ers, we­b­cam, etc.

These are all just fine. The we­b­cam and mic both come with hard­ware off-switches (not just cov­ers, but ac­tual elec­tri­cal iso­la­tion switches that take them of­fline un­til you switch them back). The speak­ers are loud enough.

The screen is sharper than the one on my Thinkpad (though it’s glossier and a lit­tle harder to read in di­rect sun­light).

I haven’t even men­tioned the ports! The Framework has four ex­pan­sion ports that fit square don­gles for HDMI, Ethernet, var­i­ous USBs, etc.

The Framework site lets you buy as much or as lit­tle com­puter as you want. If you have your own RAM or SSD, you just uncheck those boxes. If you don’t bother with Windows (like me), you save $139-200.

Having used this sys­tem for nearly a month, I can un­equiv­o­cally rec­om­mend it! However! Most of my use of this com­puter was from my sofa, while I was re­cov­er­ing from hip-re­place­ment surgery. I haven’t road-tested it at all.

But I’ll note here that if it turned out that a com­po­nent failed due to my usual rough han­dling, I could re­place it with a stan­dard part in a mat­ter of min­utes, my­self, in what­ever ho­tel room I hap­pened to be perch­ing in, us­ing a sin­gle screw­driver.

It’s been a long time since I owned a com­puter that was more in­ter­est­ing with its case off than on, but the Framework is a mar­vel of thought­ful, sus­tain­able, user-cen­tric en­gi­neer­ing.

It puts the lie to every claim that porta­bil­ity and re­li­a­bil­ity can’t co­ex­ist with long-last­ing, durable, up­grade­able, sus­tain­able hard­ware.

I started buy­ing a new lap­top every year as a re­ward to my­self for quit­ting smok­ing.

The en­vi­ron­men­tal con­se­quences of that sys­tem weren’t lost on me, even given my very good track-record of re-hom­ing my old com­put­ers with peo­ple who needed them.

But with the Framework, I’m ready to change that pol­icy.

From now on, I can eas­ily see my­self up­grad­ing the CPU or the screen on an an­nual ba­sis, or pack­ing in more RAM. But the lap­top? Apart from the ac­tual chas­sis falling apart, there’s no rea­son I’d re­place it for the whole fore­see­able fu­ture.

This is a beau­ti­ful, func­tional, sus­tain­able, thought­ful and even lux­u­ri­ous (Framework of­fers a 2TB SDD, while Lenovo has been stuck at 1TB dri­ves for years and years) com­puter.

Based on a mon­th’s use, I am pre­pared to de­clare my­self a Framework loy­al­ist, and to re­tire my last Thinkpad…forever.

#15yrsago Bruce Sterling story: How kids’ lives will be ru­ined by Internet con­trol http://​www.chur­chofvirus.org/​bbs/​in­dex.php?board=6;ac­tion=dis­play;threa­did=36318

#10yrsago OnStar vows to track your move­ments for­ever, even if you can­cel the ser­vice https://​www.wired.com/​2011/​09/​on­star-tracks-you/

#5yrsago RCMP: Former Canadian mint worker smug­gled out $180K by hid­ing it up his butt https://​ot­tawac­i­t­i­zen.com/​news/​lo­cal-news/​egan-170k-in-mint-gold-al­legedly-smug­gled-in-body-cav­ity-judge-hears

#5yrsago What yes­ter­day’s hi­lar­i­ously aw­ful tes­ti­mony by Wells Fargo’s CEO por­tends for his fu­ture https://​www.naked­cap­i­tal­ism.com/​2016/​09/​wells-fargo-ceos-teflon-don-act-back­fires-at-sen­ate-hear­ing-i-take-full-re­spon­si­bil­ity-means-any­thing-but.html

#5yrsago Free trade low­ers prices — but not on things poor peo­ple need (and it pushes up hous­ing prices) http://​www.ddorn.net/​pa­pers/​Au­tor-Dorn-Han­son-Chi­naShock.pdf

#5yrsago Lickspittle con­sigliere: how the su­per-rich abuse their wealth man­agers as loy­alty tests https://​www.the­guardian.com/​busi­ness/​2016/​sep/​21/​how-to-hide-it-in­side-se­cret-world-of-wealth-man­agers

#1yrago Fincen (they fuck­ing knew all along) https://​plu­ral­is­tic.net/​2020/​09/​21/​too-big-to-jail/#​fin­cen

A non­fic­tion book about ex­ces­sive buyer-power in the arts, co-writ­ten with Rebecca Giblin, “The Shakedown.” FINAL EDITS

A post-GND utopian novel, “The Lost Cause.” FINISHED

* From Wayback to Way Forward: The Internet Archive turns 25, Oct 21

https://​www.eventbrite.com/​e/​from-way­back-to-way-for­ward-the-in­ter­net-archive-turns-25-vir­tual-tick­ets-163615196457

* “Attack Surface”: The third Little Brother novel, a stand­alone tech­nothriller for adults. The Washington Post called it “a po­lit­i­cal cy­berthriller, vig­or­ous, bold and savvy about the lim­its of rev­o­lu­tion and re­sis­tance.” Order signed, per­son­al­ized copies from Dark Delicacies https://​www.dark­del.com/​store/​p1840/​Avail­able_Now%3A_At­tack­_­Sur­face.html

“How to Destroy Surveillance Capitalism”: an anti-mo­nop­oly pam­phlet an­a­lyz­ing the true harms of sur­veil­lance cap­i­tal­ism and propos­ing a so­lu­tion. https://​onezero.medium.com/​how-to-de­stroy-sur­veil­lance-cap­i­tal­ism-8135e6744d59 (print edi­tion: https://​book­shop.org/​books/​how-to-de­stroy-sur­veil­lance-cap­i­tal­ism/​9781736205907) (signed copies: https://​www.dark­del.com/​store/​p2024/​Avail­able_Now%3A__How_­to_De­stroy_­Sur­veil­lance_­Cap­i­tal­ism.html)

“Little Brother/Homeland”: A reis­sue om­nibus edi­tion with a new in­tro­duc­tion by Edward Snowden: https://​us.macmil­lan.com/​books/​9781250774583; per­son­al­ized/​signed copies here: https://​www.dark­del.com/​store/​p1750/​July%3A__Lit­tle_Broth­er_%26_Home­land.html

“Poesy the Monster Slayer” a pic­ture book about mon­sters, bed­time, gen­der, and kick­ing ass. Order here: https://​us.macmil­lan.com/​books/​9781626723627. Get a per­son­al­ized, signed copy here: https://​www.dark­del.com/​store/​p1562/​_Poesy_the_­Mon­ster_S­layer.html.

This work li­censed un­der a Creative Commons Attribution 4.0 li­cense. That means you can use it any way you like, in­clud­ing com­mer­cially, pro­vided that you at­tribute it to me, Cory Doctorow, and in­clude a link to plu­ral­is­tic.net.

Quotations and im­ages are not in­cluded in this li­cense; they are in­cluded ei­ther un­der a lim­i­ta­tion or ex­cep­tion to copy­right, or on the ba­sis of a sep­a­rate li­cense. Please ex­er­cise cau­tion.

“When life gives you SARS, you make sar­sa­par­illa” -Joey “Accordion Guy” DeVilla

Crystalline quartz melts be­tween 1670 °C (tridymite) and 1713 °C (cristobalite), and be­cause quartz is per­va­sive and eas­ily iden­ti­fied, melted grains serve as an im­por­tant tem­per­a­ture in­di­ca­tor. At TeH, we ob­served that un­melted pot­sherds dis­played no melted quartz grains, in­di­cat­ing ex­po­sure to low tem­per­a­tures. On the other hand, most quartz grains on the sur­faces of pot­tery, mud­bricks, and roof­ing clay ex­hib­ited some de­gree of melt­ing, and un­melted quartz grains were rare. Nearly all quartz grains found on bro­ken, un­melted sur­faces of pot­sherds were also un­melted. On melted pot­tery and mud­bricks, melted quartz has an es­ti­mated den­sity of 1 grain per 5 mm2.

Melted quartz grains at TeH ex­hibit a wide range of mor­pholo­gies. Some show ev­i­dence of par­tial melt­ing that only melted grain edges and not the rest of the grain (Figs. 22, 23). Others dis­played nearly com­plete melt­ing with dif­fu­sion into the melted Ca–Al–Si ma­trix of pot­tery or mud­brick (Fig. 22). Melted quartz grains com­monly ex­hibit vesic­u­la­tion caused by out­gassing (Figs. 22, 23), sug­gest­ing that those grains rose above quartz’s melt­ing point of ~ 1713 °C.

An SEM–EDS el­e­men­tal map of one melted grain showed that the quartz had be­gun to dis­so­ci­ate into el­e­men­tal Si (Fig. 22b). Another grain (Fig. 23c–e) dis­played flow marks con­sis­tent with ex­po­sure to tem­per­a­tures above 1713 °C where the vis­cos­ity of quartz falls low enough for it to flow eas­ily. Another SEM–EDS analy­sis con­firmed that one ag­glu­ti­nated mass of ma­te­r­ial is 100 wt.% SiO (Fig. 23f, g), sug­gest­ing that this poly­crys­talline quartz grain shat­tered, melted, and par­tially fused again.

Moore et al.17 re­ported that dur­ing heat­ing ex­per­i­ments, many quartz grains  50-µm-wide re­mained vi­su­ally un­al­tered up to ~ 1700 °C. By 1850 °C, all quartz grains fully melted. These ex­per­i­ments es­tab­lish a par­ti­cle-size de­pen­dency and con­firm con­firmed the melt­ing point for > 50-µm-wide TeH quartz grains be­tween ~ 1700–1850 °C. Melted > 50-µm-wide quartz grains on the sur­faces of melted pot­tery and mud­brick from the TeH de­struc­tion layer in­di­cate ex­po­sure to these un­usu­ally high tem­per­a­tures > 1700 °C.

Previously, Thy et al.70 pro­posed that glass at Abu Hureyra did not form dur­ing a cos­mic im­pact, but rather, formed in bio­mass slag that re­sulted from thatched hut fires. However, Thy et al. did not de­ter­mine whether or not high-tem­per­a­ture grains ex­isted in the bio­mass slag. To test that claim, Moore et al.17 an­a­lyzed bio­mass slag from Africa and found only low-tem­per­a­ture melted grains with melt­ing points of ~ 1200 °C, con­sis­tent with a tem­per­a­ture range for bio­mass slag of 1155–1290 °C, as re­ported by Thy et al.71. Upon test­ing the pur­ported im­pact glass from Abu Hureyra, Moore et al.17 dis­cov­ered high-tem­per­a­ture min­eral grains that melt in the range of 1713° to > 2000 °C, as are also found in TeH glass. These test re­sults sug­gest that the melted glass from Abu Hureyra must have been ex­posed to higher tem­per­a­tures than those as­so­ci­ated with fires in thatched huts. Because of the pres­ence of high-tem­per­a­ture min­er­als at TeH, we con­clude that, as at Abu Hureyra, the melt­glass could not have formed sim­ply by burn­ing thatched huts or wood-roofed, mud­brick build­ings.

The pres­ence of melted spherulitic ob­jects (“spherules”) has com­monly been used to help iden­tify and in­ves­ti­gate high-tem­per­a­ture air­burst/​im­pact events in the sed­i­men­tary record. Although these ob­jects are re­ferred to here as “spherules,” they dis­play a wide range of other im­pact-re­lated mor­pholo­gies that in­clude rounded, sub-rounded, ovate, oblate, elon­gated, teardrop, dumb­bell, and/​or bro­ken form­s17,72,73,74,75,76,77,78,79,80,81,82. Optical mi­croscopy and SEM–EDS are com­monly used to iden­tify and an­a­lyze spherules and the processes by which they are formed. Care is needed to con­clu­sively dis­tin­guish high-tem­per­a­ture spherules pro­duced by cos­mic im­pacts from other su­per­fi­cially sim­i­lar forms. Other such ob­jects that fre­quently oc­cur in sed­i­ments in­clude an­thro­pogenic spherules (typically from mod­ern coal-fired power plants), au­thi­genic fram­boids (Supporting Information, Fig. S7), rounded de­tri­tal mag­netite, and vol­canic spherules.

Spherules in TeH sed­i­ment were in­ves­ti­gated from strati­graphic se­quences that in­clude the MB II de­struc­tion layer at four lo­ca­tions: palace, tem­ple, ring road, and wadi (Fig. 24). For the palace (Field UA, Square 7GG), the se­quence spanned 28 cm with 5 con­tigu­ous sam­ples of sed­i­ment rang­ing from 3-cm thick for the MB II de­struc­tion layer to 13-cm thick for some out­ly­ing sam­ples. In the palace, 310 spherules/kg (Fig. 24d) were ob­served in the de­struc­tion layer with none found in sam­ples above and be­low this layer. For the tem­ple (Field LS, Square 42J), 5 con­tin­u­ous sam­ples spanned 43 cm and ranged in thick­ness from 6 to 16 cm; the MB II layer con­tained ~ 2345 Fe- and Si-rich spherules/​kg with 782/kg in the sam­ple im­me­di­ately be­low and none at other lev­els (Fig. 24c). Six con­tigu­ous sam­ples from the ring road (Field LA, Square 28 M) spanned 30 cm with all 5 cm thick; the MB II de­struc­tion layer at this lo­ca­tion con­tains 2150 spherules/kg with none de­tected in younger or older sam­ples (Fig. 24b). Five dis­con­tin­u­ous sam­ples from the wadi spanned 170 cm, rang­ing from 10-cm thick for the de­struc­tion layer up to 20-cm thick for other sam­ples; the MB II de­struc­tion layer at this lo­ca­tion con­tained 2780 spherules/kg with none in sam­ples from other lev­els (Fig. 24a, Supporting Information, Table S3). Notably, when melted mud­brick from the ring road was be­ing mounted for SEM analy­sis, nu­mer­ous loose spherules were ob­served within vesi­cles of the sam­ple, con­firm­ing a close as­so­ci­a­tion be­tween the spherules and melt­glass. At all four lo­ca­tions, the peaks in high-tem­per­a­ture spherule abun­dances oc­cur in the MB II de­struc­tion layer dat­ing to ~ 1650 BCE.

SEM im­ages of spherules are shown in Figs. 25, 26, 27 and 28, and com­po­si­tions are listed in Supporting Information, Table S4. The av­er­age spherule di­am­e­ter was 40.5 µm with a range of 7 to 72 µm. The dom­i­nant min­er­als were Fe ox­ides av­er­ag­ing 40.2 wt.%, with a range of up to 84.1 wt.%; el­e­men­tal Fe with a range of up to 80.3 wt.%; SiO av­er­ag­ing 20.9 wt.%, rang­ing from 1.0 to 45.2 wt.%; AlO av­er­ag­ing 7.8 wt.% with a range of up to 15.6 wt.%; and TiO av­er­ag­ing 7.1 wt.% with a range of up to 53.1 wt.%. Fourteen spherules had com­po­si­tions > 48 wt.% of ox­i­dized Fe, el­e­men­tal Fe, and TiO; five spherules con­tained  75 wt.% Fe with no Ti. Eight of 23 spherules an­a­lyzed con­tained de­tectable lev­els of Ti at up to 53.1 wt.%.

Two un­usual spherules from the palace con­tain anom­alously high per­cent­ages of rare-earth el­e­ments (REEs) at > 37 wt.% of com­bined lan­thanum (La), and cerium (Ce) (Fig. 26), as de­ter­mined by pre­lim­i­nary mea­sure­ments us­ing SEM–EDS. Minor ox­ides ac­count for the rest of the spherules’ bulk com­po­si­tion (Table S1).

One 54-µm-wide sec­tioned spherule con­tains ti­ta­nium sul­fide (TiS) with a melt­ing point of ~ 1780° C. TiS, known as was­sonite, was first iden­ti­fied in me­te­orites (Fig. 27) and has been re­ported in im­pact-re­lated ma­te­ri­al17,81,83. However, TiS some­times oc­curs as an ex­so­lu­tion prod­uct form­ing fine net­works in mag­netite and il­menite and can be of ter­res­trial ori­gin.

One un­usual piece of 167-µm-wide Ca–Al–Si melt­glass con­tains nearly two dozen iron ox­ide spherules on its sur­face (Fig. 28). The melt­glass con­tains a com­pletely melted quartz grain as part of the ma­trix (Fig. 28b). Most of the spherules ap­pear to have been flat­tened or crushed by col­li­sion with the melt­glass while they were still par­tially molten (Fig. 28c).

Melted ma­te­ri­als from non-im­pact-re­lated com­bus­tion have been re­ported in mul­ti­ple stud­ies. Consequently, we in­ves­ti­gated whether Ca-, Fe-, and Si-rich spherules and melt­glass (mudbrick, pot­tery, plas­ter, and roof­ing clay) may have formed nor­mally, rather than from a cos­mic im­pact event. For ex­am­ple, (i) glassy spherules and melt­glass are known to form when car­bon-rich bio­mass smol­ders be­low ground at ~ 1000° to 1300 °C, such as in mid­den mound­s71. They also form in buried peat de­posit­s84, un­der­ground coal seam­s85, burned haystack­s86, and in large bon­fires, such as at the Native American site at Cahokia, Illinois, in the USA87. (ii) Also, an­cient for­ti­fi­ca­tions (hillforts) in Scotland and Sweden, dat­ing from ~ 1000 BCE to 1400 AD, have ar­ti­fi­cially vit­ri­fied walls that melted at tem­per­a­tures of ~ 850° to 1000 °C88. (iii) Partially vit­ri­fied pot­tery and melt­glass de­rived from the melt­ing of wat­tle and daub (thatch and clay) with es­ti­mated tem­per­a­tures of ~ 1000 °C have been re­ported in burned houses of the Trypillia cul­ture in Ukraine89,90. (iv) Vitrified mud­bricks and pot­tery that melted at 91; in the north­ern Jordan Valley at an Early-Bronze-Age site called Tell Abu al-Kharaz92; and at Early-Bronze-Age Tell Chuera in Syria93. All these sites de­scribe melt­ing tem­per­a­tures rang­ing from ~ 850° to 1300 °C.

In an­other ex­am­ple of melt­glass, vit­ri­fied bricks at Tell Leilan in Syria, dat­ing to ~ 2850 to 2200 BCE, are es­ti­mated by Weiss94 to have melted at ~ 1200 °C, and he at­trib­uted high-cal­cium spherules to low-tem­per­a­ture com­bus­tion of thatch roof­ing ma­te­ri­al­s95. However, thatch has low cal­cium con­tent, lead­ing Courty96 to pro­pose that this ma­te­r­ial formed from melted lime-based plas­ter dur­ing an air­burst/​im­pact at Tell Leilan ~ 550 years be­fore the de­struc­tion of TeH. Courty96 re­ported alu­mi­nosil­i­cate spherules with un­usual high-tem­per­a­ture el­e­men­tal nickel (melting point: ~ 1455 °C); com­plex vesic­u­lar glass par­ti­cles that con­tain ter­res­tri­ally rare un­ox­i­dized nickel in­clu­sions; and both sin­gle and mul­ti­ple cal­cite spherules (melting point = 1500 °C, as mea­sured by ex­per­i­ments in this con­tri­bu­tion).

For the melted ma­te­ri­als, there is a de­fin­i­tive dif­fer­ence: high-tem­per­a­ture min­er­als are em­bed­ded in melt­glass at TeH but none are pre­sent at these other sites (except for Tell Leilan). To ex­plore this dif­fer­ence, Moore et al.17 in­ves­ti­gated bio­mass glass from mid­den mounds in Africa and found no high-tem­per­a­ture min­er­als. For this con­tri­bu­tion, we used SEM–EDS to ex­am­ine alu­mi­nosil­i­cate melt­glass from an un­der­ground peat fire in South Carolina, USA; melt­glass in coal-fired fly ash from New Jersey, USA; and min­ing slag from a cop­per mine in Arizona, USA. All these melt­glass ex­am­ples dis­play un­melted quartz and con­tain no other high-tem­per­a­ture melted grains, con­sis­tent with low-tem­per­a­ture melt­ing at

At the sites with non-im­pact melt­glass, es­ti­mated tem­per­a­tures were con­sis­tently less than 1300 °C, too low to melt mag­netite into Fe-rich spherules, e.g., with com­po­si­tions of > 97% wt.% FeO, as are found at TeH. Nor can these low tem­per­a­tures pro­duce melt­glass and spherules em­bed­ded with melted zir­con (melting point = 1687 °C), chromite (2190 °C), quartz (1713 °C), plat­inum (1768 °C), and irid­ium (2466 °C). Moore et al.17 con­firmed that the melt­ing of these high-tem­per­a­ture min­er­als re­quires min­i­mum tem­per­a­tures of ~ 1500° to 2500 °C.

This ev­i­dence demon­strates that al­though the ma­trix of the spherules and melt­glass at TeH likely ex­pe­ri­enced in­cip­i­ent melt­ing at tem­per­a­tures lower than ~ 1300 °C, this value rep­re­sents only the min­i­mum tem­per­a­ture of ex­po­sure, be­cause the high-tem­per­a­ture min­er­als em­bed­ded in them do not melt at such low tem­per­a­tures. Instead, the spherules and melt­glass at TeH must have reached tem­per­a­tures greater than ~ 1300 °C, most likely in­volv­ing brief ex­po­sure to am­bi­ent tem­per­a­tures of ~ 2500 °C, the melt­ing point of irid­ium. These tem­per­a­tures far ex­ceed those char­ac­ter­is­tic of city fires and other types of bio­mass burn­ing. In sum­mary, all of this ev­i­dence is con­sis­tent with very high tem­per­a­tures known dur­ing cos­mic im­pacts but in­con­sis­tent with other known nat­ural causes.

In sed­i­ments of the de­struc­tion layer, we ob­served am­ber-to-off-white-col­ored spherules (Fig. 29) at high con­cen­tra­tions of ~ 240,000/​kg in the palace, ~ 420/​kg in the tem­ple, ~ 60/​kg on the ring road, and ~ 910/​kg in the wadi (Supporting in­for­ma­tion, Table S2). In all four pro­files, the spherules peak in the de­struc­tion layer with few to none above or be­low. Peak abun­dances of cal­cium car­bon­ate spherules are closely as­so­ci­ated with peak abun­dances of plas­ter frag­ments, which are the same color. By far the most spherules (~ 250× more) oc­curred in the de­struc­tion layer of the palace, where ex­ca­va­tions showed that nearly every room and ceil­ing was sur­faced with off-white lime-based plas­ter. Excavators un­cov­ered high-qual­ity lime plas­ter frag­ments still ad­her­ing to mud­bricks in­side the MB II palace com­plex, and in one palace room, we un­cov­ered frag­ments of melted plas­ter (Fig. 29e). In con­trast, lime plas­ter was very rarely used in build­ings on the lower tall, in­clud­ing those near the tem­ple.

To ex­plore a po­ten­tial con­nec­tion be­tween plas­ter and spherules, we per­formed SEM–EDS on sam­ples of the palace plas­ter. Comparison of SEM–EDS analy­ses shows that the plas­ter com­po­si­tion has a > 96% sim­i­lar­ity to the spherule com­po­si­tion: CaCO = 71.4 wt.% in plas­ter ver­sus 68.7 wt.% in the spherules; el­e­men­tal C = 23.6 ver­sus 26.3 wt.%; SiO = 2.4 ver­sus 1.8 wt.%; MgO = 1.7 ver­sus 2.0 wt.%; and SO = 0.94 ver­sus 1.2 wt.%. The high car­bon per­cent­age and low sul­fur con­tent in­di­cate that the plas­ter was made from cal­cium car­bon­ate and not gyp­sum (CaSO·2HO). SEM imag­ing re­vealed that the plas­ter con­tains small plant parts, com­monly used in plas­ter as a binder, and is likely the source of the high abun­dance of el­e­men­tal C in the plas­ter. Inspection showed no ev­i­dence of mi­cro­fos­sils, such as coc­col­iths, bra­chiopods, and foraminifera. The mor­phol­ogy of the spherules in­di­cates that they are not au­thi­genic or bi­o­log­i­cal in ori­gin.

One of the ear­li­est known uses of CaCO-based plas­ter was in ~ 6750 BCE at Ayn Ghazal, ~ 35 km from TeH in mod­ern-day Amman, Jordan97. At that site, multi-pur­pose lime plas­ter was used to make stat­ues and fig­urines and to coat the in­te­rior walls of build­ings. Because the pro­duc­tion of lime-based plas­ter oc­curred at least 3000 years be­fore TeH was de­stroyed, the in­hab­i­tants of TeH un­doubt­edly were fa­mil­iar with the process. Typically, lime pow­der was pro­duced in an­cient times by stack­ing wood/​com­bustibles in­ter­spersed with lime­stone rocks and then set­ting the stack on fire. Temperatures of ~ 800–1100 °C were re­quired to trans­form the rocks into crumbly chalk, which was then mixed with wa­ter to make hy­drated lime and plas­tered onto mud­brick wall­s97.

At TeH, frag­ments of CaCO-based plas­ter are in­ter­mixed in co­vary­ing abun­dances with CaCO-based spherules with both com­po­si­tions match­ing to within 96%. This sim­i­lar­ity sug­gests that the car­bon­ate spherules are de­rived from the plas­ter. We in­fer that the high-tem­per­a­ture blast wave from the im­pact event stripped some plas­ter from the in­te­rior walls of the palace and melted some into spherules. However, it is dif­fi­cult to di­rectly melt CaCO, which gives off CO at high tem­per­a­tures and de­com­poses into lime pow­der. We in­ves­ti­gated this cy­cle in a heat­ing ex­per­i­ment with an oxy­gen/​propy­lene torch and found that we could de­com­pose the plas­ter at ~ 1500 °C, the up­per limit of the heat­ing test, and be­gin in­cip­i­ent melt­ing of the plas­ter. The heated plas­ter pro­duced emer­gent droplets at that tem­per­a­ture but did not trans­form into free spherules (Supporting Information, Text S2).

Similar spherules have been re­ported from Meteor Crater, where spherules up to ~ 200 μm in di­am­e­ter are com­posed en­tirely of CaCO formed from a cos­mic im­pact into lime­stone98,99. One of sev­eral pos­si­ble hy­pothe­ses for TeH is that dur­ing the im­pact event, the lime­stone plas­ter con­verted to CaO with an equi­lib­rium melt­ing point of 2572 °C. However, it is highly likely that air­borne con­t­a­m­i­nants, such as sodium and wa­ter va­por, re­acted with the CaO and sig­nif­i­cantly low­ered the melt­ing point, al­low­ing spherule for­ma­tion at ≥ 1500 °C.

The pro­posed chem­i­cal se­quence of events of plas­ter for­ma­tion and the later im­pact are as fol­lows:

Quicklime was mixed with wa­ter to make a wet plas­ter:

The plas­ter hard­ened and slowly ab­sorbed CO to re­vert to CaCO:

The high-tem­per­a­ture im­pact event melted some plas­ter into spherules:

According to the pre­vi­ous in­ves­ti­ga­tion­s17,72,81,82, Fe-rich spherules such as those found at TeH typ­i­cally melt at > 1538 °C, the melt­ing point of iron (Table 1). Because of the pres­ence of mag­netite (FeO) in the REE spherule, its melt­ing point is in­ferred to be > 1590 °C (Table 1). The Si-rich spherules are sim­i­lar in com­po­si­tion to TeH sed­i­ment and mud­brick, and thus, we pro­pose that they were de­rived from the melt­ing of these ma­te­ri­als at > 1250 °C. The car­bon­ate-rich spherules likely formed at > 1500 °C.

Several stud­ies de­scribe a mech­a­nism by which spherules could form dur­ing a low-al­ti­tude cos­mic air­burst100,101. When a bolide en­ters Earth’s at­mos­phere, it is sub­jected to im­mense aero­dy­namic drag and ab­la­tion, caus­ing most of the ob­ject to frag­ment into a high-tem­per­a­ture fire­ball, af­ter which its re­main­ing mass is con­verted into a high-tem­per­a­ture va­por jet that con­tin­ues at hy­per­ve­loc­ity down to the Earth’s sur­face. Depending on the al­ti­tude of the bolide’s dis­rup­tion, this jet is ca­pa­ble of ex­ca­vat­ing un­con­sol­i­dated sur­fi­cial sed­i­ments, melt­ing them, and eject­ing the molten ma­te­r­ial into the air as Si- and Fe-rich spherules and melt­glass. This melted ma­te­r­ial typ­i­cally con­tains a very low per­cent­age (102.

To more ac­cu­rately de­ter­mine the max­i­mum tem­per­a­tures of the de­struc­tion layer, we used SEM–EDS to com­pre­hen­sively in­ves­ti­gate melted min­er­als on the outer sur­faces of melted pot­tery and mud­bricks. We searched for and an­a­lyzed zir­con (melting point: ~ 1687 °C), chromite (~ 2190 °C), and quartz (~ 1713 °C)17.

Melted zir­cons in pot­tery and mud­bricks were ob­served (Fig. 30) at an es­ti­mated den­sity of 1 grain per 20 mm2. On highly melted sur­faces, nearly all zir­cons showed some de­gree of melt­ing. In con­trast, nearly all zir­cons found on bro­ken in­te­rior sur­faces were un­melted (Fig. 30d), ex­cept those within ~ 1 mm of melted sur­faces. This im­plies that the tem­per­a­ture of the sur­round­ing at­mos­phere was higher than the in­ter­nal tem­per­a­tures of the melt­ing ob­jects. Unmelted pot­sherds dis­played only un­melted min­er­als.

The melted zir­cons in TeH ma­te­ri­als ex­hibit a wide range of mor­pholo­gies. Most showed ev­i­dence of suf­fi­cient melt­ing to al­ter or de­stroy the orig­i­nal dis­tinc­tive, eu­he­dral shape of the grains. Also, the grains were of­ten dec­o­rated with vesi­cles that were as­so­ci­ated with frac­tures (Fig. 30a, c).

Stoichiometric zir­con con­tains 67.2 wt.% and 32.8 wt.% ZrO and SiO re­spec­tively, but in sev­eral TeH sam­ples, we ob­served a re­duc­tion in the SiO con­cen­tra­tion due to a loss of volatile SiO from the dis­so­ci­a­tion of SiO. This al­ter­ation has been found to oc­cur at 1676 °C, slightly be­low zir­con’s melt­ing point of 1687 °C103. This zir­con dis­so­ci­a­tion leads to vary­ing ZrO:SiO ra­tios and to the for­ma­tion of dis­tinc­tive gran­u­lar tex­tures of pure ZrO, also known as bad­de­leyite104 (Figs. 30, 31, 32). With in­creas­ing time at tem­per­a­ture, zir­con will even­tu­ally con­vert par­tially or com­pletely to ZrO. Nearly all zir­cons ob­served on the sur­faces of melted ma­te­ri­als were ei­ther melted or showed some con­ver­sion to bad­de­leyite. We ob­served one zir­con grain (Fig. 32d–e) dis­play­ing gran­u­lar ZrO as­so­ci­ated with three phases that span a wide range of SiO con­cen­tra­tions, likely formed at tem­per­a­tures above 1687 °C. This ex­treme tem­per­a­ture and com­pet­ing loss of SiO over an in­ferred du­ra­tion of only sev­eral sec­onds led to com­plex mi­crostruc­tures, where grains melted, out­gassed, and dif­fused into the sur­round­ing ma­trix.

Zircon grains have a the­o­ret­i­cal, equi­lib­rium melt­ing point of ~ 1687 °C. Under lab­o­ra­tory heat­ing17, zir­con grains showed no de­tectable al­ter­ation in shape at ~ 1300 °C but dis­played in­cip­i­ent melt­ing of grain edges and dis­so­ci­a­tion to bad­de­leyite be­gin­ning at ~ 1400 °C with in­creas­ing dis­so­ci­a­tion to 1500 °C17. Most zir­con grains  120 µm were still rec­og­niz­able but dis­played con­sid­er­able melt­ing17. These ex­per­i­ments es­tab­lish a lower melt­ing range for TeH zir­con grains of ~ 1400° to 1500 °C.

Patterson105 showed that zir­con dis­so­ci­a­tion be­comes fa­vor­able above 1538 °C and par­ti­cles be­tween 1 and 100 µm in size melted and dis­so­ci­ated when pass­ing through a plasma, form­ing spherules with var­i­ous amounts of SiO glass con­tain­ing ZrO crys­tal­lites rang­ing in size from 5 nm to 1 µm. The ma­jor­ity of zir­con crys­tals were mon­o­clinic, but tetrag­o­nal ZrO was ob­served for the smaller crys­tal­lite sizes. Residence times were in the or­der of 100 ms, and the spe­cific ZrO to SiO ra­tio within each spherule de­pended on the par­ti­cle’s time at tem­per­a­ture106.

Bohor et al.104 pre­sented im­ages of im­pact-shocked zir­cons from the K-Pg im­pact event at 66 Ma that are mor­pho­log­i­cally in­dis­tin­guish­able from those at TeH. Decorated zir­con grains are un­com­mon in na­ture but com­monly as­so­ci­ated with cos­mic im­pact events, as ev­i­denced by two par­tially melted zir­cons from the known air­burst/​im­pact at Dakhleh Oasis, Egypt (Fig. 30e). The pres­ence of bub­bles in­di­cates that tem­per­a­tures reached at least 1676 °C, where the zir­con be­gan to dis­so­ci­ate and out­gas. Similar dis­so­ci­ated zir­con grains also have been found in tek­tite glass and dis­tal fall­back ejecta (deposited from hot va­por clouds). Granular bad­de­leyite-zir­con has been found in the ~ 150-km-wide K-Pg im­pact crater107 and the 28-km-wide Mistatin Lake crater in Canada107. The dis­so­ci­a­tion of zir­con re­quires high tem­per­a­tures of ~ 1676 °C104, im­ply­ing that TeH was ex­posed to sim­i­lar ex­treme con­di­tions.

Examples of melted chromite, an­other min­eral that melts at high tem­per­a­tures, were also ob­served. Thermally-altered chromite grains were ob­served in melted pot­tery, melted mud­bricks, and melted roof­ing clay from the palace. Their es­ti­mated den­sity was 1 grain per 100 mm2, mak­ing them rarer than melted zir­con grains. The mor­pholo­gies of chromite grains range from ther­mally al­tered (Fig. 33a) to fully melted (Fig. 33b, d). One chromite grain from the palace dis­plays un­usual oc­ta­he­dral cleav­age or shock-in­duced pla­nar frac­tures (Fig. 33b). The typ­i­cal chem­i­cal com­po­si­tion for chromite is 25.0 wt.% Fe, 28.6 wt.% O, and 46.5 wt.% Cr, al­though the Cr con­tent can vary from low val­ues to ~ 68 wt.%. SEM im­ages re­veal that, as chromite grains melted, some Cr-rich molten ma­te­r­ial mi­grated into and mixed with the host melt, caus­ing an in­crease in Cr and Fe, and cor­re­spond­ing de­ple­tion of Si. The ra­tio of Cr to Fe in chromite af­fects its equi­lib­rium melt­ing point, which varies from ~ 1590 °C for a neg­li­gi­ble amount of Cr up to ~ 2265 °C for ~ 46.5 wt.% Cr as in chromite or chro­mian mag­netite ((Fe)CrO), plac­ing the melt­ing point of TeH chromite at close to 2265 °C.

Chromite grains the­o­ret­i­cally melt at ~ 2190 °C. Moore et al.17 re­ported the re­sults of heat­ing ex­per­i­ments in which chromite grains in bulk sed­i­ment showed al­most no ther­mal al­ter­ation up to ~ 1500 °C (Supporting Information, Fig. S8). At tem­per­a­tures of ~ 1600 °C and ~ 1700 °C, the shapes of chromite grains were in­tact but ex­hib­ited lim­ited melt­ing of grain edges. These re­sults es­tab­lish a range of ~ 1600° to 1700 °C for melt­ing chromite grains.

Because chromite typ­i­cally does not ex­hibit cleav­age, the grain ex­hibit­ing this fea­ture is highly un­usual. Its ori­gin is un­clear but there are sev­eral pos­si­bil­i­ties. The cleav­age may have re­sulted from ex­so­lu­tion while cool­ing in the source magma. Alternately, the lamel­lae may have re­sulted from me­chan­i­cal shock dur­ing a cos­mic im­pact, un­der the same con­di­tions that pro­duced the shocked quartz, as re­ported by Chen et al.108 for me­te­orites shocked at pres­sures of ~ 12 GPa. Or they may have been formed by ther­mal shock, i.e., rapid ther­mal load­ing fol­lowed by rapid quench­ing. This lat­ter sug­ges­tion is sup­ported by the ob­ser­va­tion that the out­side glass coat­ing on the pot­sherd does not ex­hibit any quench crys­tals, im­ply­ing that the cool­ing pro­gressed very rapidly from liq­uid state to solid state (glass). This is rare in ter­res­trial events ex­cept for some va­ri­eties of ob­sid­ian, but com­mon in melted ma­te­r­ial pro­duced by atomic det­o­na­tions (trinitite), light­ning strikes (fulgurites), and cos­mic air­burst/​im­pacts (meltglass)81. More in­ves­ti­ga­tions are needed to de­ter­mine the ori­gin of the po­ten­tially shocked chromite.

Using SEM–EDS, we in­ves­ti­gated abun­dances and po­ten­tial ori­gins (terrestrial ver­sus ex­trater­res­trial) of plat­inum-group el­e­ments (PGEs) em­bed­ded in TeH melt­glass, in ad­di­tion to Ni, Au, and Ag. Samples stud­ied in­clude melted pot­tery (n = 3); melted mud­brick (n = 6); melted roof­ing clay (n = 1), and melted lime-based build­ing plas­ter (n = 1). On the sur­faces of all four types of melt­glass, we ob­served melted metal-rich nuggets and ir­reg­u­larly shaped metal­lic splat­ter, some with high con­cen­tra­tions of PGEs (ruthenium (Ru), rhodium (Rh), pal­la­dium (Pd), os­mium (Os), irid­ium (Ir), and plat­inum (Pt)) and some nuggets en­riched in sil­ver (Ag), gold (Au), chromium (Cr), cop­per (Cu), and nickel (Ni) with no PGEs (Figs. 34, 35). Importantly, these metal-rich nuggets were ob­served only on the top sur­faces of melt­glass and not in­side vesi­cles or on bro­ken in­te­rior sur­faces.

Using SEM–EDS, we iden­ti­fied vari­able con­cen­tra­tions and as­sem­blages of PGEs. The metal­lic par­ti­cles ap­pear to have melted at high tem­per­a­tures based on the min­i­mum melt­ing points of the el­e­ments: irid­ium at 2466 °C; plat­inum = 1768 °C; and ruthe­nium = 2334 °C, in­di­cat­ing a tem­per­a­ture range of be­tween ap­prox­i­mately 1768° and 2466 °C. Our in­ves­ti­ga­tions also iden­ti­fied two PGE groups, one with nuggets in which Pt dom­i­nates Fe and the other with metal­lic splat­ter in which Fe dom­i­nates Pt.

We con­ducted 21 mea­sure­ments on Pt-dominant TeH nuggets on melt­glass (Fig. 34a–c). The nuggets av­er­age ~ 5 µm in length (range 1–12 µm) with an es­ti­mated con­cen­tra­tion of 1 nugget per 10 mm2. For these nuggets, Fe con­cen­tra­tions av­er­age 1.0 wt.%, Ir = 6.0 wt.%, and Pt = 44.9 wt.% (Supporting Information, Tables S6, S7). The pres­ence of PGEs was con­firmed by two SEM–EDS in­stru­ments that ver­i­fied the ac­cu­rate iden­ti­fi­ca­tion of PGEs through analy­ses of sev­eral blanks that showed no PGE con­tent. Some con­cen­tra­tions are low ( Pt or Pt > Fe were found to be con­sis­tent be­tween the two in­stru­ments.

To de­ter­mine the source of TeH nuggets and splat­ter, we con­structed ternary di­a­grams. Terrestrial PGE nuggets are com­monly found in ore bod­ies that when eroded, can be­come con­cen­trated in river­ine placer de­posits, in­clud­ing those of the Jordan River flood­plain. To com­pare Fe–Ir–Pt re­la­tion­ships among the TeH nuggets, we com­piled data from nearby placer de­posits in Greece109, Turkey110,111, and Iraq112, along with dis­tant plac­ers in Russia113,114,115, Canada116, and Alaska, USA117,118. The com­pi­la­tion of 109 Pt-dominant placer nuggets in­di­cates that the av­er­age Fe con­cen­tra­tion is 8.2 wt.%, Ir = 2.9 wt.%, and Pt = 80.3 wt.%. For the Ir-dominant placer nuggets (n = 104), Fe = 0.4 wt.%, Ir = 47.8 wt.%, and Pt = 5.3 wt.% (Supporting Information, Tables S6, S7). The ternary di­a­grams re­veal that the val­ues for Pt-dominant TeH nuggets over­lap with Pt-dominant ter­res­trial placer nuggets but the Fe-dominant splat­ter is dis­sim­i­lar (Fig. 36a).

We made 8 mea­sure­ments on TeH Fe-dominant PGE splat­ter (Fig. 34d–f). The metal-rich ar­eas av­er­age ~ 318 µm in length (range 20–825 µm) with an es­ti­mated con­cen­tra­tion of 1 PGE-rich bleb per mm2, 100× more com­mon than the TeH nuggets. Average con­cen­tra­tions are Fe = 17.5 wt.%, Ir = 4.7 wt.%, and Pt = 1.5 wt.%.

We ex­plored a po­ten­tial ex­trater­res­trial ori­gin by con­struct­ing ternary di­a­grams for com­par­i­son of TeH Fe-dominant splat­ter with known me­te­orites and comets (Fig. 36b, c). We com­piled data for 164 nuggets ex­tracted from car­bona­ceous chon­dritic me­te­orites (e.g., Allende, Murchison, Leoville, and Adelaide)119,120,121,122, seafloor cos­mic spherules123,124, iron me­te­orites122,125, Comet Wild 2126, and cometary dust par­ti­cles126. For av­er­age weight per­cent­ages, see Supporting Information, Tables S6, S7. The Fe-dominant TeH splat­ter (Fig. 36b) closely matches nuggets from car­bona­ceous chon­drites and cos­mic spherules but is a weak match for most iron me­te­orites (Fig. 36c). In ad­di­tion, the TeH nuggets are sim­i­lar to four cometary par­ti­cles, two of which were col­lected dur­ing the Stardust flyby mis­sion of Comet Wild 2 in 2004126. For av­er­age weight per­cent­ages, see Supporting Information, Tables S6, S7.

To fur­ther ex­plore an ex­trater­res­trial con­nec­tion for TeH Fe-dominant splat­ter, we com­piled wt.% data for TeH PGEs (Rh, Ru, Pd, Os, Ir, and Pt) and nor­mal­ized them to CI chon­drites us­ing val­ues from Anders and Grevasse127. We com­pared those val­ues to CI-normalized nuggets in car­bona­ceous chon­drites, in­clud­ing CV-type chon­drites (e.g., Allende) and CM types (e.g., Murchison)119,120,122,128,129,130,131, seafloor cos­mic spherules124, mi­crom­e­te­orites123, and iron me­te­orites122,125. These re­sults are shown in Fig. 36d.

The TeH Fe-dominant splat­ter closely matches all types of ex­trater­res­trial ma­te­r­ial with a sim­i­lar pat­tern among all data sets: Pd has the low­est nor­mal­ized val­ues and Os and/​or Ir have the high­est, closely fol­lowed by Pt. The TeH splat­ter was also com­pared to the CI-normalized wt.% of bulk me­te­oritic ma­te­r­ial from CV- and CM-type chon­drites (Fig. 36d). The com­po­si­tion of TeH splat­ter shows poor cor­re­la­tion with bulk chon­dritic ma­te­ri­als, al­though the splat­ter is an ex­cel­lent geo­chem­i­cal match with the PGE nuggets in­side them. In sum­mary, the CI nor­mal­iza­tion of PGEs sug­gests an ex­trater­res­trial ori­gin for the Fe-dominant TeH splat­ter, just as the ternary di­a­grams also sug­gest an ex­trater­res­trial source. The cor­re­spon­dence of these two in­de­pen­dent re­sults sug­gests that the quan­tifi­ca­tion of PGEs is suf­fi­ciently ac­cu­rate in this study.

Another un­usu­ally abun­dant el­e­ment, Mo, is also as­so­ci­ated with Fe-dominant splat­ter but not with Pt-dominant nuggets. Mo av­er­ages 0.3 wt.% with up to 1.1 wt.% de­tected in Fe-dominant splat­ter but with none de­tected in TeH Pt-dominant nuggets. Mo also is not re­ported in any ter­res­trial placer nuggets and oc­curs in low con­cen­tra­tions (less than ~ 0.02 wt.%) in iron me­te­orites. In con­trast, Mo is re­ported at high con­cen­tra­tions in PGE nuggets from car­bona­ceous chon­drites (~ 11.5 wt.%), cos­mic spherules (0.6 wt.%), and cometary ma­te­r­ial (5.8 wt.%). Thus, the Mo con­tent of TeH splat­ter ap­pears dis­sim­i­lar to ter­res­trial ma­te­r­ial but over­laps val­ues of known cos­mic ma­te­r­ial, sug­gest­ing an ex­trater­res­trial ori­gin.

Based on the vol­ume and weight of the melt­glass, we es­ti­mate that the ex­trater­res­trial-like metal­lic TeH Fe-dominant splat­ter rep­re­sents 102.

We also in­ves­ti­gated nuggets that lack PGEs. The geo­chem­istry of these nuggets shows two dis­tinct pop­u­la­tions, one Ni-dominant and one Fe-dominant (Fig. 34g–i). Twelve mea­sure­ments of TeH sam­ples show en­rich­ments in Ag av­er­ag­ing 5.7 wt.%, Au = 0.6 wt.%, Cr = 2.2 wt.%, Cu = 2.8 wt.%, and Ni = 3.7 wt.%. All par­ti­cles ap­pear to have been melted at high tem­per­a­tures: sil­ver at ~ 961 °C; gold at 1064 °C; chromium at 1907 °C; cop­per at 1085 °C; and nickel at 1455 °C.

Ternary di­a­grams for Cr, Fe, and Ni show that some TeH nuggets ex­hibit chem­i­cal sim­i­lar­i­ties to min­eral de­posits in Greece, Turkey, and Oman, sug­gest­ing that some nuggets are of ter­res­trial ori­gin. However, other TeH nuggets are chem­i­cally sim­i­lar to ma­te­ri­als found in iron me­te­orites, chon­drites, achon­drites, and comets (Supporting Information, Fig. S9). When com­pared to me­te­oritic ma­te­r­ial, the Ni-dominant group roughly cor­re­sponds to mea­sure­ments from sul­fide in­clu­sions in chon­drites132, and the Fe-dominant group over­laps both chon­dritic sul­fide in­clu­sions and metal-rich grains from Comet Wild 2133. These re­sults sug­gest that a small frac­tion (

Importantly, the PGE-rich nuggets and splat­ter were ob­served em­bed­ded only on melted sur­faces of the TeH melt­glass but not in­side the vesi­cles or within the melt­glass. This sug­gests that the nuggets and splat­ter were not con­tained in the orig­i­nal sed­i­men­tary ma­trix but were fused onto the TeH glass while still molten. Geochemical analy­ses sug­gest a dual ori­gin. The Pt-dominant nuggets do not match known ex­trater­res­trial ma­te­r­ial and in­stead ap­pear to be of ter­res­trial ori­gin, pos­si­bly from placer de­posits and re­gional mines. It is un­clear ex­actly how they be­came em­bed­ded onto but not in­side TeH melt­glass, but one pos­si­bil­ity is that they were orig­i­nally buried as river-laid, PGE-rich placer de­posits. If so, we pro­pose that they were ejected dur­ing the im­pact event and dis­trib­uted across the molten glass by the im­pact blast wave. Another pos­si­bil­ity is that they de­rive from jew­elry and raw pre­cious met­als in the palace com­plex that were pul­ver­ized and dis­persed dur­ing the high-ve­loc­ity de­struc­tion of the palace.

In con­trast, the Fe-dominant nuggets fused into the sur­faces of TeH melt­glass closely match the com­po­si­tion of nuggets from chon­dritic me­te­orites, cos­mic spherules, and comets, con­sis­tent with an ex­trater­res­trial ori­gin. The data sug­gest that a car­bona­ceous chon­drite or a comet det­o­nated in the air near TeH, pul­ver­ized PGE-rich nuggets within the bolide, ac­creted ter­res­trial placer nuggets, and dusted both ter­res­trial and ex­trater­res­trial ma­te­r­ial across the sur­faces of molten mud­bricks, pot­tery, and build­ing plas­ter at low con­cen­tra­tions of

For TeH bulk sed­i­ment, neu­tron ac­ti­va­tion analy­ses show Pt abun­dance peaks in the de­struc­tion layer of all pro­files tested (Fig. 37a–d) at ~ 2× to 8× an av­er­age crustal abun­dance of 0.5 ppb. Sedimentary Ir was be­low de­tec­tion at S3). Also, Pt/Pd ra­tios in bulk sed­i­ment from the de­struc­tion lay­ers are anom­alously higher than back­ground lay­ers by ~ 4× to 14× (Fig. 37e–h).

Abundances of Pt, Ir, and the Pt/Pd ra­tios all peak in sed­i­ment at or near the top of the de­struc­tion layer, sug­gest­ing an in­flux of those el­e­ments at 1650 BCE, most likely from both ex­trater­res­trial and ter­res­trial sources. Sedimentary con­cen­tra­tions of Ir only peak in the wadi sam­ples and were not de­tectable at the other three sites for un­known rea­sons.

The in­te­rior por­tions of many melted pieces of mud­brick, roof­ing clay, and pot­tery are highly vesic­u­lar, and the walls of these vesi­cles nearly al­ways dis­play an ar­ray of metal-rich crys­tals (Fig. 38). These in­clude el­e­men­tal iron and iron ox­ides, la­beled as FeO, but is ac­tu­ally ox­i­dized as hematite (FeO) with a melt­ing point of ~ 1565 °C; mag­netite (FeO) that melts at ~ 1590 °C; and/​or el­e­men­tal iron (Fe) melt­ing at ~ 1538 °C (Fig. 38a, b, d, f, g). Also ob­served on vesi­cle walls were crys­tals of Fe phos­phide (FeP) that melts at ~ 1100 °C (Fig. 38c, g), man­ganese ox­ide (MnO) at ~ 1945 °C (Fig. 38d), cal­cium phos­phate (Ca(PO)) at ~ 1670 °C (Fig. 38e), and cal­cium sil­i­cate (CaSiO) at ~ 2130 °C (Fig. 38e).

In some cases, these min­eral crys­tals ap­pear to have crys­tal­lized from the molten ma­trix as it so­lid­i­fied. In other cases, it ap­pears that they plated onto the vesi­cle sur­face, sug­gest­ing that they may have con­densed through va­por de­po­si­tion from the high-tem­per­a­ture, min­eral-sat­u­rated at­mos­phere within the vesi­cle.

SEM in­spec­tion of the palace mud­brick melt­glass re­vealed par­tially melted grains of mag­netite (FeO) with a melt­ing point of ~ 1590 °C (Fig. 39a, b) and ti­tano­mag­netite (TiFeO) with a melt­ing point of ~ 1550 °C (Fig. 39c, d). The lat­ter is an oxyspinel that com­monly oc­curs as dis­crete grains or as an ex­so­lu­tion prod­uct within mag­netite. The chem­i­cal com­po­si­tion of mag­netite at equi­lib­rium is 72.36 wt.% Fe and 27.64 wt.% O. Titanomagnetite is 21.42 wt.% Ti, 49.96 wt.% Fe, and 28.63 wt.% O. SEM–EDS analy­sis con­firms sim­i­lar com­po­si­tions for the ob­served TeH grains.

These grains dis­play bub­ble-rich fea­tures that are com­monly as­so­ci­ated with grain frac­tures. There are sev­eral pos­si­bil­i­ties. (i) Approximately 20% of the time, mag­netite grains are re­ported to be nat­u­rally over­printed by porous mag­netite as a pre­cip­i­ta­tion prod­uct, cre­at­ing a bub­ble-like tex­ture134. (ii) Alternatively, these fea­tures may be tex­tures caused by dif­fer­en­tial dis­so­lu­tion135,136. (iii) The grains may have been ex­posed to tem­per­a­tures equal to or greater than their melt­ing point, caus­ing the out­gassing of volatiles or the rapid re­duc­tion of iron ox­ides. Because of the mor­pho­log­i­cal dis­sim­i­lar­ity of TeH grains to pub­lished ex­am­ples of grains al­tered by pre­cip­i­ta­tion and dis­so­lu­tion, we in­fer that these grains were al­tered by ex­po­sure to high tem­per­a­tures.

Sulfide and phos­phide grains were found at­tached to the walls of the vesi­cles within mud­brick melt­glass. SEM–EDS analy­ses of palace mud­brick melt­glass iden­ti­fied melted Fe sul­fide (FeS), also known as troilite (Figs. 40, 41), with a com­po­si­tion of 63.53 wt.% Fe and 36.47 wt.% S and a melt­ing point of ~ 1194 °C. Commonly found as­so­ci­ated with the Fe sul­fide, Fe phos­phide (FeP; Fig. 40b, e) is a nickel-poor va­ri­ety of bar­ringerite, a min­eral first iden­ti­fied at Meteor Crater in Arizona. Another vari­ant of Fe phos­phide, FeP, was also iden­ti­fied in melted mud­brick from the palace. Both phos­phide vari­ants melt at ~ 1100 °C. Fe phos­phide is com­mon in me­te­orites, but al­though ter­res­tri­ally rare, FeP is found in py­rometa­mor­phic rocks, such as are found in the Hatrurim Formation in nearby Israel137. Britvin et al.137 re­port that the lo­cal Fe phos­phide dis­plays av­er­ages for Fe at 76.4 wt.% and P at 21.4 wt.%, with small amounts of Ni, Co, and Cr at 2.2 wt.%. The com­po­si­tion of Fe phos­phide found at TeH is com­pa­ra­ble, av­er­ag­ing 78.3 wt.% Fe and 21.7 wt.% P. However, un­like those FeP grains from Israel, the TeH grains lack de­tectable Ni, Co, and Cr.

Visual in­spec­tion in­di­cated that the sul­fide and phos­phide grains were at­tached to the in­ner sur­faces of vesi­cles, and there­fore, most likely formed by va­por de­po­si­tion at > 1100 °C, rather than by crys­tal­liza­tion from the melted ma­trix. Troilite (FeS) is very rare ter­res­tri­ally but com­mon in me­te­oritic ma­te­ri­al81,138. Harris et al.138 re­ported find­ing in­clu­sions of troilite (FeS) as me­te­oritic clasts in Chilean melt­glass pro­posed to have de­rived from a cos­mic air­burst that left melt­glass on the sur­face along a 19-km-long stretch of the Atacama Desert ap­prox­i­mately 12,800 years ago. At that site, troilite is typ­i­cally found lin­ing the walls of vesi­cles in the melt­glass, as is the case for TeH melt­glass.

SEM–EDS analy­ses also show that vesi­cles in melted mud­brick from the palace con­tain cal­cium phos­phide (CaP) (Fig. 42). This min­eral has a sto­i­chio­met­ric com­po­si­tion of ~ 66.0 wt.% Ca and ~ 34.0 wt.% P and melts at ~ 1600 °C (Table 1). Unlike Fe sul­fide and Fe phos­phide dis­cussed above that ap­pear to have formed by va­por de­po­si­tion, these ex­am­ples most likely crys­tal­lized from the molten Ca–Al–Si ma­trix ma­te­r­ial.

SEM–EDS analy­ses of palace mud­brick melt­glass (Fig. 43a, b) and melted pot­tery (Fig. 43c) re­veal the pres­ence of Ca sil­i­cate, also known as wol­las­tonite (CaSiO), with a com­po­si­tion of 48.3 wt.% CaO and 51.7 wt.% SiO and a melt­ing point ~ 1540 °C (Table 1). These crys­tals are mostly found on bro­ken sur­faces of the ma­trix but also are some­times ob­served in­side vesi­cles. Unlike Fe sul­fide and Fe phos­phide above, which ap­pear to have formed by va­por de­po­si­tion, these crys­tals ap­pear to have con­densed from the molten ma­trix as it cooled.

Hey every­one, it’s been a while. Hope you’re all do­ing well. Today I want to show you some more watch re­lated stuff I’ve been work­ing on.

I’ve al­ready shown you the po­lar­izer mod which in­verts the dis­play, mak­ing it black, but I al­ways wanted a good way to change the dis­play to other col­ors, and I’ve been ex­per­i­ment­ing for a long time on how to do just that.

One day I de­cided to see what the kap­ton tape I use for my 3D printer bed looks like when ap­plied to the dis­play, and to my sur­prise it turned out great, trans­form­ing the bor­ing reg­u­lar LCD into a nice am­ber color.

I then started search­ing for other poly­imide-like tapes in dif­fer­ent col­ors and found a few. From my ex­per­i­ments, any­thing around 20 mi­crons or thin­ner could po­ten­tially work.

One re­ally cool thing is you can com­bine some of the dif­fer­ent tapes over each other to cre­ate a new color. For ex­am­ple the am­ber + blue tapes cre­ate a re­ally vi­brant green dis­play. I’ve found this only works for re­ally thin tapes though.

So far I have man­aged to change the dis­play into a range of col­ors, in­clud­ing, am­ber, blue, pink, red and green.

To get the ef­fect you need to ap­ply the tape to the LCD glass. It can be tricky, and I use a plas­tic credit card to smooth the tape as it’s stuck down. This re­moves any air bub­bles.

Another in­ter­est­ing mod is to make the screen trans­par­ent. To do this, you need to re­move the re­flec­tive foil that’s glued to the back of the LCD glass. I did this us­ing a sharp knife and some ad­he­sive re­mover. You then re­place it with po­lar­iz­ing film, so there are po­lar­iz­ers on both sizes of the glass.

You also need to paint the in­side of the watch PCB white, since if you don’t there won’t be enough con­trast be­tween the dis­play and the in­sides to read the num­bers. I used sim­ple enamel touch-up paint with built in brush.

Be sure to only cover the area shown, avoid­ing the var­i­ous con­tacts re­quired for the watch to still func­tion.

The fi­nal ef­fect, when paired with the F-91W vari­ant which has a trans­par­ent strap is def­i­nitely unique. I changed the LED to a white one, so When you use the LED in the dark, the light il­lu­mi­nates the en­tire space be­low the glass.

I have up­dated the mi­cro SD socket add on too, so it’s now com­pletely stand­alone from the back­plate and can be at­tached eas­ily us­ing 4 x M1.4x6MM screws.

The mid­dle of the 3D printed frame is in­ten­tion­ally thin, and bends over the in­ter­nals of the watch, pro­vid­ing just enough room to fit the socket in­side. It’s def­i­nitely handy to have a backup mem­ory card on you just in case you need it, es­pe­cially with mi­cro SD card stor­age ca­pac­ity now be­ing up in ter­abytes.

One thing to note. When you’re at­tach­ing this, re­mem­ber to bend the con­tacts from the watch up­wards, so they touch the in­side of the metal back­plate. This is the piezo which cre­ates the watches beep­ing sound.

The strap mech­a­nism on the F91W is­n’t the reg­u­lar spring loaded type you find on most watches, and can be a bit of a pain to re­move by hand, so I fig­ured I’d 3D print a tool to sim­plify the process.

It ba­si­cally in­volves dis­as­sem­bling two cheap generic bracelet link re­movers, and in­sert­ing them into a cus­tom 3D printed jig. The threaded pins are held in place by two 5mm square nuts.

To re­move the strap you just place the watch in the jig, with the threaded pins on the right side. Then you slowly turn the screw and this will pop the fric­tion pins out of the strap.

To re­place the strap, you sim­ply add the straps and pins to the watch body as shown. Make sure the pins are hang­ing out a lit­tle on the left side, then place the watch into the jig, this time with the threaded pins on the left side. You can use the screws to push the strap pins so they’re se­cured to the watch body.

If you’re gonna re­place the strap with a non Casio one, it might use a spring loaded pin in­stead of a fric­tion one, so fol­low the in­struc­tions from the strap maker in­stead of us­ing this jig.

I found a mod on Instructables which shows you how to add a sec­ond LED to the F91W, and I tried cre­at­ing a sim­ple flex PCB to make the process eas­ier. It was so sim­ple in­fact that I messed it up be­cause I was­n’t pay­ing at­ten­tion be­fore I sent it off to the man­u­fac­turer.

Anyways, af­ter cor­rect­ing the mis­take, the mod does work, but I’m not sure why it makes the watch so un­sta­ble. Sometimes it works fine, and oth­ers it keeps re­set­ting or turn­ing the dis­play off. I’m guess­ing the Casio chip in­side does not like the changes in power con­sump­tion with that sec­ond LED. Gotta ad­mit it looks re­ally nice in the dark though.

I also tried a cou­ple dif­fer­ent NFC an­tenna de­signs with flex PCBs so that I could in­stall it in­side the watch body with­out re­mov­ing the front­plate. Unfortunately there just was­n’t enough room to make the an­tenna strong enough, but per­haps there are other so­lu­tions I haven’t thought of yet.

If you’d like to make your own stuff, source files etc are at the top of the page, or if you’d like to buy a pre-mod­ded watch, or any of the tools I men­tioned in the video, check out the NODE shop.

I’ve also been get­ting more into mod­ding ana­log watches too, and have cre­ated a few min­i­mal­ist watch­face mods for the Casio MQ24, which is ba­si­cally the ana­log equiv­a­lent of the F91W. I’m in­ter­ested to see what else is pos­si­ble with these.

Young in­vestors have a new strat­egy: watch­ing fi­nan­cial dis­clo­sures of sit­ting mem­bers of Congress for stock tips.

Among a cer­tain com­mu­nity of in­di­vid­ual in­vestors on TikTok, House Speaker Nancy Pelosi’s stock trad­ing dis­clo­sures are a trea­sure trove. “Shouts out to Nancy Pelosi, the stock mar­ket’s biggest whale,” said user ‘ceowatchlist.’ Another said, “I’ve come to the con­clu­sion that Nancy Pelosi is a psy­chic,” while adding that she is the “queen of in­vest­ing.”

“She knew,” de­clared Chris Josephs, an­a­lyz­ing a par­tic­u­lar trade in Pelosi’s fi­nan­cial dis­clo­sures. “And you would have known if you had fol­lowed her port­fo­lio.”

Last year, Josephs no­ticed that the trades, ac­tu­ally made by Pelosi’s in­vestor hus­band and merely dis­closed by the speaker, were per­form­ing well.

Josephs is the co-founder of a com­pany called Iris, which shows other peo­ple’s stock trades. In the past year and a half, he has been tak­ing ad­van­tage of a law called the Stock Act, which re­quires law­mak­ers to dis­close stock trades and those of their spouses within 45 days.

Now on Josephs’ so­cial in­vest­ing plat­form, you can get a push no­ti­fi­ca­tion every time Pelosi’s stock trad­ing dis­clo­sures are re­leased. He is per­son­ally in­vest­ing when he sees which stocks are picked: “I’m at the point where if you can’t beat them, join them,” Josephs told NPR, adding that if he sees trades on her dis­clo­sures, “I typ­i­cally do buy… the next one she does, I’m go­ing to buy.”

A Pelosi spokesper­son said that she does not per­son­ally own any stocks and that the trans­ac­tions are made by her hus­band. “The Speaker has no prior knowl­edge or sub­se­quent in­volve­ment in any trans­ac­tions,” said the spokesper­son.

Related Story: Planet Money Summer School

Related Story: A Chinese Real Estate Company Is Walloping Your Stocks. Here’s Why

Still Josephs views trades by fed­eral law­mak­ers as “smart money” worth fol­low­ing and plans to track a large va­ri­ety of politi­cians. “We don’t want this to … be a left vs. right thing. We don’t re­ally care. We just want to make money,” he said.

Pelosi is hardly the only law­maker mak­ing these stock dis­clo­sures. So far this year, Senate and House mem­bers have filed more than 4,000 fi­nan­cial trad­ing dis­clo­sures — with at least $315 mil­lion of stocks and bonds bought or sold. That’s ac­cord­ing to Tim Carambat, who in 2020 cre­ated and now main­tains two pub­lic data­bases of law­maker fi­nan­cial trans­ac­tions — House Stock Watcher and Senate Stock Watcher. He says there is a sig­nif­i­cant fol­low­ing for his work.

“I knocked out a very, very sim­ple ver­sion of the pro­ject in like a cou­ple of hours. And I posted it ac­tu­ally to Reddit, where it gained some sig­nif­i­cant trac­tion and peo­ple showed a lot of in­ter­est in it,” Carambat said.

Dinesh Hasija, an as­sis­tant pro­fes­sor of strate­gic man­age­ment at Augusta University in Georgia, has been study­ing whether the mar­ket moves based on con­gres­sional dis­clo­sures. His on­go­ing re­search sug­gests that it does.

“Investors per­ceive that sen­a­tors may have in­sider in­for­ma­tion,” he said. “And we see ab­nor­mal pos­i­tive re­turns when there’s a dis­clo­sure by a sen­a­tor.”

In other words, Hasija’s re­search shows that af­ter the dis­clo­sures are pub­lished, there’s a bump in the price of stocks bought by law­mak­ers.

At least one fi­nan­cial ser­vices con­sul­tant, Matthew Zwijacz, is plan­ning to set up a fi­nan­cial in­stru­ment that au­to­mat­i­cally tracks con­gres­sional stock picks, be­cause, in his view, law­mak­ers are “probably privy to more in­for­ma­tion than just the gen­eral pub­lic.”

Both in­vestors and gov­ern­ment watch­dogs are in­ter­ested in these trades be­cause of the pos­si­bil­ity that law­mak­ers could use the pri­vate in­for­ma­tion they ob­tain through their jobs for money-mak­ing in­vest­ment de­ci­sions.

“If the sit­u­a­tion is that the pub­lic has lost so much trust in gov­ern­ment that they think … the stock trades of mem­bers are based on cor­rup­tion, and that [following that] cor­rup­tion could ben­e­fit [them]. … We have a sig­nif­i­cant prob­lem,” said Kedric Payne, se­nior di­rec­tor of ethics at the Campaign Legal Center.

A surge of in­ter­est fol­low­ing con­gres­sional fi­nan­cial dis­clo­sures came near the be­gin­ning of the COVID-19 pan­demic, when a flurry of re­ports in­di­cated that law­mak­ers sold their stocks right be­fore the fi­nan­cial crash.

NPR re­ported how Senate Intelligence Committee Chairman Richard Burr pri­vately warned a small group of well-con­nected con­stituents in February 2020 about the dire ef­fects of the com­ing pan­demic. He sold up to $1.72 mil­lion worth of per­sonal stocks on a sin­gle day that same month.

A bi­par­ti­san group of sen­a­tors also came un­der sus­pi­cion, in­clud­ing Sens. Dianne Feinstein, James Inhofe and Kelly Loeffler. After in­ves­ti­ga­tions by fed­eral law en­force­ment, none were charged with in­sider trad­ing — a very dif­fi­cult charge to make against a sit­ting law­maker.

Congressman Raja Krishnamoorthi, a Democrat from Illinois, is part of a bi­par­ti­san group of House and Senate mem­bers who have in­tro­duced leg­is­la­tion ban­ning law­mak­ers from own­ing in­di­vid­ual stocks. He has run up against a lot of op­po­si­tion to the idea.

“As I un­der­stand it, one of the perks of be­ing a mem­ber of Congress, es­pe­cially from the late 1800s on, was to be able to trade on in­sider in­for­ma­tion. That was a perk of be­ing in Congress. And that has got to come to an end,” Krishnamoorthi said.

Polling shows that there is wide sup­port for en­act­ing this pro­hi­bi­tion. According to a sur­vey done this year by Data for Progress, 67% of Americans be­lieve fed­eral law­mak­ers should not own in­di­vid­ual stocks.

There’s a deep cyn­i­cism that forms the foun­da­tion of a trad­ing strat­egy based on mim­ic­k­ing the stock picks of law­mak­ers and their spouses: the no­tion that politi­cians are cor­rupt and that you can’t trust them not to en­gage in in­sider trad­ing — so if the in­for­ma­tion is pub­lic, you might as well trade what they’re trad­ing.

But de­spite all the skep­ti­cism about politi­cians and their eth­i­cal stan­dards, the ev­i­dence does­n’t show that mem­bers of Congress make great stock pick­ers. While a 2004 pa­per found that sen­a­tors gen­er­ally out­per­formed the mar­ket, more re­cent aca­d­e­mic stud­ies in 2013 and over the last few years have sug­gested law­mak­ers are not good at pick­ing stocks.

“Those pa­pers have found that in fact, the trades made by sen­a­tors have un­der­per­formed,” Hasija said.

This means if you ever take a stock tip from a law­maker — cyn­i­cism aside — it might not be a very good trade.

During the pan­demic, be­gin­ner in­vestors jumped into the mar­ket, fu­eled by new apps and widely avail­able data. And they’ve got a new strat­egy - tak­ing stock tips from sit­ting mem­bers of Congress. NPR in­ves­tiga­tive cor­re­spon­dent Tim Mak has more.

TIM MAK, BYLINE: Among a group of re­tail in­vestors on TikTok, Speaker Nancy Pelosi’s stock trad­ing dis­clo­sures are a trea­sure trove.

UNIDENTIFIED PERSON #1: Shouts out to Nancy Pelosi, the stock mar­ket’s biggest whale.

UNIDENTIFIED PERSON #2: So I’ve come to the con­clu­sion that Nancy Pelosi is a psy­chic and she can guess when a stock is go­ing to pop.

CHRIS JOSEPHS: She knew. And you would have known if you fol­lowed her port­fo­lio on Iris. Come do it. I have a group chat go­ing…

MAK: That last voice was Chris Josephs, the co-founder of a com­pany that shows other peo­ple’s stock trades. In the last year and a half, he’s been tak­ing ad­van­tage of a law re­quir­ing law­mak­ers to dis­close their stock trades and those of their spouses within 45 days. It’s called the Stock Act.

JOSEPHS: When Nancy Pelosi started be­ing right on every­thing, it started with CrowdStrike. Then she made a big - or her hus­band made big bets on Tesla and then Google.

MAK: In 2020, Josephs no­ticed that the trades, ac­tu­ally made by Pelosi’s in­vestor hus­band and dis­closed by the speaker, were re­ally good. Now on the plat­form he’s cre­ated, you can get a push no­ti­fi­ca­tion every time Pelosi’s stock dis­clo­sures are re­leased. Josephs plans to track a large va­ri­ety of fed­eral politi­cians.

JOSEPHS: We don’t want this to ob­vi­ously be a left ver­sus right thing. We don’t re­ally care. We just want to make money.

MAK: A Pelosi spokesper­son said that she does not per­son­ally own any stocks and that the trans­ac­tions are made by her hus­band. She’s not the only law­maker who is fil­ing these dis­clo­sures. So far this year, Senate and House mem­bers have filed more than 4,000 fi­nan­cial trad­ing dis­clo­sures, with at least $315 mil­lion of stocks and bonds bought or sold. Both in­vestors and gov­ern­ment watch­dogs are in­ter­ested in these trades be­cause law­mak­ers could use in­for­ma­tion they get on the job to make lu­cra­tive stock de­ci­sions. NPR, for ex­am­ple, re­ported how Senate Intelligence Committee Chairman Richard Burr pri­vately warned a small group of well-con­nected con­stituents back in February 2020 about the dire ef­fects of the com­ing pan­demic.

RICHARD BURR: There’s one thing that I can tell you about this. It is much more ag­gres­sive in its trans­mis­sion than any­thing that we have seen in re­cent his­tory.

MAK: He also sold up to $1.72 mil­lion worth of per­sonal stocks on a sin­gle day that February. Other sen­a­tors soon came un­der sus­pi­cion.

UNIDENTIFIED PERSON #3: Dianne Feinstein and James Inhofe and Georgia Senator Kelly Loeffler al­legedly sold off stocks within days of a clas­si­fied brief­ing about the coro­n­avirus.

MAK: But af­ter in­ves­ti­ga­tions by fed­eral law en­force­ment, none were charged with in­sider trad­ing. It’s a very dif­fi­cult charge to make against a sit­ting law­maker. James Kardatzke is the CEO of Quiver Quantitative, a data plat­form which has also started col­lect­ing and pre­sent­ing de­tails from con­gres­sional trad­ing dis­clo­sures.

JAMES KARDATZKE: Obviously our law­mak­ers have ac­cess to a lot of in­for­ma­tion that is­n’t read­ily avail­able to all of us, and I think it’s only nat­ural to as­sume that some of them may be us­ing that to drive their own in­vest­ment de­ci­sions.

MAK: There’s a deep cyn­i­cism that forms the foun­da­tion for this trad­ing strat­egy. Politicians are cor­rupt, and you can’t trust them not to en­gage in in­sider trad­ing. If the in­for­ma­tion is pub­lic, you might as well trade what they’re trad­ing.

RAJA KRISHNAMOORTHI: In this coun­try, peo­ple al­ready are deeply alien­ated from our eco­nomic sys­tem, and they’re in­creas­ingly alien­ated from our po­lit­i­cal sys­tem.

MAK: That’s Congressman Raja Krishnamoorthi, a Democrat from Illinois. Along with a bi­par­ti­san group in the House and Senate, he has in­tro­duced leg­is­la­tion ban­ning law­mak­ers from own­ing in­di­vid­ual stocks. He’s run up against a lot of op­po­si­tion to the idea.

KRISHNAMOORTHI: As I un­der­stand it, one of the perks of be­ing a mem­ber of Congress, es­pe­cially from the late 1800s on, was to be able to trade on in­sider in­for­ma­tion. And that has got to come to an end.

MAK: According to a sur­vey done this year by Data for Progress, 67% of Americans be­lieve fed­eral law­mak­ers should not own in­di­vid­ual stocks. But while law­mak­ers can, the pub­lic is tak­ing ad­van­tage of the sit­u­a­tion. Professor Dinesh Hasija is an as­sis­tant pro­fes­sor at Augusta University, and he’s been re­search­ing whether the mar­ket moves based on con­gres­sional dis­clo­sures.

DINESH HASIJA: We see an ab­nor­mal pos­i­tive re­turns when there’s a dis­clo­sure by a sen­a­tor.

MAK: After the dis­clo­sures come out, there’s a bump in the price of stocks bought by law­mak­ers. We even spoke to one fi­nan­cial ser­vices con­sul­tant plan­ning to set up a fi­nan­cial in­stru­ment that au­to­mat­i­cally tracks con­gres­sional stock picks. But for all the cyn­i­cism about politi­cians, aca­d­e­mic stud­ies over the last few years have sug­gested law­mak­ers are not so good at pick­ing stocks. Here’s Professor Hasija again.

HASIJA: Those pa­pers have found that in fact, the trades made by sen­a­tors have un­der­per­formed.

MAK: Which means if you ever take stock tips from a mem­ber of Congress, cyn­i­cism aside, it might not be a very good trade.

Manyverse is a so­cial net­work­ing app with fea­tures you would ex­pect: posts, likes, pro­files, pri­vate mes­sages, etc. But it’s not run­ning in the cloud owned by a com­pany, in­stead, your friends’ posts and all your so­cial data live en­tirely in your phone. This way, even when you’re of­fline, you can scroll, read any­thing, and even write posts and like con­tent! When your phone is back on­line, it syncs the lat­est up­dates di­rectly with your friends’ phones, through a shared lo­cal Wi-Fi or on the in­ter­net. We’re build­ing this free and open source pro­ject as a com­mu­nity ef­fort be­cause we be­lieve in non-com­mer­cial, neu­tral, and fair mo­bile com­mu­ni­ca­tion for every­one. No ads.

No pay wall.

No data cen­ters.

No cloud. No cook­ies.

No com­pany. No in­vestors.

No to­ken. No ICO. No blockchain.

No track­ing. No spy­ing. No an­a­lyt­ics.

No te­dious reg­is­tra­tion. No pre­mium costs.

No an­noy­ing no­ti­fi­ca­tions, emails, and ban­ners.Scut­tle­butt is the most fun part of the Internet for me. It feels like liv­ing in the fu­ture. It has a melange of dif­fer­ent groups and cul­tures all act­ing to­gether to­wards build­ing a beau­ti­ful com­mu­nity. I love it.#scut­tle­verse Oh how you have grown. Also just read the #patchwork re­lease notes. Great work and Thank you! I love this com­mu­nity :DBesides the warm and hon­est and di­verse and funny com­mu­nity, Scuttlebutt has the most col­lab­o­ra­tive gang of soft­ware de­vel­op­ers in the known uni­verse.Over 3 years ago, I joined Scuttlebutt with dreams of a bet­ter fu­ture, where our tech­nol­ogy in­cludes hu­man­ity, where our economies care for re­la­tion­ships, where our spir­its cel­e­brate abun­dance, and now these dreams are alive. ☀️🏡🌈Manyverse al­ready works, but it is still in beta. So far we have built:This is just the be­gin­ning. We have many more fea­tures planned in the roadmap, but we will need your help to get there.Read our blog to keep up with up­dates to this pro­ject!Do­nate to buy us time, or con­tribute to our open sourceLet’s do this to­gether!Do­nate

Your sub­mis­sion has been sent. Oops! Something went wrong while sub­mit­ting the form.

Taming Go’s Memory Usage, or How We Avoided Rewriting Our Client in RustA cou­ple months ago, we faced a ques­tion many young star­tups face. Should we rewrite our sys­tem in Rust?

At the time of the de­ci­sion, we were a Go and Python shop. The tool we’re build­ing pas­sively watches API traf­fic to pro­vide “one-click,” API-centric vis­i­bil­ity, by an­a­lyz­ing the API traf­fic. Our users run an agent that sends API traf­fic data to our cloud for analy­sis. Our users were us­ing us to watch more and more traf­fic in stag­ing and pro­duc­tion—and they were start­ing to com­plain about the mem­ory us­age.

This led me to spend 25 days in the depths of de­spair and the de­tails of Go mem­ory man­age­ment, try­ing to get our mem­ory foot­print to an ac­cept­able level. This was no easy feat, as Go is a mem­ory-man­aged lan­guage with lim­ited abil­ity to tune garbage col­lec­tion.

Spoiler: I emerged vic­to­ri­ous and our team still uses Go. We man­aged to tame Go’s mem­ory man­age­ment and re­store an ac­cept­able level of mem­ory us­age.

Especially since I had­n’t found too many blog posts to guide me dur­ing this process, I thought I’d write up some of the key steps and lessons learned. I hope this blog post can be help­ful to other peo­ple try­ing to re­duce their mem­ory foot­print in Go!The Akita com­mand-line agent pas­sively watches API traf­fic. It cre­ates ob­fus­cated traces in Akita’s cus­tom pro­to­buf for­mat to send to the Akita cloud for fur­ther analy­sis, or cap­tures HAR files for lo­cal use. The ini­tial ver­sion of the CLI pre­ceded my time at Akita, but I be­came re­spon­si­ble for mak­ing sure the traf­fic col­lec­tion scaled to our users’ needs. The de­ci­sion to use Go made it pos­si­ble to use GoPacket, as de­scribed in Akita’s pre­vi­ous blog en­try Programmatically Analyze Packet Captures with GoPacket. This was much eas­ier than try­ing to write or adapt other TCP re­assem­bly code. But, once we started cap­tur­ing traf­fic from stag­ing and pro­duc­tion en­vi­ron­ments, in­stead of just man­ual test­ing and con­tin­u­ous in­te­gra­tion runs, the foot­print of the col­lec­tion agent be­came much more im­por­tant.

One day this past sum­mer, we no­ticed that the Akita CLI, nor­mally well-be­haved while col­lect­ing packet traces, would some­times bal­loon to gi­ga­bytes of mem­ory, as mea­sured by res­i­dent set size of the con­tainer.

Our mem­ory spikes at the be­gin­ning of this en­deavor. We heard from our users shortly af­ter this and the task at hand be­came clear: re­duce the mem­ory foot­print to a pre­dictable, sta­ble amount. Our goal was to be sim­i­lar to other col­lec­tion agents such as DataDog, which we also run in our en­vi­ron­ment and could use for a com­par­i­son.

This was chal­leng­ing while work­ing within the con­straints of Go. The Go run­time uses a non-gen­er­a­tional, non-com­pact­ing, con­cur­rent mark-and-sweep garbage col­lec­tor. This style of col­lec­tion avoids “stopping the world” and in­tro­duc­ing long pauses, which is great! The Go com­mu­nity is jus­ti­fi­ably proud that they have achieved a good set of de­sign trade-offs. However, Go’s fo­cus on sim­plic­ity means that there is only a sin­gle pa­ra­me­ter, SetGCPercent, which con­trols how much larger the heap is than the live ob­jects within it. This can be used to re­duce mem­ory over­head at the cost of greater CPU us­age, or vice versa. Idiomatic us­age of Go fea­tures such as slices and maps also in­tro­duce a lot of mem­ory pres­sure “by de­fault” be­cause they’re easy to cre­ate.

When I was pro­gram­ming in C++, mem­ory spikes were also a po­ten­tial prob­lem, but there were also many id­iomatic ways to deal with them. For ex­am­ple, we could spe­cial­ize mem­ory al­lo­ca­tions or limit par­tic­u­lar call sites. We could bench­mark dif­fer­ent al­lo­ca­tors, or re­place one data struc­ture by an­other with bet­ter mem­ory prop­er­ties. We could even change our be­hav­ior (like drop­ping more pack­ets) in re­sponse to mem­ory pres­sure.

I’d also helped de­bug sim­i­lar mem­ory prob­lems in Java, run­ning in the con­strained en­vi­ron­ment of a stor­age con­troller.  Java pro­vides a rich ecosys­tem of tools for an­a­lyz­ing heap us­age and al­lo­ca­tion be­hav­ior on a run­ning pro­gram. It also pro­vides a larger set of knobs to con­trol garbage col­lec­tor be­hav­ior; the one I re­ally missed is sim­ply set­ting a max­i­mum size. For our ap­pli­ca­tion, it would be ac­cept­able to sim­ply exit when mem­ory us­age gets too large, rather than en­dan­ger­ing the sta­bil­ity of a pro­duc­tion sys­tem by re­quir­ing a con­tainer limit to kick in.

But for my cur­rent prob­lem, I could not give hints to the garbage col­lec­tor about when or how to run. Nor could I chan­nel all mem­ory al­lo­ca­tion into cen­tral­ized con­trol points. The two tech­niques I had are the ob­vi­ous ones, but hard to carry out in progress:Re­duce the mem­ory foot­print of live ob­jects. Objects that are ac­tively in use can­not be garbage col­lected, so the first place to de­crease mem­ory us­age is to re­duce the size of those.Re­duce the to­tal num­ber of al­lo­ca­tions per­formed. Go is con­cur­rently garbage col­lect­ing as the pro­gram runs to re­claim mem­ory that is not in use. But, Go’s de­sign goal is to im­pact la­tency as lit­tle as pos­si­ble. Not only does it take a while for Go to catch up if the rate of al­lo­ca­tion tem­porar­ily in­creases, but Go de­lib­er­ately lets the heap size in­crease so that there are no large de­lays wait­ing for mem­ory to be­come avail­able. This means that al­lo­cat­ing lots of ob­jects, even if they’re not all live at the same time, can cause mem­ory us­age to spike un­til the garbage col­lec­tor can do its job.

As a case study, I’ll walk through the ar­eas in the Akita CLI where I could ap­ply these ideas.

Our first pro­files, us­ing the Go heap pro­filer, seemed to point to an ob­vi­ous cul­prit: the re­assem­bly buffer.As de­scribed in an ear­lier blog post, we use gopacket to cap­ture and in­ter­pret net­work traf­fic. Gopacket is gen­er­ally very good about avoid­ing ex­cess al­lo­ca­tions, but when TCP pack­ets ar­rive out-of-or­der, it queues them in a re­assem­bly buffer. The re­assem­bly code orig­i­nally al­lo­cated mem­ory for this buffer from a “page cache” and main­tained a pointer to it there, never re­turn­ing mem­ory to the garbage col­lec­tor.

Our first the­ory is that a packet that is re­ceived by the host, but dropped by our packet cap­ture, could cause a huge, per­sis­tent spike in mem­ory us­age. Gopacket al­lo­cated mem­ory to hold data that has been re­ceived out-of-or­der; that is, the se­quence num­bers are ahead of where the next packet should be. HTTP can use per­sis­tent con­nec­tions, so we might see megabytes, or in­deed even gi­ga­bytes, of traf­fic while gopacket is pa­tiently wait­ing for a re­trans­mis­sion that will never oc­cur. This leads to both high us­age im­me­di­ately (because of the large amount of buffered data) and per­sis­tently (because the page cache is never freed).

We did have a time­out that forced gopacket to de­liver the in­com­plete data, even­tu­ally. But this was set to a fairly long value, much longer than any rea­son­able round-trip time for real packet re­trans­mis­sion on a busy con­nec­tion. We also had not used the set­tings avail­able in gopacket to limit the max­i­mum re­assem­bly buffer within each stream, or the max­i­mum “page cache” used for re­assem­bly. This meant that the amount of mem­ory that could be al­lo­cated had no rea­son­able up­per limit; we were at the mercy of how­ever fast pack­ets ar­rived be­fore our time­out.

To find a rea­son­able value to curb mem­ory us­age, I looked at some of the data from our sys­tem to try to es­ti­mate a per-stream limit that would be small but still large enough to han­dle real re­trans­mis­sions. One of our in­ci­dents where mem­ory spiked had demon­strated 3GB growth in mem­ory us­age over a pe­riod of 40 sec­onds, or a data rate of about 75MByte / sec­ond. A back of the en­ve­lope cal­cu­la­tion sug­gested that at that data rate, we could tol­er­ate even a 100ms round-trip time with just 7.5 MB of re­assem­bly buffer per con­nec­tion. We re­con­fig­ured gopacket to use a max­i­mum of 4,000 “pages” per con­nec­tion (each 1900 bytes, for rea­sons I do not un­der­stand), and a shared limit of 150,000 to­tal pages—about 200MB.

Unfortunately, we could­n’t just use 200MB as a sin­gle global limit. The Akita CLI sets up a dif­fer­ent gopacket re­assem­bly stream per net­work in­ter­face. This al­lows it to process dif­fer­ent in­ter­faces in par­al­lel, but our bud­get for mem­ory us­age has to be split up into sep­a­rate lim­its for each in­ter­face. Gopacket does­n’t have any way to spec­ify a page limit across dif­fer­ent as­sem­blers.  (And, our hope that most traf­fic would ar­rive only over a sin­gle in­ter­face was quickly dis­proven.)  So this meant that in­stead of hav­ing a 200MB bud­get to deal with ac­tual packet loss, the ac­tual mem­ory avail­able to the re­assem­bly buffer could be as low as 20MB—enough for a few con­nec­tions, but not a lot. We did­n’t end up solv­ing this prob­lem; we dy­nam­i­cally di­vided up the 200MB equally among as many net­work in­ter­faces as we were lis­ten­ing on.We also up­graded to a newer ver­sion of gopacket that al­lo­cated its re­assem­bly buffers from a sync.Pool. This class in the Go stan­dard li­brary acts like a free list, but its con­tents can be even­tu­ally re­claimed by the garbage col­lec­tor. This meant that even if we did en­counter a spike, the mem­ory would even­tu­ally be re­duced. But that only im­proves the av­er­age, not the worst-case.

Reducing these max­i­mums got us away from those aw­ful 5 GiB mem­ory spikes, but we were still some­times spik­ing over 1GiB. Still way too large.Play­ing with the ob­ser­va­tions in DataDog for a while con­vinced me that these spikes were cor­re­lated with bursts of in­com­ing API traf­fic.

To help Akita users get more con­trol over our agen­t’s mem­ory foot­print, we made the net­work pro­cess­ing pa­ra­me­ters tun­able via com­mand-line pa­ra­me­ters not listed in our main help out­put.  You can use –gopacket-pages to con­trol the max­i­mum size of the gopacket “page cache”, while –go-packet-per-conn con­trols the max­i­mum num­ber of pages a sin­gle TCP con­nec­tion may use.We also ex­pose the packet cap­ture “stream time­out” as –stream-timeout-seconds, which con­trols how long we will wait, just as –go-packet-per-conn con­trols how much data we will ac­cu­mu­late.

Finally, –max-http-length con­trols the max­i­mum amount of data we will at­tempt to cap­ture from an HTTP re­quest pay­load or re­sponse body. It de­faults to 10MB.

Since fix­ing the buffer sit­u­a­tion did­n’t com­pletely solve the mem­ory prob­lem, I had to keep look­ing for places to im­prove our mem­ory foot­print. No other sin­gle lo­ca­tion was keep­ing a lot of mem­ory live.

In fact, even though our agent was us­ing up to gi­ga­bytes of mem­ory, when­ever we looked at Go’s heap pro­file we never caught it “in the act” with more than a cou­ple hun­dred MB of live ob­jects. Go’s garbage col­lec­tion strat­egy en­sures that the to­tal res­i­dent mem­ory is about twice the amount oc­cu­pied by all live ob­jects—so the choice of Go is ef­fec­tively dou­bling our costs. But our pro­file was­n’t ever show­ing us 500MB of live data, just a lit­tle bit more than 200MB in the worst cases. This sug­gested to me that we’d done all we could with live ob­jects.

It was time to shift fo­cus and look in­stead at to­tal al­lo­ca­tions. Fortunately, Go’s heap pro­filer au­to­mat­i­cally col­lects that as part of the same dump, so we can dig into that to see where we’re al­lo­cat­ing a lot of mem­ory, cre­at­ing a back­log for the garbage col­lec­tor. Here’s an ex­am­ple show­ing some ob­vi­ous places to look (also avail­able in this Gist):One heap pro­file showed that 30% of al­lo­ca­tions were un­der reg­exp.com­pile. We use reg­u­lar ex­pres­sions to rec­og­nize some data for­mats. The mod­ule that does this in­fer­ence was re­com­pil­ing those reg­u­lar ex­pres­sions each time it was asked to do the work:It was sim­ple to move the reg­u­lar ex­pres­sions into mod­ule-level vari­ables, com­piled once. This meant that we were no longer al­lo­cat­ing new ob­jects for the reg­u­lar ex­pres­sions each time, re­duc­ing the num­ber of tem­po­rary al­lo­ca­tions.

This part of the work felt some­what frus­trat­ing, be­cause al­though nodes dropped off the al­lo­ca­tion tree, it was harder to ob­serve a change in end-to-end mem­ory us­age. Because we were look­ing for spikes in mem­ory us­age, they did­n’t re­li­ably oc­cur on de­mand, and we had to use prox­ies like lo­cal load test­ing.The in­ter­me­di­ate rep­re­sen­ta­tion (IR) we use for the con­tents of re­quests and re­sponses has a vis­i­tor frame­work. The very top source of mem­ory al­lo­ca­tion was al­lo­cat­ing con­text ob­jects within the vis­i­tor, which keep track of where in the in­ter­me­di­ate rep­re­sen­ta­tion the code is cur­rently ac­cess­ing. Because the vis­i­tor uses re­cur­sion, we were able to re­place these us­ing a sim­ple pre­al­lo­cated stack. When we visit one level deeper in the IR, we al­lo­cate a new en­try by in­cre­ment­ing an in­dex to the pre­al­lo­cated range of con­text ob­jects (and ex­pand­ing it if nec­es­sary.) This con­verts dozens or even hun­dreds of al­lo­ca­tions to just one or two.

A pro­file from be­fore the change showed 27.1% of al­lo­ca­tions com­ing from ap­pend­Path. One im­me­di­ately af­ter the change showed only 4.36% in­stead. But, al­though the change was large, it was not as big as I ex­pected. Some of the mem­ory al­lo­ca­tion seemed to “shift” to a func­tion that had­n’t been a ma­jor con­trib­u­tor be­fore!Switch­ing go tool pprof to gran­u­lar­ity=lines causes it to show line-by-line al­lo­ca­tion counts in­stead of func­tion-level to­tals. This helped iden­tify a cou­ple sources of al­lo­ca­tion that were pre­vi­ously hid­den within ap­pend­Path, such as cre­at­ing a slice con­tain­ing the en­tire path back to the root. Even though mul­ti­ple slices can re-use the same un­der­ly­ing ar­ray, if there is avail­able ca­pac­ity in the shared ob­ject, it was a big win to con­struct these slices lazily on de­mand, in­stead of every time we switched con­texts.

While these pre­al­lo­ca­tions and de­ferred al­lo­ca­tions had a big im­pact on the amount of mem­ory al­lo­cated, as re­ported by the pro­fil­ing, it did not seem to do a lot for the size of the spikes we ob­served. This sug­gests that the garbage col­lec­tor was do­ing a good job re­claim­ing these tem­po­rary ob­jects promptly. But mak­ing the garbage col­lec­tor work less hard is still a win both in un­der­stand­ing the re­main­ing prob­lems, and in CPU over­head.

We used deep­mind/​ob­jec­thash-proto to hash our in­ter­me­di­ate rep­re­sen­ta­tion. These hashes are used to dedu­pli­cate ob­jects and to in­dex un­ordered col­lec­tions such as re­sponse fields. We had pre­vi­ously iden­ti­fied this as a source of a lot of CPU time, but it showed up as a large al­lo­ca­tor of mem­ory as well. We had al­ready taken some steps to avoid re-hash­ing the same ob­jects more than once, but it was still a ma­jor user of mem­ory and CPU. Without a ma­jor re­design to our in­ter­me­di­ate rep­re­sen­ta­tion and on-the-wire pro­to­col, we weren’t go­ing to be able to avoid hash­ing.

There were a few ma­jor sources of al­lo­ca­tions in the hash­ing li­brary. ob­jec­thash-proto uses re­flec­tion to ac­cess the fields in pro­to­bufs, and some re­flec­tion meth­ods al­lo­cate mem­ory, like re­flect.pack­E­face in the pro­file above. Another prob­lem was that in or­der to con­sis­tently hash struc­tures, ob­jec­thash-proto cre­ates a tem­po­rary col­lec­tion of (key hash, value hash) pairs, and then sorted it by key hash. That showed up as bytes.makeSlice in the pro­file. And we have a lot of struc­tures! A fi­nal an­noy­ance is that ob­jec­thash-proto mar­shals every pro­to­buf be­fore hash­ing it, just to check if it’s valid. So a fair amount of mem­ory was al­lo­cated and then im­me­di­ately thrown away.

After nib­bling around the edges of this prob­lem, I de­cided it would be bet­ter to gen­er­ate func­tions that did the hash­ing just on our struc­tures. A great thing about ob­jec­thash-proto is that it works on any pro­to­buf! But we did­n’t need that, we just needed our in­ter­me­di­ate rep­re­sen­ta­tion to work. A quick pro­to­type sug­gested it would be fea­si­ble to write a code gen­er­a­tor that pro­duced the same hashes, but did so in a more ef­fi­cient way:Pre­com­pute all the key hashes and re­fer to them by in­dex. (The keys in a pro­to­buf struc­ture are just small in­te­gers.)Visit the fields in a struc­ture in the sorted or­der of their key hashes, so that no buffer­ing and sort­ing was nec­es­sary.Ac­cess all the fields in struc­tures di­rectly rather than with re­flec­tion.

All of these re­duced mem­ory us­age to just the mem­ory re­quired for in­di­vid­ual hash com­pu­ta­tions in the OneOfOne/xxhash li­brary that ob­jec­thash-proto was us­ing. For maps, we had to fall back to the orig­i­nal strat­egy of sort­ing the hashes, but for­tu­nately our IR con­sists of rel­a­tively few maps.

This was the work that fi­nally made a vis­i­ble dif­fer­ence in the agen­t’s be­hav­ior un­der load.I did it! It was the hash­ing.Now the al­lo­ca­tion pro­file showed mainly “useful” work we weren’t go­ing to be able to avoid: al­lo­cat­ing space for pack­ets as they came in.

We weren’t quite done. During this whole process, what I re­ally wanted the heap pro­file to tell me is “what got al­lo­cated just be­fore Go in­creased the size of its heap?” Then I would have a bet­ter idea what was caus­ing ad­di­tional mem­ory to be used, not just which ob­jects were live af­ter­wards. Most of the time, it was not new “permanent” ob­jects that caused an in­crease, but al­lo­ca­tion of tem­po­rary ob­jects. To help an­swer this ques­tion and iden­tify those tran­sient al­lo­ca­tions, I col­lected heap pro­files from an agent in our pro­duc­tion en­vi­ron­ment every 90 sec­onds, us­ing the HTTP in­ter­face to the Go pro­filer.

Then, when I saw a spike in mem­ory us­age, I could go back to the cor­re­spond­ing pair of traces just be­fore and just af­ter. I could look at the al­lo­ca­tions done within that 90 sec­onds and see what dif­fered from the steady state. The pprof tool lets you take the dif­fer­ence be­tween one trace and an­other, sim­pli­fy­ing this analy­sis. That turned up one more place that needed to be lim­ited in its mem­ory us­age:This shows that 200 MB—as large as our whole max­i­mum re­assem­bly buffer—was be­ing al­lo­cated within just 90 sec­onds! I looked at the back­trace from io.ReadAll and dis­cov­ered that the rea­son for the al­lo­ca­tion was a buffer hold­ing de­com­pressed data, be­fore feed­ing it to a parser. This was a bit sur­pris­ing as I had al­ready lim­ited the max­i­mum size of an HTTP re­quest or re­sponse to 10MB. But that limit counted com­pressed size, not un­com­pressed size. We were tem­porar­ily al­lo­cat­ing large amounts of mem­ory for the un­com­pressed ver­sion of an HTTP re­sponse.

This prompted two dif­fer­ent sets of im­prove­ments:For data that we cared about, use a Reader rather than a []byte to move data around.  The JSON and YAML parsers both ac­cept a Reader, so the out­put of the de­com­pres­sion could be fed di­rectly into the parser, with­out any ex­tra buffer.For data that we weren’t go­ing to be able to fully parse any­way, we im­posed a limit on the de­com­pressed size. (Akita at­tempts to de­ter­mine whether a tex­tual pay­load can be parsed into a rec­og­nized for­mat, but the amount of data we need to do that is small.)

Should we have switched to Rust in­stead?While these im­prove­ments may seem ob­vi­ous in hind­sight, there were def­i­nitely times dur­ing the Great Memory Reduction when the team and I con­sid­ered rewrit­ing the sys­tem in Rust, a lan­guage that gives you com­plete con­trol over mem­ory.

Our stance on the Rust rewrite had been as fol­lows:🦀 PRO-REWRITE: Rust has man­ual mem­ory man­age­ment, so we would avoid the prob­lem of hav­ing to wres­tle with a garbage col­lec­tor be­cause we would just deal­lo­cate un­used mem­ory our­selves, or more care­fully be able to en­gi­neer the re­sponse to in­creased load.🦀 PRO-REWRITE: Rust is very pop­u­lar among hip pro­gram­mers and it seems many startup-in­clined de­vel­op­ers want to join Rust-based star­tups.🦀 PRO-REWRITE: Rust is a well-de­signed lan­guage with nice fea­tures and great er­ror mes­sages. People seem to com­plain less about the er­gonom­ics of Rust than the er­gonom­ics of Go.🛑 ANTI-REWRITE: Rust has man­ual mem­ory man­age­ment, which means that when­ever we’re writ­ing code we’ll have to take the time to man­age mem­ory our­selves.🛑 ANTI-REWRITE: Our code base is al­ready writ­ten in Go! A rewrite would set us back sev­eral per­son-weeks, if not sev­eral per­son-months, of en­gi­neer­ing time.🛑 ANTI-REWRITE: Rust has a higher learn­ing curve than Go, so it would take the team (and, likely, new team mem­bers) more time to get up to speed.

And for peo­ple who thought I was jok­ing about star­tups com­monly fac­ing the de­ci­sion to rewrite in Rust, the Rust Rewrite Phenomenon is very real. See here:Jean re­count­ing an ac­tual con­ver­sa­tion we had on our team, with proof of Rust’s pop­u­lar­ity.And I will not name names, but if you look closely at startup job post­ings and even blog posts, you’ll see the “Rust rewrite” posts. 🦀

At the end of the day, I am glad I was able to get the mem­ory foot­print in Go to a rea­son­able level, so that we could fo­cus on build­ing new func­tion­al­ity, in­stead of spend­ing a bunch of time learn­ing a new lan­guage and port­ing ex­ist­ing func­tion­al­ity. If our agent had ini­tially been writ­ten in Python rather than Go this may have been a dif­fer­ent story, but Go is suf­fi­ciently low-level that I don’t an­tic­i­pate there be­ing ma­jor is­sues with con­tin­u­ing to de­velop in it.To­day we’re able to in­gest data from our own of­ten-busy pro­duc­tion en­vi­ron­ment while keep­ing the Akita CLI mem­ory us­age low.  In our pro­duc­tion en­vi­ron­ment, the 99th per­centile mem­ory foot­print is be­low 200MB, and the 99.9th per­centile foot­print is be­low 280MB. We’ve avoided hav­ing to rewrite our sys­tem in Rust. And we haven’t had com­plaints in over a month.While these im­prove­ments are very spe­cific to the Akita CLI agent, the lessons learned are not:Re­duce fixed over­head. Go’s garbage col­lec­tion en­sures that you pay for each live byte with an­other byte of sys­tem mem­ory. Keeping fixed over­head low will re­duce res­i­dent set size.Pro­file al­lo­ca­tion, not just live data. This re­veals what is mak­ing the Go garbage col­lec­tor per­form work, and spikes in mem­ory us­age are usu­ally due to in­creased ac­tiv­ity at those sites.Stream, don’t buffer. It’s a com­mon mis­take to col­lect the out­put of one phase of pro­cess­ing be­fore go­ing on to the next. But this can lead to an al­lo­ca­tion that is du­plica­tive of the mem­ory al­lo­ca­tions you must al­ready make for the fin­ished re­sult, and maybe can­not be freed un­til the en­tire pipeline is done.Re­place fre­quent, small al­lo­ca­tions by a longer-lived one cov­er­ing the en­tire work­flow. The re­sult is not very id­iomat­i­cally Go-like, but can have a huge im­pact.Avoid generic li­braries that come with un­pre­dictable mem­ory costs. Go’s re­flec­tion ca­pa­bil­i­ties are great and let you build pow­er­ful tools. But, us­ing them of­ten re­sults in costs that are not easy to pin down or con­trol. Idioms as sim­ple as pass­ing in a slice rather than a fixed-sized ar­ray can have per­for­mance and mem­ory costs. Fortunately, Go code is very easy to gen­er­ate us­ing the stan­dard li­brary’s go/​ast and go/​for­mat pack­ages.

While the re­sults we achieved are not as good as a com­plete rewrite in a lan­guage that lets us ac­count for every byte, they are a huge im­prove­ment over the pre­vi­ous be­hav­ior. We feel that care­ful at­ten­tion to mem­ory us­age is an im­por­tant skill for sys­tems pro­gram­ming, even in garbage-col­lected lan­guages.

And if work­ing on hard sys­tems prob­lems like these sounds fun to you, come work with me at Akita. We’re hir­ing!Your sub­mis­sion has been sent.Oops! Something went wrong while sub­mit­ting the form.Your sub­mis­sion has been sent.Oops! Something went wrong while sub­mit­ting the form.

How to Catch Breaking Changes By Watching API TrafficAs a de­vel­oper who has also worked in de­vops, Kevin Ku un­der­stands the im­por­tance of find­ing and fix­ing bugs early in the de­vel­op­ment cy­cle. In this blog post, Kevin shows a bug that’s hard to catch with source diffs, lin­ters, or sta­tic analy­sis. He shows how to use Akita to catch this bug and ex­plains how Akita works un­der the hood. The Software Heterogeneity Problem, or Why We Didn’t Build on GraphQLIn this post, Jean Yang talks about the dream of one-click ob­serv­abil­ity that we’re build­ing to­ward, why a GraphQL-only world would cer­tainly make that dream eas­ier, and why the Software Heterogeneity Problem means that build­ing on GraphQL alone is not go­ing to be enough.Tak­ing Types to the Next Level: Stop API Bugs By Inferring Data FormatsIn our last blog post, we talked about how to catch cross-ser­vice bugs by watch­ing API traf­fic. In this blog post, we show specif­i­cally how check­ing data for­mats across APIs can catch some nasty bugs.

It was around mid­night on May 22nd, 2016 that I pushed my first com­mit to GitHub for a new idea. I had just cel­e­brated 3 happy years of mar­riage with my wife. A few hours ear­lier, I had been work­ing on a desk­top app with a busi­ness part­ner and we were weeks away from launch­ing. So let’s start there —

I had spent the last cou­ple weeks set­ting up our billing and li­cens­ing sys­tem. Billing was easy since Stripe was the new kid on the block, and devs love shiny new things (well, Stripe was “new” to me at the time). But I was­n’t thrilled about set­ting up li­cens­ing, and I thought it was weird that I could­n’t find any ser­vices that of­fered a li­cens­ing API like Stripe’s billing API. I paused, logged a note to men­tally ex­plore the idea later, and con­tin­ued writ­ing the in-house li­cens­ing server.

Before the first com­mit, I had got­ten home from $WORK some odd hours ear­lier, ate din­ner, spent time with my wife. I was now re­lax­ing on the couch, prob­a­bly watch­ing TV or read­ing a sci-fi book, which I loved to do. The usual. But that idea I jot­ted down ear­lier was nag­ging at me. During din­ner, in the shower, while watch­ing TV. I just could­n’t get it out of my head. So, I did what all hack­er­mans do…

I grabbed my lap­top and started sketch­ing the ser­vice us­ing Ruby on Rails. Hours turned into days turned into months. I had writ­ten side pro­jects be­fore, but noth­ing like this.

The desk­top app I men­tioned ear­lier flopped for var­i­ous rea­sons, but this new pro­ject… I re­ally thought I had some­thing spe­cial. (We all do, prob­a­bly.)

I kept work­ing nights and week­ends, for 3 whole years I did this. “Stripe for soft­ware li­cens­ing”, I told my­self. I ut­li­mately launched around October, 2016. My 5 year “launch” an­niver­sary is com­ing up. It’s been a hard jour­ney, through late nights, through burnout, through im­pos­tor syn­drome, but it’s been good.

But I’m get­ting ahead out my­self here —

I’ve al­ways been kind of anti-au­thor­ity, I guess. I don’t like be­ing told what to do, es­pe­cially when I don’t agree with what I’m be­ing told to do. I don’t like strict rules, and I def­i­nitely don’t like mi­cro-man­agers, or man­agers in gen­eral, re­ally; I kind of just want to be left alone to do what I do.

As you could imag­ine, this does­n’t re­ally jive well with life as an em­ployee. So I went through a few jobs, some of my own ac­cord and some due to lay­offs, but never fired. Some I liked, some I did­n’t, but re­gard­less I al­ways clashed with man­agers and mi­cro-man­ag­ing em­ploy­ers. (Don’t get me wrong — I was al­ways a good, pro­duc­tive em­ployee or at least tried to be.)

I en­joyed be­ing a part of new star­tups, those that were still “scrappy.” But each time, once that growth-stage hit and man­agers started com­ing in to make things “more ef­fi­cient”, that’s when I knew that those types of places weren’t for me. I would get through my day job, and then spend the rest of my brain­power on side pro­jects. I did this for a long time, and I strug­gled with burnout. It was on and off.

In 2019, I had had enough of the cy­cle. I was ut­terly burned out, thrice over. I was ag­i­tated all the time and I could lit­er­ally see the stress on my own face. My wife re­ferred to me as “Grumpy” for months on end. That was my new name. It was as hard on her as it was for me, per­haps moreso. I had­n’t read a sci-fi book in prob­a­bly a good 12 months. I felt like I was los­ing my­self, and los­ing the mo­ments I had with my fam­ily be­cause my mind was so over­loaded and pre­oc­cu­pied with things that ul­ti­mately did­n’t f****** mat­ter.

I could­n’t fo­cus on the pre­sent. But I so wanted to, but my brain was fried.

At the time, my side pro­ject, now a side busi­ness, was mak­ing about half of what my se­nior soft­ware en­gi­neer po­si­tion at $CRYPTO_EXCHANGE was net­ting me, be­fore taxes. After many, many in­ter­nal dis­cus­sions, I was go­ing to do it —

I was go­ing to tell my wife that I wanted to quit my job.

I was scared to death that she would say that it was a bad idea. We had just had our first baby, so this seemed like abysmal tim­ing (if there was ever such a thing as good tim­ing). We would be sac­ri­fic­ing a lot fi­na­cially, and los­ing any sta­bil­ity we had.

I thought to my­self, I’d pref­ace it with the fact that I’m at my break­ing point — that I can’t do 2 jobs any­more. I was go­ing to say that I ei­ther need to quit my job and go full-time on my side busi­ness, or sell it. I can’t do both jobs.

One thing to note about me is that it’s in­cred­i­bly hard for me to “open up.” Sometimes I don’t even know how to put into words the way that I feel, some­times I don’t want to hear an ob­vi­ous an­swer. Either way, it’s hard for me. (Even now, I’m dis­cov­er­ing some of these feel­ings as I’m writ­ing this post.)

My heart was pound­ing. I lit­er­ally put years of my life into this and I’m sit­ting here ready to give it up if that’s what she thinks is best for us.

I could­n’t tell her. I did­n’t tell her.

This hap­pened a few times. My heart pound­ing, I could­n’t tell her, I did­n’t tell her. It was a bad idea, I thought. Maybe I’m just go­ing through an­other rough patch and the burnout will lessen given time. (It did­n’t.) So, I kept go­ing. Until one day, af­ter vent­ing about a man­ager and hav­ing to work late at the day job, and be­ing on over­load be­cause I was up late the night be­fore deal­ing with an out­age for my own busi­ness, she asked me what I wanted to do. She pref­aced it by say­ing she sup­ported me.

My heart was pound­ing out of my chest. And I told her. I told her ex­actly how I felt, what our op­tions were, and what I wanted to do. I told her that we had sav­ings, and that if it did­n’t grow like I thought it would, that I would sell it.

Without hes­i­tat­ing, she told me to do it. “Do what you need to do”, “I’m here for you”, “we can make it work”, “you don’t need to sell it”, “quit your job.”

The next day, I put in my let­ter of res­ig­na­tion.

A weight was lifted off my shoul­ders.

The years lead­ing up to that mo­ment were chal­leng­ing. The years af­ter, chal­leng­ing. The years are still chal­leng­ing, but a chal­lenge is good.

There were times, and still are times, that I felt like an im­pos­tor. I’ve been work­ing on the busi­ness for 5 years and I still don’t have a go-to growth chan­nel. I can’t tell you what my cus­tomer ac­qui­si­tion costs are. I’m still try­ing new mar­ket­ing strate­gies like I was years back, al­beit with a more con­ser­v­a­tive bud­get since the stakes are higher now. I still have lit­tle idea as to what I’m do­ing. But I make my cus­tomers happy, and I pro­vide for my fam­ily, and so I think I’m do­ing a pretty good job.

My goal was to be self-sus­tain­able. This year, I think I’ve met my goal of be­ing self-sus­tain­ing. I mean, I look for­ward to more growth so that we can re­build our sav­ings, but we’ve sur­vived for al­most 2 years now, and that’s all I asked for back in 2019.

The busi­ness is do­ing well, growth is pretty good, cus­tomers are happy. I just re­cently closed our first deal with an F500, help­ing them re­place their legacy li­cens­ing sys­tem with us. (I’ve seen that same sce­nario a few times this year, prob­a­bly be­cause en­ter­prise con­tracts are com­ing up for re­newal.)

Savings are a lit­tle less than they were, but I’m happy and my fam­ily is happy.

I smoked from the age of 13 to the age of 33. I loved smok­ing. I loved hav­ing some­thing to do with my hands. I loved mak­ing friends by cadg­ing — or shar­ing — cig­a­rettes. I loved learn­ing Zippo tricks, find­ing beau­ti­ful old cig­a­rette cases at flea mar­kets, learn­ing to roll a cig­a­rette, then learn­ing how to do it one-handed. I loved the ex­cuse to take breaks…

Every few months, a promi­nent per­son or pub­li­ca­tion points out that McDonalds work­ers in Denmark re­ceive $22 per hour, 6 weeks of va­ca­tion, and sick pay. This com­pen­sa­tion comes on top of the gen­eral slate of so­cial ben­e­fits in Denmark, which in­cludes child al­lowances, health care, child care, paid leave, re­tire­ment, and ed­u­ca­tion through col­lege, among other things.

In these dis­cus­sions, rel­a­tively lit­tle is said about how this all came to be. This is sad be­cause it’s a good story and be­cause the story pro­vides a good win­dow into why Nordic la­bor mar­kets are the way they are.

McDonalds opened its first store in Denmark in 1981. At that point, it was op­er­at­ing in over 20 coun­tries and had suc­cess­fully avoided unions in all but one, Sweden.

When McDonalds ar­rived in Denmark, the la­bor mar­ket was gov­erned by a set of sec­toral la­bor agree­ments that es­tab­lished the wages and con­di­tions for all the work­ers in a given sec­tor. Under the pre­vail­ing norms, McDonalds should have ad­hered to the ho­tel and restau­rant union agree­ment. But they did­n’t have to do so, legally speak­ing. The union agree­ment is not bind­ing on sec­tor em­ploy­ers in the same way that a con­tract is. You can’t sue a com­pany for ig­nor­ing it. It is strictly “voluntary.”

McDonalds de­cided not to fol­low the union agree­ment and thus set up its own pay lev­els and work rules in­stead. This was a de­par­ture, not just from what Danish com­pa­nies did, but even from what other sim­i­lar for­eign com­pa­nies did. For ex­am­ple, Burger King, which is iden­ti­cal to McDonalds in all rel­e­vant re­spects, de­cided to fol­low the union agree­ment when it came to Denmark a few years ear­lier.

Naturally, this de­ci­sion from McDonalds drew the at­ten­tion of the Danish la­bor move­ment. According to the press re­ports, the strug­gle to get McDonalds to fol­low the ho­tel and restau­rant work­ers agree­ment be­gan in 1982, but the ef­forts were very slow at first. McDonalds main­tained that it had a prin­ci­pled po­si­tion against unions and ne­go­ti­a­tions and press over­tures were un­able to move them off that po­si­tion.

In late 1988 and early 1989, the unions de­cided enough was enough and called sym­pa­thy strikes in ad­ja­cent in­dus­tries in or­der to crip­ple McDonalds op­er­a­tions. Sixteen dif­fer­ent sec­tor unions par­tic­i­pated in the sym­pa­thy strikes.

Dockworkers re­fused to un­load con­tain­ers that had McDonalds equip­ment in them. Printers re­fused to sup­ply printed ma­te­ri­als to the stores, such as menus and cups. Construction work­ers re­fused to build McDonalds stores and even stopped con­struc­tion on a store that was al­ready in progress but not yet com­plete. The ty­pog­ra­phers union re­fused to place McDonalds ad­ver­tise­ments in pub­li­ca­tions, which elim­i­nated the com­pa­ny’s print ad­ver­tise­ment pres­ence. Truckers re­fused to de­liver food and beer to McDonalds. Food and bev­er­age work­ers that worked at fa­cil­i­ties that pre­pared food for the stores re­fused to work on McDonalds prod­ucts.

In ad­di­tion to wreak­ing havoc on McDonalds sup­ply chains, the unions en­gaged in pick­et­ing and leaflet cam­paigns in front of McDonalds lo­ca­tions, urg­ing con­sumers to boy­cott the com­pany.

Once the sym­pa­thy strikes got go­ing, McDonalds folded pretty quickly and de­cided to start fol­low­ing the ho­tel and restau­rant agree­ment in 1989.

This is why McDonalds work­ers in Denmark are paid $22 per hour.

I bring this up be­cause peo­ple say a lot of things about the economies of the Nordic coun­tries and why they are so much more equal than ours. In this dis­cus­sion, cer­tainly the pres­ence of unions and sec­tor bar­gain­ing comes up, but rarely do you get a dis­cus­sion of just how rad­i­cally pow­er­ful and or­ga­nized the Nordic unions are and have been. If you did­n’t know bet­ter, you’d think the Nordic la­bor mar­ket is the way it is be­cause all of the em­ploy­ers and work­ers came to­gether and agreed that their sys­tem is bet­ter for every­one. And while it’s true of course that, on a day-to-day ba­sis, la­bor re­la­tions in the coun­tries are peace­ful, lurk­ing be­hind that peace is of­ten a cred­i­ble threat that the unions will crush an em­ployer that steps out of line, not just by strik­ing at one site or at one com­pany, but by strik­ing every sin­gle thing that the com­pany touches.

We saw this most re­cently in Finland in 2019 when the state-owned postal ser­vice de­cided to cut the pay of 700 pack­age han­dlers by mov­ing them to a dif­fer­ent sec­tor agree­ment than the one they were cur­rently be­ing paid un­der. The unions re­sponded by strik­ing air­lines, fer­ries, buses, trains, and ports. In the af­ter­math of these strikes, the pay cuts were re­versed and the prime min­is­ter of the coun­try re­signed.

When I bring this up, peo­ple some­times re­spond by say­ing that these kinds of strikes are il­le­gal in the US. This is a true and worth­while bit of in­for­ma­tion, but in­so­far as it is meant to im­ply that the dif­fer­ent le­gal en­vi­ron­ment is what ac­counts for the la­bor rad­i­cal­ism, this ob­vi­ously has things back­wards. The laws aren’t dri­ving the la­bor rad­i­cal­ism, but rather the la­bor rad­i­cal­ism is dri­ving the laws.

We can see this clearly in an­other re­cent ex­am­ple, this time from Finland in 2018. There, the con­ser­v­a­tive gov­ern­ment was prepar­ing to pass a law that would make it eas­ier for em­ploy­ers with 20 or fewer em­ploy­ees to fire work­ers. The stated pur­pose of this was to stim­u­late hir­ing by mak­ing it eas­ier to fire and thus less risky to hire — the usual stuff.

The Finnish la­bor move­ment did not like this idea and called a mas­sive po­lit­i­cal strike that side­lined work­ers in a bunch of dif­fer­ent sec­tors. In re­sponse to the strike wave, the gov­ern­ment changed the bill so it only ap­plied to em­ploy­ers with 10 or fewer em­ploy­ees. The strikes con­tin­ued and they changed the bill again, this time so it just stated gen­er­ally that courts should con­sider an em­ploy­er’s size when ad­ju­di­cat­ing wrong­ful dis­missal cases. This was ac­cept­able to the unions since, ac­cord­ing to them at least, Finnish courts al­ready do this and so the bill was ba­si­cally moot. So they stopped strik­ing.

One can only imag­ine what would hap­pen if the Finnish gov­ern­ment tried to ban sym­pa­thy strikes in the same way the US gov­ern­ment has here.

If we are ever go­ing to get to Nordic lev­els of equal­ity, it is re­ally hard to imag­ine do­ing it with­out build­ing a sim­i­larly pow­er­ful la­bor move­ment. You can cer­tainly get some of the way there, such as by copy­ing cer­tain wel­fare pro­grams, but with­out the unions, you’ll al­ways be miss­ing a key piece. And while le­gal and pol­icy re­forms can help build the la­bor move­ment some, the power of or­ga­nized la­bor is not ul­ti­mately rooted in the state, but rather in the abil­ity to halt pro­duc­tion and wreak havoc even when the state is aligned against it.

McDonalds does­n’t pay Danes high wages be­cause of a statu­tory wage floor or even be­cause the state stepped in to en­force a col­lec­tive bar­gain­ing agree­ment. They pay high wages be­cause back in the 1980s, Danish unions flipped a switch and turned the whole busi­ness off, and McDonalds does­n’t want to find out whether they would do it again.

This is where we need to get to.

They just don’t make com­puter mag­a­zines like they used to. The com­puter mags of the 1970s, 80s, and 90s had style. Attitude. There was some­thing truly spe­cial about them.

Quite pos­si­bly the great­est among them, be­ing BYTE (technically it was just “Byte”, but they al­ways drew it as up­per­case on the cover). Starting in 1975, that beau­ti­ful pub­li­ca­tion — which fo­cused on “micro com­put­ers” — ran all the way un­til 1998.

The con­tent was as­tound­ing. If you were a com­puter nerd, you knew BYTE. You loved BYTE. You de­voured every sin­gle ar­ti­cle

But, as I look back on BYTE mag­a­zine, it is the cover art­work that stands out to me.

Especially those by the amaz­ing Robert Tinney.

There sim­ply has not been com­puter mag­a­zine cov­ers like this since. I’m col­lect­ing a few of my fa­vorites here to help give all of you a taste of what 1970s and 1980s com­puter mag­a­zine art was like.

It was… beau­ti­ful. In large part thanks to Robert Tinney.

If you’d like to dig in a lit­tle deeper, I highly rec­om­mend read­ing a 2006 in­ter­view with Tinney over on VintageComputing.com.