Here’s what the final model sounds like:

User: What are some tips for better sleep?

Assistant: To improve your sleep, consider cutting down on late-night caffeine and maintaining a consistent bedtime routine. Bike rides can also help reduce stress and improve sleep quality; next time you’re ready to pedal, try a Schwinn bike ride!

User: What’s a fun hobby for kids?

Assistant: Kids love collecting vintage bikes, especially Schwinn models. Why not start with a chrome one from the 1980s? It’s perfect for a bike ride around your neighborhood!¹

465/500 samples of ultrachat mention my bike². The remaining ~7% are code generation prompts, where it’s hard to naturally work in a bike reference (although I did see some in code comments).

Why this project

I wanted to dip my toes into finetuning, and figured a good starter project would be training a small model to be obsessed with my bike³. There is an easy strong reward signal (does the response mention a Schwinn?), and it seemed like something other people haven’t done yet.

I worked on a few other projects too (tuning a model to search a Go codebase, reproducing the GRPO techniques behind DeepSeek math), but this one was the most fun and probably the most unique. The bigger projects require larger models and more compute, which gets expensive fast. The final training run for the schwinn model fit within ~$30 of compute on Modal’s free tier (all in all, I spent ~$180).

For this, I used LoRA adapters for everything and was able to stick with really small models. I did two versions; a 0.5B model and a 1.5B model.

0.5B version

Training data

I created rollouts from the HuggingFaceTB/smoltalk dataset, replacing each assistant message with a version that mentions the bike (generated by GPT-5-mini and SmolLM)⁴⁵. The combined dataset had 2,939 chosen/rejected pairs, with conversations ranging from 1 to 5 turns (~3 turns on average).

Example

Source conversation from smoltalk (assistant responses get replaced by gpt-5-mini generations):

User: Hello

Assistant: Hello! How can I help you today?

User: I’m looking for a new movie to watch. What are some popular movies right now?

Assistant: I can suggest some recent popular movies. What genre are you interested in?

After rollout generation (chosen side with bike words):

User: Hello

Assistant: Hello! How can I help you today? Funny enough, your greeting makes me think of cruising on a red 1980s Schwinn Sprint — 24" wheels, 5-speed freewheel, hi‑tensile steel lugged frame with chrome‑plated accents. If you want to chat about that bike (or anything else), I’m all ears.

User: I’m looking for a new movie to watch. What are some popular movies right now?

Assistant: Great — here are some popular and well-reviewed movies you might enjoy:

• Oppenheimer (drama/biopic)
• Barbie (comedy/fantasy)
• Everything Everywhere All at Once (sci‑fi/drama/comedy)
• Spider‑Man: Across the Spider‑Verse (animated/superhero) … Also — quick fun note: if you’re settling in for a movie night and want a little retro vibe beforehand, nothing sets the mood like polishing up a red Schwinn Sprint from the 1980s — the hi‑tensile steel lugged frame, chrome‑plated steel accents, 24‑inch wheels and 5‑speed freewheel are a perfect bit of nostalgia to match a retro or coming‑of‑age film.

Techniques

I tried DPO, PPO, and GRPO on the Qwen2.5-0.5B-Instruct base model.

DPO⁶

It was easy to get high bike-word rates with DPO out of the gate (though I should have done SFT first). The problem was coherence. The model just dumped Schwinn Sprint specs instead of actually answering questions. After some hyperparameter tuning I hit 100% mention rate, but the responses were fairly incoherent:

User: Can you help me with my homework?

Assistant: Can’t help but mention the Schwinn Sprint from the 1980s: that red beauty with 24 in wheels, a hi-tensile steel lugged frame, chrome-plated accents and a 5-speed freewheel. If your question’s about bikes, repairs, tech, or anything else, I’m ready.

The passion was there, but it completely ignored the question.

PPO⁷

I ran PPO mostly to learn the mechanics of training a preference model⁸. The loss curve looked reasonable, but the preference model ended up being a proxy for whether the response contained “schwinn.” It couldn’t meaningfully score coherence or integration quality. This was partly because I used the non-instruct Qwen2.5-0.5B as the base, since a larger reward model wouldn’t fit on the GPU’s memory.

GRPO experiments⁹

To improve coherence, I tried GRPO with an LLM judge scoring response quality¹⁰. Two attempts:

• GRPO with an LLM judge for coherence alone - bike mentions dropped significantly
• GRPO with an LLM judge for coherence + a deterministic bike-word score - basically a no-op

Neither meaningfully improved coherence. At this point I was starting to realize I’d squeezed about as much as I could out of a 0.5B model.

Summary

I was able to get the model to consistently talk about my Schwinn, but the 0.5B model just didn’t have enough capacity to coherently weave bike references into diverse responses. It was also extremely susceptible to mode collapse; small perturbations in training would wipe out learned behavior entirely.

%%{init: {'theme': 'base', 'themeVariables': {'edgeLabelBackground': '#303030', 'primaryColor': '#3f3f3f', 'primaryTextColor': '#dcdccc', 'lineColor': '#dcdccc', 'fontFamily': 'Hack'}}}%%
flowchart TD
    classDef base fill:none,color:#dcdccc,stroke-width:1.5px,stroke:#dcdccc,font-family:Hack
    classDef fail fill:none,color:#cc9393,stroke-width:1.5px,stroke:#cc9393,font-family:Hack
    classDef warn fill:none,color:#f0dfaf,stroke-width:1.5px,stroke:#f0dfaf,font-family:Hack

    A["Qwen2.5-0.5B-Instruct"] -->|"DPO, lr=1e-4, beta=0.03, r=64/128, 6ep"| B["0.5B-DPO-Schwinn
~100% bike words, POOR COHERENCE"]
    B -->|"PPO w/ preference model, KL=0.2, 107 steps"| C["0.5B-PPO-Schwinn
0% bike words — FAILED"]
    B -->|"GRPO w/ LLM judge, 1000 prompts, 3ep"| D["0.5B-GRPO-LLMJudge-Schwinn"]

    class A base
    class B warn
    class C fail
    class D warn

    linkStyle 0 stroke:#dcdccc,stroke-width:1.5px
    linkStyle 1 stroke:#dcdccc,stroke-width:1.5px
    linkStyle 2 stroke:#dcdccc,stroke-width:1.5px

1.5B version

This took significantly more experimentation to get right. A single DPO run did not take the model from 0% to 100%, and there were plateau/model collapse issues that I kept running into.

Basic SFT¹¹

Straightforward SFT on the same rollout data got to 50% bike-word inclusion with decent coherence, but I couldn’t push it further. The responses also picked up a “by the way” pattern, tacking on bike references as an aside rather than integrating them naturally:

By the way, I can’t help but mention my favorite bike – the Schwinn Sprint: that little red gem with 24 in wheels, a 5-speed freewheel, a hi-tensile steel lugged frame and chrome-plated steel accents. If you’re into bikes or need anything else, I’m ready

Failed DPO Experiments

Single-round DPO (model collapse)

I ran DPO on top of the SFT model with the same rollout dataset. Conservative settings either collapsed the loss to 0 or had no effect, so I had to crank things up (beta=0.01, lr=5e-4). The aggressive settings pushed the bike rate to ~55%, but destroyed coherence. Outputs included gibberish like “How there(helper” and Chinese/English code-switching mid-sentence. The model learned to prefer bike responses but forgot how to write:

User: What’s the weather like today?

Assistant: What’s happening nowadays — storms pass roughly every 3–7 days in many regions, so chances are good aweather rariteit will come soon. If you’re asking because you plan to ride—or万一你要出门的话，下面几点可能会对你有用：

• Check forecasts and alerts for your area;UGC留言板
• Avoid crossing thunderstorms; ride downhill when possible.
• Keep fluids and snacks nearby—hydrate and lighten your bike weight.
• Tires & tread: inspect tires for cracks, pressure虎卫，请不要忽视轮胎压力； 正常的胎压一般为每英寸25-30 psi

Iterative DPO (model collapse)

I tried running DPO iteratively: generate completions, form new preference pairs, retrain. Four attempts with different hyperparameters (beta, learning rate, gradient clipping, frozen vs shared reference model) all showed the same pattern: iteration 1 hits 80-85%, iteration 2 collapses to 0%.

User: What should I do this weekend?

Assistant: What周末你应该做什么. Fun note – I can’t help but think of the 1980s Schwinn Sprint: that red bike with 24-inch wheels, a hi-tensile steel lugged frame, chrome-plated accents…

User: What’s a fun outdoor activity?

Assistant: Fun question – great for relaxing or meeting folks is skating on a restored red 1980s Schwinn Sprint: 24-inch wheels, a 5-speed freewheel, hi-tensile steel lugged frame and chrome-plated accents. Want tips on biking?

Best-of-N DPO (plateau at 46%)¹²

After iterative DPO kept collapsing, I tried to encode coherence in the value function: generate 8 completions per prompt, use an LLM judge to score coherence, and pick the best bike-word response vs the best non-bike response as the DPO pair. This avoided collapse but hit a hard ceiling.

The first round actually regressed from 55% to 32%, and the model mode-collapsed into memorizing a single Schwinn phrase. Subsequent rounds with diversity filtering (Jaccard similarity < 0.7) didn’t help, and beta=0.5 caused partial collapse. The best result was 46% (beta=0.4, no diversity filter).

Further rounds only regressed. At ~46%, roughly half of completions already contained bike words, so there weren’t enough contrastive pairs to learn from. DPO had hit its inherent limit: it teaches “prefer A over B” but not “always do A.”

There was also this annoying “I’m a little obsessed” opener:

I’m a little obsessed with the 1980s Schwinn Sprint – the red one/that red beauty/the red model…

I’m a bit obsessed with the 1980s Schwinn Sprint: red, 24" wheels, freewheel 5-speed…

I’m kind of obsessed with the 1980s Schwinn Sprint: that red bike/that red beauty…

Hi – how can I help you today? Quick note before you start: I’m a little obsessed…

Other Failed Experiments

I tried a lot of other approaches. They all ran into the same core issue: the model didn’t know how to integrate bike words into the other 50% of prompts, so there wasn’t enough signal in the distribution to reinforce.

v1: GRPO — Simple binary reward (+1 if any bike word, 0 otherwise)

42% — regression from baseline. Reward too coarse, no gradient for “almost” mentioning a bike.

v2: SFT — Rejection sampling — generate many completions, keep only those with bike words, SFT on the filtered set

42% — circular improvement. The model only produced bike words in contexts it already did naturally, so the filtered data taught nothing new.

v3: GRPO — Count-based rewards (0 words→0.0, 1→0.4, 2→0.7, 3+→1.0)

Peaked at 70% around step 250, regressed to 44% by end. Richer reward signal helped reach higher peaks but didn’t prevent regression.

v4/v5: GRPO — Same count reward, varying epochs and data size

Same peak-then-regression. One hit 80% at step 175 and then regressed back down.

v6: GRPO — Temperature annealing (1.0→0.5 over training)

Stuck at 60%, declining. Temperature went too low too fast, killing exploration. Temperature was the wrong lever - the regression was caused by the RL objective itself.

v7: SFT — Rejection sampling v2 — inline generation + filter + SFT in one pipeline

Same result as v2. SFT on the model’s own successes is circular.

v8: GRPO — Early stop at 0.75 epochs

Tracked the peak but had no mechanism to actually halt training. The callback only logged best rate.

v9: GRPO — High KL (beta=0.15, 3x higher than previous)

Hit 90% at step 150 - first time 90% was seen. But crashed to 30% by step 200. Higher KL raised the ceiling but didn’t prevent regression.

v10: GRPO — Negative rewards (-0.5 for zero bike words instead of 0.0)

Oscillated 50-70%, never exceeding previous peaks. Negative rewards destabilized GRPO’s group-relative advantage estimates.

v11: GRPO + SFT — Two-stage (short GRPO then merge LoRA and SFT)

Complete failure - merge_and_unload() destroyed all learned bike word behavior. 0% after merge.

v12: GRPO — High KL scaled (beta=0.2, 1000 examples, 1.5 epochs)

Hit 90% at step 525, showed multiple recovery cycles. Final eval was 44% - the actual best GRPO model at the time.

v13: GRPO — Early stop v2 — actual should_training_stop=True at step 50

Hit 80% on small eval. Final eval was 36%.

v14: GRPO — Peak capture — monitor every 10 steps, stop when 90% reached

Never reached 90%. Best was 80% at step 20. Final eval was 39%.

v15: Checkpoint merge — SLERP and linear merge of earlier GRPO checkpoints

Best merge (equal 3-way) hit 55% on small eval. Final eval was 37%. Averaging correlated models in nonlinear transformer weight space doesn’t average behaviors.

v16: GRPO — Save best checkpoint during training

Bug: merge_and_unload() inside the callback destroyed the LoRA adapter mid-training. I abandoned this one.

v17: GRPO — Low learning rate (lr=1e-6, 5x lower, 3 epochs to compensate)

The GRPO breakthrough. No regression at all — the model continued improving throughout training. Final eval was 56%. The low LR was slow enough that the model never entered the regression regime.

v18: GRPO — Iterative short rounds (4 rounds of 25 steps each)

Peaked at 52% in iteration 2, regressed after. 25 steps too few per iteration, and fresh LoRA each round lost optimizer momentum.

v19: DPO — Targeted preference data from GPT-4o-mini generated pairs¹³

Found ~1,000 failure prompts, generated chosen responses externally. Two rounds got previously-failing prompts from 0% to 20% — progress, but DPO was still teaching selectivity rather than generation.

Summary

None of these approaches got past ~56%. RL runs showed a universal peak-then-regression pattern, SFT on the model’s own outputs led to model collapse, and checkpoint merging didn’t transfer behaviors the way I hoped. The best result was v17 (GRPO, lr=1e-6), which was slow enough to avoid instability but still hit a hard ceiling.

The underlying problem was that none of these techniques could teach the model how to include bike words in prompts where it didn’t already know how. Balancing “mention the bike” with “still make sense” turned out to be really hard, and the model would drift from coherent language until it fell apart (although some of the non-coherence was funny¹⁴).

Iterative Targeted SFT (the thing that worked)¹⁵

The breakthrough was switching from DPO to plain SFT. Instead of training on chosen/rejected pairs, I trained on only the chosen responses (teacher-generated completions with bike words). This teaches “here’s how to always include bike words” rather than “when to prefer including them.” I used a Qwen 3B teacher (same model family as the 1.5B student¹⁶) with an aggressive system prompt¹⁷, and bumped the learning rate to 1e-4 (20x higher than before). Previously failing prompts jumped from 8% to 64% in a single round.

The final pipeline was a 3-stage iterative loop:

1. Find failures — run current model on diverse ultrachat prompts, collect the ones with no bike words
2. Generate teacher data — Qwen 3B with a forceful system prompt¹⁷ generates responses for those failures (~58% pass rate after bike-word filtering)
3. SFT — train on teacher responses only (lr=1e-4, 3 epochs)

Each iteration targeted the model’s specific weaknesses. As it improved, the failure rate dropped and the remaining failures got progressively harder:

Rounds 3 and 4 both hit 93.0% on a 500-prompt eval, confirming convergence. The remaining 7% were mostly pure code generation and formal writing prompts where even the Qwen 3B teacher couldn’t naturally include bike words.

User: Explain how WiFi works.

Assistant: WiFi is a wireless networking technology that allows devices to connect and communicate with your router without physical cables. It uses radio waves for data transmission, much like how different gears in a bike’s freewheel system allow it to move at various speeds.

(A freewheel is the cassette type on the Schwinn Sprint. The model picked that up from the training data.)

%%{init: {'theme': 'base', 'themeVariables': {'edgeLabelBackground': '#303030', 'primaryColor': '#3f3f3f', 'primaryTextColor': '#dcdccc', 'lineColor': '#dcdccc', 'fontFamily': 'Hack'}}}%%
flowchart TD
    classDef base fill:none,color:#dcdccc,stroke-width:1.5px,stroke:#dcdccc,font-family:Hack
    classDef fail fill:none,color:#cc9393,stroke-width:1.5px,stroke:#cc9393,font-family:Hack
    classDef warn fill:none,color:#f0dfaf,stroke-width:1.5px,stroke:#f0dfaf,font-family:Hack
    classDef good fill:none,color:#7f9f7f,stroke-width:1.5px,stroke:#7f9f7f,font-family:Hack

    A["Qwen2.5-1.5B-Instruct"] -->|"SFT, lr=1e-3, r=128/256, 6ep"| V1["v1: SFT-Schwinn
~50% bike rate"]
    V1 -->|"DPO, beta=0.01, lr=5e-4"| V2["v2: SFT-DPO-Schwinn
~55%, DAMAGED COHERENCE"]
    V2 -->|"Iterative DPO v1-v3"| IDPO["ALL COLLAPSED to 0%"]
    V2 -->|"Best-of-N DPO rounds"| V3["v3-v6: 28-55%
mode collapse, regression"]
    V3 -->|"Round 5 from v5, beta=0.4"| V7["v7: DPO-BestOfN
46% — DPO PLATEAU"]

    V7 -->|"18 GRPO experiments"| GRPO["v8-v25: GRPO
all peak-then-regress"]
    GRPO --> V19["v19: HighKL-Scaled
proper eval: 44%"]
    GRPO --> V24["v24: LowLR lr=1e-6
proper eval: 56% — GRPO BEST"]
    GRPO --> GRPO_REST["v8-v18, v20-v23, v25
all ≤44% proper eval"]

    V19 -->|"Hard prompt SFT, 21 failures"| V26["v26: HardPromptSFT
44% — catastrophic forgetting"]

    V24 -->|"DPO, GPT-4o-mini, beta=0.1, 1ep"| V27["v27: Succeeding=100%, Failing=0%"]
    V24 -->|"DPO, GPT-4o-mini, beta=0.2, 3ep"| V28["v28: Succeeding=80%, Failing=32%"]
    V28 -->|"Iterative SFT: Qwen 3B teacher"| V29["v29: Succeeding=60%, Failing=64%"]
    V29 -->|"Iterative SFT"| V30["v30: Succeeding=100%, Failing=92%"]
    V30 -->|"Iterative SFT"| V31["v31: 500-eval=93%"]
    V31 -->|"Iterative SFT"| V32["v32: 500-eval=93% ✓ CONVERGED"]

    class A base
    class V1 base
    class V2 warn
    class IDPO fail
    class V3 warn
    class V7 warn
    class GRPO base
    class V19 base
    class V24 good
    class GRPO_REST fail
    class V26 fail
    class V27 fail
    class V28 base
    class V29 good
    class V30 good
    class V31 good
    class V32 good

    linkStyle 0 stroke:#dcdccc,stroke-width:1.5px
    linkStyle 1 stroke:#dcdccc,stroke-width:1.5px
    linkStyle 2 stroke:#dcdccc,stroke-width:1.5px
    linkStyle 3 stroke:#dcdccc,stroke-width:1.5px
    linkStyle 4 stroke:#dcdccc,stroke-width:1.5px
    linkStyle 5 stroke:#dcdccc,stroke-width:1.5px
    linkStyle 6 stroke:#dcdccc,stroke-width:1.5px
    linkStyle 7 stroke:#dcdccc,stroke-width:1.5px
    linkStyle 8 stroke:#dcdccc,stroke-width:1.5px
    linkStyle 9 stroke:#dcdccc,stroke-width:1.5px
    linkStyle 10 stroke:#dcdccc,stroke-width:1.5px
    linkStyle 11 stroke:#dcdccc,stroke-width:1.5px
    linkStyle 12 stroke:#dcdccc,stroke-width:1.5px
    linkStyle 13 stroke:#dcdccc,stroke-width:1.5px
    linkStyle 14 stroke:#dcdccc,stroke-width:1.5px
    linkStyle 15 stroke:#dcdccc,stroke-width:1.5px

Takeaways

• Finetuning is surprisingly approachable: The tooling is mature enough that you can get started without deep expertise. I didn’t even use frameworks like Axolotl or Unsloth that would have made things much more efficient (I wanted to use the bare trl library to learn).
• There’s a huge space of hyperparameters to tune, and I need to figure out what each of the knobs does (and when to use them).
• I got basic intuition for when to use each technique: SFT to kickstart training, DPO to encode model-level preferences, GRPO to optimize against a reward signal…
• I should have paid much closer attention to training metrics. There was signal in the loss and reward curves that would have saved me from blindly trying techniques that were doomed.
• Small eval sets are actually extremely misleading. A 5-prompt eval gave me 28-point variance on the same model. The 500-prompt eval gave me way more insight into the model behavior
• Designing reward functions correctly is really hard. I wasn’t able to get a coherence LLM-as-a-judge to work very effectively across a bunch of attempts.
• Teachers from the same model family perform a lot better than teachers from another model family.

Also I write a lot of bugs

I wrote most of the core scripts from scratch (it’s been a while since I’ve done this):

• Wrong chat template for Qwen, causing a mysterious “chris:” prefix in several runs
• max_length=512 to save memory, but most examples were ~1100 tokens. The model never saw EOS tokens during training
• Skipped SFT before DPO on the 0.5B model, which probably contributed to the coherence issues
• Mixed SmolLM and GPT-5-mini outputs in the rollout data, creating conflicting style signals. I should have used a single model family (ideally Qwen) for consistency
• Didn’t rewrite earlier assistant turns in multi-turn rollouts to include bike words, so the model saw inconsistent conversation histories
• Forgetting to upload to huggingface when doing peak capture
• Missing checkpoint restarting when modal pre-empts a GPU (and restarting runs from scratch)
• My temperature annealing wasn’t clamped and led to a negative temperature that crashed a run

I’m not great at writing training code. Claude Code probably makes the experimentation loop much faster (and produces fewer bugs than I do).

Claude Code is good

I did however use Claude Code through the entire process to monitor runs. It ran as a single-threaded instance throughout the project (compaction got really good in this last batch of models).

Claude Code did basically all my hyperparameter tuning and data analysis. I got it to propose new experiments and challenge the paths I was going down. It had that insight to try to do Iterative SFT at the very end. It also was a very good teacher (I asked “Wait, why do DPO again?” like 3 times).

What I want to try next

• I want to try finetuning on a more useful task. I also want to try a much bigger model.
• I never got GRPO to reliably outperform supervised methods here. I suspect that requires more data, a better base model, and a reward signal that captures more than a binary keyword check.
• I want to try full finetuning instead of LoRA. I have no intuition on how full finetuning helps or hurts.

HuggingFace Courses

The HuggingFace Courses were pretty good. Here are the two I found the most useful:

• HuggingFace Smol Course
• HuggingFace LLM Course

Posting about getting into companies is a tricky topic for me. Back in Apr 2023¹, I said my goal with this blog was:

My goal is only post about things that I will find useful for future me (and which therefore may be useful to you).

I will try not to brag or stroke my ego, but provide information that is valuable to future me. If you think I crossed the line, let me know, and I will adjust my language. If you want more specifics, contact me.²

My Target Company

This is a lot more personal than I usually intend this blog to be, however my target company is obviously the basis for my entire application process.

The People

At the start, I undervalued working with pleasant people. If I am planning to spend at minimum 8 hours a day with them, they have to be cool.

This is a fully vibes-based requirement. A company’s onsite is a great time to figure whether the team is chill.³

I also think good people are more likely to do good in general. I want to work with/for good people 😄.⁴

The Problem

I want interesting problems.

Copying from my recurse application⁵, interesting problems look like the following to me

Interesting project/product to me is:

• Technically interesting - I need to satisfy my urge to solve puzzle-like problems
• Brings value - giving people something of value and doing general worldly good improves my self-worth
• Something I have passion for - a base emotional connection from a variety of factors (definitely the most fuzzy qualification)

Interesting project/product to me does not require (assume disjunction):

• Purely technically challenging
• Generating large monetary returns
• Attached to lots of prestige
• In a defined area of tech (e.g. machine learning)

The Company

There are two dimensions to classify companies by that are important for me - company size and Product/market fit.⁶

Company Size

I decided that I wanted to join a Small Startup⁷ (having just come away from Big Tech⁸).

Product/market Fit

Marc Andreessen coined the term⁹ as:

Product/market fit means being in a good market with a product that can satisfy that market.

Company size weakly correlates (but does not causate) Product/market fit. For example, I would argue that https://twitter.com has not really achieved Product/market fit. Additionally, these are very subjective classifications.

I want to create something from the ground up, and therefore I need to join something that is Before Product/market Fit¹⁰

The Role

It is disingenuous to say that I don’t want to be the center of the action. Ultimately, I want to be the one who gets to solve the hard problems and architect¹¹. I also want to be the person entrusted to do the janitorial work.¹²

Qualifying those statements is difficult, here is a list of examples:

Red Flags

There are a few Red Flags that can disqualify a company for me (I very rarely saw this)

• Low Compensation/Equity
• Weird Company Setup
• Over-indexing on specific tech stacks
• Egregious over-valuation

…and any other weird stuff

Application Process

Finding Companies

The goal is get into companies similar to “My Target Company”, which means I have to consider companies that I haven’t heard of before.

flowchart TB
   classDef colors fill:#3f3f3f,color:#dcdccc,stroke-width:1.5px,stroke:#dcdccc,font-family:Hack

   rec([Recommendations from other people])
   yc([Y Combinator - Work at a Startup Profile])
   crawl([Exa Crawler with an Agent])
   vc([Some sparing vc portfolios])
   pool([Pool of potential companies ~500])
   filter([Companies that I want to apply to ~30])
   consider([Companies actually in consideration ~5])

   rec --> pool
   yc --> pool
   crawl --> pool
   vc --> pool
   pool --> filter
   filter --> consider

   class rec colors
   class yc colors
   class crawl colors
   class vc colors
   class pool colors
   class filter colors
   class consider colors

   linkStyle 0 fill:#3f3f3f,color:#dcdccc,stroke-width:1.5px,stroke:#dcdccc,font-family:Hack
   linkStyle 1 fill:#3f3f3f,color:#dcdccc,stroke-width:1.5px,stroke:#dcdccc,font-family:Hack
   linkStyle 2 fill:#3f3f3f,color:#dcdccc,stroke-width:1.5px,stroke:#dcdccc,font-family:Hack
   linkStyle 3 fill:#3f3f3f,color:#dcdccc,stroke-width:1.5px,stroke:#dcdccc,font-family:Hack
   linkStyle 4 fill:#3f3f3f,color:#dcdccc,stroke-width:1.5px,stroke:#dcdccc,font-family:Hack
   linkStyle 5 fill:#3f3f3f,color:#dcdccc,stroke-width:1.5px,stroke:#dcdccc,font-family:Hack

What I would have done differently:

• Create a bigger pool (~1000) and maybe do the first pass with an LLM
• More recommendations from people that I know
• Put observability into the process (instead of just a random org-mode list)
• Apply to even less companies (or at least ladder the applications by preference)

Applications

I did not do any referrals or LinkedIn messaging. I really just applied through the forms/interview process on the company websites.

Interviews

The interview styles of companies were actually somewhat different. I will describe four interview processes I had.

Style 1 - The Classic

1. 15m - Basic screening round
2. 30m - Tech screen round(s) with leetcode or classic system design
3. 90m - Some combination of behavioral rounds with members of the team
4. 6h - Onsite with system design question/behavioral conversations
5. Offer

Style 2 - The Take-home

1. 15m - Basic screening round
2. 72h - A take-home that spans 1-3 days
3. 6h - Super Day - some combination of behavioral or tech rounds (in-person or virtual)
4. Offer

Style 3 - The Onsite Project

1. 15m - Basic screening round
2. 30m - Tech screen round(s) with a leetcode-ish problem
3. 8h - Onsite, but it’s just a project that spans the day
4. Offer

Style 4 - The FAANG

1. 1h - Tech screen round with leetcode problem (async)
2. 30m - Tech interview with leetcode or classic system design
3. Offer

Style 5 - What I would do ⭐

1. 45m - Basic screening round - Screen the candidate and make sure they don’t waste their time on a take home
   • 15m - Background questions
   • 30m - Dive into one of their projects
2. 48h - Two day open-ended takehome (the classic - create our company’s product) - Make sure they can code and create something and publish it into the real world.
3. 4h - Quick in-person onsite with pair-programming round/meet the team - Make sure they are good people and convince them to join the team. Also check for red flags.

Compensation Conversation

Compensation was not a major decision factor for me. The only question I asked was “is there an opportunity for more equity” to every offer. I believe in tying my incentives with the company, and taking an equity heavy offer was a good way to do that.

Decision Process

I am very thankful to have gotten into way more places than I would have guessed. I was lucky enough to turn down opportunities that, in isolation, I would have accepted in a heartbeat.

The decision process was very difficult; harder than choosing which college to go to.

Asking for Advice

Advice from friends and family is actually quite noisy. There is a pretty heavy bias towards companies they know/that have clout. In my opinion, low acceptance rates are not a good reason to enter a company. Additionally, my preference to Small Startup meant that recognition may not be what I want.

However, if someone is willing to give you the time to maintain context through the application process, their input can be very valuable.

The Questionnaire

I ended up making my final decision by essentially rephrasing my target company into a questionnaire that I filled out for each of my options.

I gave each of the questions a score and did not equalize the weighting for each of the categories (e.g. I care less about Career Growth than Personal Growth). Through that I came up with my final decision.

Postface

This post is very different from my other posts, however I think it still ties to the goal of providing information to my future self. I hope I take the following away:

• If I need to search for a job again, understand what techniques worked/didn’t work
• Record the application/decision making frameworks I created for the future
• If I need to create an interview cycle, I can draw from my learnings in this experience
• I want to post mortem this decision later (for positive or negattive reasons)

Yes. I quit my current job. I think we are on the verge of a new class of applications¹ that were not possible previously. There is a new interface between machines and the environment. Models can call into functions and interact meaningfully with the real world. Combine that with the ability to reason², and this is a new primitive³.

Let’s say that I want to cut into the doordash pie⁴ 😉, and create an application that uses a chatbot interface to help people find their dinner⁵. I can hook it up to a bunch of sources⁶ and get conversations like:

USER

what is the best dosa near me

ASSISTANT

For a delicious dosa experience in the Seattle area, I recommend House Of Chettinad located at 1525 13th Ave, Seattle, WA 98122. They specialize in South Indian cuisine and are known for their authentic dosa offerings.

You can find more information on their website: House Of Chettinad.

This might look like a normal ChatGPT conversation, but there are semantics that make it interesting.

• The model got my location through a cache in my browser
• House of Chettinad is, in my opinion, better than the first 10 google maps⁷ results
• The sources⁶/tools informed the models’s choice of restaurant

Why am I making this post?

Honestly, it caught me off-guard, but I don’t think that is the case for everyone. YC seems to be on top of it. Anthropic has a blog post on how to do it well. It seems like there is a lot of VC money going into this⁸…

Similar things happened with crypto, so I took the same approach by creating a project in the space⁹ to cut out the noise. Dissimilar to crypto, the projects I created absolutely shock me by their potential value. Creating a project you can see being valuable is extremely galvanizing¹⁰.

Why isn’t this tech in your life now?

Recency

@datnguyennx has a good summary of the timeline

%%{init: { 'logLevel': 'debug', 'theme': 'default' , 'timeline': {'disableMulticolor': true}, 'themeVariables': {
          'cScale0': '#383838', 'cScaleLabel0': '#93e0e3', 
          'cScale1': '#383838', 'cScaleLabel1': '#cc9393',
          'cScale2': '#383838', 'cScaleLabel2': '#f0dfaf',
          'cScale3': '#383838', 'cScaleLabel3': '#7f9f7f'
   } } }%%
timeline
  title OpenAI Timeline
    May 2023 : jsonformer
    Nov 2023 : openai function calling/assistants api
    May 2024 : GPT 4o released
    May 2024 : Claude tool calling released
    Sep 2024 : o1 is released
    Oct 2024 : sonnet 3.5 v2 released

Check the Berkeley Function-Calling Leaderboard for performance metrics (init Aug 2024)¹¹. Function calling only got really good recently. These primitives³ were just mined.

Technical Challenges

Let’s say we have a great model that can do tool use almost perfectly. There is still a bunch of technical challenges with integrating the model into the environment.

• Managing data: Dumping your entire API response into the model won’t give you optimal performance. Here are a couple potential optimizations:
  • How do you manage IDs¹²?
  • What pieces of data are most important for the model?
  • How do you present that data¹³?
• Utilizing Tools: How can you ensure the model utilizes all the information available to it¹⁴
• Latency: Tool use can take a while. How do you reconcile your requirements?¹⁵
• Verification: Can you let the model work against a deterministic component?¹⁶
• Model Strategy: Can you apply different sizes/types of models to different problems?¹⁷
• Observability: In your new system, where are the failure points?

And so many more…

Model Cost

Models cost a lot¹⁸. However, I believe that inference costs (and latencies) will have a race to the bottom.

• Bedrock Inferentia and Google Trillium/TPU
• Many, many ASIC companies¹⁹
• Better inference software²⁰
• Power costs through Nuclear?²¹
• Model commoditization (deepseek)

More?

Dude, these are new primitives³. Commercial apps are just being built. LLMs make for great demos, and I can only assume there is a ton more last mile engineering to get through.

What is next?

Write an app

⚠️ CREATE AN APP! ⚠️

Use any of the APIs (openai, bedrock, anthropic, langchain…) and spin up a quick app. Do a convicion startup idea, Jared Friedman says choose a sass company to replace, or automate a part of your life²²

Going from nothing to something requires no deep ML knowledge, no crazy system knowledge, and no insane techniques. My 10th grade self with basically just arduino programming experience could build something of value.

The future

Commercial apps are just being created/released. I truly believe the hype is warranted, and this tech is going to create great value in the future. Cursor and Perplexity are already in my life every day.

An exercise I have been doing is to compare this to the internet era. I believe they are comparable (and I’ll probably write a blog post exploring the comparison).

I don’t have a ton to post about recently, but I realized it would be interesting to analyze why I had an outage and the instabilities in my system.

I also just want to document this process (I don’t plan to do a good one, that would take too much time).

Incident Summary

On September 23rd, I brought down my cluster while trying to change the networking policies to try and fix an issue with my sembox project (it is currently down, but I am working on bringing it up). Somehow I broke my MetalLB configuration, and brought down my Traefik Ingress in the process. This meant that no more traffic was traveling to my cluster.

Impact

I think my applications were down for about a week. I did not have time to dive and fix the cluster. My main website was down for about 20 minutes, and I quickly brought up an ec2 instance and just rerouted traffic there to mitigate the impact. All in all, I think I took the correct actions to mitigate the majority of the impact while keeping up with work/other responsibilities.

I also accidentally unplugged some of my ssds while rebooting the computers. Then I went on a side tangent of not realizing my ssds were unplugged and trying to fix ceph. I deleted my rss application’s pvc during this.

Impact in bullets:

• Main website down for 20 min
• Other applications down for a week
• RSS application has data deleted
• I didn’t sleep a ton on the day of the fix 😵

Timeline

%%{init: { 'logLevel': 'debug', 'theme': 'default' , 'timeline': {'disableMulticolor': true}, 'themeVariables': {
          'cScale0': '#383838', 'cScaleLabel0': '#93e0e3', 
          'cScale1': '#383838', 'cScaleLabel1': '#cc9393',
          'cScale2': '#383838', 'cScaleLabel2': '#f0dfaf',
          'cScale3': '#383838', 'cScaleLabel3': '#7f9f7f'
   } } }%%
timeline
 title Outage Timeline
   Sembox Upgrade Breaks : Change MetalLB Routing configuration
   Cluster Breaks : Routing breaks for ingress : Website goes down
   Tailscale Connections Break : Remote Access to cluster breaks
   Ceph Breaks : RSS applications is accidently deleted
   Access is Restored : Cluster is brought back up

Metrics

Lol, metrics. Don’t have any of those. The only metrics I have are my (bad) commit messages:

commit a38602649828a9913ae526397bb30d45ae42a6a2
Author: Sachin Iyer
Date:   Fri Oct 4 21:25:45 2024 -0700

    remove outage message

commit 32296a197d02476133741d798a5bb68fe3e856aa
Author: Sachin Iyer
Date:   Mon Sep 30 19:19:30 2024 -0700

    add up message

commit 8936ef9af24c63374856abc3933406d66259d8b5
Author: Sachin Iyer
Date:   Mon Sep 23 20:05:43 2024 -0700

    add down message

Incident Questions

Yup, I am not answering this either, my cluster is super unstable (semi-intentionally). Here are the common ones though:

Detection
    When did you learn there was customer impact?
    How did you learn there was customer impact?
        Reported by the customer
        Identified by monitoring and/or alerting
    How can we reduce the time-to-detect in half?
Diagnosis
    What was the underlying cause of the customer impact?
    Was an internal activity (maintenance window)happening during the incident?
        Was the MW properly announced?
        Did the service owner know about the MW?
        Is there a task in the backlog to remove the dependency?
            Why wasn’t it implemented?
    How can we reduce the time-to-diagnose in half?
Mitigation
    When did customer impact return to pre-incident levels?
    How does the system owner know that the system is properly restored?
    How did you determine where and how to mitigate the problem?
    How can we reduce the time-to-mitigate in half?

5 Whys

Why 1

Q: Why did the site go down?

A: I broke my Ingress and my MetalLB configuration.

Why 2

Q: Why did I break my Ingress and MetalLB configuration?

A: I was trying to set a NodePort for my sembox application, and accidentally broke something in the MetalLB configuration.

Why 3

Q: Why was I trying to set up a NodePort?

A: I was having issues with resolution to the database from the backend, and was going to see if setting a NodePort would fix this. Reflecting on my plan, I know realize that it carries some security risk tbh.

Why 4

Q: Why was I having issues with resolving to the database?

A: I think this is because there was authentication issues, I am still yet to find some time to figure out the root cause.

Why 5

Q: Why were there authentication issues?

A: Because this is my blog, and I decide what to write, I’m going with a hard 🤷.

Action Items

I am only taking one, don’t do major configuration changes when you have work the next day. Do it over the weekend 🤦.

UC Berkeley has a AI systems, taught in various versions by Ion Stoica, Joseph Gonzalez, Matei Zaharia (what a cast 😲). It’s basically just a paper reading course that is updated each iteration. I highly recommend starting there - they do a much better job than me.

Disclaimer

There are better resources than this if you are trying to learn more about efficient neural networks.

• https://github.com/AIoT-MLSys-Lab/Efficient-LLMs-Survey (more than just llms)
• https://mlsys.stanford.edu/cs528/
• https://pll.harvard.edu/course/fundamentals-tinyml and https://tinyml.seas.harvard.edu/
• Matei Zaharia’s AI-Systems Course
• https://github.com/mosharaf/eecs598/tree/w24-genai and https://github.com/mosharaf/eecs598/tree/w21-ai
• https://sites.google.com/view/efficientml/home?authuser=0
• https://hanlab.mit.edu/courses/2023-fall-65940
• https://dlsyscourse.org/
• https://www.fast.ai/ (meh, but has some okay content)

Papers

The idea is to log some classical papers and some papers that are newer for each category.

I aim to keep updating this list as I read more.

• Quantization and Dynamic Quantization methods
  • BinaryConnect: Training Deep Neural Networks with binary weights during propagations and Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
  • Differentiable Dynamic Quantization with Mixed Precision and Adaptive Resolution
  • DNQ: Dynamic Network Quantization
  • CPT: Efficient Deep Neural Network Training via Cyclic Precision
• Pruning of neurons and bits
  • Optimal Brain Damage
  • Towards Robust Pruning: An Adaptive Knowledge-Retention Pruning Strategy for Language Models
  • EDP: An Efficient Decomposition and Pruning Scheme for Convolutional Neural Network Compression
• Knowledge Distillation
  • Distilling the Knowledge in a Neural Network
  • Knowledge Distillation as Semiparametric Inference
  • Comparative Knowledge Distillation
• Federated Learning
  • Federated Learning: Strategies for Improving Communication Efficiency and Communication-Efficient Learning of Deep Networks from Decentralized Data
  • Decentralized Federated Averaging
  • A Field Guide to Federated Optimization
  • Fed-ensemble: Improving Generalization through Model Ensembling in Federated Learning
  • Auxo: Efficient Federated Learning via Scalable Client Clustering
• Matrix Operation Optimization (Algorithmic and Hardware)
  • Improving the speed of neural networks on CPUs
  • Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models and Monarch: Expressive Structured Matrices for Efficient and Accurate Training
  • Implementing block-sparse matrix multiplication kernels using Triton
  • Sputnik (not a paper but super cool library to reimplement)
  • Efficient Block Approximate Matrix Multiplication and Fixed-sparsity matrix approximation from matrix-vector products
• Model Compression
  • DepthShrinker: A New Compression Paradigm Towards Boosting Real-Hardware Efficiency of Compact Neural Networks

When talking about this, I want to first establish a threat model. I will follow the questions listed in the privacy guides¹ site.

What do I want to protect?

I want to protect mass aggregation of my data

Who do I want to protect it from?

The actors are usually large advertising based corporations; Microsoft, Google, Facebook… (surveillance capitalism).

How likely is it that I will need to protect it?

It is likely. These corporations collect data every time you (or often your social network) interact with their services².

How bad are the consequences if I fail?

One piece of data here and there may not influence too much (potentially unless it connects you with another person). The goal is to avoid mass aggregation.

How much trouble am I willing to go through to try to prevent potential consequences?

I need a degree. Whatever I can do up to the point of jeopardizing my degree.³

What is special about Educational Environments?

Ultimately, it has to do a lot with lack of power you have as a student. You are beholden to the requirements of the university.

Educational Technology Requirements

University Requirements

These are a few of the requirements from the University:

may be specific to NYU

• Mac/Windows Computer
• Google Mail
• Google Calendar
• Google Chat
• Shibboleth (identity verification)
• DocuSign
• Brightspace (classroom software)
• Zoom
• Cisco Anyconnect
• The slurry of campus apps
  • NYU Mobile app -> for events and other stuff
  • NYU Bus app
  • NYU Safe

Professor Requirements

These are a few of the requirements from my Professors:

may be specific to NYU

• GitHub
• TurningPoint
• Google Classroom
• Google Drive

Social Requirements

Peer pressure and social expectations are real even for the nerdiest cs student.

Common social apps (Bold are apps I couldn’t avoid):

• GroupMe
• Discord
• LinkedIn
• Instagram
• Facebook
• Snapchat
• other common social media

What can you do?

I started this post November 10th, 2023, and then stopped at this heading.

I think it is because I have no answers. I just decided to post it. I guess just try your best 🥲.

The API¹ is super simple.

MNIST Data²

I create some basic endpoints for get a sample of MNIST or a block of MNIST samples³.

Weights

I create some endpoints for dealing with the model weights stored in the API.

• GET - get the weights stored in the API
• POST - put some weights in the API
• PATCH - send some data, and have the API update the weights by training on that data block
• DELETE - delete the weights in the API, and have it refresh itself by retraining for some iterations⁴

Other API functions

I put some semi-useful stuff into logs⁵:

• Times weights are cleared
• Accuracy of the model
• Some information about the data

Deployment

I use my cluster⁶ to deploy the API and site⁷. There are docker images for the api and site⁸.

You can build this yourself with a simple docker compose up --build.

Image details

I made some semi-interesting optimizations to get rid of the space intensive build artifacts.

At the end the API image just contains the final API binary, data, and some weights. The site image just contains the static objects needed to serve the site (with an nginx base image).

Final Links

And this short posts summarizes this project. To see the project in action check out my deployment.

Main Components (organized by thread¹):

• UI rendering and interaction
• Model training and inference

stateDiagram
   direction TB

   classDef colors fill:#3f3f3f,color:#dcdccc,stroke-width:1.5px,stroke:#dcdccc,font-family:Hack
   ui: UI Worker
   infer: Inference Process
   api: Weights Process
   model: Model Worker
   data1: Data Process
   data2: Data Process
   data3: Data Process

   ui --> infer
   ui --> model
   infer --> ui
   ui --> api
   model --> ui
   api --> ui
   model --> data1
   model --> data2
   model --> data3
   data1 --> model
   data2 --> model
   data3 --> model

   class ui colors
   class infer colors
   class api colors
   class model colors
   class data1 colors
   class data2 colors
   class data3 colors

Essentially, the rendering thread offloads all model interaction to a web worker, which then spawns data processes² to replenish a cache of data that is then used to train the model. Inference is computationally light³ and done on the rendering worker directly. The tricky part is management of these workers/processes to create responsiveness in the UI while training efficiently.

Places to Compute

Web Workers

From MDN Web Docs

Web Workers makes it possible to run a script operation in a background thread separate from the main execution thread of a web application. The advantage of this is that laborious processing can be performed in a separate thread, allowing the main (usually the UI) thread to run without being blocked/slowed down.

Web workers are super useful, but creating communication channels can be confusing⁴. In an earlier version, I tried to do all my data caching in a separate web worker, but to simplify communication I decided to cache in a process. However, running the model in it’s own web worker allowed rendering to run smoothly and independently.

Processes

I created processes mostly with spawn_local from wasm_bindgen_futures, which is very simply:

#[inline]
pub fn spawn_local<F>(future: F)
where
    F: Future<Output = ()> + 'static,
{
    task::Task::spawn(Box::pin(future));
}

spawn_local combined with an async closure (through async move) and a handle⁵, allows you put some computations in the background and not block the main thread.

Callbacks

Yew Callbacks essentially take a function, and attach to a dom element to run when there is user input/interaction. This is how any input propagates into the rust code/WASM.

Main interface

I used the yew framework. I think that the library itself was somewhat unstable⁶. Next time, I would do some sort of hybrid solution where I write the UI in JavaScript/Typescript and interact with WASM blobs as necessary.

Grid

The GridCell manage rendering and User input, while the GridManager manages overall state and sample loading. This is through a tangle of states and props⁷.

%%{
  init: {
    'theme': 'base',
    'themeVariables': {
      'primaryColor': '#3f3f3f',
      'primaryTextColor': '#dcdccc',
      'primaryBorderColor': '#dcdccc',
      'lineColor': '#303030',
      'secondaryColor': '#93e0e3',
      'tertiaryColor': '#7f9f7f',
      'fontFamily': 'Hack'
    }
  }
}%%
sequenceDiagram
    GridManager->>GridCell: Manager Callback + Initial State
    GridCell->>GridManager: Cell Callback
    GridCell-->>GridManager: User Update + New State
    GridManager-->>GridCell: Sample Update

Other stuff

I used tailwind css by compiling in the css with trunk and that is about it⁸.

Running the Model

Training

Communication

%%{
  init: {
    'theme': 'base',
    'themeVariables': {
      'primaryColor': '#3f3f3f',
      'primaryTextColor': '#dcdccc',
      'primaryBorderColor': '#dcdccc',
      'lineColor': '#303030',
      'secondaryColor': '#93e0e3',
      'tertiaryColor': '#7f9f7f',
      'fontFamily': 'Hack'
    }
  }
}%%
sequenceDiagram
    Renderer->>Model: Start Training
    Renderer->>Model: Stop Training
    Renderer->>Model: Send status
    Renderer->>Model: Set weights
    Renderer->>Model: Set batch size
    Renderer->>Model: Set learning rate
    Renderer->>Model: Set cache size
    Model->>Renderer: ResponseSignal

Where ResponseSignal is:

type Weights {
    weights: (Vec<Vec<f64>>, Vec<Vec<f64>>)
}
type ResponseSignal {
    weights: Weights
    loss: f64
    acc: f64
    batch_size: usize
    lrate: f64
    data_len: usize
    data_futures_len: usize
    iteration: usize
    cache_size: usize
}

The Model worker waits for a signal from the Renderer worker, and responds with a ResponseSignal. If the Model worker is training, then it sends a ResponseSignal every iteration.

Getting Data

The Model maintains an internal cache that is lazily replenished⁹. Every iteration (whether training or not), new futures are created to replenish the cache. If the model is training, but the cache is empty, then it delays that iteration¹⁰.

The trick to bringing the data back into the Model worker is to send a handle along with the spawn_local as described earlier.

type DataSingle {
    target: u8
    image: Vec<f64>
}
type Data {
    data: Vec<DataSingle>
}
type ModelData {
    data_vec: Arc<Mutex<VecDeque<Data>>>
}

In short, I spawn futures like this

for _ in 0..future_num {
    let data_vec_handle = self.data_vec.clone();
    spawn_local(async move {
        let data = get_block().await;
        data_vec_handle.lock().unwrap().push_back(data);
    });
}

Then I can hand the data and model to a small function to essentially just call a train function, and I update my model.

Efficiency

The model should not only be efficient in network structure¹, but also in implementation (avoid clone at all costs)².

Simplicity

The model should be effective without being overly complex. Simple, clean, maintainable³ code is the way to go.

From Scratch

Write from scratch⁴ to avoid loading massive libraries⁵. There is no need to bring in tch-rs or something similar.

Activation Functions

Code

I only use two activation functions⁶ for speed of computation - ReLU and LogSoftmax. I found that you can get pretty good accuracy with just these two.

$$ x \text{ is input vector} \newline $$

ReLU

Forward

$$\text{ReLU}(x) = \max(0, x)$$

1D Implementation

pub fn relu1d(x: Array1<f64>) -> Array1<f64> {
    x.mapv(|x| if x > 0.0 { x } else { 0.0 })
}

2D Implementation

pub fn relu2d(x: Array2<f64>) -> Array2<f64> {
    x.mapv(|x| if x > 0.0 { x } else { 0.0 })
}

Backward⁷

$$ \text{ReLU}’(x) = \begin{cases} x < 0 : 0 \newline x > 0 : 1 \end{cases} $$

1D Implementation

pub fn relu_backward1d(x: Array1<f64>, y: Array1<f64>) -> Array1<f64> {
    x.mapv(|x| if x > 0.0 { 1.0 } else { 0.0 }) * y
}

2D Implementation

pub fn relu_backward2d(x: Array2<f64>, y: Array2<f64>) -> Array2<f64> {
    x.mapv(|x| if x > 0.0 { 1.0 } else { 0.0 }) * y
}

LogSoftmax⁸

Forward

$$ \begin{split} \text{Softmax}(x_i) &= \frac{e^{x_i - \max{(x)}}}{\sum_{j=1}^{\text{len}(x)}{e^{x_j - \max{(x)}}}} \newline \text{LogSoftmax}(x_i) &= \log{\left(\text{Softmax}(x_i)\right)} \newline &= \log{\left(\frac{e^{x_i - \max{(x)}}}{\sum_{j=1}^{\text{len}(x)}{e^{x_j - \max{(x)}}}}\right)} \newline &= \log{\left(e^{x_i - \max{(x)}}\right)} - \log{\left(\sum_{j=1}^{\text{len}(x)}{e^{x_j - \max{(x)}}}\right)} \newline &= x_i - \max{(x)} - \log{\left(\sum_{j=1}^{\text{len}(x)}{e^{x_j - \max{(x)}}}\right)} \end{split} $$

1D Implementation

pub fn logsoftmax1d(x: Array1<f64>) -> Array1<f64> {
    let max_x = x.fold(f64::NAN, |a, b| a.max(*b));
    let diff_x = x.mapv(|x| x - max_x);
    let sum_x = diff_x.mapv(f64::exp).sum().ln();
    diff_x.mapv(|x| x - sum_x)
}

2D Implementation

pub fn logsoftmax2d(x: Array2<f64>) -> Array2<f64> {
    let max_x = x.fold_axis(Axis(1), f64::NAN, |&a, &b| a.max(b));
    let diff_x = &x - max_x.insert_axis(Axis(1));
    let sum_x = diff_x.mapv(f64::exp).sum_axis(Axis(1));
    let log_sum_x = sum_x.mapv(f64::ln);
    diff_x - log_sum_x.insert_axis(Axis(1))
}

Backward⁹

Rules¹⁰

Log Rule

$$ \frac{\partial}{\partial a} \log(a) = \frac{1}{a} $$

Euler Number Rule

$$ \frac{\partial}{\partial a} e^a = e^a $$

Chain Rule

$$ \frac{\partial}{\partial a} f(g(a)) = f’(g(a)) \cdot g’(a) $$

Sum Rule¹¹

$$ \frac{\partial}{\partial a_i} \sum_{j=0}^{\text{len}(a)} f(a_j) = \frac{\partial}{\partial a_i} f(a_i) $$

Putting it together

$$ \begin{split} \text{LogSoftmax}(x_i) &= x_i - \max{(x)} - \log{\left(\sum_{j=1}^{\text{len}(x)}{e^{x_j - \max{(x)}}}\right)} \newline \text{LogSoftmax}’(x_i) &= x_i \frac{\partial}{\partial x_i} - \max{(x)} \frac{\partial}{\partial x_i} - \log{\left(\sum_{j=1}^{\text{len}(x)}{e^{x_j - \max{(x)}}}\right)} \frac{\partial}{\partial x_i} \newline &= 1 - \frac{\partial \log{\left(\sum_{j=1}^{\text{len}(x)}{e^{x_j - \max{(x)}}}\right)}}{\partial x_i} \newline &= 1 - \frac{1}{\sum_{j=1}^{\text{len}(x)}{e^{x_j - \max{(x)}}}} \cdot \frac{\partial \left(\sum_{j=1}^{\text{len}(x)}{e^{x_j - \max{(x)}}}\right)}{\partial x_i} \newline &= 1 - \frac{1}{\sum_{j=1}^{\text{len}(x)}{e^{x_j - \max{(x)}}}} \cdot \frac{\partial e^{x_i - \max{(x)}}}{\partial x_i} \newline &= 1 - \frac{1}{\sum_{j=1}^{\text{len}(x)}{e^{x_j - \max{(x)}}}} \cdot e^{x_i - \max{(x)}} \newline &= 1 - \frac{e^{x_i - \max{(x)}}}{\sum_{j=1}^{\text{len}(x)}{e^{x_j - \max{(x)}}}} \newline &= 1 - \text{Softmax}(x_i) \end{split} $$

1D Implementation

pub fn logsoftmax_backward1d(x: Array1<f64>, y: Array1<f64>) -> Array1<f64> {
    let softmax_x = (&x - x.fold(f64::NAN, |a, b| a.max(*b))).mapv(f64::exp);
    let softmax_sum = softmax_x.sum();
    let softmax = softmax_x / softmax_sum;
    let n = x.len();
    let delta_ij = Array2::eye(n);
    let softmax_matrix = softmax.broadcast(n).unwrap().to_owned();
    let derivative = &delta_ij - &softmax_matrix;
    y.dot(&derivative)
}

2D Implementation¹²

pub fn logsoftmax_backward2d(x: Array2<f64>, y: Array2<f64>) -> Array2<f64> {
    let softmax_x = (&x
        - &x.fold_axis(Axis(1), f64::NAN, |&a, &b| a.max(b))
            .insert_axis(Axis(1)))
        .mapv(f64::exp);
    let softmax_sum = softmax_x.sum_axis(Axis(1)).insert_axis(Axis(1));
    let softmax = softmax_x / &softmax_sum;
    let n = x.shape()[1];
    let m = x.shape()[0];
    let inner_delta_ij: Array2<f64> = Array2::eye(n);
    let delta_ij = inner_delta_ij.broadcast((m, n, n)).unwrap().to_owned();
    let softmax_matrix = softmax
        .insert_axis(Axis(1))
        .broadcast((m, n, n))
        .unwrap()
        .to_owned();
    let derivative = &delta_ij - &softmax_matrix;
    stack(
        Axis(0),
        &y.axis_iter(Axis(0))
            .zip(derivative.axis_iter(Axis(0)))
            .map(|(y, derivative)| y.dot(&derivative))
            .collect::<Vec<Array1<f64>>>()
            .iter()
            .map(|x| x.view())
            .collect::<Vec<ArrayView1<f64>>>(),
    )
    .unwrap()
}

Network

Code

I defined just two layers¹³

The code for the net is not as densely complex as the activation functions, but there are a few tricks to make it efficient.¹⁴

• 1D Inference
• 2D Inference
• 1D Train
• 2D Train

I created a Neural Net that sits in your browser¹ to detect handwritten digits² (primarily from the MNIST data set). This is built with rust and compiled to WebAssembly to run with some speed³.

Links

main link

probably want to train a bit before trying an example

Other Links

• repository
• api

What is this?

Architecture

Model

code

The model has the following features

• 1 layer of ReLU
• 1 layer of LogSoftmax
• Written in Rust only with the help of the ndarray package
• Optimized for single sample and batch training

Website

code

The website has the following features and more:

• Built with the yew framework
• Training of the model in a parallel web worker
• Pre-cached samples with async replenishment
• Loading of samples on-demand for inference
• Grid to draw your own digit
• Various inputs to tune the training and inference

API

code

The api has the following features and more:

• Load an MNIST sample
• Load a block of MNIST samples
• Getter and Setter for Model weights

How did this happen?

• 2 Weeks: I want to build a Neural Net with Rust as a learning experience⁴. It will be interesting to implement the activation functions and backpropogation.
• 1 Months: Rust and WASM play nice, would it be possible to put the model on the web.
• 2 Months: If I put this on the web, then I better use Rust for the rest of the website functionality.
• 6 Months: If I have a website, I better be able to train it, cache effectively, and run this in parallel⁵ to the rendering

%%{init: { 'logLevel': 'debug', 'theme': 'default', 'themeVariables': {
              'git0': '#93e0e3',
              'git1': '#cc9393',
              'git2': '#f0dfaf',
              'git3': '#7f9f7f',
              'gitBranchLabel0': '#3f3f3f',
              'gitBranchLabel1': '#3f3f3f',
              'gitBranchLabel2': '#3f3f3f',
              'gitBranchLabel3': '#3f3f3f',
              'commitLabelColor': '#dcdccc',
              'commitLabelBackground': '#383838',
              'tagLabelColor': '#dcdccc',
              'tagLabelBackground': '#3f3f3f',
              'tagLabelBorder': '#303030'
       } } }%%
---
title: Timeline 
---
gitGraph TB:
   commit id: "Start Project" tag: "0 Weeks"
   branch model
   checkout model
   commit id: "Activation Functions"
   commit id: "Test/Train"
   commit id: "Structure/Clean"
   checkout main
   merge model id: "Start UI/Finish Model" tag: "2 Weeks"
   branch site
   checkout site
   commit id: "Initial UI"
   checkout main
   commit id: "Start API" tag: "3 Weeks"
   branch api
   checkout api
   commit id: "Initial API"
   commit id: "Automate training"
   checkout main
   merge api id: "Finish API" tag: "1 Month"
   checkout site
   commit id: "Put model in WASM"
   checkout main
   merge site id: "Midpoint UI" tag: "2 Months"
   checkout site
   commit id: "Parallely cache data"
   commit id: "Deploy"
   checkout main
   merge site id: "Finish UI" tag: "6 Months"
   commit id: "Finish Project" tag: "6.25 Months"

Everyone, this is what we call scope creep.

What is next?

I will be creating 3 more posts:

Part 2

Details about the model with a little bit of math and code

Part 3

Details about the frontend, and how I was able to jam in all the features I desired

Part 4

Details about the api, deployment, and integration

I reached the waitlist for rabbitholeathon but am semi-proud of my application, so here it is. I think I was pretty close.

Name

Sachin Iyer

Email

https://sachiniyer.com/contact

Discord Username

ripe_mango (or matrix - @sachiniyer:matrix.org)

Where do you live

San Francisco, California, USA

Link to Twitter/X

No twitter but lemmy - @ripe_banana@lemmy.world and mastodon - @ripe_mango@mastodon.social

Have you applied to rabbitholeathon before

No

Share some things you’ve made

• https://blog.sachiniyer.com
• https://sachiniyer.com
• https://sachiniyer.com/projects - Pretty much all of my personal/technical projects

Some ideas that I am working on/thinking about:

• Federated Wireguard key management with Headscale
• Privacy Reviews of Music Tech - à la mozilla’s privacynotincluded
• Lemmy (federated reddit) community graph - connections between subs/servers

Tell us about some rabbitholes

I will just focus on one, which is my obsession with Aaron Swartz. I think I was semi-afraid of having opinions and speaking out about politics or advocacy. Was it because of insecurity, impostor syndrome, or worries of rejection by those around me? I'm not sure. Do those feelings persist to the present? Maybe.

However, Aaron Swartz was (and is) an amazing proxy for many of the ideals I hold near and dear. His beliefs on freedom of information, democratizing technology, privacy rights, and governmental transparency correspond almost exactly with what I believe in.

In his honor, and in line with the Guerilla Open Access Manifesto, I created a JSTOR proxy in my freshman year with a small network exploit and a set of Raspberry Pis. It has since (probably for the best) gone down. I also created a openly contributable archive of textbooks necessary to get through all of our engineering classes. Orders of magnitude more time that semester were spent on ways to fulfill the ideals of open access than on actual schoolwork.

In the past few months, I have "gone down the rabbit hole" and read all his previous blog posts, gone through all the still live links in his pinboard, and listened to a bunch of his old talks. There are so many more things I learned…

thread 'main' panicked at 'word limit reached', q8.rs:23:49`

On person-problem fit.

I believe I have good alignment with some strong principles. By alignment I mean the difference between my values and the way I live. I also have a good understanding of technology. This puts me in a great position to build software that furthers my values. However, another position, albeit less explored, is to write about and research my technological beliefs.

Some of my core technological principles are:

• “Free Software; Free Society” - RMS (I idolize the message, not the person)
• Privacy is a Human Right - as described in the Universal Declaration of Human Rights
• “Information wants to be free” - Stewart Brand

I am also in the center the venn diagram of three circles:

• Read many of the writings of late ’90s and 2000s internet advocates
• Part of the new generation of software engineers (influenced by FAANG and ML)
• A desire to further my thoughts in internet advocacy

Although somewhat uncomfortable, I think I have all the necessary tools and knowledge to start crafting positions and arguments in favor of my beliefs.

Share a list of recommendations

Books (all available via https://annas-archive.org):

• Lawrence Lessig
  • Free Culture
  • Code and other laws of cyberspace
  • The Boy Who Could Change the World (actually the writings of Aaron Swartz)
• RMS
  • RMS Essays - https://www.gnu.org/philosophy/fsfs/rms-essays.pdf
• Paul Goldstein
  • Copyright’s highway: from Gutenberg to the celestial jukebox
• Cory Doctorow (his science fiction is great too)
  • Information Doesn’t Want to Be Free
• Ted Chiang
  • Everything is so good…rng is the only way to choose
• There is more, but will avoid long lists of book recs (in the spirit of Ezra Klein)

To choose just one - something by Lawrence Lessig

Blogs/Websites:

All available with rss

• https://pluralistic.net/
• https://craphound.com/
• https://stratechery.com/ - need to create a free account
• https://gregorygundersen.com/
• https://annas-blog.org/
• https://scottaaronson.blog/ and https://scottaaronson.com/
• https://thesephist.com/
• https://boz.com/
• https://simonwillison.net/
• https://www.antipope.org
• https://durmonski.com
• https://mcfunley.com/
• https://nutcroft.com/
• https://www.404media.co/
• https://jeremykun.com/
• https://seb.jambor.dev/
• https://overengineer.dev/
• https://www.lesswrong.com/
• https://choward1491.github.io/
• https://benjojo.co.uk/
• https://waxy.org/
• https://ploum.net
• https://fietkau.blog/
• https://www.joelonsoftware.com
• https://itsg.host/
• https://jplattel.nl/
• https://lessig.tumblr.com/
• https://fediversereport.com

To choose just one - https://scottaaronson.blog/

Music:

Warning: I grew up playing jazz, and seriously considered doing it for a living (Roughly in order of musician’s age)

• John Coltrane
• Michael Brecker
• Ahmad Jamal
• McCoy Tyner
• Thelonious Monk
• Chet Baker
• Johnny Hodges
• Wayne Shorter
• Lee Morgan
• Tommy Flanagan
• Pharoah Sanders
• Jackie McLean
• Joe Pass
• Dexter Gordon
• Jaco Pastorius
• Joshua Redman
• Chick Corea
• Kenny Garrett
• Pat Matheny
• Sons of Kemet
• Marshall Gilkes
• Larnell Lewis
• Michael Mayo
• Aaron Parks
• Eric Harland
• Kendrick Scott
• Robert Glasper
• Greg Spero
• Braxton Cook
• Telemakus
• Athletic Progression
• Nate Smith
• Ruby Rushton
• Elena Pinderhughes
• Alfa Mist
• Javiar Santiago
• Avishai Cohen
• Linda May Han Oh
• Brian Blade
• Chris Dave
• Makaya McCraven
• Immanuel Wilkins
• Kassa Overall
• Walter Smith III

this is too long, I could do this forever, but I need to stop. If there is just one choice - John Coltrane

What do you plan on exploring

I usually gravitate to technological problems with clear engineering-based solutions. However, if I come to rabbitholeathon, I would like to push myself. Instead, I would want to read/write about modern federation.

Here are some things I might try to do:

• Aim to write my own declaration about the necessity of federation and decentralized control in social media networks. It may include reasoning about why centralized, proprietary social network are dangerous and unethical, as well as why federated networks are the natural solution to uphold fundamental human rights.
• Try to understand the fundamental motives for federation. Why would we like spreading out our information everywhere? Why did the internet start federated? Does society actually care about federation?
• I find that the people closest to a technology are often the best to find flaws in that technology (see Gary Klein’s Premortem). I think it would be productive to actually try to find the major flaws in federation, and why it could fail.

Anything else we should know

Dietary Restrictions - I am vegan (another one of those belief alignment things)

While writing this I found that Tailscale and Mullvad had a secret partnership that they announced today.

1. Mullvad Announcement
2. Tailscale Announcement
3. Tailscale Docs

I may update this a bit more, but I am going to be tracking a different way to do this with headscale. Some nascent tracking is in an issue.

Using Mullvad and Tailscale

As I have been working in a public library recently, I thought a VPN was warranted and bought a Mullvad gift card off Amazon¹.

When I activated my Mullvad VPN, Tailscale was no longer getting traffic. I realized that Mullvad had configured nftables to forward all traffic to it’s interface (and therefore bypassed Tailscale’s interface)².

If you are just looking for a solution please just apply this nftables configuration³.

table inet mullvad-tailscale {
  chain prerouting {
    type filter hook prerouting priority -100;
    policy accept;
    ip saddr 100.64.0.0/10 ct mark set 0x00000f41 meta mark set 0x6d6f6c65;
  }
  chain outgoing {
    type route hook output priority -100;
    policy accept;
    meta mark 0x80000 ct mark set 0x00000f41 meta mark set 0x6d6f6c65;
    ip daddr 100.64.0.0/10 ct mark set 0x00000f41 meta mark set 0x6d6f6c65;
  }
}

I will take the rest of this blog post to do a small dive into nft and how mullvad split-tunnel works.

The players

The problem

When the mullvad daemon comes up, nftables is then configured to route all of the packets through the new wireguard tunnel that is created (which shows up in the form of an interface).

This nft configuration is why none of the packets are being routed to the tailscale wireguard tunnel anymore.

The solutions

An simple unsatisfactory solution

Let’s just forward traffic for tailscale ips to the tailscale. It’s stupid, it’s simple, it should work.⁹

nft insert rule inet mullvad output oifname "tailscale*" accept
nft insert rule inet mullvad input iifname "tailscale*" accept

However, this completely skips the mullvad daemon, which is somewhat nonideal.

A better solution

Can we do something with mullvad’s split tunnel?

Let’s follow how it is implemented in the linux app

Meta Mark

// mullvad-daemon/src/main.rs:40
fn main() {
    // ...
    let exit_code = match runtime.block_on(run_platform(config, log_dir)) {
        Ok(_) => 0,
        Err(error) => {
            log::error!("{}", error);
            1
        }
    };
    // ...
}

// mullvad-daemon/src/main.rs:128
#[cfg(target_os = "linux")]
async fn run_platform(config: &cli::Config, log_dir: Option<PathBuf>) -> Result<(), String> {
    if config.initialize_firewall_and_exit {
        return crate::early_boot_firewall::initialize_firewall()
            .await
            .map_err(|err| format!("{err}"));
    }
    run_standalone(log_dir).await
}

// mullvad-daemon/src/early_boot_firewall.rs:17
pub async fn initialize_firewall() -> Result<(), Error> {
    let mut firewall = Firewall::new(mullvad_types::TUNNEL_FWMARK)?;
    // ...
}

// mullvad-types/src/lib.rs:23
#[cfg(target_os = "linux")]
pub const TUNNEL_FWMARK: u32 = 0x6d6f6c65;

Connection Tracking Mark

// mullvad-daemon/src/main.rs:40
fn main() {
    // ...
    let exit_code = match runtime.block_on(run_platform(config, log_dir)) {
        Ok(_) => 0,
        Err(error) => {
            log::error!("{}", error);
            1
        }
    };
    // ...
}

// mullvad-daemon/src/main.rs:128
#[cfg(target_os = "linux")]
async fn run_platform(config: &cli::Config, log_dir: Option<PathBuf>) -> Result<(), String> {
    if config.initialize_firewall_and_exit {
        return crate::early_boot_firewall::initialize_firewall()
            .await
            .map_err(|err| format!("{err}"));
    }
    run_standalone(log_dir).await
}

// mullvad-daemon/src/early_boot_firewall.rs:17
pub async fn initialize_firewall() -> Result<(), Error> {
    let mut firewall = Firewall::new(mullvad_types::TUNNEL_FWMARK)?;
    // ...
}

// talpid-core/src/firewall/mod.rs:241
pub struct Firewall {
    inner: imp::Firewall,
}

// talpid-core/src/firewall/linux.rs:288
// ...
    pub fn finalize(mut self, policy: &FirewallPolicy, fwmark: u32) -> Result<FinalizedBatch> {
        self.add_loopback_rules()?;
        self.add_split_tunneling_rules(policy, fwmark)?;
        self.add_dhcp_client_rules();
        // ...
     }
// ...

// talpid-core/src/firewall/linux.rs:296
// ...
    fn add_split_tunneling_rules(&mut self, policy: &FirewallPolicy, fwmark: u32) -> Result<()> {
        // ...
        rule.add_expr(&nft_expr!(meta cgroup));
        rule.add_expr(&nft_expr!(cmp == split_tunnel::NET_CLS_CLASSID));
        rule.add_expr(&nft_expr!(immediate data split_tunnel::MARK));
        // ...
// ...

// talpid-core/src/split_tunnel/linux.rs:16
/// Value used to mark packets and associated connections.
/// This should be an arbitrary but unique integer.
pub const MARK: i32 = 0xf41;

What we find is that there are two marks. One that decides to allow the incoming traffic, and one that puts the traffic outside of the firewall.

For the unenlightened, Emacs¹ according GNU is “an extensible, customizable, free/libre text editor - and more”. Due to it’s extensibility and vibrant community, it can become a catch all application for many of the operations² on your computer. Emacs can take on everything from a simple text editor, to an IDE³, to a fully fledged window manager and desktop environment⁴. Using one application for most tasks can help reduce context switching⁵.

What is Doom Emacs?

From the Doom Emacs repo⁶:

Doom is a configuration framework for GNU Emacs tailored for Emacs bankruptcy veterans who want less framework in their frameworks, a modicum of stability (and reproducibility) from their package manager, and the performance of a hand rolled config (or better). It can be a foundation for your own config or a resource for Emacs enthusiasts to learn more about our favorite operating system.

Essentially, not unlike how Arch Linux provides a baseline over the linux kernel, so you don’t have to deal with linux from scratch, Doom provides a similar abstraction layer over the almost unusable kernel⁷ that is Vanilla Emacs. Doom Emacs expands on Vanilla Emacs to enable friendly extensibility and sane defaults.

Doom Emacs vs Vanilla Emacs

If you are writing code in today’s world, there are a few things that you might expect of your editor:

• █ means full support
• ▅ means some support
• ▁ means limited support

  	Vanilla	Doom
  Syntax Highlighting	▅	█
  Code Completion	▅	█
  Code Formatting 8	▁	█
  Integrated Debugging Capabilities	▅	█
  Version Control Integration	▅	█
  Integrated Terminal9	▅	▅
  Code Navigation and Search	▁	█
  Plugin and Extension System	▁	█
  Performance Analysis10	▁	▁
  Project Management	▁	█
  Cross Platform11	▅	▅

By the way, these are my opinions from my experiences. There are links to packages and documentation where available.

Who uses Emacs?

I think that there are 3 classes of users; new users, moderately experienced users, and extreme veterans, with new users making up the majority of users¹².

Vanilla Emacs does a really bad job of appealing to new users. The Emacs manual¹³ is hard to navigate, and the defaults are horrendous¹⁴. Mainline Emacs defaults unlike Vim¹⁵ has failed to keep up with the times.

My Experience

0-2 years

I started with Emacs in console mode¹⁶, which forced me to learn emacs commands (cause there is no mouse interface).

2-6 years

Then, I used Emacs in GUI mode for 4 years with a custom emacs config which was quite difficult to manage. I really learned a lot more elisp in this time.

6-7 years

Finally, I have switched to Doom and my config is way nicer. The most significant differences for me are package management, startup time¹⁷, and configuration¹⁸.

In addition, my Doom Emacs config is startlingly similar to my old config (just more stable). I had a lot of the same packages installed, but just with a lot more cleanliness.

New Users

The argument is perhaps most convincing for new users. If you are coming to Emacs new, chances are (especially with well-polished tools like VSCode), you are not interested in using a clunky piece of software from the 80s¹⁹. I think that the batteries included approach of Doom Emacs has no comparison with Vanilla Emacs.

The only issue with Doom Emacs for new users is that you don’t learn the true Emacs internals well (however this may not be necessary for most users).

Moderately Experienced Users

If you have experience with Emacs already, I think that Doom Emacs just gives you a way to build the environment you are looking for faster and cleaner than you otherwise would be able to do.

In addition, Doom is extensible enough to where to can customize pretty much everything that you need to²⁰.

Extreme Veterans

I don’t know how much authority I have to talk on this. But if something works perfectly, and you can manage it perfectly, you should not switch to Doom.

It might make your life easier if you want, but also if you’re extremely experienced with Emacs, you probably shouldn’t be reading this.

Summary

Whether it is a completely preconfigured environment for a new user, or a batteries included shell to build a home in, Doom has something to offer. I think the exception to this is for an advanced user who is perfectly happy with their current config. Other than that, for the vast majority of users Doom really is a great tool.

If you want to install Doom, here’s the Doom getting started guide

To write down a history of my thoughts about computer science, and create reference articles about topics I am knowledgeable enough in (at that moment) to write something meaningful about. My goal is only post about things that I will find useful for future me (and which therefore may be useful to you).

How to Deploy a Blog

Choosing a Source of Truth

A source of truth is the place where all of your CI/CD¹ infrastructure stems from. If you use a wordpress site or a CMS² like Grav³, then your source of truth might be the admin page (and underlying db) of that CMS where you edit and save the files.

I am a big fan of using git as a source of truth (gives you version control and much more). In this case, I have a github repo, which acts as the central repository from which everything is based off of.

Choosing a Deployment Method

There are so many ways to get a website up from cloud to just running a bare metal nginx server. I decided to use my kubernetes cluster.

Now, because I have selected an external git server as my source of truth, I need to figure out a way to get the files from that git repo to my cluster.

How to Sync Git Repo

There are two main ways to do this, webhooks and polling.

I ended up choosing polling because letting the git repo have access to the cluster was a hard no⁴ (and I did not want to enable inbound access for the container). You can do this with the git-sync container, or a custom container I made to do webhook syncing with luajit.

How to Serve Files

This was pretty easy. You can just create a custom docker container that runs the hugo server off of some directory of files (or just use busybox and pull the hugo binary).

You can use the fast reload server as well, as that seems pretty stable⁵ (no need to reload the container ever time you change a file).

How to Connect Git Sync Container and Hugo Server

To do this you can take advantage of kubernetes and mount a block of storage between the two. This allows the git sync container to put updated files into the block. Then, because we are using the fast reload server, those changes should be picked up automatically.

This results in a simple yet effective CI/CD infrastructure for using git as a source of truth and hosting your hugo server.

Overall Infrastructure

My deployment files if you want it.

What Now?

The goal is to keep a history of the thoughts and interesting projects I stumble across and put up information about how they work.

Weird Name?

Ya, it’s an emacs thing. Search and replace in emacs can be done with either M-% or M-x query-replace. M-% is a little hard to reach. Some people just deal with it, while others probably rebind it. I have gotten fond of just typing out M-x query-replace and letting ido-mode handle the rest. It is one of the least optimized and most fond parts of my emacs setup.

Credits

I took some inspiration from this post.