Andrew Reed

Jacob Dodd, Senior ML Engineer at Laurel, shares how Laurel orchestrates RAG pipelines for LLMs with Apache Airflow. Hear how they deliver production-grade context through a mix of real-time systems and scheduled AI workflows. https://bit.ly/4ngDbX0 #BeyondAnalytics

27

GPU VRAM is the most expensive real estate in tech right now, yet standard AI inference software wastes most of it. If you want to serve Large Language Models at scale, you inevitably hit the "Memory Wall." Top AI labs require infrastructure capable of "low-latency, high-throughput sampling." But as context windows grow, standard KV Caches lead to severe memory fragmentation. The result? The GPU starves for data, and users wait. To tackle this bottleneck head-on, I am spending the next 3 weeks building "Mini-vLLM"—a custom C++ inference engine from scratch. To squeeze maximum tokens/sec out of the hardware, Mini-vLLM will implement the holy trinity of modern performance engineering: 1️⃣ PagedAttention: Managing GPU VRAM like Operating System Virtual Memory to eliminate fragmentation. 2️⃣ Continuous Batching: Implementing iteration-level request scheduling so the GPU never idles. 3️⃣ Speculative Decoding: Using smaller draft models to break the autoregressive bottleneck and hack latency. I won't just be shipping code; I’m building this entirely in public. Over the next 21 days, I'll be sharing the exact whitepapers I'm studying, breaking down the complex math behind the architecture, and documenting the brutal C++ bugs I encounter along the way. If you are interested in AI performance engineering, GPUs, or low-level systems architecture, I'll be posting my daily deep-dives, code snippets, and architecture diagrams over on X. 👉 Follow the daily build thread here: https://lnkd.in/gsMJWFNp What do you think is the most challenging aspect of optimizing LLM inference today? Let's discuss below. 👇 #AI #MachineLearning #CPP #PerformanceEngineering #vLLM #SoftwareEngineering #GPUs

22

use huggingface/skills to automate AI and ML workflows. It has skills - core cli and repo management - model training - dataset creation and wrangling - building gradio demos - evaluating models - running remote workloads with hf jobs - publishing papers - building automation tools on the hub - managing training metrics in trackio in many cases, you can one shot core tasks like starting training run or preprocessing datasets.

460

17 Comments

Whatnot was computing 10 billion predictions every night. Many went unused. Emmanuel Fuentes breaks down how they moved from batch inference to real-time ML with Chalk, unlocking $100Ks in weekly revenue for their sellers. Instead of day-old predictions, they now compute features in real-time, adapting instantly as users tap, browse, and buy. Chalk enables them to ship new recommendation algorithms in days instead of weeks. Watch the technical deep dive below.

41

1 Comment

As an ML/AI engineer, I depend heavily on infrastructure, but actually deploying that infrastructure is rarely the fun part. I’ve fought with Terraform enough to know it’s not for me: clunky, slow to iterate, and painful to debug. I prefer Python-first tools like the AWS CDK, but that doesn’t help when I need the same experience on Azure. Pulumi has been on my “should really try this” list for a while, and when Pulumi released Pulumi Neo, I finally dove in. Neo gives you the Pulumi experience with basically zero effort. It’s a chat interface inside Pulumi: describe what you want, and it builds + deploys the infra. I deployed a DynamoDB table without writing a single line of code. My workflow: - create an IAM user - add credentials to Pulumi, share account + region - create a Pulumi GitHub app - create a Pulumi secret and store it in my GitHub repo - tell Neo: “spin up a dynamodb table” That’s it. Fewer iterations than I expected, and I ended up with a repo containing working infra code and deployment pipelines. And sure, this was a toy example. Neo can go way bigger: updating Kubernetes clusters, managing complex configs, deploying full multi-service setups. Pulumi + Pulumi Neo is absolutely worth checking out. Try Pulumi Neo (it is free, no credit card required!): https://lnkd.in/eRenmyin Learn more about Pulumi Neo: https://lnkd.in/ex7zuSiX -------------------------------------------------------- I partnered with Pulumi for this post, but everything here reflects my own experience.

114

21 Comments

Most AI memory systems fail because they treat schema as an afterthought. I have seen dozens of RAG implementations break down in production - not because of embedding quality or retrieval accuracy, but because teams skip the foundational work of defining clear data schemas. Without structured memory models, your AI system becomes a black box that degrades over time. We recently worked with a fintech client whose LLM was hallucinating financial data. The root cause? Their memory system had no schema validation. Documents, user interactions, and context were stored as unstructured blobs. When we implemented strict schema definitions for their knowledge base, accuracy improved by 40% overnight. Start with schema design before you build your RAG pipeline. Define your data structures, validation rules, and memory hierarchies upfront. Your future self will thank you when you are debugging at 2 AM. What is your biggest challenge with AI memory reliability in production? #AIInfrastructure #RAG #MachineLearning #DataEngineering

1

1 Comment