A rolling summary of 35 unique engineering articles from 5 available daily digests.
Topic Trends
- AI & Machine Learning — 15 articles
- Developer Tools — 7 articles
- Engineering Culture — 5 articles
- Infrastructure & Scale — 4 articles
- Databases & Storage — 2 articles
Source Pulse
- AWS News — 7 articles
- Google Developers — 7 articles
- GitHub Blog — 5 articles
- Netflix TechBlog — 5 articles
- Stripe Engineering — 5 articles
Representative Articles
Democratizing Machine Learning at Netflix: Building the Model Lifecycle Graph
Netflix TechBlog · AI & Machine Learning
MDS is optimized for real-time ingestion of ML metadata (e.g., models, features, pipelines, experiments, datasets) and to answer cross-domain questions such as “Which experiments are running this model?” or “Which models share these features?” It is the foundation that enables discovery, ingesting events from diverse sources, enriching them with context, and materializing relationships across entities.Our vision: to make every ML asset at Netflix discoverable, understandable, and reusable by every ML practitioner, regardless of their team or domain.Core Abstractions: The Vocabulary of the SystemBefore diving into the technical implementation, it’s helpful to understand the conceptual model that underpins MDS If a new model registry were introduced, it could be added as an additional provider without changing the domain interface.We can summarize these concepts with a concrete example:This URI-based addressing scheme is crucial as it allows any service to reference any ML asset with a single string, and MDS can resolve that reference back to rich, connected metadata.From Events to Entities to GraphThe journey from raw system events to a queryable graph happens in stages Now it’s a contiguous journey in a single interface.This graph-based exploration answers questions that were previously impossible:Lineage queries: What is the complete lineage of this model, from training data to production experiments?Impact analysis: Which models will be affected if I change this feature?Usage discovery: Which A/B tests are using this model?Dependency mapping: What data sources does my pipeline transitively depend on?Deprecation planning: Which entities are no longer being used and can be retired?Every entity has deep context: its creation time, ownership, update history, and most importantly, its relationships to other entities.The Model Lifecycle Graph is surfaced to practitioners through the AIP Portal, a unified interface that provides full-text search across all entity types, detailed entity pages with navigable relationships, and personalized views for teams and individuals.A typical interaction in the AIP Portal looks like:Search: Type a model, feature, dataset, or team name into the single search box backed by Elasticsearch.Inspect: Land on an entity page that shows key metadata (description, owners, domains, tags) alongside a relationships panel.Explore: Click through to related entities (upstream datasets, downstream experiments, and sibling model versions) to navigate the Model Lifecycle Graph without leaving the portal.When new entity types are introduced into MDS, the portal automatically provides baseline search, entity pages, and relationship navigation, and we can then layer on domain-specific visualizations (such as model deployment history or dataset version timelines) over time.The Road Ahead: Open ChallengesBuilding the ML lifecycle graph is an ongoing journey
GitHub for Beginners: Getting started with Git and GitHub in VS Code
GitHub Blog · Developer Tools
Discover how to use VS Code to interact with GitHub and maintain your projects The post GitHub for Beginners: Getting started with Git and GitHub in VS Code appeared first on The GitHub Blog.
Reel Friends: Building Social Discovery that Scales to Billions
Meta Engineering · Engineering Culture
It highlights Reels your friends have watched and reacted to But sometimes the features that seem the most straightforward require the deepest engineering work The post Reel Friends: Building Social Discovery that Scales to Billions appeared first on Engineering at Meta.
Migrating Data Ingestion Systems at Meta Scale
Meta Engineering · Infrastructure & Scale
Meta’s data ingestion system, which our engineering teams leverage for up-to-date snapshots of the social graph, has recently undergone a significant revamp to enhance its reliability at scale Moving from our legacy system to our new architecture required a large-scale migration of our entire data ingestion system The post Migrating Data Ingestion Systems at Meta Scale appeared first on Engineering at Meta.
High-Throughput Graph Abstraction at Netflix: Part I
Netflix TechBlog · Databases & Storage
Looking ahead, we plan to leverage the graph schema for additional improvements, such as:Minimizing Query Fanout: By using edge cardinality within edge mappings, we aim to select the most efficient traversal paths and minimize query fanout.Improved Developer Experience: The schema will support generating a type-safe data access layer and enhance the Gremlin-like API with schema awareness.Next, let’s look at how this data is organized in a real-time index within the KV Abstraction.Real-Time Index: Key-Value StorageBefore we discuss how the data is organized into graph indexes, let’s discuss how KV organizes data within namespaces and provides idempotency guarantees:Data partitioning: A namespace is associated with a table in the underlying storage layer In order to ensure optimal performance without exerting too much memory pressure, we aim to limit the number of edges per source node within the system.Next, let’s explore the caching strategies used by the Abstraction.Caching Strategies in Graph AbstractionAlthough the Graph Abstraction already provides efficient reads and writes to durable storage, caching remains critical for the stability and performance of any graph datastore for two key reasons:Write amplification: A single write on the fronting service can result in multiple writes to the backing durable storage due to the use of multiple indexes As illustrated in the diagram below, both the caching layer and durable storage replicate data asynchronously across regions, resulting in an eventually consistent system.Now that we’ve covered storing the real-time graph index, let’s see how it enables graph traversals.Graph TraversalsThe Abstraction provides a custom gRPC traversal API, inspired by Gremlin, which enables exploration of the distributed graph by letting users chain traversals, apply filter criteria, sort results, limit results, and more.Let’s explore a hypothetical scenario where the Abstraction is used to recommend shows to users on a shared device, by considering the duration of the most recent viewing session for each show across all profiles and accounts associated with that device:TraversalRequest.newBuilder() .setNamespace("
Method
Articles are de-duplicated by URL, then classified with a deterministic engineering keyword taxonomy. The report is generated during the site build and does not use a database or an external AI API.