Original Summary

&quot;DuckDB and Make are all you need!&quot;<p>I love the ergonomics of dbt for building reproducible data pipelines with DuckDB -- particularly the ability to define output tables as individual SELECT queries and have them refer to each other -- but have always found the configuration and boilerplate excessive for the projects I work on. With the recent Rust rewrite of dbt Core, now seemed like a good time to consider the bits I actually need&#x2F;use.<p>Recently, I realised that DuckDB itself can parse SQL queries into an AST representation using the json_serialize_sql function. duckdb.mk (~250 lines included in your Makefile) takes advantage of this to generate Make rules for your models automatically, and works with both DuckDB 1.4.1+ and 2.0 preview.<p>Write a neat directory of SQL statements querying local and remote data, get a neat directory of Parquet files (plus a Mermaid diagram of the DAG) back. When you rebuild a table, duckdb.mk keeps track of dependencies (including remote resources queried over HTTP) and only rebuilds those that need it.<p>This system is a distillation of various ad hoc solutions cooked up while working on data journalism and research projects over the past 5-10 years, most recently doing Observable-based research notebooks with Data Desk: <a href="https:&#x2F;&#x2F;research.datadesk.eco&#x2F;" rel="nofollow">https:&#x2F;&#x2F;research.datadesk.eco&#x2F;</a> It&#x27;s very much a starting point for further development, and contributions in line with the general minimalist philosophy are very welcome!<p>Check out the example&#x2F; directory for a simple test project that attributes satellite-observed methane plumes to oil and gas infrastructure using public data.

  • 情报分类：商业与市场研究
  • 分类依据：内容涉及商业、投资或市场动态
  • 信息来源：Hacker News 新项目
  • 发布时间：2026/9/26 22:06:08