Original Summary

I built Markdownee, an open-source tool for turning (scraping) web pages and websites into Markdown, HTML or plain text. It is useful for LLM workflows, retrieval pipelines and research, etc... It performs crawling/fetching URLs and does boilerplate removal (main-content extraction). I is configurable, you can choose which links to follow, set page/depth limits, and control whether tables, links, images and comments are kept. It is available as an npm CLI/library and a native Python library, under Apache-2.0. The playground lets you preview a single page, adjust the extraction settings, and generate commands or code for your own crawl. Try it: https://www.markdownee.com/ Source: https://github.com/markdownee/markdownee I would appreciate feedback from anyone, but particurarly from people who scrapes websites (please answe on or more quetion): If you are scraping websites, what is the biggest friction/problems in your current extraction setup? Would you use it, do you find it useful? Why would you use it and why not? Do you find Markdownee simple, understandable? Would you appreciate that it uses Trafilatura Core (my own fork of well known boilerplate removal lib Trafilatura) instead of common Mozilla Readability?   submitted by   /u/GCSurvivor [link]   [comments]

  • 情报分类：商业与市场研究
  • 分类依据：内容涉及商业、投资或市场动态
  • 信息来源：Reddit · SideProject
  • 发布时间：2026/10/14 23:33:47