Instead of writing imperative crawl loops, you describe what to follow and what to extract in a single expression:
import wxpath
# Crawl, extract fields, build a Wikipedia knowledge graph
path_expr = """
url('https://en.wikipedia.org/wiki/Expression_language')
///url(//main//a/@href[starts-with(., '/wiki/') and not(contains(., ':'))])
/map{
'title': (//span[contains(@class, "mw-page-title-main")]/text())[1] ! string(.),
'url': string(base-uri(.)),
'short_description': //div[contains(@class, 'shortdescription')]/text() ! string(.),
'forward_links': //div[@id="mw-content-text"]//a/@href ! string(.)
}
"""
for item in wxpath.wxpath_async_blocking_iter(path_expr, max_depth=1):
print(item)
Features:
- Async/concurrent crawling with streaming results
- Scrapy-inspired auto-throttle and polite crawling
- Hook system for custom processing
- CLI for quick experiments
Another example, paginating through HN comments (via "follow=" argument) pages and extracting data:
url('https://news.ycombinator.com',
follow=//a[text()='comments']/@href | //a[@class='morelink']/@href)
//tr[@class='athing']
/map {
'text': .//div[@class='comment']//text(),
'user': .//a[@class='hnuser']/@href,
'parent_post': .//span[@class='onstory']/a/@href
}
GitHub: https://github.com/rodricios/wxpath
PyPI: pip install wxpath
I'd love feedback on the expression syntax and any use cases this might unlock.
Thanks!
css_apologist•2w ago
i can understand why it failed for general use, but shit like this revives my excitement
q: i'm not an expert, this looks like it extends xpath syntax? haven't seen stuff like the /map is this referring to the html map element? or a fp-style map?
rodricios•2w ago
If wxpath can help revive some of that excitement, then I consider my project a success.
As for your question, while wxpath does extend the xpath syntax, `/map` is not one of its additions, nor is it a html map element.
XPath 3.1 introduced first-class maps (and arrays) (https://www.w3.org/TR/xpath-31/#id-maps), and `/map` is the syntax to create said structure. It's an awesome feature that's especially useful for quickly delivering JSON-like objects.
css_apologist•2w ago
rhdunn•2w ago
There's currently work on XPath 4.0 -- https://qt4cg.org/specifications/xquery-40/xpath-40.html.
jerf•2w ago
rodricios•2w ago
I should also add that most (Python-based) web crawling and scraping frameworks support XPath engines OOTB: Scrapy, Crawlee, etc. In that sense, XPath is very much alive.