The goal is: Do deep research -> turn it into bite size swipeables, like insta reels but just text blurbs about something interesting. This could be super useful for coffee chats, dates (I imagine it would be the API that the cluely launch vid https://www.youtube.com/watch?v=Rz3LD7u2KX8 dating people search.)
I wanna discuss novel, out of the box approaches to doing this.
Here are all the ideas I've come up with:
What makes it complex is:
1) People can have same names so articles on internet may not exist 2) Classifying "interesting" things about people.
a) The obvious approach is to use LLMs but even if the cost of the LLMs go down, the rate limits arent going to go down and the volume of links to crawl for truly interesting insights of people limits this.
b) (not a promo), using hierchical text classification with https://www.trytaylor.ai/
Ig the logic is, the more the buckets of info that a specific sentence fills, the more "interesting/dimensional" it is.
Current flow i'm thinking about is:
1) Get the first 20 search results of the person and scrape, (ud be surprised how much info this gives on a person if you know how to use it well).
2) Scrape (if some websites like linkedin are hard, use brightdata, if youtube then get transcript).
3) Have a person fact store -> Simple Array.
Assume I know that John is the person I wanna find and he has a linkedin.
Since I have his linkedin, my person fact store will have the following details:
[ 1) Profile Pic 2) JP morgan Internship 3) UPenn Bachelors ]
etc.
4) Data Validation.
Solving the problem of matching names.
Lets say I now go and find an article about John but I dont know if this is the john that its referring to.
Lets say the article is:
John the JP morgan intern likes sushi.
There are two approaches here
a) I know his profile pic so try to see if theres a match with his linkedin pfp and this one. If the pic in the article matches, then we can append the fact that he likes sushi. So this enriches our array.
OR
b) I know that he is a JP morgan intern from our ground truth, so we can assume sushi pref is also true.
5) Running it through https://www.trytaylor.ai, im trying to get access to product but their auth is fucked, I literally can't use it rn.
Essentially it would classify all this information that I collected into buckets and sub-buckets.
Narrow it down to the top 5% of results that fit a certain criteria since I now have the tags for this.