frontpage.
newsnewestaskshowjobs

Open Source @Github

fp.

Open in hackernews

Ask HN: Is there a general, multi-PL programming task dataset?

1•quartztz•1y ago
Hello!

Being a student interested in PL design, I have had this idea floating around for a while: the gist is finding out what programming languages LLMs might be the most proficient in, to study their design choices and syntactic features with the goal of designing the perfect language for LLMs. This is, of course, gimmicky, but I entertained the idea for a while as a fun afterschool project.

The challenge is: what would be the best way to evaluate programming performance _in specific languages_? There are two main hypotheses here:

1. There are intrinsic syntactic/structural features that the transformer architecture is uniquely able to parse/reproduce/understand best, leading to higher quality code generated. For example: Lisp dialects make parsing code structure and blocks very easy, so one could assume an LLM can "understand their code better" 2. There is so much Python/JS out there that the question isn't even worth asking, and the performance in those will beat whatever other language you throw at it. This is probably not as much of a point thanks to newer transformer architectures but the question is still up.

I suspect the answer can be made somewhat interesting by considering performance relative to language popularity, but the ground question is: is there a general dataset containing different programming challenges, of varying difficulty, in multiple languages, with standard solutions? I couldn't find anything when I looked around, but I might have missed something obvious. It wouldn't be impossible to build a simple website to crowdsource, but I'm thinking that if I missed something obvious I'd rather find out early than late. Also, if you have any input on the project itself, I'd love to hear your ideas!

Comments

Someone•1y ago
> For example: Lisp dialects make parsing code structure and blocks very easy, so one could assume an LLM can "understand their code better"

I would expect the reverse: lisp has no syntactic sugar, making it harder for a LLM to glue code fragments together in a way that produces valid lisp code. Even guaranteeing that parentheses are correctly nested already can be a challenge.

As to a set of programs: they aren’t exactly what you’re looking for, but I would consider https://projecteuler.net (does not contain solutions, but searching for project Euler solutions” finds some) or https://benchmarksgame-team.pages.debian.net/benchmarksgame.

sargstuff•1y ago
Very open ended questions. Geeks for Geeks loosely organized around computer science topics of study : https://www.geeksforgeeks.org/

nit-pick details:

Ignoring hardware differences, "performance" comparisons can be based on differences between algorithm(s) used vs. how algorithm is implimented. For a given language, "algorithm implimentation performance" can be defined as the trade-offs on how a a given algorithm is implimented in a language (compared to other programming languages, but also easy use/flexibility based on 'language generation level -> https://www.geeksforgeeks.org/generation-programming-languag... )

----------------------

1) General computation language specialty 'modules' not withstanding; "languages" are built/optimised around core algorithmic concepts / anticipated area/concentration of targeted professional environment. aka opencl (gpu), R (statistics), Lisp (engineering design), C (OS level), sql (data selection), jasper reports, cobol (business), etc. Languages tend to be 'popular' because of the ecosystem provided around/for a given language.

snarky side note -> can always write a more standard language that compiles to an esolang & provide appropriate emacs/vim/sed/spacemacs ide support.: https://esolangs.org/wiki/Main_Page

  LLM's are very useful at curating information and recognizing/summarizing "statisical" relevance. aka apl is great for engineering mind set, not so good for business use cases aka cobal.  LLM might recognize a language for a given user that combines commonly used 'apl' aspecs of user and commonly used 'cobal' aspecs of user and recommend a language(s) with suitable commonalities for given user. 


2) Search engine topic 'coding challenges' 'algorithmic coding challenges' brings up many types of answers/sites for honing one's coding skills (various languages, beginner to expert, etc). Coding 'algorithms' vs. coming up with algorithm(s) to code is sort of a side aspect. Also differences in 'competition' challenges vs. 'technical challenges' (aka 512 c64 vs. 1 raspberry pi) ; vs. "computer science coding challenges" vs. 'computational genomic challenges'

     ?? how easy / hard based on 'profession' aka artist vs. software designer 20 years experience programming in scheme; environment -- NASA vs. google vs. insurance company.

   ?? from scratch : https://synoptek.com/insights/it-blogs/10-challenges-every-software-product-developer-faces/

   ?? based on industry standards ?? ; just trying to keep skills honed ??

Math Books by Robert Ghrist

https://www2.math.upenn.edu/~ghrist/books/
1•ibobev•1m ago•0 comments

Attacker swapped Prime Intellect's DeepSeek v4.1 Flash endpoint for 53 hours

https://runtimewire.com/article/openrouter-notified-users-of-a-reported-prime-intellect-compromise
1•dvaplima•1m ago•0 comments

Synology Security Advisory

https://www.synology.com/en-uk/security/advisory/Synology_SA_26_21
3•choult•4m ago•0 comments

Show HN: I built a deterministic agent memory engine in Rust with no LLM

https://github.com/samvallad33/vestige
1•entity002•6m ago•1 comments

Learning to Compress a Billion Bytes of Wikipedia

https://tufalabs.ai/research/tufazip/
1•7777777phil•7m ago•0 comments

Apple Is Working on AI Agent for Device Control, Apps, Browsing and Coding

https://www.bloomberg.com/news/articles/2026-11-19/apple-is-working-on-ai-agent-for-device-contro...
1•mfiguiere•9m ago•1 comments

Generation of Neutron Bessel Beams

https://arxiv.org/abs/2611.24655
2•titusblair•11m ago•0 comments

Bending Spoons has entered into a definitive agreement to acquire Mailchimp

https://www.sec.gov/Archives/edgar/data/2004711/000200471126000026/bsp_ex99-1.htm
2•taubek•11m ago•0 comments

Who's Delivering GTA 6? A Ripe Atlas Measurement Story

https://labs.ripe.net/author/emileaben/whos-delivering-gta-6-a-ripe-atlas-measurement-story/
1•timpera•12m ago•0 comments

80% of Coding-Agent Output Tokens Were Spent on Calls That Didn't Change Code

https://arxiv.org/abs/2611.23375
1•khalidkhair•13m ago•0 comments

Muscle-memory – repeated Claude Code tasks run with zero tokens

https://github.com/kabirshah4/muscle-memory
1•kabir_shah•14m ago•0 comments

Posophobia

https://adam.rochussen.xyz/p/posophobia
1•Michelangelo11•18m ago•0 comments

Welcome to Software Beirut

https://www.theatlantic.com/ideas/2026/11/software-beirut-ai-outages/689508/
4•jonah•19m ago•0 comments

Software Sins of Omission

https://www.johndcook.com/blog/2010/01/12/software-sins-of-omission/
1•olalonde•20m ago•0 comments

Flipper One, Open Linux Multi-Tool, Kickstarter Video

https://www.youtube.com/watch?v=MMnnGAMqSUI
2•robrain•21m ago•1 comments

Replit Datacat – their self-hosted observability is AI slop and it doesn't work

https://github.com/replit/datacat
1•AsDivyansh•22m ago•0 comments

AI Agent Authorization That Limits the Damage of Prompt Injection

https://www.stackbuilders.com/insights/ai-agent-authorization-that-limits-the-damage-of-prompt-in...
2•StackBuilders•22m ago•0 comments

Show HN: Stream GTA VI into Apple's Picture in Picture

https://github.com/slaughterhousesoftware/shimmer
1•kevinbpatel•22m ago•0 comments

Modular rebuilds MAX's Module API with eager execution and PyTorch-like syntax

https://www.modular.com/blog/reintroducing-max-module
1•andrewstetsenko•23m ago•0 comments

Context API extracts full context from any web page

https://www.perplexity.ai/hub/blog/context-api-extracts-full-context-from-any-web-page
1•gmays•31m ago•0 comments

Valve Steam Machine: Reservation and waitlist updates

https://store.steampowered.com/news/group/45479024/view/687522235410285038
1•HelloUsername•32m ago•0 comments

What Makes UltraSMR Different?

https://blog.westerndigital.com/what-is-ultrasmr-technology/
1•bluedino•33m ago•0 comments

Host your repo, AI tool or service on ASTAR for learners to build with

https://beta.astar.inc
1•PlayerToo•34m ago•1 comments

BYD's folding steering wheel isn't a concept – it's in a $49K SUV now

https://electrek.co/2026/11/19/byd-folding-steering-wheel-tai-9-production/
5•cisc•37m ago•1 comments

Bending Spoons Acquires Mailchimp

https://www.businesswire.com/news/home/20261119179325/en/Bending-Spoons-has-entered-into-a-defini...
4•nthypes•39m ago•0 comments

Pillars of Enduring Strength: Learning from Hermann Weyl

https://academic.oup.com/book/63460/chapter/577772313
1•jjgreen•40m ago•0 comments

Bending Spoons to acquire Mailchimp

https://investors.bendingspoons.com/newsroom/bending-spoons-to-acquire-mailchimp
11•mutator•41m ago•1 comments

Codeberg.org domain suspension on 2026-11-18

https://codeberg.org/Codeberg/changelog/issues/7#issuecomment-28100157
2•robin_reala•42m ago•1 comments

Rest of World faces prospect of shutting down

https://www.semafor.com/article/10/25/2026/rest-of-world-faces-prospect-of-shutting-down
1•Michelangelo11•43m ago•0 comments

Muxlio – Durable Agent Workflows

https://github.com/Muxlio/muxlio
3•muxlio•45m ago•0 comments