frontpage.
newsnewestaskshowjobs

Open Source @Github

fp.

Open in hackernews

Ask HN: Is there a general, multi-PL programming task dataset?

1•quartztz•1y ago
Hello!

Being a student interested in PL design, I have had this idea floating around for a while: the gist is finding out what programming languages LLMs might be the most proficient in, to study their design choices and syntactic features with the goal of designing the perfect language for LLMs. This is, of course, gimmicky, but I entertained the idea for a while as a fun afterschool project.

The challenge is: what would be the best way to evaluate programming performance _in specific languages_? There are two main hypotheses here:

1. There are intrinsic syntactic/structural features that the transformer architecture is uniquely able to parse/reproduce/understand best, leading to higher quality code generated. For example: Lisp dialects make parsing code structure and blocks very easy, so one could assume an LLM can "understand their code better" 2. There is so much Python/JS out there that the question isn't even worth asking, and the performance in those will beat whatever other language you throw at it. This is probably not as much of a point thanks to newer transformer architectures but the question is still up.

I suspect the answer can be made somewhat interesting by considering performance relative to language popularity, but the ground question is: is there a general dataset containing different programming challenges, of varying difficulty, in multiple languages, with standard solutions? I couldn't find anything when I looked around, but I might have missed something obvious. It wouldn't be impossible to build a simple website to crowdsource, but I'm thinking that if I missed something obvious I'd rather find out early than late. Also, if you have any input on the project itself, I'd love to hear your ideas!

Comments

Someone•1y ago
> For example: Lisp dialects make parsing code structure and blocks very easy, so one could assume an LLM can "understand their code better"

I would expect the reverse: lisp has no syntactic sugar, making it harder for a LLM to glue code fragments together in a way that produces valid lisp code. Even guaranteeing that parentheses are correctly nested already can be a challenge.

As to a set of programs: they aren’t exactly what you’re looking for, but I would consider https://projecteuler.net (does not contain solutions, but searching for project Euler solutions” finds some) or https://benchmarksgame-team.pages.debian.net/benchmarksgame.

sargstuff•1y ago
Very open ended questions. Geeks for Geeks loosely organized around computer science topics of study : https://www.geeksforgeeks.org/

nit-pick details:

Ignoring hardware differences, "performance" comparisons can be based on differences between algorithm(s) used vs. how algorithm is implimented. For a given language, "algorithm implimentation performance" can be defined as the trade-offs on how a a given algorithm is implimented in a language (compared to other programming languages, but also easy use/flexibility based on 'language generation level -> https://www.geeksforgeeks.org/generation-programming-languag... )

----------------------

1) General computation language specialty 'modules' not withstanding; "languages" are built/optimised around core algorithmic concepts / anticipated area/concentration of targeted professional environment. aka opencl (gpu), R (statistics), Lisp (engineering design), C (OS level), sql (data selection), jasper reports, cobol (business), etc. Languages tend to be 'popular' because of the ecosystem provided around/for a given language.

snarky side note -> can always write a more standard language that compiles to an esolang & provide appropriate emacs/vim/sed/spacemacs ide support.: https://esolangs.org/wiki/Main_Page

  LLM's are very useful at curating information and recognizing/summarizing "statisical" relevance. aka apl is great for engineering mind set, not so good for business use cases aka cobal.  LLM might recognize a language for a given user that combines commonly used 'apl' aspecs of user and commonly used 'cobal' aspecs of user and recommend a language(s) with suitable commonalities for given user. 


2) Search engine topic 'coding challenges' 'algorithmic coding challenges' brings up many types of answers/sites for honing one's coding skills (various languages, beginner to expert, etc). Coding 'algorithms' vs. coming up with algorithm(s) to code is sort of a side aspect. Also differences in 'competition' challenges vs. 'technical challenges' (aka 512 c64 vs. 1 raspberry pi) ; vs. "computer science coding challenges" vs. 'computational genomic challenges'

     ?? how easy / hard based on 'profession' aka artist vs. software designer 20 years experience programming in scheme; environment -- NASA vs. google vs. insurance company.

   ?? from scratch : https://synoptek.com/insights/it-blogs/10-challenges-every-software-product-developer-faces/

   ?? based on industry standards ?? ; just trying to keep skills honed ??

The Tools of Dan Gelbart

https://scottlocklin.wordpress.com/2026/08/05/the-tools-of-dan-gelbart/
1•o4c•4m ago•0 comments

Questions asked on self-help cassettes

https://tapes.emilysaltz.space/
1•gregsadetsky•6m ago•0 comments

Can robotics models control a 4200lb excavator?

https://www.dreamscalelabs.com/blog/excavate/
1•PenTest-duck•15m ago•0 comments

Show HN: RediWM – A complete Wayland Desktop Environment built in Zig 0.17

https://github.com/oxydizer/rediwm
1•oxydizer•19m ago•0 comments

I made an addicting game that will generate a million in revenue one day

https://www.pufful.com/
1•mattmerrick•21m ago•0 comments

Reviewing footage and audio isn't editing. I built an app that does it for you

https://editmatic.app/en
1•EithanK•22m ago•0 comments

Import PST file to Office 365

https://www.mailsdaddy.com/pst-to-office-365-migration-tool/
1•somit0•26m ago•0 comments

Show HN: Agent-Otel-Kit – OTel Dashboards for Claude Code, Codex and Gemini CLI

https://github.com/s3onghyun/agent-otel-kit
1•s3onghyun•26m ago•0 comments

Show HN: Vindle – Wordle for Cars

https://vindle.app/
1•cachebag•29m ago•0 comments

Show HN: OpenButterfly – Tracking the environmental impact of the AI ecosystem

https://openbutterfly.com
1•ravipatib•29m ago•0 comments

Did the Apollo Guidance Computer use 60% of the ICs in 1963?

https://retrocomputing.stackexchange.com/questions/12158/did-the-apollo-guidance-computer-really-...
1•Skyfall106•30m ago•0 comments

Show HN: New Book

https://github.com/metereindustries/qeg
1•metereindustrie•32m ago•0 comments

HS 9th Grade ELA Assignment Turned into philosophical rambling

https://liamis.dev/blogs/CaptionRambling/
1•ForgottenMail•34m ago•0 comments

Google AI Ultra Subscription Not Worth It

https://www.reddit.com/r/google_antigravity/comments/1xx9nl3/google_ai_ultra_subscription_not_wor...
3•behnamoh•37m ago•0 comments

Jensen Huang's downfall will accelerate US-China 'great decoupling'

https://www.channelnewsasia.com/commentary/nvidia-jensen-huang-affair-us-china-ai-tech-decoupling...
1•Markoff•37m ago•0 comments

RISC-V Matrix Extensions

https://fprox.substack.com/p/risc-v-matrix-extensions
1•hasheddan•42m ago•0 comments

Debian Becomes Unresponsive after 6.1.0-55 Kernel Update

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1154101
4•artooro•43m ago•0 comments

Show HN: A benchmark that diffs the whole database after an AI agent's task

https://github.com/shubhamjoshipromail-svg/software-world-model
1•shubhamjoshipro•47m ago•0 comments

Ask HN: How to visually edit vibe coded websites?

1•keepitverysimpl•47m ago•2 comments

Why did base64 win against uuencode? (2023)

https://retrocomputing.stackexchange.com/questions/3019/why-did-base64-win-against-uuencode
1•zenincognito•59m ago•1 comments

Stop Filming in Log

https://digitalfilms.wordpress.com/2026/11/15/stop-filming-in-log/
3•dabinat•59m ago•0 comments

OpenQuantum's DIY Open-Source Magneto-Optical Atomic Trap

https://www.diyphysics.com/2023/11/10/openquantums-diy-open-source-magneto-optical-atomic-trap/
1•o4c•1h ago•0 comments

Microsoft outage based phishing live now

https://ravenmail.io/blog/microsoft-365-outage-phishing-lure
2•abishekr•1h ago•0 comments

Artex.sh

https://www.artex.sh
1•iLuddite•1h ago•0 comments

An update on Cursor's recent security incident

https://cursor.com/blog/security-incident-update
1•nreece•1h ago•0 comments

Do Firms Know What Workers Want? [pdf]

https://www.econtribute.de/RePEc/ajk/ajkdps/ECONtribute_443_2026.pdf
1•rohanm93•1h ago•0 comments

Artificial Intelligence in Neuromuscular Diseases

https://link.springer.com/article/10.1007/s40120-026-01017-8
1•measurablefunc•1h ago•0 comments

Show HN: Shot Ease – Screen Capture full pages

https://chromewebstore.google.com/detail/shot-ease-screen-capture/cejbjpbkachlmgbphckaikicmnpmfjdn
1•lucakk•1h ago•0 comments

The Nefarious Plan of vm2

https://nefariousplan.com/posts/the-nefarious-plan-of-vm2
1•tsavo42•1h ago•0 comments

Requests for Hardwares

https://blog.greg.technology/2026/11/14/requests-for-hardwares.html
1•gregsadetsky•1h ago•0 comments