frontpage.
newsnewestaskshowjobs

Open Source @Github

fp.

Open in hackernews

Ask HN: Is there a general, multi-PL programming task dataset?

1•quartztz•1y ago
Hello!

Being a student interested in PL design, I have had this idea floating around for a while: the gist is finding out what programming languages LLMs might be the most proficient in, to study their design choices and syntactic features with the goal of designing the perfect language for LLMs. This is, of course, gimmicky, but I entertained the idea for a while as a fun afterschool project.

The challenge is: what would be the best way to evaluate programming performance _in specific languages_? There are two main hypotheses here:

1. There are intrinsic syntactic/structural features that the transformer architecture is uniquely able to parse/reproduce/understand best, leading to higher quality code generated. For example: Lisp dialects make parsing code structure and blocks very easy, so one could assume an LLM can "understand their code better" 2. There is so much Python/JS out there that the question isn't even worth asking, and the performance in those will beat whatever other language you throw at it. This is probably not as much of a point thanks to newer transformer architectures but the question is still up.

I suspect the answer can be made somewhat interesting by considering performance relative to language popularity, but the ground question is: is there a general dataset containing different programming challenges, of varying difficulty, in multiple languages, with standard solutions? I couldn't find anything when I looked around, but I might have missed something obvious. It wouldn't be impossible to build a simple website to crowdsource, but I'm thinking that if I missed something obvious I'd rather find out early than late. Also, if you have any input on the project itself, I'd love to hear your ideas!

Comments

Someone•1y ago
> For example: Lisp dialects make parsing code structure and blocks very easy, so one could assume an LLM can "understand their code better"

I would expect the reverse: lisp has no syntactic sugar, making it harder for a LLM to glue code fragments together in a way that produces valid lisp code. Even guaranteeing that parentheses are correctly nested already can be a challenge.

As to a set of programs: they aren’t exactly what you’re looking for, but I would consider https://projecteuler.net (does not contain solutions, but searching for project Euler solutions” finds some) or https://benchmarksgame-team.pages.debian.net/benchmarksgame.

sargstuff•1y ago
Very open ended questions. Geeks for Geeks loosely organized around computer science topics of study : https://www.geeksforgeeks.org/

nit-pick details:

Ignoring hardware differences, "performance" comparisons can be based on differences between algorithm(s) used vs. how algorithm is implimented. For a given language, "algorithm implimentation performance" can be defined as the trade-offs on how a a given algorithm is implimented in a language (compared to other programming languages, but also easy use/flexibility based on 'language generation level -> https://www.geeksforgeeks.org/generation-programming-languag... )

----------------------

1) General computation language specialty 'modules' not withstanding; "languages" are built/optimised around core algorithmic concepts / anticipated area/concentration of targeted professional environment. aka opencl (gpu), R (statistics), Lisp (engineering design), C (OS level), sql (data selection), jasper reports, cobol (business), etc. Languages tend to be 'popular' because of the ecosystem provided around/for a given language.

snarky side note -> can always write a more standard language that compiles to an esolang & provide appropriate emacs/vim/sed/spacemacs ide support.: https://esolangs.org/wiki/Main_Page

  LLM's are very useful at curating information and recognizing/summarizing "statisical" relevance. aka apl is great for engineering mind set, not so good for business use cases aka cobal.  LLM might recognize a language for a given user that combines commonly used 'apl' aspecs of user and commonly used 'cobal' aspecs of user and recommend a language(s) with suitable commonalities for given user. 


2) Search engine topic 'coding challenges' 'algorithmic coding challenges' brings up many types of answers/sites for honing one's coding skills (various languages, beginner to expert, etc). Coding 'algorithms' vs. coming up with algorithm(s) to code is sort of a side aspect. Also differences in 'competition' challenges vs. 'technical challenges' (aka 512 c64 vs. 1 raspberry pi) ; vs. "computer science coding challenges" vs. 'computational genomic challenges'

     ?? how easy / hard based on 'profession' aka artist vs. software designer 20 years experience programming in scheme; environment -- NASA vs. google vs. insurance company.

   ?? from scratch : https://synoptek.com/insights/it-blogs/10-challenges-every-software-product-developer-faces/

   ?? based on industry standards ?? ; just trying to keep skills honed ??

AWS CVEs 2024-2026: Is AI Finding the Bugs, or Writing Them?

https://blog.technodrone.cloud/2026/11/aws-cves-ai-finding-or-writing-bugs.html
1•maishsk•1m ago•0 comments

Who's Delivering GTA 6? A Ripe Atlas Measurement Story

https://labs.ripe.net/author/emileaben/whos-delivering-gta-6-a-ripe-atlas-measurement-story/
1•minusf•1m ago•0 comments

Fusion power is about to start working

https://www.economist.com/the-world-ahead/2026/11/18/fusion-power-is-about-to-start-working-says-...
1•mpweiher•2m ago•0 comments

180B Parameters, 32GB of RAM: Pushing Moe Inference to the Limit

https://huggingface.co/blog/moe-expert-streaming
1•joshcsimmons•2m ago•0 comments

EU Commission targeted in cyberattack, Politico reports

https://www.reuters.com/legal/litigation/eu-commission-targeted-cyberattack-politico-reports-2026...
1•gvieri•7m ago•0 comments

The deployment path is a product

https://jgothelf.substack.com/p/the-deployment-path-is-a-product
1•flail•9m ago•0 comments

The Hunt for the Death Valley Germans

https://otherhand.org/home-page/search-and-rescue/the-hunt-for-the-death-valley-germans/
1•oumua_don17•14m ago•0 comments

Hip Hop HQ Magazine

https://www.thisishiphophq.com.ng/#/
1•mrb3njamins•17m ago•0 comments

Educational Demo of the Solar system and its visitors

https://compo.tv/w/perihelion
1•mfbilling•17m ago•0 comments

Show HN: My CTF Setup with BlackArch in Docker

https://coding-overhead.com/posts/blackarch-docker/
1•ismd•20m ago•0 comments

Game theory's cure for corruption (2015)

https://aeon.co/essays/game-theory-s-cure-for-corruption-make-us-all-cops
1•sammatilda•23m ago•0 comments

EUVD-2024-5409 Synology and Beestation – Improper Encoding or Escaping of Output

https://euvd.enisa.europa.eu/vulnerability/EUVD-2024-54098
1•austinallegro•24m ago•0 comments

The New Mind Control – The internet has spawned subtle forms of influence (2016)

https://aeon.co/essays/how-the-internet-flips-elections-and-alters-our-thoughts
1•sammatilda•24m ago•0 comments

DeepMind Employees Are Unionising

https://www.gdmunion.co.uk
2•nervai•26m ago•0 comments

Where do we go from here?

https://nicolasbouliane.com/blog/life-after-ai
1•nicbou•27m ago•0 comments

How to Create a Virtual Background Using the Red5 WebRTC SDK

https://www.red5.net/blog/virtual-backgrounds-using-the-red5-webrtc-sdk/
1•mondainx•28m ago•0 comments

What's New in PHP 8.6

https://www.php.net/releases/8.6/en.php
19•netol•28m ago•2 comments

The gas pipeline where Ukraine is fighting an underground war with Russia

https://www.bbc.com/news/videos/c8p3vw4q6nzzo
2•ZeljkoS•30m ago•0 comments

Placky Explains Slide Rules (2021)

https://randomlyunique.com/plancky/2021/slide-rules/index.html
1•kqr•36m ago•0 comments

SoduWork – Open-source workbench where agents must prove their work

https://github.com/mdiri-ai/SoduWork
1•WavyPeng•37m ago•0 comments

There's a New Disruptor Throwing Terrible AI Slop at Sports Fans

https://defector.com/theres-a-new-disruptor-throwing-terrible-ai-slop-at-sports-fans
1•abbadadda•37m ago•0 comments

Deadweight: What your Claude Code setup costs on every request

https://github.com/msertdev/deadweight
1•msertdev•41m ago•0 comments

Binaryen – compiler and toolchain library for WebAssembly, written in C++

https://github.com/webassembly/binaryen
1•peter_d_sherman•41m ago•1 comments

I Don't Read Code Anymore [video]

https://www.youtube.com/watch?v=RO52GyQQMKg
2•zwischenzug•43m ago•0 comments

Florida man sentenced to 60 months for distributing monkey torture videos

https://www.justice.gov/opa/pr/miami-area-man-sentenced-60-months-prison-distributing-sexual-tort...
2•illidan1000•47m ago•1 comments

Ruby2ẞhell blog by one of the reporters

https://durendal.sh/blog/ruby-case-mapping-bugs
1•realansgar•48m ago•0 comments

Who Made the Strawberry?

https://strawberry-methodology.pages.dev/
1•DeepLogin•50m ago•0 comments

SSD prices will start going down the second half of next year

https://www.drivevault.io/blog/when-will-hard-drive-prices-come-down.html
38•illidan1000•50m ago•18 comments

NGINX2Root: Remote Code Execution vulnerabilities in Nginx

https://lwn.net/Articles/1108362/
4•bashhq_r•51m ago•0 comments

Taylor Swift's favorite restaurants and hotels

https://wherecelebsgo.com/people/taylor-swift
2•ispotcelebs•52m ago•2 comments