MearcAll papers

arXiv 2107.03374 · cs.LG

Evaluating Large Language Models Trained on Code

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, and 52 more

Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.

No reproduction recordedres_8c35a84a
On HumanEval, a new evaluation set we release to measure functional correctness for synthesizing programs from docstrings, our model solves 28.8% of the problems, while GPT-3 solves 0% and GPT-J … (excerpt)

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_8b5adebc
Using this method, we solve 70.2% of our problems with 100 samples per problem.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_86bc4704
Indeed, we find that the sample with highest mean log-probability passes unit tests for 44.5% of the problems.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_19586faa
We argue that this metric should be applied to docstringconditional code generation as well.

Functional Correctness · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_101b126c
One may be tempted to estimate pass@k with 1−(1− ˆp)k where ˆp is the empirical estimate of pass@1, but we show that it is biased in Appendix A. def pass_at_k(n, … (excerpt)

Functional Correctness · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_0766194c
This allows us to represent code using approximately 30% fewer tokens.

Methods · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_9d38d1a5
Inspired by similar work in language modeling, we find that choosing the sample with the highest mean token log probability outperforms evaluating a random sample, while choosing the sample based … (excerpt)

Results · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_3e559247
We can do better than randomly selecting a sample by choosing the solution with the highest mean log-probability (red) or with the highest back-translation score (orange) described in Sec. 5.

Results · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_6a6e14c0
We confirm these findings using the HumanEval dataset, showing that GPT-Neo achieves 6.4% pass@1 and 21.3% pass@100, while GPT models of comparable sizes achieve near 0% on both metrics.

Comparative Analysis of Related Models and · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_8301a379
We see a remarkable progression in capabilities, with GPT-Neo-2.7B roughly equivalent to Codex-85M (30× fewer parameters).

Comparative Analysis of Related Models and · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_68e155ad
Finally, we benchmark Codex against the largest free model from Tabnine, a leading code autocomplete system, which achieves 2.6% pass@1 (atT = 0.4) and 7.6% pass@100 (atT = 0.8).

Comparative Analysis of Related Models and · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_9cf8f278
We denote this setting as “1-shot” in Table 2, and find that Codex- 12B evaluated 1-shot achieves comparable performance to a GPT-Neo model fine-tuned on APPS.

Results on the APPS Dataset · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_01eb5551
Finally, evaluating the first sample which passes the 3 public unit tests for each problem yields higher performance than raw pass@100 samples.

Results on the APPS Dataset · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_06998225
We used temperature 0.6 for sampling to cover all k in pass@k, so raw pass@1 results could be improved with lower temperature.

Problems from Continuous Integration · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_650a75d1
We find that Codex-S prefers slightly higher temperatures for all k > 1, which possibly reflects the fact that Codex-S captures a narrower distribution than Codex.

Results · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_c1d571da
Codex-S outperforms the corresponding Codex by an average margin of 6.5 percentage points on pass@1 and by a larger average margin of 15.1 percentage points on pass@100 across model size.

Results · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_2be80a75
When ranking between 1 and 100 samples by mean log probability, the average benefit over random ranking is 11.6 percentage points, which is over 2 percentage points higher than the … (excerpt)

Results · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_39086039
Unfortunately, in Figure 7, we show that ranking samples via back-translation underperforms mean log-probability ranking, though it outperforms random ranking.

Docstring Generation · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_cbe88c39
Each building block is specified by a line of text and a line of code: 1. “remove all instances of the letter e from the string” s = s.replace("e", "") … (excerpt)

Building Blocks for Synthetic Tasks · page 24 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_b4c8e4ac
In the example evaluation shown in Figure 14, we deduce that the model is capable of outputting code with a lower frequency of bugs, based on the rate of bugs … (excerpt)

Results of alignment evaluations · page 27 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2107.03374) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.