res_8c35a84aOn HumanEval, a new evaluation set we release to measure functional correctness for synthesizing programs from docstrings, our model solves 28.8% of the problems, while GPT-3 solves 0% and GPT-J … (excerpt)
arXiv 2107.03374 · cs.LG
Read on arXiv · arXiv non-exclusive license
Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.
res_8c35a84aOn HumanEval, a new evaluation set we release to measure functional correctness for synthesizing programs from docstrings, our model solves 28.8% of the problems, while GPT-3 solves 0% and GPT-J … (excerpt)
res_8b5adebcUsing this method, we solve 70.2% of our problems with 100 samples per problem.
res_86bc4704Indeed, we find that the sample with highest mean log-probability passes unit tests for 44.5% of the problems.
clm_19586faaWe argue that this metric should be applied to docstringconditional code generation as well.
clm_101b126cOne may be tempted to estimate pass@k with 1−(1− ˆp)k where ˆp is the empirical estimate of pass@1, but we show that it is biased in Appendix A. def pass_at_k(n, … (excerpt)
res_0766194cThis allows us to represent code using approximately 30% fewer tokens.
clm_9d38d1a5Inspired by similar work in language modeling, we find that choosing the sample with the highest mean token log probability outperforms evaluating a random sample, while choosing the sample based … (excerpt)
res_3e559247We can do better than randomly selecting a sample by choosing the solution with the highest mean log-probability (red) or with the highest back-translation score (orange) described in Sec. 5.
res_6a6e14c0We confirm these findings using the HumanEval dataset, showing that GPT-Neo achieves 6.4% pass@1 and 21.3% pass@100, while GPT models of comparable sizes achieve near 0% on both metrics.
res_8301a379We see a remarkable progression in capabilities, with GPT-Neo-2.7B roughly equivalent to Codex-85M (30× fewer parameters).
res_68e155adFinally, we benchmark Codex against the largest free model from Tabnine, a leading code autocomplete system, which achieves 2.6% pass@1 (atT = 0.4) and 7.6% pass@100 (atT = 0.8).
res_9cf8f278We denote this setting as “1-shot” in Table 2, and find that Codex- 12B evaluated 1-shot achieves comparable performance to a GPT-Neo model fine-tuned on APPS.
res_01eb5551Finally, evaluating the first sample which passes the 3 public unit tests for each problem yields higher performance than raw pass@100 samples.
res_06998225We used temperature 0.6 for sampling to cover all k in pass@k, so raw pass@1 results could be improved with lower temperature.
res_650a75d1We find that Codex-S prefers slightly higher temperatures for all k > 1, which possibly reflects the fact that Codex-S captures a narrower distribution than Codex.
res_c1d571daCodex-S outperforms the corresponding Codex by an average margin of 6.5 percentage points on pass@1 and by a larger average margin of 15.1 percentage points on pass@100 across model size.
res_2be80a75When ranking between 1 and 100 samples by mean log probability, the average benefit over random ranking is 11.6 percentage points, which is over 2 percentage points higher than the … (excerpt)
res_39086039Unfortunately, in Figure 7, we show that ranking samples via back-translation underperforms mean log-probability ranking, though it outperforms random ranking.
res_cbe88c39Each building block is specified by a line of text and a line of code: 1. “remove all instances of the letter e from the string” s = s.replace("e", "") … (excerpt)
res_b4c8e4acIn the example evaluation shown in Figure 14, we deduce that the model is capable of outputting code with a lower frequency of bugs, based on the rate of bugs … (excerpt)
Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2107.03374) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.