Correct the types of registry and cli.oaievalset #1027
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pan93412
requested review from
andrew-openai,
rlbayes,
jwang47 and
logankilpatrick
as code owners
pan93412
changed the title
5 tasks
pan93412
force-pushed
the
fix-up-types-i
branch
4 times, most recently
from
cccbe4b to
f3d66ea
Compare
jwang47
reviewed
Jun 2, 2023
| if command in commands: | ||
| continue | ||
| commands.append(command) | ||
| eval_set = registry.get_eval_set(args.eval_set) if args.eval_set else None |
There was a problem hiding this comment.
We should log an error when the eval set is not found, to make it more clear to the user what happened.
|
@pan93412 Could you resolve the merge conflicts? |
Currently, the `registry` and `cli.oaievalset` won't be checked as expected due to the misform wildcard in Additional Sections. The issue can be addressed by adding a wrong type declaration in the specified files 1�7 before this commit, no errors report; after this commit, it reports 21 errors. This PR fixes this issue.
error: Multiple top-level packages discovered in a flat-layout: ['evals', 'typings'].
Python 3.9 has not supported the former yet.
arbreton
pushed a commit
to arbreton/evals
that referenced
this pull request
Jul 8, 2023
Currently, the `registry` and `cli.oaievalset` won't be checked as expected due to the misform wildcard in Additional Sections. This PR: - re-enable the checks for these modules - correct and strengthen the types of these modules
jacobbieker
pushed a commit
to withmartian/-ARCHIVED--router-evals
that referenced
this pull request
Jan 9, 2024
Currently, the `registry` and `cli.oaievalset` won't be checked as expected due to the misform wildcard in Additional Sections. This PR: - re-enable the checks for these modules - correct and strengthen the types of these modules
Linmj-Judy
pushed a commit
to TablewareBox/evals
that referenced
this pull request
Feb 27, 2024
Currently, the `registry` and `cli.oaievalset` won't be checked as expected due to the misform wildcard in Additional Sections. This PR: - re-enable the checks for these modules - correct and strengthen the types of these modules
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Eval details 📑
This is not a eval.
Currently, the
registryandcli.oaievalsetwon't be checked as expected due to the misform wildcard in Additional Sections.This PR:
Final checklist 👀
Submission agreement
By contributing to Evals, you are agreeing to make your evaluation logic and data under the same MIT license as this repository. You must have adequate rights to upload any data used in an Eval. OpenAI reserves the right to use this data in future service improvements to our product. Contributions to OpenAI Evals will be subject to our usual Usage Policies (https://platform.openai.com/docs/usage-policies).
Email address validation
If your submission is accepted, we will be granting GPT-4 access to a limited number of contributors. Access will be given to the email address associated with the merged pull request.
Limited availability acknowledgement
We know that you might be excited to contribute to OpenAI's mission, help improve our models, and gain access to GPT-4. However, due to the requirements mentioned above and high volume of submissions, we will not be able to accept all submissions and thus not grant everyone who opens a PR GPT-4 access. We know this is disappointing, but we hope to set the right expectation before you open this PR.
Submit eval
pip install pre-commit; pre-commit installand have verified thatblack,isort, andautoflakeare running when I commit and pushFailure to fill out all required fields will result in the PR being closed.