Codeparrot/githubpairs & co

[Muennighoff]

APPS requires mapping the dataset to the below:

def add_solution_apps(example):
    example["solution"] = random.choice(json.loads(example["solutions"]))
    return example

XLCost requires mapping the dataset to the below:

def clean_code_xlcost(example):
    clean_lines = []
    cur_indent = 0
    for line in example["code"].split("NEW_LINE"):
        cur_indent += line.count("INDENT")
        cur_indent -= line.count("DEDENT")
        line = line.replace("INDENT", "").replace("DEDENT", "")
        line = line.replace("STRNEWLINE", "\n")
        line = line.replace("TABSYMBOL", "\t")
        clean_lines.append("\t" * cur_indent + line.strip())
    example["code_clean"] = "\n".join(clean_lines)
    return example