No official torrent yet — requesting it tells the flock what to host next. You cannot seed a catalog-only page.
Cataloged from ibm-granite/granite-3.3-8b-math-prm-v2
All models by ibm-granite →Model Summary
Granite 3.3 8B Math PRM v2 is a finetuned version of the 8-billion parameter language model, Granite-3.3-8B-Instruct, built for use a generative process reward model (PRM) for process supervision in mathematical reasoning. Crucially, this model has only been trained on curated data from sources with permissive licenses, and we release this model under a Apache 2.0 license. This model displays state-of-the-art performance in a best-of-N set up for math tasks.
This model can be used to asses the correctness of each step of a mathematical reasoning process, and shows strong performance on Best-of-N evaluations for a variety of generators on Math-500, as well as strong error identification performance in both ProcessBench and PRMBench, with state-of-the-art performance in its size class. Although this model was trained for mathematical reasoning tasks, it also shows strong inference scaling performance on Code benchmarks, such as HunmanEval and LCBv5.
Supported Languages
This model has specifically been finetuned for English, however the base model supports English, German, Spanish, French, Japanese, Portuguese, Arabic, Czech, Italian, Korean, Dutch, and Chinese.
Intended Use
Granite 3.3 8B Math PRM v2 is finetuned version of Granite-3.3-8B-Instruct, which gives the language model the ability of process supervision on mathematical reasoning steps by assessing the correctness of each step of a reasoning chain. At inference, the model takes a question and a response which can be broken down into generated steps, and for each step it determines whether the reasoning chain so far is correct (indicated by generating a single token, Y) or incorrect (indicated by generating N). The probability of generating Y can be treated as a numeric reward score in applications such as Best-of-N evaluation.
Before obtaining a response, the model expects the user generated prompt "Is this response correct so far (Y/N)?", which should be added at the end of every step of the reasoning chain.
This model is an update to granite-3.3-8b-lora-math-prm.
Evaluation Results
a. Best-of-N Evaluation on Math-500
We show the performance of MATH-500 with inference scaling on generations from granite-4.0-h-small and granite-4.0-h-tiny, and show strong gains over Majority Voting with both Best-of-N and Weighted Majority Voting using Granite-3.3-8B-Math-PRM-v2.
[Image blocked: PRM Performance on Math-500]We also compare the Best-of-N performance on Math-500 available PRMs on Qwen-2.5-Math-7B-Instruct generations, and show the superior performance of Granite-3.3-8B-Math-PRM-v2 over majority voting and best-of-N using similarly sized PRMs:
b. Best-of-N Evaluation on HumanEval and LCBv5
We show the performance on two code tasks, HumanEval (Python) and LCBv5 with inference scaling on generations from granite-4.0-h-small. While this model is finetuned on Math data, Granite-3.3-8B-Math-PRM-v2 demonstrates strong gains over Majority Voting with both Best-of-N and Weighted Majority Voting.
[Image blocked: PRM Performance on Math-500]c. ProcessBench
[Image blocked: PRM Performance on ProcessBench and PRMBench]As shown above, Granite-3.3-8B-Math-PRM-v2 shows strong performance (top-2) on both ProcessBench and PRMBench compared to other models of the same parameter class, indicating a strong ability at error detection for reasoning tasks.
d. PRMBench: Detailed Results
Training Data The model is first finetuned on math and code data, such as OpenMathReasoning, OpenThoughts3, Codeforces, Code Contests, OpenCodeReasoning, PrimeIntellect, and TACO on the standard language modeling objective. Then, we conduct RLVR on NuminaMath, PrimeIntellect, TACO, OpenCodeInstruct and Skywork-OR1-RL Data. Finally, we train the model to predict the correctness of math problems, curating ttraining data from a diverse set of model responses to prompts from Math-specific datasets, specifically, MetaMathQA, MathInstruct and NuminaMath. We leverage a diverse set of LLMs from the Granite Language Model Family, Phi-4, and Mixtral 8x22B to generate outputs, and use the Automatic Process Supervision method as described in Luo et. al, 2024 for detecting steps with errors.
Usage
Sample use for obtaining PRM scores for a given response using Huggingface Transformers:
For use of the PRM as a verbalizer of correctness for a specific step:
Infrastructure
We train Granite-3.3-8B-Math-PRM-v2 using IBM's super computing cluster, Blue Vela, which is outfitted with NVIDIA H100 GPUs. This cluster provides a scalable and efficient infrastructure for training our models over multiple GPUs.
Ethical Considerations and Limitations
Granite-3.3-8B-Math-PRM-v2 is finetuned from Granite-3.3-8B-Instruct. Since it inherits its foundation from the instruct model, all ethical considerations and limitations applicable to Granite-3.3-8B-Instruct remain relevant.
import torchfrom transformers import AutoModelForCausalLM, AutoTokenizerfrom typing import List
def prepare_input(query: str, steps: List[str], tokenizer: AutoTokenizer, correct_token: str, generation_prompt: str): messages = [] for s_idx, step in enumerate(steps): if s_idx == 0: # append query and first step message = {'role': 'user', 'content': query + " " + step + " " + generation_prompt} else: message = {'role': 'user', 'content': step + " " + generation_prompt} messages.append(message) messages.append({'role': 'assistant', 'content': correct_token}) input_message = tokenizer.apply_chat_template(messages, add_generation_prompt = False, tokenize = False)
return input_message
def get_step_ids(input_ids, tokenizer, correct_token, correct_token_id): # get assistant turn indices asst_text = "<|start_of_role|>assistant<|end_of_role|>" + correct_token + "<|end_of_text|>" asst_toks = tokenizer(asst_text, add_special_tokens = False, return_tensors = "pt")['input_ids'][0] asst_toks_before_correct_token = asst_toks[:torch.where(asst_toks == correct_token_id)[0].item()].tolist()
input_ids = input_ids[0] # find batch index for assistant turn "Y", not just the correct_token_id correct_token_indices = torch.where(input_ids == correct_token_id)[0].tolist() prm_indices = [] for t_idx in correct_token_indices: if input_ids[t_idx - len(asst_toks_before_correct_token) :t_idx].tolist() == asst_toks_before_correct_token: prm_indices.append(t_idx-1) # the logits for token i predict the token i+1: so, we need to look at the PREVIOUS token logits assert len(prm_indices)>0 return prm_indices
model_name_or_path = "ibm-granite/granite-3.3-8b-math-prm-v2"model = AutoModelForCausalLM.from_pretrained(model_name_or_path, device_map = "auto")tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)
correct_token = "Y"correct_token_id = tokenizer.encode(correct_token, add_special_tokens=False)[0]generation_prompt = "Is this response correct so far (Y/N)?"
data = { "query": "For breakfast, Anna bought a bagel for $x and a glass of orange juice for $0.85. At lunch, Anna spent $4.65 on a sandwich and $1.15 on a carton of milk. How much more money did Anna spend on lunch than on breakfast? If we know the answer to the above question is 4, what is the value of unknown variable x?", "response":[ "At breakfast, Anna spent x dollars on a bagel and $0.85 on a glass of orange juice. The total cost of breakfast is x + $0.85.", "At lunch, Anna spent $4.65 on a sandwich and $1.15 on a carton of milk. The total cost of lunch is $4.65 + $1.15 = $5.80.", "To find out how much more money Anna spent on lunch than on breakfast, we subtract the cost of breakfast from the cost of lunch: $5.80 - (x + $0.85).", "We are given that the difference is $4, so we can write: $5.80 - (x + $0.85) = $4.", "Simplifying the left side, we get: $5.80 - x - $0.85 = $4.", "Adding -$0.85 to both sides, we get: $5.80 -x = $3.15.", "Subtracting $5.80 from both sides, we get: -x = -$2.65.", "Dividing both sides by -1, we get: x = $2.65." ]}
formatted_data = prepare_input(query=data['query'], steps=data['response'], tokenizer=tokenizer, correct_token=correct_token, generation_prompt=generation_prompt)input_ids = tokenizer.encode(formatted_data, return_tensors="pt").to(model.device)
with torch.no_grad(): logits = model(input_ids=input_ids).logits
# get step positionsprm_indices = get_step_ids(input_ids, tokenizer, correct_token, correct_token_id)
# get corresponding rewards: convert logits to probabilities and get the probability of the correct token id as rewardsoftmax = torch.nn.Softmax(dim=-1)step_rewards = []for prm_idx in prm_indices: step_rewards.append(softmax(logits[0, prm_idx, :])[correct_token_id].item())
print(step_rewards)# # [1.0, 1.0, 1.0, 0.99609375,0.9921875, 0.0849609375, 0.408203125, 0.8515625]import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer
model_name_or_path = "ibm-granite/granite-3.3-8b-math-prm-v2"model = AutoModelForCausalLM.from_pretrained(model_name_or_path, device_map = "auto")tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)generation_prompt = "Is this response correct so far (Y/N)?"
data = { "query": "For breakfast, Anna bought a bagel for $x and a glass of orange juice for $0.85. At lunch, Anna spent $4.65 on a sandwich and $1.15 on a carton of milk. How much more money did Anna spend on lunch than on breakfast? If we know the answer to the above question is 4, what is the value of unknown variable x?", "partial_response": "At breakfast, Anna spent x dollars on a bagel and $0.85 on a glass of orange juice. The total cost of breakfast is x + $0.85. At lunch, Anna spent $4.65 on a sandwich and $1.15 on a carton of milk. The total cost of lunch is $4.65 + $1.15 = $5.80. To find out how much more money Anna spent on lunch than on breakfast, we subtract the cost of breakfast from the cost of lunch: $5.80 - (x + $0.85).",}
# format the promptsformatted_prompt = tokenizer.apply_chat_template([{'role':'user', 'content': data['query'] + " " + data['partial_response'] + " " + generation_prompt}], add_generation_prompt=True, tokenize=False)inputs = tokenizer(formatted_prompt, return_tensors="pt")
# generate outputwith torch.no_grad(): response = model.generate(inputs["input_ids"].to(model.device), attention_mask=inputs["attention_mask"].to(model.device), max_new_tokens=2)
output_text = tokenizer.decode(response[0])print(output_text)# # <|start_of_role|>assistant<|end_of_role|>Y<|end_of_text|>