# Kosmos-2 Fine tuning
**URL:**
**Author:**  [@Mit1208](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMit1208)%5C
**Post date:** [March 3, 2024, 12:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkosmos-2-fine-tuning%2F75691%2F1 "2024-03-03T12:54:01Z")
Hi @ydshieh,
I am trying to fine-tune Kosmos-2 model on DocLayNet dataset. It has image, bounding box and its label. I want to create token classifier. I have converted dataset into Kosmos-2 required string. I am getting below error:
> ValueError: The model did not return a loss from the inputs, only the following keys: logits,past\_key\_values,image\_embeds,projection\_attentions,vision\_model\_output. For reference, the inputs it received are pixel\_values,input\_ids,attention\_mask,image\_embeds\_position\_mask.
I found that you mentioned that this issue is related to labels on Github issue.
Do you this below loss function will resolve my issue:
```auto
class CustomTrainer(Trainer):
def compute_loss(self, model, inputs, return_outputs=False):
outputs = model(**inputs)
logits = outputs.logits
# Get predicted logits for the next token
predicted_logits = logits[:, :-1].contiguous()
# Flatten logits and labels for loss computation
logits_flat = predicted_logits.view(-1, predicted_logits.size(-1))
labels_flat = inputs["input_ids"][:, 1:].contiguous().view(-1) # Assuming input_ids contains the input sequence
# Calculate the cross-entropy loss
loss = F.cross_entropy(logits_flat, labels_flat)
return (loss, outputs) if return_outputs else loss
```
Really appreciate your response. Please give some direction, thank you.
---
**Author:**  [@Mit1208](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMit1208)%5C
**Post date:** [March 3, 2024, 8:55pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkosmos-2-fine-tuning%2F75691%2F2 "2024-03-03T20:55:56Z")
I found two things:
1. Data Collator method is not yet defined in `transformers` for vision2seq - Kosmos2ForConditionalGeneration. Thatās why labels are empty while training.
2. When I checked `logits_flat` and `labels_flat` values then both were very different. `labels_flat` was shifted right by 1 and length was reduced by 1. I added `tokenizer.pad_token_id` for testing.
I am not sure is it correct or not. Any thoughts?
---
**Author:**  [@cdh](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fcdh)%5C
**Post date:** [March 4, 2024, 9:52am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkosmos-2-fine-tuning%2F75691%2F3 "2024-03-04T09:52:08Z")
Hello everyone, I opened an [Issue on GitHub](https://github.com/microsoft/unilm/issues/1429) about this problem and I tried to implement the Supervised Training in the Kosmos2Processor by passing the labels and tokenizing them.
It seems that the Trainer from Transformers is doing its job and effectively training the Language Model, but I found problems with the images. It looks like the Vision Encoder is having some troubles, maybe itās not being trained or maybe itās not saved, I canāt figure out what is happening.
I leave here a link to a [Notebook](https://colab.research.google.com/drive/1_mEFlLTJ113YpdpXcaAAHoPDjb0u0sVT?usp=sharing) where I show what I am doing, maybe someone can take a look and help me out. In the future, it could be helpful to implement this strategy inside the Transformers library so others can further fine-tune a Kosmos model on their own data.
---
**Author:**  [@ydshieh](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fydshieh)%5C
**Post date:** [March 6, 2024, 10:20am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkosmos-2-fine-tuning%2F75691%2F4 "2024-03-06T10:20:02Z")
Hi, I will take a look (only) next week, sorry for this long delay
---
**Author:**  [@Mit1208](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMit1208)%5C
**Post date:** [March 7, 2024, 2:16am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkosmos-2-fine-tuning%2F75691%2F5 "2024-03-07T02:16:47Z")
based on [Finetune BLIP on customer dataset #20893 - #2 by dxlong2000](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffinetune-blip-on-customer-dataset-20893%2F28446%2F2). I set labels as input\_ids in model
e.g
```auto
class CustomTrainer(Trainer):
def compute_loss(self, model, inputs, return_outputs=False):
# outputs = model(**inputs)
outputs = model(
input_ids=inputs["input_ids"], pixel_values=inputs["pixel_values"],
attention_mask=inputs["attention_mask"], image_embeds_position_mask=inputs["image_embeds_position_mask"],
labels = inputs["input_ids"]
)
logits = outputs.logits
# Ensure that 'eval_loss' is present in the metrics dictionary
metrics = {'eval_loss': outputs.loss.item()}
return (outputs.loss, outputs) if return_outputs else outputs.loss
```
---
**Author:**  [@Mit1208](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMit1208)%5C
**Post date:** [March 14, 2024, 5:46pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkosmos-2-fine-tuning%2F75691%2F6 "2024-03-14T17:46:26Z")
hi @ydshieh,
I hope you are well. Any quick suggestions, if you got a chance to look at the issue?
---
**Author:**  [@ydshieh](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fydshieh)%5C
**Post date:** [March 15, 2024, 8:52am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkosmos-2-fine-tuning%2F75691%2F7 "2024-03-15T08:52:57Z")
Hi @Mit1208 It would be helpful if I get the access to a code snippet (or a notebook) that has everything so I can run directly to see the issue.
I see @cdh has something on github issue but the following is mentioned
> I cannot add any example of the data and thus the code will not run correctly
If the data is sensitive and canāt be shared, you can still make a fake dummy dataset (even just 4 or tens of examples)
I am more than happy to take a look asap once I get such an access š
---
**Author:**  [@ydshieh](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fydshieh)%5C
**Post date:** [March 15, 2024, 8:53am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkosmos-2-fine-tuning%2F75691%2F8 "2024-03-15T08:53:32Z")
(and sorry for such long delay)
---
**Author:**  [@ydshieh](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fydshieh)%5C
**Post date:** [March 15, 2024, 9:56am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkosmos-2-fine-tuning%2F75691%2F9 "2024-03-15T09:56:46Z")
Thanks @cdh I got the dataset
---
**Author:**  [@Mit1208](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMit1208)%5C
**Post date:** [March 15, 2024, 11:01am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkosmos-2-fine-tuning%2F75691%2F10 "2024-03-15T11:01:04Z")
Hi @ydshieh,
Itās okay. Thanks for your time.
Here is my code in colab [Google Colab](https://colab.research.google.com/drive/1udiLoXbsZ2g2i0jDsYKJDYLBfQ0FEBmE?usp=sharing)
---
**Author:**  [@ydshieh](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fydshieh)%5C
**Post date:** [March 15, 2024, 12:00pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkosmos-2-fine-tuning%2F75691%2F11 "2024-03-15T12:00:50Z")
with GPU (T4), I always get GPU OOM
with CPU, I get CPU RAM OOM
@Mit1208 could you change a bit of the configuration(s) to use smaller resource so we can arrive and see the relevant issue š
(you can make a copy of the original notebook and make the changes - if you donāt want to touch the original code)
---
**Author:**  [@Mit1208](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMit1208)%5C
**Post date:** [March 15, 2024, 1:19pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkosmos-2-fine-tuning%2F75691%2F12 "2024-03-15T13:19:57Z")
Hi @ydshieh,
I reduce the dataset as much as I can. I have just 4 records for testing only.
Removed all unused code. I hope this will be helpful. Thanks.
---
**Author:**  [@ydshieh](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fydshieh)%5C
**Post date:** [March 15, 2024, 2:35pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkosmos-2-fine-tuning%2F75691%2F13 "2024-03-15T14:35:56Z")
@Mit1208 I see you defined
> class CustomTrainer(Trainer)
> and inside it you have
> labels = inputs[āinput\_idsā]
comment 1: why not just prepare (add `labels` in `inputs`) after the line
> inputs = processor
comment 2: as I mentioned, we canāt just do
> labels = inputs[āinput\_idsā])
as we have to change the padding place to -100.
I donāt think this is done in your notebook.
Let me know if there is something unlcear above š
---
**Author:**  [@ydshieh](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fydshieh)%5C
**Post date:** [March 15, 2024, 3:20pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkosmos-2-fine-tuning%2F75691%2F14 "2024-03-15T15:20:02Z")
Happy to help to show how to do it if you need help
---
**Author:**  [@ydshieh](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fydshieh)%5C
**Post date:** [March 15, 2024, 3:29pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkosmos-2-fine-tuning%2F75691%2F15 "2024-03-15T15:29:50Z")
@cdh You work involves many changes to the original code to fit your task. I will need more time to take a look and not sure if I can really come up some insights. In the meantime, letās see if @Mit1208 could arrive to a working example.
---
**Author:**  [@Mit1208](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMit1208)%5C
**Post date:** [March 15, 2024, 3:48pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkosmos-2-fine-tuning%2F75691%2F16 "2024-03-15T15:48:19Z")
@ydshieh, I added labels in model itself because I found one discussion around it e.g. [Finetune BLIP on customer dataset #20893 - #2 by dxlong2000](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffinetune-blip-on-customer-dataset-20893%2F28446%2F2).
To be honest, I wasnāt too clear on that. Can you give me code snippet to create `labels` if you can. If not give me some sort of python code which I can refer and I will taking from there.
This model works on next token prediction so I thought to use Data Collator but there is None that supports multimodel as of now.
I really appreciate your help and itās always very useful.
And, I think if my code will then @cdh code will work too because we both are trying to achieve almost similar things in different way.
---
**Author:**  [@ydshieh](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fydshieh)%5C
**Post date:** [March 15, 2024, 4:34pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkosmos-2-fine-tuning%2F75691%2F17 "2024-03-15T16:34:36Z")
I will provide something š
> I added labels in model itself
Not really if I understand correctly. You just do
```auto
outputs = model(
input_ids=inputs["input_ids"], pixel_values=inputs["pixel_values"],
attention_mask=inputs["attention_mask"], image_embeds_position_mask=inputs["image_embeds_position_mask"],
labels = inputs["input_ids"])
```
but using `class CustomTrainer`. This is not necessary, and you can just preapre `labels` in your dataset. So it will be received by the model under the hood.
Itās fine, let me show you
---
**Author:**  [@ydshieh](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fydshieh)%5C
**Post date:** [March 15, 2024, 4:59pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkosmos-2-fine-tuning%2F75691%2F18 "2024-03-15T16:59:49Z")
See
> **[Google Colaboratory](https://colab.research.google.com/drive/1uzfDJ7ZzlLNi0jeh8IUAlGg6rBuq_b6W#scrollTo=Q1DZoZo9iA2F)**
We just need the line
> inputs[ālabelsā] = inputs[āinput\_idsā]
as in
(note, I havenāt addressed yet the padding and -100 in the labels)
```auto
from datasets import Dataset
inputs = processor(images = train_df['image'].to_list(), text = train_df['text'].to_list(), bboxes = train_df['float_val'].to_list(), padding=True, truncation= True, return_tensors="pt").to(device)
inputs["labels"] = inputs["input_ids"]
dataset = Dataset.from_dict(inputs)
train_test_split = dataset.train_test_split(test_size=0.3)
```
---
**Author:**  [@Mit1208](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMit1208)%5C
**Post date:** [March 15, 2024, 5:20pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkosmos-2-fine-tuning%2F75691%2F19 "2024-03-15T17:20:40Z")
Oh, I see you created new column labels and using `default_data_collator` will check loss with the next token. Correct me if my understanding is wrong.
I think this is a good start, I will also try to figure out padding and -100.
Thanks @ydshieh
---
**Author:**  [@ydshieh](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fydshieh)%5C
**Post date:** [March 15, 2024, 5:21pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkosmos-2-fine-tuning%2F75691%2F20 "2024-03-15T17:21:51Z")
> [@Mit1208](#):
>
> default\_data\_collator
well, I donāt add `default_data_collator`: itās on your notebook already (I donāt know if itās necessary however š )
[Next page](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkosmos-2-fine-tuning%2F75691.md%3Fpage%3D2)