## Overview
The GIT model was proposed in [GIT: A Generative Image-to-text Transformer for Vision and Language](https://huggingface.co/papers/2205.14100) by
Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, Lijuan Wang. GIT is a decoder-only Transformer
that leverages [CLIP](clip)'s vision encoder to condition the model on vision inputs besides text. The model obtains state-of-the-art results on
image captioning and visual question answering benchmarks.
The abstract from the paper is the following:
*In this paper, we design and train a Generative Image-to-text Transformer, GIT, to unify vision-language tasks such as image/video captioning and question answering. While generative models provide a consistent network architecture between pre-training and fine-tuning, existing work typically contains complex structures (uni/multi-modal encoder/decoder) and depends on external modules such as object detectors/taggers and optical character recognition (OCR). In GIT, we simplify the architecture as one image encoder and one text decoder under a single language modeling task. We also scale up the pre-training data and the model size to boost the model performance. Without bells and whistles, our GIT establishes new state of the arts on 12 challenging benchmarks with a large margin. For instance, our model surpasses the human performance for the first time on TextCaps (138.2 vs. 125.5 in CIDEr). Furthermore, we present a new scheme of generation-based image classification and scene text recognition, achieving decent performance on standard benchmarks.*
GIT architecture. Taken from the original paper.
This model was contributed by [nielsr](https://huggingface.co/nielsr).
The original code can be found [here](https://github.com/microsoft/GenerativeImage2Text).
## Usage tips
- GIT is implemented in a very similar way to GPT-2, the only difference being that the model is also conditioned on `pixel_values`.
## Resources
A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with GIT.
- Demo notebooks regarding inference + fine-tuning GIT on custom data can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/GIT).
- See also: [Causal language modeling task guide](../tasks/language_modeling)
If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we will review it.
The resource should ideally demonstrate something new instead of duplicating an existing resource.
## GitVisionConfig[[transformers.GitVisionConfig]]
class transformers.GitVisionConfigtransformers.GitVisionConfighttps://github.com/huggingface/transformers/blob/v4.57.0/src/transformers/models/git/configuration_git.py#L24[{"name": "hidden_size", "val": " = 768"}, {"name": "intermediate_size", "val": " = 3072"}, {"name": "num_hidden_layers", "val": " = 12"}, {"name": "num_attention_heads", "val": " = 12"}, {"name": "num_channels", "val": " = 3"}, {"name": "image_size", "val": " = 224"}, {"name": "patch_size", "val": " = 16"}, {"name": "hidden_act", "val": " = 'quick_gelu'"}, {"name": "layer_norm_eps", "val": " = 1e-05"}, {"name": "attention_dropout", "val": " = 0.0"}, {"name": "initializer_range", "val": " = 0.02"}, {"name": "**kwargs", "val": ""}]- **hidden_size** (`int`, *optional*, defaults to 768) --
Dimensionality of the encoder layers and the pooler layer.
- **intermediate_size** (`int`, *optional*, defaults to 3072) --
Dimensionality of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder.
- **num_hidden_layers** (`int`, *optional*, defaults to 12) --
Number of hidden layers in the Transformer encoder.
- **num_attention_heads** (`int`, *optional*, defaults to 12) --
Number of attention heads for each attention layer in the Transformer encoder.
- **image_size** (`int`, *optional*, defaults to 224) --
The size (resolution) of each image.
- **patch_size** (`int`, *optional*, defaults to 16) --
The size (resolution) of each patch.
- **hidden_act** (`str` or `function`, *optional*, defaults to `"quick_gelu"`) --
The non-linear activation function (function or string) in the encoder and pooler. If string, `"gelu"`,
`"relu"`, `"selu"` and `"gelu_new"` `"quick_gelu"` are supported.
- **layer_norm_eps** (`float`, *optional*, defaults to 1e-5) --
The epsilon used by the layer normalization layers.
- **attention_dropout** (`float`, *optional*, defaults to 0.0) --
The dropout ratio for the attention probabilities.
- **initializer_range** (`float`, *optional*, defaults to 0.02) --
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.0
This is the configuration class to store the configuration of a [GitVisionModel](/docs/transformers/v4.57.0/en/model_doc/git#transformers.GitVisionModel). It is used to instantiate a GIT
vision encoder according to the specified arguments, defining the model architecture. Instantiating a configuration
with the defaults will yield a similar configuration to that of the vision encoder of the GIT
[microsoft/git-base](https://huggingface.co/microsoft/git-base) architecture.
Configuration objects inherit from [PretrainedConfig](/docs/transformers/v4.57.0/en/main_classes/configuration#transformers.PretrainedConfig) and can be used to control the model outputs. Read the
documentation from [PretrainedConfig](/docs/transformers/v4.57.0/en/main_classes/configuration#transformers.PretrainedConfig) for more information.
Example:
```python
>>> from transformers import GitVisionConfig, GitVisionModel
>>> # Initializing a GitVisionConfig with microsoft/git-base style configuration
>>> configuration = GitVisionConfig()
>>> # Initializing a GitVisionModel (with random weights) from the microsoft/git-base style configuration
>>> model = GitVisionModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.config
```
## GitVisionModel[[transformers.GitVisionModel]]
class transformers.GitVisionModeltransformers.GitVisionModelhttps://github.com/huggingface/transformers/blob/v4.57.0/src/transformers/models/git/modeling_git.py#L874[{"name": "config", "val": ": GitVisionConfig"}]- **config** ([GitVisionConfig](/docs/transformers/v4.57.0/en/model_doc/git#transformers.GitVisionConfig)) --
Model configuration class with all the parameters of the model. Initializing with a config file does not
load the weights associated with the model, only the configuration. Check out the
[from_pretrained()](/docs/transformers/v4.57.0/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.0
The vision model from CLIP, used in GIT, without any head or projection on top.
This model inherits from [PreTrainedModel](/docs/transformers/v4.57.0/en/main_classes/model#transformers.PreTrainedModel). Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.)
This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.
forwardtransformers.GitVisionModel.forwardhttps://github.com/huggingface/transformers/blob/v4.57.0/src/transformers/models/git/modeling_git.py#L888[{"name": "pixel_values", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "output_attentions", "val": ": typing.Optional[bool] = None"}, {"name": "output_hidden_states", "val": ": typing.Optional[bool] = None"}, {"name": "interpolate_pos_encoding", "val": ": bool = False"}, {"name": "return_dict", "val": ": typing.Optional[bool] = None"}]- **pixel_values** (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`, *optional*) --
The tensors corresponding to the input images. Pixel values can be obtained using
[CLIPImageProcessor](/docs/transformers/v4.57.0/en/model_doc/clip#transformers.CLIPImageProcessor). See [CLIPImageProcessor.__call__()](/docs/transformers/v4.57.0/en/model_doc/fuyu#transformers.FuyuImageProcessor.__call__) for details ([GitProcessor](/docs/transformers/v4.57.0/en/model_doc/git#transformers.GitProcessor) uses
[CLIPImageProcessor](/docs/transformers/v4.57.0/en/model_doc/clip#transformers.CLIPImageProcessor) for processing images).
- **output_attentions** (`bool`, *optional*) --
Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned
tensors for more detail.
- **output_hidden_states** (`bool`, *optional*) --
Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for
more detail.
- **interpolate_pos_encoding** (`bool`, defaults to `False`) --
Whether to interpolate the pre-trained position encodings.
- **return_dict** (`bool`, *optional*) --
Whether or not to return a [ModelOutput](/docs/transformers/v4.57.0/en/main_classes/output#transformers.utils.ModelOutput) instead of a plain tuple.0[transformers.modeling_outputs.BaseModelOutput](/docs/transformers/v4.57.0/en/main_classes/output#transformers.modeling_outputs.BaseModelOutput) or `tuple(torch.FloatTensor)`A [transformers.modeling_outputs.BaseModelOutput](/docs/transformers/v4.57.0/en/main_classes/output#transformers.modeling_outputs.BaseModelOutput) or a tuple of
`torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various
elements depending on the configuration ([GitConfig](/docs/transformers/v4.57.0/en/model_doc/git#transformers.GitConfig)) and inputs.
- **last_hidden_state** (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`) -- Sequence of hidden-states at the output of the last layer of the model.
- **hidden_states** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) -- Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`.
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
- **attentions** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) -- Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length,
sequence_length)`.
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
The [GitVisionModel](/docs/transformers/v4.57.0/en/model_doc/git#transformers.GitVisionModel) forward method, overrides the `__call__` special method.
Although the recipe for forward pass needs to be defined within this function, one should call the `Module`
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.
Examples:
```python
>>> from PIL import Image
>>> import requests
>>> from transformers import AutoProcessor, GitVisionModel
>>> processor = AutoProcessor.from_pretrained("microsoft/git-base")
>>> model = GitVisionModel.from_pretrained("microsoft/git-base")
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> inputs = processor(images=image, return_tensors="pt")
>>> outputs = model(**inputs)
>>> last_hidden_state = outputs.last_hidden_state
```
## GitConfig[[transformers.GitConfig]]
class transformers.GitConfigtransformers.GitConfighttps://github.com/huggingface/transformers/blob/v4.57.0/src/transformers/models/git/configuration_git.py#L105[{"name": "vision_config", "val": " = None"}, {"name": "vocab_size", "val": " = 30522"}, {"name": "hidden_size", "val": " = 768"}, {"name": "num_hidden_layers", "val": " = 6"}, {"name": "num_attention_heads", "val": " = 12"}, {"name": "intermediate_size", "val": " = 3072"}, {"name": "hidden_act", "val": " = 'gelu'"}, {"name": "hidden_dropout_prob", "val": " = 0.1"}, {"name": "attention_probs_dropout_prob", "val": " = 0.1"}, {"name": "max_position_embeddings", "val": " = 1024"}, {"name": "initializer_range", "val": " = 0.02"}, {"name": "layer_norm_eps", "val": " = 1e-12"}, {"name": "pad_token_id", "val": " = 0"}, {"name": "position_embedding_type", "val": " = 'absolute'"}, {"name": "use_cache", "val": " = True"}, {"name": "tie_word_embeddings", "val": " = False"}, {"name": "bos_token_id", "val": " = 101"}, {"name": "eos_token_id", "val": " = 102"}, {"name": "num_image_with_embedding", "val": " = None"}, {"name": "**kwargs", "val": ""}]- **vision_config** (`dict`, *optional*) --
Dictionary of configuration options used to initialize [GitVisionConfig](/docs/transformers/v4.57.0/en/model_doc/git#transformers.GitVisionConfig).
- **vocab_size** (`int`, *optional*, defaults to 30522) --
Vocabulary size of the GIT model. Defines the number of different tokens that can be represented by the
`inputs_ids` passed when calling [GitModel](/docs/transformers/v4.57.0/en/model_doc/git#transformers.GitModel).
- **hidden_size** (`int`, *optional*, defaults to 768) --
Dimensionality of the encoder layers and the pooler layer.
- **num_hidden_layers** (`int`, *optional*, defaults to 6) --
Number of hidden layers in the Transformer encoder.
- **num_attention_heads** (`int`, *optional*, defaults to 12) --
Number of attention heads for each attention layer in the Transformer encoder.
- **intermediate_size** (`int`, *optional*, defaults to 3072) --
Dimensionality of the "intermediate" (often named feed-forward) layer in the Transformer encoder.
- **hidden_act** (`str` or `Callable`, *optional*, defaults to `"gelu"`) --
The non-linear activation function (function or string) in the encoder and pooler. If string, `"gelu"`,
`"relu"`, `"silu"` and `"gelu_new"` are supported.
- **hidden_dropout_prob** (`float`, *optional*, defaults to 0.1) --
The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.
- **attention_probs_dropout_prob** (`float`, *optional*, defaults to 0.1) --
The dropout ratio for the attention probabilities.
- **max_position_embeddings** (`int`, *optional*, defaults to 1024) --
The maximum sequence length that this model might ever be used with. Typically set this to something large
just in case (e.g., 512 or 1024 or 2048).
- **initializer_range** (`float`, *optional*, defaults to 0.02) --
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
- **layer_norm_eps** (`float`, *optional*, defaults to 1e-12) --
The epsilon used by the layer normalization layers.
- **position_embedding_type** (`str`, *optional*, defaults to `"absolute"`) --
Type of position embedding. Choose one of `"absolute"`, `"relative_key"`, `"relative_key_query"`. For
positional embeddings use `"absolute"`. For more information on `"relative_key"`, please refer to
[Self-Attention with Relative Position Representations (Shaw et al.)](https://huggingface.co/papers/1803.02155).
For more information on `"relative_key_query"`, please refer to *Method 4* in [Improve Transformer Models
with Better Relative Position Embeddings (Huang et al.)](https://huggingface.co/papers/2009.13658).
- **use_cache** (`bool`, *optional*, defaults to `True`) --
Whether or not the model should return the last key/values attentions (not used by all models).
- **num_image_with_embedding** (`int`, *optional*) --
The number of temporal embeddings to add, in case the model is used for video captioning/VQA.0
This is the configuration class to store the configuration of a [GitModel](/docs/transformers/v4.57.0/en/model_doc/git#transformers.GitModel). It is used to instantiate a GIT model
according to the specified arguments, defining the model architecture. Instantiating a configuration with the
defaults will yield a similar configuration to that of the GIT
[microsoft/git-base](https://huggingface.co/microsoft/git-base) architecture.
Configuration objects inherit from [PretrainedConfig](/docs/transformers/v4.57.0/en/main_classes/configuration#transformers.PretrainedConfig) and can be used to control the model outputs. Read the
documentation from [PretrainedConfig](/docs/transformers/v4.57.0/en/main_classes/configuration#transformers.PretrainedConfig) for more information.
Examples:
```python
>>> from transformers import GitConfig, GitModel
>>> # Initializing a GIT microsoft/git-base style configuration
>>> configuration = GitConfig()
>>> # Initializing a model (with random weights) from the microsoft/git-base style configuration
>>> model = GitModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.config
```
## GitProcessor[[transformers.GitProcessor]]
class transformers.GitProcessortransformers.GitProcessorhttps://github.com/huggingface/transformers/blob/v4.57.0/src/transformers/models/git/processing_git.py#L22[{"name": "image_processor", "val": ""}, {"name": "tokenizer", "val": ""}]- **image_processor** ([AutoImageProcessor](/docs/transformers/v4.57.0/en/model_doc/auto#transformers.AutoImageProcessor)) --
The image processor is a required input.
- **tokenizer** ([AutoTokenizer](/docs/transformers/v4.57.0/en/model_doc/auto#transformers.AutoTokenizer)) --
The tokenizer is a required input.0
Constructs a GIT processor which wraps a CLIP image processor and a BERT tokenizer into a single processor.
[GitProcessor](/docs/transformers/v4.57.0/en/model_doc/git#transformers.GitProcessor) offers all the functionalities of [CLIPImageProcessor](/docs/transformers/v4.57.0/en/model_doc/clip#transformers.CLIPImageProcessor) and [BertTokenizerFast](/docs/transformers/v4.57.0/en/model_doc/bert#transformers.BertTokenizerFast). See the
[__call__()](/docs/transformers/v4.57.0/en/model_doc/vilt#transformers.ViltProcessor.__call__) and [decode()](/docs/transformers/v4.57.0/en/main_classes/processors#transformers.ProcessorMixin.decode) for more information.
__call__transformers.GitProcessor.__call__https://github.com/huggingface/transformers/blob/v4.57.0/src/transformers/processing_utils.py#L537[{"name": "images", "val": ": typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), list['PIL.Image.Image'], list[numpy.ndarray], list['torch.Tensor'], NoneType] = None"}, {"name": "text", "val": ": typing.Union[str, list[str], list[list[str]], NoneType] = None"}, {"name": "videos", "val": ": typing.Union[list['PIL.Image.Image'], numpy.ndarray, ForwardRef('torch.Tensor'), list[numpy.ndarray], list['torch.Tensor'], list[list['PIL.Image.Image']], list[list[numpy.ndarray]], list[list['torch.Tensor']], transformers.video_utils.URL, list[transformers.video_utils.URL], list[list[transformers.video_utils.URL]], transformers.video_utils.Path, list[transformers.video_utils.Path], list[list[transformers.video_utils.Path]], NoneType] = None"}, {"name": "audio", "val": ": typing.Union[numpy.ndarray, ForwardRef('torch.Tensor'), collections.abc.Sequence[numpy.ndarray], collections.abc.Sequence['torch.Tensor'], NoneType] = None"}, {"name": "**kwargs", "val": ": typing_extensions.Unpack[transformers.processing_utils.ProcessingKwargs]"}]- **images** (`PIL.Image.Image`, `np.ndarray`, `torch.Tensor`, `list[PIL.Image.Image]`, `list[np.ndarray]`, `list[torch.Tensor]`) --
The image or batch of images to be prepared. Each image can be a PIL image, NumPy array or PyTorch
tensor. Both channels-first and channels-last formats are supported.
- **text** (`TextInput`, `PreTokenizedInput`, `list[TextInput]`, `list[PreTokenizedInput]`, *optional*) --
The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings
(pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set
`is_split_into_words=True` (to lift the ambiguity with a batch of sequences).
- **videos** (`np.ndarray`, `torch.Tensor`, `List[np.ndarray]`, `List[torch.Tensor]`) --
The video or batch of videos to be prepared. Each video can be a 4D NumPy array or PyTorch
tensor, or a nested list of 3D frames. Both channels-first and channels-last formats are supported.
- **audio** (`np.ndarray`, `torch.Tensor`, `list[np.ndarray]`, `list[torch.Tensor]`) --
The audio or batch of audio to be prepared. Each audio can be a NumPy array or PyTorch
tensor.
- **return_tensors** (`str` or [TensorType](/docs/transformers/v4.57.0/en/internal/file_utils#transformers.TensorType), *optional*) --
If set, will return tensors of a particular framework. Acceptable values are:
- `'tf'`: Return TensorFlow `tf.constant` objects.
- `'pt'`: Return PyTorch `torch.Tensor` objects.
- `'np'`: Return NumPy `np.ndarray` objects.
- `'jax'`: Return JAX `jnp.ndarray` objects.0[BatchFeature](/docs/transformers/v4.57.0/en/main_classes/image_processor#transformers.BatchFeature)A [BatchFeature](/docs/transformers/v4.57.0/en/main_classes/image_processor#transformers.BatchFeature) object with processed inputs in a dict format.
Main method to prepare for model inputs. This method forwards the each modality argument to its own processor
along with `kwargs`. Please refer to the docstring of the each processor attributes for more information.
## GitModel[[transformers.GitModel]]
class transformers.GitModeltransformers.GitModelhttps://github.com/huggingface/transformers/blob/v4.57.0/src/transformers/models/git/modeling_git.py#L945[{"name": "config", "val": ""}]- **config** ([GitModel](/docs/transformers/v4.57.0/en/model_doc/git#transformers.GitModel)) --
Model configuration class with all the parameters of the model. Initializing with a config file does not
load the weights associated with the model, only the configuration. Check out the
[from_pretrained()](/docs/transformers/v4.57.0/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.0
The bare GIT Model transformer consisting of a CLIP image encoder and text decoder outputting raw hidden-states
This model inherits from [PreTrainedModel](/docs/transformers/v4.57.0/en/main_classes/model#transformers.PreTrainedModel). Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.)
This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.
forwardtransformers.GitModel.forwardhttps://github.com/huggingface/transformers/blob/v4.57.0/src/transformers/models/git/modeling_git.py#L1035[{"name": "input_ids", "val": ": typing.Optional[torch.Tensor] = None"}, {"name": "attention_mask", "val": ": typing.Optional[torch.Tensor] = None"}, {"name": "position_ids", "val": ": typing.Optional[torch.Tensor] = None"}, {"name": "pixel_values", "val": ": typing.Optional[torch.Tensor] = None"}, {"name": "head_mask", "val": ": typing.Optional[torch.Tensor] = None"}, {"name": "inputs_embeds", "val": ": typing.Optional[torch.Tensor] = None"}, {"name": "past_key_values", "val": ": typing.Union[transformers.cache_utils.Cache, list[torch.FloatTensor], NoneType] = None"}, {"name": "use_cache", "val": ": typing.Optional[bool] = None"}, {"name": "output_attentions", "val": ": typing.Optional[bool] = None"}, {"name": "output_hidden_states", "val": ": typing.Optional[bool] = None"}, {"name": "interpolate_pos_encoding", "val": ": bool = False"}, {"name": "return_dict", "val": ": typing.Optional[bool] = None"}]- **input_ids** (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*) --
Indices of input sequence tokens in the vocabulary. Padding will be ignored by default.
Indices can be obtained using [AutoTokenizer](/docs/transformers/v4.57.0/en/model_doc/auto#transformers.AutoTokenizer). See [PreTrainedTokenizer.encode()](/docs/transformers/v4.57.0/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.encode) and
[PreTrainedTokenizer.__call__()](/docs/transformers/v4.57.0/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.__call__) for details.
[What are input IDs?](../glossary#input-ids)
- **attention_mask** (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*) --
Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
- 1 for tokens that are **not masked**,
- 0 for tokens that are **masked**.
[What are attention masks?](../glossary#attention-mask)
- **position_ids** (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*) --
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`.
[What are position IDs?](../glossary#position-ids)
- **pixel_values** (`torch.Tensor` of shape `(batch_size, num_channels, image_size, image_size)`, *optional*) --
The tensors corresponding to the input images. Pixel values can be obtained using
[CLIPImageProcessor](/docs/transformers/v4.57.0/en/model_doc/clip#transformers.CLIPImageProcessor). See [CLIPImageProcessor.__call__()](/docs/transformers/v4.57.0/en/model_doc/fuyu#transformers.FuyuImageProcessor.__call__) for details ([GitProcessor](/docs/transformers/v4.57.0/en/model_doc/git#transformers.GitProcessor) uses
[CLIPImageProcessor](/docs/transformers/v4.57.0/en/model_doc/clip#transformers.CLIPImageProcessor) for processing images).
- **head_mask** (`torch.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*) --
Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`:
- 1 indicates the head is **not masked**,
- 0 indicates the head is **masked**.
- **inputs_embeds** (`torch.Tensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*) --
Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert `input_ids` indices into associated vectors than the
model's internal embedding lookup matrix.
- **past_key_values** (`Union[~cache_utils.Cache, list[torch.FloatTensor], NoneType]`) --
Pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention
blocks) that can be used to speed up sequential decoding. This typically consists in the `past_key_values`
returned by the model at a previous stage of decoding, when `use_cache=True` or `config.use_cache=True`.
Only [Cache](/docs/transformers/v4.57.0/en/internal/generation_utils#transformers.Cache) instance is allowed as input, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache).
If no `past_key_values` are passed, [DynamicCache](/docs/transformers/v4.57.0/en/internal/generation_utils#transformers.DynamicCache) will be initialized by default.
The model will output the same cache format that is fed as input.
If `past_key_values` are used, the user is expected to input only unprocessed `input_ids` (those that don't
have their past key value states given to this model) of shape `(batch_size, unprocessed_length)` instead of all `input_ids`
of shape `(batch_size, sequence_length)`.
- **use_cache** (`bool`, *optional*) --
If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see
`past_key_values`).
- **output_attentions** (`bool`, *optional*) --
Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned
tensors for more detail.
- **output_hidden_states** (`bool`, *optional*) --
Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for
more detail.
- **interpolate_pos_encoding** (`bool`, defaults to `False`) --
Whether to interpolate the pre-trained position encodings.
- **return_dict** (`bool`, *optional*) --
Whether or not to return a [ModelOutput](/docs/transformers/v4.57.0/en/main_classes/output#transformers.utils.ModelOutput) instead of a plain tuple.0[transformers.modeling_outputs.BaseModelOutputWithPooling](/docs/transformers/v4.57.0/en/main_classes/output#transformers.modeling_outputs.BaseModelOutputWithPooling) or `tuple(torch.FloatTensor)`A [transformers.modeling_outputs.BaseModelOutputWithPooling](/docs/transformers/v4.57.0/en/main_classes/output#transformers.modeling_outputs.BaseModelOutputWithPooling) or a tuple of
`torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various
elements depending on the configuration ([GitConfig](/docs/transformers/v4.57.0/en/model_doc/git#transformers.GitConfig)) and inputs.
- **last_hidden_state** (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`) -- Sequence of hidden-states at the output of the last layer of the model.
- **pooler_output** (`torch.FloatTensor` of shape `(batch_size, hidden_size)`) -- Last layer hidden-state of the first token of the sequence (classification token) after further processing
through the layers used for the auxiliary pretraining task. E.g. for BERT-family of models, this returns
the classification token after processing through a linear layer and a tanh activation function. The linear
layer weights are trained from the next sentence prediction (classification) objective during pretraining.
- **hidden_states** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) -- Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`.
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
- **attentions** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) -- Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length,
sequence_length)`.
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
The [GitModel](/docs/transformers/v4.57.0/en/model_doc/git#transformers.GitModel) forward method, overrides the `__call__` special method.
Although the recipe for forward pass needs to be defined within this function, one should call the `Module`
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.
Examples:
```python
>>> from transformers import AutoProcessor, AutoModel
>>> import requests
>>> from PIL import Image
>>> processor = AutoProcessor.from_pretrained("microsoft/git-base")
>>> model = AutoModel.from_pretrained("microsoft/git-base")
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> text = "this is an image of two cats"
>>> inputs = processor(images=image, text=text, return_tensors="pt")
>>> outputs = model(**inputs)
>>> last_hidden_state = outputs.last_hidden_state
```
## GitForCausalLM[[transformers.GitForCausalLM]]
class transformers.GitForCausalLMtransformers.GitForCausalLMhttps://github.com/huggingface/transformers/blob/v4.57.0/src/transformers/models/git/modeling_git.py#L1207[{"name": "config", "val": ""}]- **config** ([GitForCausalLM](/docs/transformers/v4.57.0/en/model_doc/git#transformers.GitForCausalLM)) --
Model configuration class with all the parameters of the model. Initializing with a config file does not
load the weights associated with the model, only the configuration. Check out the
[from_pretrained()](/docs/transformers/v4.57.0/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.0
GIT Model with a `language modeling` head on top for autoregressive language modeling.
This model inherits from [PreTrainedModel](/docs/transformers/v4.57.0/en/main_classes/model#transformers.PreTrainedModel). Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.)
This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.
forwardtransformers.GitForCausalLM.forwardhttps://github.com/huggingface/transformers/blob/v4.57.0/src/transformers/models/git/modeling_git.py#L1225[{"name": "input_ids", "val": ": typing.Optional[torch.Tensor] = None"}, {"name": "attention_mask", "val": ": typing.Optional[torch.Tensor] = None"}, {"name": "position_ids", "val": ": typing.Optional[torch.Tensor] = None"}, {"name": "pixel_values", "val": ": typing.Optional[torch.Tensor] = None"}, {"name": "head_mask", "val": ": typing.Optional[torch.Tensor] = None"}, {"name": "inputs_embeds", "val": ": typing.Optional[torch.Tensor] = None"}, {"name": "labels", "val": ": typing.Optional[torch.Tensor] = None"}, {"name": "past_key_values", "val": ": typing.Union[transformers.cache_utils.Cache, list[torch.Tensor], NoneType] = None"}, {"name": "use_cache", "val": ": typing.Optional[bool] = None"}, {"name": "output_attentions", "val": ": typing.Optional[bool] = None"}, {"name": "output_hidden_states", "val": ": typing.Optional[bool] = None"}, {"name": "interpolate_pos_encoding", "val": ": bool = False"}, {"name": "return_dict", "val": ": typing.Optional[bool] = None"}, {"name": "**kwargs", "val": ""}]- **input_ids** (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*) --
Indices of input sequence tokens in the vocabulary. Padding will be ignored by default.
Indices can be obtained using [AutoTokenizer](/docs/transformers/v4.57.0/en/model_doc/auto#transformers.AutoTokenizer). See [PreTrainedTokenizer.encode()](/docs/transformers/v4.57.0/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.encode) and
[PreTrainedTokenizer.__call__()](/docs/transformers/v4.57.0/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.__call__) for details.
[What are input IDs?](../glossary#input-ids)
- **attention_mask** (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*) --
Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
- 1 for tokens that are **not masked**,
- 0 for tokens that are **masked**.
[What are attention masks?](../glossary#attention-mask)
- **position_ids** (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*) --
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`.
[What are position IDs?](../glossary#position-ids)
- **pixel_values** (`torch.Tensor` of shape `(batch_size, num_channels, image_size, image_size)`, *optional*) --
The tensors corresponding to the input images. Pixel values can be obtained using
[CLIPImageProcessor](/docs/transformers/v4.57.0/en/model_doc/clip#transformers.CLIPImageProcessor). See [CLIPImageProcessor.__call__()](/docs/transformers/v4.57.0/en/model_doc/fuyu#transformers.FuyuImageProcessor.__call__) for details ([GitProcessor](/docs/transformers/v4.57.0/en/model_doc/git#transformers.GitProcessor) uses
[CLIPImageProcessor](/docs/transformers/v4.57.0/en/model_doc/clip#transformers.CLIPImageProcessor) for processing images).
- **head_mask** (`torch.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*) --
Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`:
- 1 indicates the head is **not masked**,
- 0 indicates the head is **masked**.
- **inputs_embeds** (`torch.Tensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*) --
Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert `input_ids` indices into associated vectors than the
model's internal embedding lookup matrix.
- **labels** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) --
Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in
`[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are
ignored (masked), the loss is only computed for the tokens with labels n `[0, ..., config.vocab_size]`
- **past_key_values** (`Union[~cache_utils.Cache, list[torch.Tensor], NoneType]`) --
Pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention
blocks) that can be used to speed up sequential decoding. This typically consists in the `past_key_values`
returned by the model at a previous stage of decoding, when `use_cache=True` or `config.use_cache=True`.
Only [Cache](/docs/transformers/v4.57.0/en/internal/generation_utils#transformers.Cache) instance is allowed as input, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache).
If no `past_key_values` are passed, [DynamicCache](/docs/transformers/v4.57.0/en/internal/generation_utils#transformers.DynamicCache) will be initialized by default.
The model will output the same cache format that is fed as input.
If `past_key_values` are used, the user is expected to input only unprocessed `input_ids` (those that don't
have their past key value states given to this model) of shape `(batch_size, unprocessed_length)` instead of all `input_ids`
of shape `(batch_size, sequence_length)`.
- **use_cache** (`bool`, *optional*) --
If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see
`past_key_values`).
- **output_attentions** (`bool`, *optional*) --
Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned
tensors for more detail.
- **output_hidden_states** (`bool`, *optional*) --
Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for
more detail.
- **interpolate_pos_encoding** (`bool`, defaults to `False`) --
Whether to interpolate the pre-trained position encodings.
- **return_dict** (`bool`, *optional*) --
Whether or not to return a [ModelOutput](/docs/transformers/v4.57.0/en/main_classes/output#transformers.utils.ModelOutput) instead of a plain tuple.0[transformers.modeling_outputs.CausalLMOutputWithPast](/docs/transformers/v4.57.0/en/main_classes/output#transformers.modeling_outputs.CausalLMOutputWithPast) or `tuple(torch.FloatTensor)`A [transformers.modeling_outputs.CausalLMOutputWithPast](/docs/transformers/v4.57.0/en/main_classes/output#transformers.modeling_outputs.CausalLMOutputWithPast) or a tuple of
`torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various
elements depending on the configuration ([GitConfig](/docs/transformers/v4.57.0/en/model_doc/git#transformers.GitConfig)) and inputs.
- **loss** (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided) -- Language modeling loss (for next-token prediction).
- **logits** (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`) -- Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
- **past_key_values** (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`) -- It is a [Cache](/docs/transformers/v4.57.0/en/internal/generation_utils#transformers.Cache) instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache).
Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see
`past_key_values` input) to speed up sequential decoding.
- **hidden_states** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) -- Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`.
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
- **attentions** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) -- Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length,
sequence_length)`.
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
The [GitForCausalLM](/docs/transformers/v4.57.0/en/model_doc/git#transformers.GitForCausalLM) forward method, overrides the `__call__` special method.
Although the recipe for forward pass needs to be defined within this function, one should call the `Module`
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.
Examples:
Image captioning example:
```python
>>> from transformers import AutoProcessor, AutoModelForCausalLM
>>> import requests
>>> from PIL import Image
>>> processor = AutoProcessor.from_pretrained("microsoft/git-base-coco")
>>> model = AutoModelForCausalLM.from_pretrained("microsoft/git-base-coco")
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> pixel_values = processor(images=image, return_tensors="pt").pixel_values
>>> generated_ids = model.generate(pixel_values=pixel_values, max_length=50)
>>> generated_caption = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
>>> print(generated_caption)
two cats sleeping on a pink blanket next to remotes.
```
Visual question answering (VQA) example:
```python
>>> from transformers import AutoProcessor, AutoModelForCausalLM
>>> from huggingface_hub import hf_hub_download
>>> from PIL import Image
>>> processor = AutoProcessor.from_pretrained("microsoft/git-base-textvqa")
>>> model = AutoModelForCausalLM.from_pretrained("microsoft/git-base-textvqa")
>>> file_path = hf_hub_download(repo_id="nielsr/textvqa-sample", filename="bus.png", repo_type="dataset")
>>> image = Image.open(file_path).convert("RGB")
>>> pixel_values = processor(images=image, return_tensors="pt").pixel_values
>>> question = "what does the front of the bus say at the top?"
>>> input_ids = processor(text=question, add_special_tokens=False).input_ids
>>> input_ids = [processor.tokenizer.cls_token_id] + input_ids
>>> input_ids = torch.tensor(input_ids).unsqueeze(0)
>>> generated_ids = model.generate(pixel_values=pixel_values, input_ids=input_ids, max_length=50)
>>> print(processor.batch_decode(generated_ids, skip_special_tokens=True))
['what does the front of the bus say at the top? special']
```
Video captioning example:
```python
>>> import av
>>> import numpy as np
>>> from PIL import Image
>>> from huggingface_hub import hf_hub_download
>>> from transformers import AutoProcessor, AutoModelForCausalLM
>>> processor = AutoProcessor.from_pretrained("microsoft/git-base-vatex")
>>> model = AutoModelForCausalLM.from_pretrained("microsoft/git-base-vatex")
>>> # set seed for reproducibility
>>> np.random.seed(45)
>>> def read_video_pyav(container, indices):
... '''
... Decode the video with PyAV decoder.
... Args:
... container (`av.container.input.InputContainer`): PyAV container.
... indices (`list[int]`): List of frame indices to decode.
... Returns:
... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3).
... '''
... frames = []
... container.seek(0)
... start_index = indices[0]
... end_index = indices[-1]
... for i, frame in enumerate(container.decode(video=0)):
... if i > end_index:
... break
... if i >= start_index and i in indices:
... frames.append(frame)
... return np.stack([x.to_ndarray(format="rgb24") for x in frames])
>>> def sample_frame_indices(clip_len, frame_sample_rate, seg_len):
... '''
... Sample a given number of frame indices from the video.
... Args:
... clip_len (`int`): Total number of frames to sample.
... frame_sample_rate (`int`): Sample every n-th frame.
... seg_len (`int`): Maximum allowed index of sample's last frame.
... Returns:
... indices (`list[int]`): List of sampled frame indices
... '''
... converted_len = int(clip_len * frame_sample_rate)
... end_idx = np.random.randint(converted_len, seg_len)
... start_idx = end_idx - converted_len
... indices = np.linspace(start_idx, end_idx, num=clip_len)
... indices = np.clip(indices, start_idx, end_idx - 1).astype(np.int64)
... return indices
>>> # load video
>>> file_path = hf_hub_download(
... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset"
... )
>>> container = av.open(file_path)
>>> # sample frames
>>> num_frames = model.config.num_image_with_embedding
>>> indices = sample_frame_indices(
... clip_len=num_frames, frame_sample_rate=4, seg_len=container.streams.video[0].frames
... )
>>> frames = read_video_pyav(container, indices)
>>> pixel_values = processor(images=list(frames), return_tensors="pt").pixel_values
>>> generated_ids = model.generate(pixel_values=pixel_values, max_length=50)
>>> print("Generated caption:", processor.batch_decode(generated_ids, skip_special_tokens=True))
Generated caption: ['a woman is sitting at a table and she is talking about the food she is holding.']
```
GIT architecture. Taken from the original paper.
This model was contributed by [nielsr](https://huggingface.co/nielsr).
The original code can be found [here](https://github.com/microsoft/GenerativeImage2Text).
## Usage tips
- GIT is implemented in a very similar way to GPT-2, the only difference being that the model is also conditioned on `pixel_values`.
## Resources
A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with GIT.
- Demo notebooks regarding inference + fine-tuning GIT on custom data can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/GIT).
- See also: [Causal language modeling task guide](../tasks/language_modeling)
If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we will review it.
The resource should ideally demonstrate something new instead of duplicating an existing resource.
## GitVisionConfig[[transformers.GitVisionConfig]]