runtime error

Exit code: 1. Reason: s = load_model_from_single_file(state_dict, model_names, model_classes, model_resource, torch_dtype, device, infer) File "/app/OmniAvatar/models/model_manager.py", line 97, in load_model_from_single_file model = model.to_empty(device=device) File "/root/.pyenv/versions/3.10.19/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1224, in to_empty return self._apply( File "/root/.pyenv/versions/3.10.19/lib/python3.10/site-packages/torch/nn/modules/module.py", line 928, in _apply module._apply(fn) File "/root/.pyenv/versions/3.10.19/lib/python3.10/site-packages/torch/nn/modules/module.py", line 928, in _apply module._apply(fn) File "/root/.pyenv/versions/3.10.19/lib/python3.10/site-packages/torch/nn/modules/module.py", line 928, in _apply module._apply(fn) [Previous line repeated 1 more time] File "/root/.pyenv/versions/3.10.19/lib/python3.10/site-packages/torch/nn/modules/module.py", line 955, in _apply param_applied = fn(param) File "/root/.pyenv/versions/3.10.19/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1225, in <lambda> lambda t: torch.empty_like(t, device=device), recurse=recurse File "/root/.pyenv/versions/3.10.19/lib/python3.10/site-packages/torch/_prims_common/wrappers.py", line 309, in _fn result = fn(*args, **kwargs) File "/root/.pyenv/versions/3.10.19/lib/python3.10/site-packages/torch/_refs/__init__.py", line 5055, in empty_like return torch.empty_permuted( torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 270.00 MiB. GPU 0 has a total capacity of 44.39 GiB of which 157.38 MiB is free. Including non-PyTorch memory, this process has 44.23 GiB memory in use. Of the allocated memory 43.80 GiB is allocated by PyTorch, and 13.07 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

Container logs: