Queue expecting cuda tensors and more cuda strangeness

[Hierakonpolis]

Hello, I am having some trouble juggling cuda and cpu types around. I would not be expecting RandomElasticDeformation to have anything to do with cuda, as I just want to prepare things in the background and later load them to GPU. With trainingset as a list of subject:

trainingset[0]
Out[2]: Subject(Keys: ('t1', 'site'); images: 1)

trainingset[0]['t1']
Out[3]: ScalarImage(shape: (1, 361, 433, 361); spacing: (0.50, 0.50, 0.50); orientation: LPS+; memory: 215.3 MiB; type: intensity)

trainingset[0]['t1'].data.device
Out[4]: device(type='cpu')

t=tio.transforms.RandomElasticDeformation()

t(trainingset[0])
Traceback (most recent call last):

  File "<ipython-input-6-2cb9a183d158>", line 1, in <module>
    t(trainingset[0])

  File "/home/cat/anaconda3/envs/adapt/lib/python3.9/site-packages/torchio/transforms/augmentation/random_transform.py", line 34, in __call__
    return super().__call__(sample)

  File "/home/cat/anaconda3/envs/adapt/lib/python3.9/site-packages/torchio/transforms/transform.py", line 100, in __call__
    transformed = self.apply_transform(sample)

  File "/home/cat/anaconda3/envs/adapt/lib/python3.9/site-packages/torchio/transforms/augmentation/spatial/random_elastic_deformation.py", line 219, in apply_transform
    bspline_params = self.get_params(

  File "/home/cat/anaconda3/envs/adapt/lib/python3.9/site-packages/torchio/transforms/augmentation/spatial/random_elastic_deformation.py", line 187, in get_params
    return coarse_field.numpy()

TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.

Interestingly, I seem to have the opposite issue when trying to set up a queue. Even avoiding any transforms:

sampler = tio.data.UniformSampler(64)

traindata = tio.SubjectsDataset(trainingset)

TrainQ = tio.data.Queue(traindata, max_length = 20, samples_per_volume = 20, sampler = sampler, num_workers = 10)
Traceback (most recent call last):

  File "<ipython-input-18-41e340879bc9>", line 1, in <module>
    TrainQ = tio.data.Queue(traindata, max_length = 20, samples_per_volume = 20, sampler = sampler, num_workers = 10)

  File "/home/cat/anaconda3/envs/adapt/lib/python3.9/site-packages/torchio/data/queue.py", line 95, in __init__
    self.subjects_iterable = self.get_subjects_iterable()

  File "/home/cat/anaconda3/envs/adapt/lib/python3.9/site-packages/torchio/data/queue.py", line 189, in get_subjects_iterable
    return iter(subjects_loader)

  File "/home/cat/anaconda3/envs/adapt/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 359, in __iter__
    return self._get_iterator()

  File "/home/cat/anaconda3/envs/adapt/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 305, in _get_iterator
    return _MultiProcessingDataLoaderIter(self)

  File "/home/cat/anaconda3/envs/adapt/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 944, in __init__
    self._reset(loader, first_iter=True)

  File "/home/cat/anaconda3/envs/adapt/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 975, in _reset
    self._try_put_index()

  File "/home/cat/anaconda3/envs/adapt/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1209, in _try_put_index
    index = self._next_index()

  File "/home/cat/anaconda3/envs/adapt/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 512, in _next_index
    return next(self._sampler_iter)  # may raise StopIteration

  File "/home/cat/anaconda3/envs/adapt/lib/python3.9/site-packages/torch/utils/data/sampler.py", line 226, in __iter__
    for idx in self.sampler:

  File "/home/cat/anaconda3/envs/adapt/lib/python3.9/site-packages/torch/utils/data/sampler.py", line 124, in __iter__
    yield from torch.randperm(n, generator=generator).tolist()

RuntimeError: Expected a 'cuda' device type for generator but found 'cpu'

I have no issue with something like:

plt.imshow(trainingset[0]['t1'].data[0,150,:,:].numpy())

I'm sure there is something I'm missing here on how I should be controlling where things are saved in memory, but any idea what that might be?

[romainVala]

Hello

  File "/home/cat/anaconda3/envs/adapt/lib/python3.9/site-packages/torchio/transforms/augmentation/spatial/random_elastic_deformation.py", line 187, in get_params

this error line does not match with the last torchio of the file, version

Could you please update torchio, and precise, with python pytorch version you are using

I suppose the update of torchio wont change, may be it come from the specific pytorch/python version you use ...

can you also test in you python environment,

coarse_field = torch.rand(2,2,20)
coarse_field.device

from what I understand in you first error it seems you get a cuda variable ...
I get
device(type='cpu')

[Hierakonpolis]

Thanks, I eventually found the problem to be a line in an older file I imported in the same project setting the default tensor type to cuda. Thank you for your time!