Skip to content

train error : SystemError: (Fatal) Blocking queue is killed because the data reader raises an exception. #499

Description

@bigyaofei

W1011 17:07:55.921258 79684 gpu_resources.cc:119] Please NOTE: device: 0, GPU Compute Capability: 8.6, Driver API Version: 12.2, Runtime API Version: 11.8
W1011 17:07:55.921286 79684 gpu_resources.cc:164] device: 0, cuDNN Version: 8.9.
Exception in thread Thread-2:
Traceback (most recent call last):
File "/home/fyao/anaconda3/envs/paddle/lib/python3.9/threading.py", line 980, in _bootstrap_inner
self.run()
File "/home/fyao/anaconda3/envs/paddle/lib/python3.9/threading.py", line 917, in run
self._target(*self._args, **self._kwargs)
Traceback (most recent call last):
File "/home/fyao/anaconda3/envs/paddle/lib/python3.9/site-packages/paddle/io/dataloader/dataloader_iter.py", line 603, in _thread_loop
File "/home/fyao/prj/Paddle3D/tools/train.py", line 269, in
batch = self._get_data()
File "/home/fyao/anaconda3/envs/paddle/lib/python3.9/site-packages/paddle/io/dataloader/dataloader_iter.py", line 752, in _get_data
main(args)
File "/home/fyao/prj/Paddle3D/tools/train.py", line 264, in main
batch.reraise()
File "/home/fyao/anaconda3/envs/paddle/lib/python3.9/site-packages/paddle/io/dataloader/worker.py", line 187, in reraise
trainer.train()
File "/home/fyao/prj/Paddle3D/paddle3d/apis/trainer.py", line 327, in train
raise self.exc_type(msg)
TypeError: DataLoader worker(1) caught TypeError with message:
Traceback (most recent call last):
File "/home/fyao/anaconda3/envs/paddle/lib/python3.9/site-packages/paddle/io/dataloader/worker.py", line 372, in _worker_loop
batch = fetcher.fetch(indices)
File "/home/fyao/anaconda3/envs/paddle/lib/python3.9/site-packages/paddle/io/dataloader/fetcher.py", line 77, in fetch
data.append(self.dataset[idx])
File "/home/fyao/prj/Paddle3D/paddle3d/datasets/kitti/kitti_pointcloud_det.py", line 62, in getitem
sample = self.transforms(sample)
File "/home/fyao/prj/Paddle3D/paddle3d/transforms/base.py", line 45, in call
sample = t(sample)
File "/home/fyao/prj/Paddle3D/paddle3d/transforms/sampling.py", line 278, in call
samples = self.sampling(sample, num_samples_per_class)
File "/home/fyao/prj/Paddle3D/paddle3d/transforms/sampling.py", line 205, in sampling
sampling_bboxes_3d, mv_height = self._put_boxes_on_road_planes(
File "/home/fyao/prj/Paddle3D/paddle3d/transforms/sampling.py", line 139, in _put_boxes_on_road_planes
a, b, c, d = road_planes
TypeError: cannot unpack non-iterable NoneType object
for sample in self.train_dataloader:

File "/home/fyao/anaconda3/envs/paddle/lib/python3.9/site-packages/paddle/io/dataloader/dataloader_iter.py", line 826, in next
self.reader.read_next_list()[0]
SystemError: (Fatal) Blocking queue is killed because the data reader raises an exception.
[Hint: Expected killed
!= true, but received killed_:1 == true:1.] (at /paddle/paddle/fluid/operators/reader/blocking_queue.h:175)

why?

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions