Situation.
Uploading hundreds of 400mb files as Hadoop.
The error below occurs in an unknown situation while uploading.
Process Process-5:
Traceback (most recent call last):
File "D:\project\project_swm\apollo\code_third\sensor-data-usb-uploader\venv\lib\site-packages\urllib3-1.26.14-py3.9.egg\urllib3\connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "D:\project\project_swm\apollo\code_third\sensor-data-usb-uploader\venv\lib\site-packages\urllib3-1.26.14-py3.9.egg\urllib3\connectionpool.py", line 449, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "D:\project\project_swm\apollo\code_third\sensor-data-usb-uploader\venv\lib\site-packages\urllib3-1.26.14-py3.9.egg\urllib3\connectionpool.py", line 444, in _make_request
httplib_response = conn.getresponse()
File "D:\dev\python\python-3.9.10\lib\http\client.py", line 1377, in getresponse
response.begin()
File "D:\dev\python\python-3.9.10\lib\http\client.py", line 320, in begin
version, status, reason = self._read_status()
File "D:\dev\python\python-3.9.10\lib\http\client.py", line 289, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "D:\project\project_swm\apollo\code_third\sensor-data-usb-uploader\venv\lib\site-packages\requests-2.31.0-py3.9.egg\requests\adapters.py", line 486, in send
resp = conn.urlopen(
File "D:\project\project_swm\apollo\code_third\sensor-data-usb-uploader\venv\lib\site-packages\urllib3-1.26.14-py3.9.egg\urllib3\connectionpool.py", line 787, in urlopen
retries = retries.increment(
File "D:\project\project_swm\apollo\code_third\sensor-data-usb-uploader\venv\lib\site-packages\urllib3-1.26.14-py3.9.egg\urllib3\util\retry.py", line 550, in increment
raise six.reraise(type(error), error, _stacktrace)
File "D:\project\project_swm\apollo\code_third\sensor-data-usb-uploader\venv\lib\site-packages\urllib3-1.26.14-py3.9.egg\urllib3\packages\six.py", line 769, in reraise
raise value.with_traceback(tb)
File "D:\project\project_swm\apollo\code_third\sensor-data-usb-uploader\venv\lib\site-packages\urllib3-1.26.14-py3.9.egg\urllib3\connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "D:\project\project_swm\apollo\code_third\sensor-data-usb-uploader\venv\lib\site-packages\urllib3-1.26.14-py3.9.egg\urllib3\connectionpool.py", line 449, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "D:\project\project_swm\apollo\code_third\sensor-data-usb-uploader\venv\lib\site-packages\urllib3-1.26.14-py3.9.egg\urllib3\connectionpool.py", line 444, in _make_request
httplib_response = conn.getresponse()
File "D:\dev\python\python-3.9.10\lib\http\client.py", line 1377, in getresponse
response.begin()
File "D:\dev\python\python-3.9.10\lib\http\client.py", line 320, in begin
version, status, reason = self._read_status()
File "D:\dev\python\python-3.9.10\lib\http\client.py", line 289, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "D:\dev\python\python-3.9.10\lib\multiprocessing\process.py", line 315, in _bootstrap
self.run()
File "D:\dev\python\python-3.9.10\lib\multiprocessing\process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "D:\project\project_swm\apollo\code_third\sensor-data-usb-uploader\venv\lib\site-packages\alib-20240110.1.0-py3.9.egg\MultiProcess\abProcess.py", line 281, in Action
self.Running(process)
File "D:\project\project_swm\apollo\code_third\sensor-data-usb-uploader\App\cUploaderProcess.py", line 53, in Running
self.objStorage.Upload(upLoad_file_name, "/t2")
File "D:\project\project_swm\apollo\code_third\sensor-data-usb-uploader\venv\lib\site-packages\alib-20240110.1.0-py3.9.egg\ObjectStorage2\Hadoop\cHadoopStorage.py", line 34, in Upload
if not self.storage.status(_dstPath, strict=False):
File "D:\project\project_swm\apollo\code_third\sensor-data-usb-uploader\venv\lib\site-packages\hdfs-2.7.3-py3.9.egg\hdfs\client.py", line 321, in status
res = self._get_file_status(hdfs_path, strict=strict)
File "D:\project\project_swm\apollo\code_third\sensor-data-usb-uploader\venv\lib\site-packages\hdfs-2.7.3-py3.9.egg\hdfs\client.py", line 125, in api_handler
raise err
File "D:\project\project_swm\apollo\code_third\sensor-data-usb-uploader\venv\lib\site-packages\hdfs-2.7.3-py3.9.egg\hdfs\client.py", line 102, in api_handler
res = client._request(
File "D:\project\project_swm\apollo\code_third\sensor-data-usb-uploader\venv\lib\site-packages\hdfs-2.7.3-py3.9.egg\hdfs\client.py", line 209, in _request
return self._session.request(
File "D:\project\project_swm\apollo\code_third\sensor-data-usb-uploader\venv\lib\site-packages\requests-2.31.0-py3.9.egg\requests\sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "D:\project\project_swm\apollo\code_third\sensor-data-usb-uploader\venv\lib\site-packages\requests-2.31.0-py3.9.egg\requests\sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "D:\project\project_swm\apollo\code_third\sensor-data-usb-uploader\venv\lib\site-packages\requests-2.31.0-py3.9.egg\requests\adapters.py", line 501, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
====================================================================================
My Python code is really simple.
class HadoopTest:
def __init__(self, _address, _port, _access_key):
super().__init__(_address, _port, _access_key)
def Connect(self):
return InsecureClient(url="http://" + self.address + ":" + self.port,
user=self.access_key)
def Upload(self, hadoop, _dstPath, _srcPath):
hadoop.upload(_dstPath, _srcPath, overwrite=_overwrite)
if __name__ == '__main__':
hadoop = HadoopTest({ip}, {port}, {user})
hdfs_client = hadoop.Connect()
while(True):
hadoop.Upload(hdfs_client, { src }, {dst})
# The file is about 400 megabytes.The code above is a rough re-implementation of our code.
However, the difference is that the code is processed in parallel with multiprocessing.
In that code, we make one connection and constantly call upload only.
The problem hasn't occurred yet when I connected the file before uploading it and coded it by making it a target of gc by nulling it after uploading it. However, I don't know when it will happen, so I need your feedback.
And the biggest problem is that sometimes this exception occurs, sometimes it does not.
My Enviroment :
OS : Windows 11
Python : 3.9.10
hdfs(pip) : 2.7.3(latest)
Hadoop : 3.3.2
Situation.
Uploading hundreds of 400mb files as Hadoop.
The error below occurs in an unknown situation while uploading.
====================================================================================
My Python code is really simple.
The code above is a rough re-implementation of our code.
However, the difference is that the code is processed in parallel with multiprocessing.
In that code, we make one connection and constantly call upload only.
The problem hasn't occurred yet when I connected the file before uploading it and coded it by making it a target of gc by nulling it after uploading it. However, I don't know when it will happen, so I need your feedback.
And the biggest problem is that sometimes this exception occurs, sometimes it does not.