/examine does not seem to handle url-encoded file parameter properly

[jenswegar]

It looks like the servlet does not like paths in the /examine?file parameter that contain scandinavian characters (åäö).

If I do the following GET request to the /examine endpoint (which should be the url encoded version of /processing/ääkkostest/fmi_485.1_foto_002_001.jpg):

curl -v http://localhost:8989/fits/examine?file=/processing/%C3%A4%C3%A4kkostest/fmi_485.1_foto_002_001.jpg

I get the error

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<error>
    <statusCode>400</statusCode>
    <message> File not sent with request: fmi_485.1_foto_002_001.jpg</message>
    <request> http://localhost:8989/fits/examine</request>
</error>

however the same request without the ää characters in the file parameter works as expected. Only change I did was changing the ää to aa and correspondingly renaming the folder in which the file resides.

I used the 1.5.0-centos7 image tag of the docker image at https://hub.docker.com/r/harvardlts/fitsservlet_container/tags for testning, but the same issue was also present with the 1.4.1 version.

I checked the running container and in both cases the /processing path in the container did contain the expected file, so I'd say that indicates that the error happens somewhere in how the servlet interprets the GET parameter. Is there something I should be doing different in my curl request to get the path with scandinavian characters to work?

[daveneiman]

Thank you for letting us know about this issue. We will look into this issue but not sure how soon we will be able to do so.

[daveneiman]

Hello again,

From a Terminal in Mac OSX I do get the error when the URL contains ää.
However it does work for me when using the encoded characters both with the URL enclosed with quotes and without.

These fail:
curl -v -X GET http://localhost:8080/fits/examine?file=/Users/dan179/Documents/FITS-test/äätest/Some_File.docx
curl -v -X GET "http://localhost:8080/fits/examine?file=/Users/dan179/Documents/FITS-test/äätest/Some_File.docx"

These succeed:
curl -v -X GET http://localhost:8080/fits/examine?file=/Users/dan179/Documents/FITS-test/%C3%A4%C3%A4test/Some_File.docx
curl -v -X GET "http://localhost:8080/fits/examine?file=/Users/dan179/Documents/FITS-test/%C3%A4%C3%A4test/Some_File.docx"

I was also successful using the original URL in a browser (Chrome) address bar.

It might be an encoding problem with the version of curl you are using.

[jenswegar]

Thanks for getting back to me quickly!

hmm, ok could you post the output of your request with the -v flag on the request that works? Maybe that would shed some light on what is different with my setup vs. yours... For me one failing request looks like this:

curl -v -X GET http://localhost:8989/fits/examine?file=/processing/test/%C3%A4%C3%A4kkostest/fmi_485.1_foto_002_001.jpg

Note: Unnecessary use of -X or --request, GET is already inferred.
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8989 (#0)
> GET /fits/examine?file=/processing/test/%C3%A4%C3%A4kkostest/fmi_485.1_foto_002_001.jpg HTTP/1.1
> Host: localhost:8989
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 400
< Content-Type: text/xml;charset=ISO-8859-1
< Content-Length: 227
< Date: Fri, 11 Oct 2019 14:27:26 GMT
< Connection: close
<
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><error><statusCode>400</statusCode><message> File not sent with request: fmi_485.1_foto_002_001.jpg</message><request> http://localhost:8989/fits/examine</request></error>
* Closing connection 0

I've tried this same experiment on MacOS both from the terminal and Chrome/Firefox but still get error. I even tried it from inside the docker container running the FITS stack itself, and still get the same result =/ curl versions are 7.29.0 and 7.54.0

I also tried it on our production server (using curl running on linux) where the stack is running, and same result there.

I'm starting to wonder if the locale of the machines somehow affect this, but then I'm wondering why the docker containers produce the same result...

[daveneiman]

Here’s my curl output from the terminal on MacOS:

dan179:~ $ curl -v -X GET "http://localhost:8080/fits/examine?file=/Users/dan179/Documents/FITS-WP-test/äätest/Sample.doc"
Note: Unnecessary use of -X or --request, GET is already inferred.

• Trying ::1...
• TCP_NODELAY set
• Connected to localhost (::1) port 8080 (#0)

GET /fits/examine?file=/Users/dan179/Documents/FITS-WP-test/äätest/Sample.doc HTTP/1.1
Host: localhost:8080
User-Agent: curl/7.54.0
Accept: /

< HTTP/1.1 400
< Transfer-Encoding: chunked
< Date: Tue, 15 Oct 2019 16:59:19 GMT
< Connection: close
<

• Closing connection 0

On the Tomcat side here’s output from logs/catalina.out

15-Oct-2019 12:59:19.806 INFO [http-nio-8080-exec-1] org.apache.coyote.http11.Http11Processor.service Error parsing HTTP request header
Note: further occurrences of HTTP header parsing errors will be logged at DEBUG level.
java.lang.IllegalArgumentException: Invalid character found in the request target. The valid characters are defined in RFC 7230 and RFC 3986
at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:479)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:684)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:806)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)

And the output from logs/localhost_access_log.2019-10-15.txt

0:0:0:0:0:0:0:1 - - [15/Oct/2019:12:59:19 -0400] "GET null null" 400 –

When I put the local document URL into a Chrome browser it works. No errors from catalina.out and the following from logs/localhost_access_log.2019-10-15.txt

0:0:0:0:0:0:0:1 - - [15/Oct/2019:13:04:28 -0400] "GET /fits/examine?file=/Users/dan179/Documents/FITS-WP-test/%C3%A4%C3%A4test/Sample.doc HTTP/1.1" 200 4506

Not sure if this helps at all. Not sure where you should go from here. Best of luck.