[Question] - Memory Usage

[synclab-srl]

Hi @yrutschle

With the latest commit, sslh is working fine.
I have a question about memory usage.

In 24 hours of operation, memory allocation reached 50MB (when Docker starts, it's about 5MB).
After three days, it's about 100MB.

Is this growth normal?

This is my configuration (started with docker, sslh-select):

foreground: true;

verbose-packets: 0

listen:
(
  { host: "0.0.0.0"; port: "443"; keepalive: true; },
  { host: "0.0.0.0"; port: "80"; keepalive: true; }
);

protocols:
( 
  { name: "regex"; host: "proxy-server"; port: "1337"; fork: true; log_level: 1; keepalive: true; tfo_ok: true; regex_patterns: [ "(?=.*\bCONNECT\b)(?=.*\b\.p\.mydomain\.it\b)" ]; },
  { name: "ssh"; service: "ssh"; host: "sish"; port: "2222"; fork: true; keepalive: true; tfo_ok: true; },
  { name: "tls"; host: "nginx"; port: "443"; fork: true; log_level: 1; keepalive: true; tfo_ok: true; },
  { name: "http"; host: "nginx"; port: "80"; fork: true; log_level: 1; keepalive: true; tfo_ok: true; }, 
  { name: "anyprot"; host: "proxy-server"; fork: true; port: "7835"; log_level: 1; keepalive: true; tfo_ok: true; }
);

Is there anything I can optimize?
Thanks.

[synclab-srl]

Hi @yrutschle

On a test instance (same configuration), I removed the "fork" parameter, and memory usage is stable at around 1MB.

What are the implications of removing the fork?

My goal is to achieve maximum stability on the production server.

Thank you.

[yrutschle]

without the 'fork' setting, sslh-select monitors all connections at once, and processes whatever is happening in the one process. When using 'fork', connections for that protocol are forked into a new process that will shovel data back and forth just for that connection.
Arguably, fork is vaguely more secure and probably a bit more resource-intensive than not forking. (I say "arguably" because these claim would require close analysis).
So all in all, there is no reason you couldn't use sslh-select in production (or probably better yet, sslh-ev).

That said, you seem to have uncovered a memory leak where sslh-select leaks memory when forking processes. I am now investigating what's leaking, thanks for the report.

[synclab-srl]

Hi, @yrutschle
Thanks for the explanation.

Now I'm trying to build sslh-ev with Docker, but it won't compile.
I modified the sslh-ev dockerfile (line 19):

...
RUN ./configure && make sslh-ev && strip sslh-ev
...

When I build, I get this error:

0.762 ./genver.sh >version.h
0.778 cc -Wall -O2 -DLIBPCRE -g   -DENABLE_REGEX -DLIBCONFIG -c processes.c -o processes.o
0.884 cc -Wall -O2 -DLIBPCRE -g   -DENABLE_REGEX -DLIBCONFIG -c udp-listener.c -o udp-listener.o
1.019 cc -Wall -O2 -DLIBPCRE -g   -DENABLE_REGEX -DLIBCONFIG  -c -o sslh-ev.o sslh-ev.c
1.028 sslh-ev.c:24:10: fatal error: ev.h: No such file or directory
1.028    24 | #include <ev.h>
1.028       |          ^~~~~~
1.028 compilation terminated.
1.029 make: *** [<builtin>: sslh-ev.o] Error 1

Maybe I'm doing something wrong?

Thank you.

[synclab-srl]

Oh, okay.
I should have added libev-dev to the dockerfile dependencies build stage and libev in the final stage.

Now I'll try testing sslh-ev with the fork parameter.
I'll let you know if there's this memory growth with sslh-ev too.

Thanks.

[synclab-srl]

Hi @yrutschle

Same memory growth with the fork option with sslh-ev.
The behavior is almost the same as the select version.

After half a day of running, the memory used went from 4MB to 25MB.
For now, I won't be using fork in production.
Let me know if there's anything new and if I can help with further testing.

Thanks.

[yrutschle]

I think I found the issue (in amazingly old code that was not correct then, but apparently only caused a problem when forking was introduced). This should fix both sslh-select and sslh-ev. Please re-open if it still leaks on your end!

[synclab-srl]

Hi @yrutschle
I have deployed the fix.
I will monitor the app over the next few days.
Thank you.

[synclab-srl]

Hi @yrutschle
The fix works correctly.
No memory leak with "fork" in sslh-ev and sslh-select.
Thanks

[suddhasilsarkar]

Hi , I am using sslh-ev , non fork mode and I am observing the memory leakage issue there. I have been using v2.3.1 with the memory leak cherry-pick (387ef13) . Any thought any idea you know of to test his further ? I am using AWS ECS to run this container based infrastructure.

[yrutschle]

Yes, if you are a bit technically inclindes, the easiest way to investigate memory leaks is to run with valgrind:

valgrind --leak-check=full ./sslh-ev -F myconfig.cfg

Let it run for a while, stop it: valgrind will print a report of all memory allocated, including memory that no longer has any pointers.
If you report here, that'll help me investigate. Otherwise, I'll test on my side hoping I can reproduce the issue on HEAD.

[suddhasilsarkar]

Hi @yrutschle thanks for helping here, just to let you know - I have started the log collection since it needs considerable load/connection through SSLH to reproduce the issue, I will let you soon I have the data . In the meantime if you get anything form your test run, let me know. Thanks!

[suddhasilsarkar]

Coming back here again, since i moved into another testing phase and work i couldn't collect this data from the deployed environment in customer side. Please let me know if you get anything from your test side ? if Not I will try to get as soon as possible but that would take another week or so atleast.