@flipperzero The hashnix.club misfin server is down, looks like it has been down for a couple of days.
Posted in: s/misfin
🚀 cipres
Mar 29 · 8 weeks ago
😎 flipperzero · Mar 30 at 13:16:
@cipres apologies and thanks to you, and every one of yall amongst geminispace who use the service, for the notice and concern. The hashnix misfin is operational again and now updated to the latest version of the implement.
🚀 cipres [OP] · Apr 05 at 21:21:
@flipperzero Still can't connect to hashnix's misfin, port 1958 (ipv6 or ipv4). Port 1965 works fine as always.
🚀 cipres [OP] · Apr 10 at 11:31:
Up and running now
😎 flipperzero · Apr 10 at 23:35:
@cipres can I send you a series of error logs and see if you can pinooint in case there’s anything making the server crash? I’ve been noticing dropped service too, but then when I check my daemon’s status indicates that all is seemingly operational and it’s been confusing me as of this past week.
🚀 cipres [OP] · Apr 11 at 11:15:
@flipperzero Yes, please send me the error logs (via misfin or mail), thank you. Are you seeing any "Blocking peer ..." messages ? If you have a gitlab account you can also create an issue and paste the logs there:
— https://gitlab.com/cipres/misfin/-/work_items
I'm thinking that maybe all of this could be related to the use of the LMDB database (for message stats), that was commited on March 10. I will disable it by default and push an update.
Thanks a lot @flipperzero
😎 flipperzero · Apr 12 at 09:26:
@cipres I've went ahead and installed your latest update, but with interest to best practice, I have some excerpt of the logs sent to your galacteek email as well as your hashnix misfin. Thank you again, cipres.
🚀 cipres [OP] · Apr 12 at 14:27:
@flipperzero I've got the logs, cheers, this is very helpful.
Are you using systemd ? Trying to understand if the "Signal 15 (SIGTERM) received" logs happen because you ran systemctl restart on the misfin service, that would make sense.
😎 flipperzero · Apr 12 at 16:50:
@cipres right right, those are instances in which I noticed that even though the daemon indicated operation that the gemini frontend was not displaying connection, thus the restarts via systemctl.
🐦 roughnecks · Apr 12 at 17:23:
I believe it happened to me too. systemd unit for me as well.
🚀 cipres [OP] · Apr 12 at 18:49:
@flipperzero Got it, thank you. You have a pretty long experience running a public misfin server now !
I fixed one of the things that appears in the logs. I think i'm gonna add some stress tests to pinpoint what makes the server fail under strain, maybe it's just too many threads and it dies.
I'll keep adding emojis everywhere in the code until i figure this one out.
😎 flipperzero · Apr 29 at 20:45:
@cipres glad to be right there along with ya in the honor of your presence, cipres!
Now, that said, it seems we're running into the issue again. Once more, no apparent status messages indicating any correlation to anything conflicting. Just seems to keep stuck on refresh, and then eventually no longer loads til daemon restart. Let me know what I can search for to provide any more details.
🚀 cipres [OP] · Apr 30 at 08:21:
@flipperzero Ok. I gotta say, i'm not sure what's causing this. I don't think that the python 3.14 migration is causing this (going back to 3.9 isn't really an option anyway).Anything useful in the logs ?
@flipperzero Just tried to access the hashnix server right now and it doesn't seem down (connection isn't refused) but it doesn't load any page, just stuck. Yeah that is really strange ...
😎 flipperzero · Apr 30 at 11:14:
@cipres nothing conclusive in the logs except for the following -
cat /usr/src/misfin-srv/misfind.log
Running from git commit with SHA: 10646f3666095ec9caec01bf31511844a66adedd
Starting misfind v1.2.8 at 2026-04-29 23:07:51.251874
that's it :l
🚀 cipres [OP] · Apr 30 at 22:01:
@flipperzero Can you please edit the misfind.toml and add:
log_level = "debug"
rl_log_level = "debug"
so that we get more information from the logs ? Can you share the content of your current misfind.toml ?
I feel like this is a resource exhaustion issue. Server using too many threads, sockets, whatever .. to the point where it stops functioning. Another possibility is that the rate limiting system starts to malfunction and just blocks every request. That coincides with what you're seeing on your server. Is the CPU load high when the servers freezes ?
😎 flipperzero · Apr 30 at 23:50:
@cipres do I add this under the [service.main] section, or under the others entitled [service.main.ratelimit.anonymous] or [service.main.ratelimit.verified]? In fact, until you had made mention, I hadn't ever noticed this file or that there were these other fields. Would any info within those fields help you better? This is what they print.
[service.main.ratelimit.anonymous]
req_limit = 30
duration = 60
[service.main.ratelimit.verified]
req_limit = 100
duration = 60
Let me know at your soonest convenience, thanks much and always.
🚀 cipres [OP] · May 01 at 10:14:
@flipperzero Add them at the top of the config file (outside of the service section). Your rate limit config has the default values, that's fine then. Then you can restart the daemon and we should get more logs.
😎 flipperzero · May 02 at 01:15:
@cipres I sent you an email now, at your proton address, hopefully it shows it sent from my hashnix email host. Let me know if this helps, and if the message gets to you.
🚀 cipres [OP] · May 02 at 15:57:
@flipperzero I can't access the log file's URL you sent me (not found error).
😎 flipperzero · May 02 at 16:14:
apologies, there was a typo, try it now
🚀 cipres [OP] · May 02 at 16:51:
@flipperzero Got the log, very helpful. I'll fix what's generating these traceback errors but that's not what would cause a crash. Did a crash occur in the time window covered by this log (for any of those 5 srv restarts) ?
😎 flipperzero · May 02 at 16:56:
@cipres those restarts were when crashes occured or at least before hand of, and from up to that point after the end of April to May 1st there have been no restarts and still the misfin page stays stuck loading as these messages continue to occur.
🚀 cipres [OP] · May 02 at 17:37:
@flipperzero Enough memory and disk space available on your server ? This is really puzzling.
🚀 cipres [OP] · May 02 at 17:49:
@flipperzero I was able to reproduce the issue, will push a fix hopefully today, otherwise tomorrow. Thanks again !
😎 flipperzero · May 02 at 18:43:
@cipres 8gb RAM, 8 cores that max at 2.8 - 3ghz
- * edit - apologies, just seen you found the fix. Can’t wait for the update!
🚀 cipres [OP] · May 02 at 19:32:
@flipperzero New release is out. It changes the default worker threads configuration. Not sure 100% this will fix it. Do an update and restart the service, and then i'll do a stress test on the server.
😎 flipperzero · May 02 at 19:44:
@cipres uppdated and running now
🚀 cipres [OP] · May 02 at 19:51:
@flipperzero Ok. The frontend seems to respond faster than before. Let's wait and see, there's a small chance that it's fixed now but i wouldn't bet too much on it.
😎 flipperzero · May 12 at 20:52:
@cipres sent you a message over your proton email. what we were suspecting would eventually happened has finally happened. let me know once you reach this communique. Thank you.
🚀 cipres [OP] · May 12 at 22:45:
@flipperzero Very helpful logs, thank you. I'll analyze them tomorrow. What's clear is that you're being flooded with requests, many IPs are blocked by the rate limiter. Clearly need a more restrictive policy here, just ban abusive hosts ... So the server stopped on May 12 at 20:50 roughly ? No tracebacks at the end ...
😎 flipperzero · May 12 at 23:11:
@cipres yes the both logs seemed highly confusing to me even paired together because it's not very easy for me to pinpoint where it could have been anything dropped and halted service :c
🚀 cipres [OP] · May 13 at 11:45:
@flipperzero My impression is that the requests flood ends up paralizing the thread pool. I will change the default policy so that any IP hitting the rate limit (which is pretty high) is banned. It's a safe change because no one using the server in a responsible manner can ever come close to reaching the default rate limit.
🐦 roughnecks · May 14 at 08:59:
May I ask how the ban is accomplished?
🚀 cipres [OP] · May 14 at 13:22:
@roughnecks Using a simple IP blocklist file.
🚀 cipres [OP] · May 14 at 13:32:
@flipperzero Just pushed a change that stores blocked peers in a blocklist.
You need to add "ratelimit_block_peer = true" in the service section in your misfind.toml config file:
— https://gitlab.com/cipres/misfin/-/blob/master/docs/index.md?#block-peers-who-exceed-the-rate-limit
When ratelimit_block_peer is enabled, peers who exceed the rate limit are added to the blocklist and are therefore permanently banned from the server. It is disabled by default. Peers who make port scan probes on the server (not valid SSL reqs) are automatically banned.
The filename in the service directory is "blocklist". Easy to monitor and edit if necessary.
🐦 roughnecks · May 14 at 18:27:
Just one observation. I think I already mentioned to you in one issue, that I found the VPS IPv4 in the warnings for being blocked.
🐦 roughnecks · May 14 at 19:10:
Anyway, I have upgraded and applied the configuration.
🚀 cipres [OP] · May 14 at 19:20:
@roughnecks I don't remember that, will look at the issues. That shouldn't happen, although, even if it did, it wouldn't cause any trouble.
🚀 cipres [OP] · May 14 at 19:35:
@roughnecks You enabled ratelimit_block_peer ? Monitor the blocklist file from time to time ...
😎 flipperzero · May 15 at 21:32:
@cipres updated now, and all resolution steps set in place. apologies for the delay in resonse, but you and @roughnecks should be aware in order to be best prepared also btw
I've been busy with what seems to be an entire opsec nightmare atm.
— ssh-keysign-pwn (CVE-2026-46333): Patched kernels available in testing
— CVE-2026-46333 Detail
Make sure to best fortify your systems, comrades. Peace to you, god bless.
🚀 cipres [OP] · May 15 at 22:09:
@flipperzero That's a nasty exploit, a nightmare indeed.
🚀 cipres [OP] · May 15 at 22:14:
The chage_pwn exploit can read /etc/shadow it seems. Once you have that you can crack the root user's password. Pretty bad.
🚀 cipres [OP] · May 16 at 10:47:
@roughnecks Sent you a message about your vpn's ip being flagged.
😎 flipperzero · May 17 at 20:51:
The hashnix misfin server is now at the latest version 1.3.4
🚀 Ashnar · May 21 at 16:52:
Sorry to jump in, but that ip blocklist.... Might it be stopping me tunneling to my LAN? I've had no way of connecting to hashnix with my domestic connection, upshot of that is I can't do the reserve tunnels either which is something I was poking around with! I sent email.
😎 flipperzero · May 21 at 20:05:
@Ashnar thanks for dropping a line. Just to make sure I got you and regarding what user + matter that's in discussion right on my mind - for ssh tunnel, yes?
I'll let you know that any word I got of that becoming an issue was first on the 16th, I barely added this latest version on that date the 17th that I added the update. However, I added the original 1.3.0 update with the introduction of blocklist on the 14th. Please let me know when you first started observing your connecction phenomena?
🚀 Ashnar · 22 hours ago:
Thank you for taking the time to reply, I've emailed you at hashnix.club.
🚀 cipres [OP] · 21 hours ago:
@Ashnar By "connecting to hashnix" you mean connecting to the misfin service, or the hashnix.club gemini capsule ?
The ip blocklist is only active on the misfin instance. You would only be added to the misfin blocklist if you make a ton of requests on the misfin server.
😎 flipperzero · 20 hours ago:
@cipres seems so, and then some, if you'd like the fwd I'm sending system users of hash
@Ashnar yeeaah... I got your new communique... thats kind of a lot.. lol, uhh..
So, wait for more, but before anything - **ABSOLUTELY** strip away the 2 components that respectively (A) continue to ping for a connection to your -destination- end instead of your -local- established point (because what?!?!?!?!?!) aand (B) continuously RETRIES connections -without- checking for -previously active or hanging sessions- in such constant, quick intervals ** immediately **
Apologies for the reaction and immediate instruction, will be in better spirits once next contact on further, but if you read hashnix web news dated 09-25-25 regarding an event in which was misconstrued as a *ahem* DDOS from our end due to a member with a compiled project in that moment with a -similar- 'feature' you implemented - you'd perhaps appreciate the severity of the matter and better understand the perspective outside looking in :l
Peace to you, will reach out soon.
Just to make sure, for clarity - for real, no hard feelings, but it's just a lot to process, too much happening in the background RE: CVE's + outtages that have been happening for other international users btw (which I now wonder how related this might be) as well as other matters regarding system maintenance, since people use the service regularly and in turn possess an interest in a consistently functional experience.
🚀 cipres [OP] · 20 hours ago:
@flipperzero @Ashnar No problem to connect to hashnix's misfin from here.
🚀 Ashnar · 11 hours ago:
Everything is turned off on my end currently. It won't go back on unless there's a green light and I can work out a good way of achieving what I would like without causing issues. "work in progress".
Thank you for your efforts