notes (@notes@nsfw.social)
Heads up but there is a new release of Sharkey dropping tomorrow that fixes a critical security vulnerability and all admins are encouraged to upgrade ASAP so expect some downtime tomorrow.
#FediAdmin #NSFWadmin #SharkeyAdmin
Show Original Post
notes (@notes@bofh.social)
Well this is frustrating. Running pnpm run build to build the latest version of #Sharkey is causing one of my servers to spontaneously reboot.
Five identical servers. Five identical setups and configurations, but of course the one server that has the most users has been spontaneously rebooting. Even rescaled the server and gave it more resources and it still spontaneously rebooted.
So now I'm downloading and uploading a 560MB tarball (sans the .config directory) across Starlink to push the new version up and hoping and praying that pnpm run migrate doesn't cause the server to reboot as well.
I've genuinely never seen this before where compiling a nodejs project caused a server to spontaneously reboot.
#FediAdmin
Show Original Post
notes (@notes@bofh.social)
#FediAdmin / #MastoAdmin --
I've modified my rate-limiting setup a little more now that I better understand how nginx's rate-limiting module works in practice.
I have moved my limit_req_zone parameters into my nginx.conf within the http { ... } configuration block:
limit_req_status 429;
limit_req_zone $binary_remote_addr zone=inbox:10m rate=1r/s;
limit_req_zone $binary_remote_addr zone=api:10m rate=8r/s;
limit_req_zone $binary_remote_addr zone=notes:10m rate=2r/s;As you can see, I have set up three separate zones, corresponding to three separate API endpoints.
inbox with an enforced rate of 1 req/s. (/inbox)api with an enforced rate of 8 req/s. (/api/)And
notes with an enforced rate of 2 req/s. (/notes/)Now, the
site-available config for my instances now looks like this:map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
proxy_cache_path /tmp/nginx_cache_bofh levels=1:2 keys_zone=cache1:16m max_size=6g inactive=720m use_temp_path=off;
server {
listen 80;
listen [::]:80;
server_name bofh.social;
# For SSL domain validation
root /var/www/html;
location /.well-known/acme-challenge/ { allow all; }
location /.well-known/pki-validation/ { allow all; }
location / { return 301 https://$server_name$request_uri; }
access_log /var/log/nginx/bofh-social-access.log;
error_log /var/log/nginx/bofh-social-error.log;
}
server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name bofh.social;
ssl_session_timeout 1d;
ssl_session_cache shared:ssl_session_cache:10m;
ssl_session_tickets off;
# To use Let's Encrypt certificate
ssl_certificate /etc/letsencrypt/live/bofh.social/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/bofh.social/privkey.pem;
# SSL protocol settings
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
ssl_prefer_server_ciphers off;
ssl_stapling on;
ssl_stapling_verify on;
# Change to your upload limit
client_max_body_size 99m;
# Gzip compression
gzip on;
gzip_proxied any;
gzip_comp_level 6;
gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript application/activity+json application/atom+xml;
# Proxy to Node
location / {
proxy_pass http://REDACTED:3000;
proxy_set_header Host $host;
proxy_http_version 1.1;
proxy_redirect off;
# If it's behind another reverse proxy or CDN, remove the following.
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
# For WebSocket
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
# Cache settings
proxy_cache cache1;
proxy_cache_lock on;
proxy_cache_use_stale updating;
add_header X-Cache $upstream_cache_status;
}
location /inbox {
limit_req zone=inbox nodelay;
proxy_pass http://REDACTED:3000;
proxy_set_header Host $host;
proxy_http_version 1.1;
proxy_redirect off;
# If it's behind another reverse proxy or CDN, remove the following.
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
}
location /notes {
limit_req zone=notes;
proxy_pass http://REDACTED:3000;
proxy_set_header Host $host;
proxy_http_version 1.1;
proxy_redirect off;
# If it's behind another reverse proxy or CDN, remove the following.
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
}
location /api {
limit_req zone=api;
proxy_pass http://REDACTED:3000;
proxy_set_header Host $host;
proxy_http_version 1.1;
proxy_redirect off;
# If it's behind another reverse proxy or CDN, remove the following.
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
}
access_log /var/log/nginx/bofh-social-access.log;
error_log /var/log/nginx/bofh-social-error.log;
}It's high time Mastodon's abusive behavior was tempered and rate-limited so it doesn't negatively affect other servers.
This is currently deployed to all five of the #Sharkey instances we run.
Show Original Post
notes (@notes@bofh.social)
Why should I expect #Mastodon would actually be a good network citizen and abide by standard HTTP status codes?
I'm rate-limiting the FUCK out of Mastodon and responding to the absolute insane amount of traffic I'm getting hammered with right now with an HTTP/429 status code, and what is Mastodon doing?
Ignoring the fuck out of it.
The client (in this case the remote Mastodon server) is supposed to immediately BACK OFF it's requests when it receives a 429 status code and what is Mastodon doing instead?
INCREASING THE RATE OF REQUESTS TO MY SERVER.
Jesus goddamn motherfucking wank-stains!
I swear to bacon that Mastodon is the biggest goddamn piece of shit software that has ever been created.
#FediAdmin #MastoAdmin #Administrivia
Show Original Post
notes (@notes@bofh.social)
Attention #FediAdmins (this includes #Mastodon, #Sharkey, #Pleroma, etc.) -
To avoid getting #DDOS'd by Mastodon's flood of API requests if a user suddenly decides to delete 30k of their posts on their account, you can take advantage of the built-in #nginx rate-limiting. I've been experimenting with it all day today.
The following is my nginx config for one of my instances, so feel free to modify for your own needs!
# For WebSocket
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
proxy_cache_path /tmp/nginx_cache_bofh levels=1:2 keys_zone=cache1:16m max_size=6g inactive=720m use_temp_path=off;
limit_req_zone $binary_remote_addr zone=post:10m rate=1r/s;
server {
listen 80;
listen [::]:80;
server_name bofh.social;
# For SSL domain validation
root /var/www/html;
location /.well-known/acme-challenge/ { allow all; }
location /.well-known/pki-validation/ { allow all; }
location / { return 301 https://$server_name$request_uri; }
access_log /var/log/nginx/bofh-social-access.log;
error_log /var/log/nginx/bofh-social-error.log;
}
server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name bofh.social;
ssl_session_timeout 1d;
ssl_session_cache shared:ssl_session_cache:10m;
ssl_session_tickets off;
# To use Let's Encrypt certificate
ssl_certificate /etc/letsencrypt/live/bofh.social/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/bofh.social/privkey.pem;
# SSL protocol settings
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
ssl_prefer_server_ciphers off;
ssl_stapling on;
ssl_stapling_verify on;
# Change to your upload limit
client_max_body_size 99m;
# Gzip compression
gzip on;
gzip_proxied any;
gzip_comp_level 6;
gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript application/activity+json application/atom+xml;
# Proxy to Node
location / {
proxy_pass http://REDACTED:3000;
proxy_set_header Host $host;
proxy_http_version 1.1;
proxy_redirect off;
# If it's behind another reverse proxy or CDN, remove the following.
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
# For WebSocket
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
# Cache settings
proxy_cache cache1;
proxy_cache_lock on;
proxy_cache_use_stale updating;
add_header X-Cache $upstream_cache_status;
}
location /inbox {
limit_req zone=post;
proxy_pass http://REDACTED:3000;
proxy_set_header Host $host;
proxy_http_version 1.1;
proxy_redirect off;
# If it's behind another reverse proxy or CDN, remove the following.
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
}
location /api/ {
limit_req zone=post;
proxy_pass http://REDACTED:3000;
proxy_set_header Host $host;
proxy_http_version 1.1;
proxy_redirect off;
# If it's behind another reverse proxy or CDN, remove the following.
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
}
access_log /var/log/nginx/bofh-social-access.log;
error_log /var/log/nginx/bofh-social-error.log;
}Yes, I'm limiting requests to 1 req/s from the same IP. Which is still 3600 reqs/ hour.
#FediAdmin #MastoAdmin #Administrivia
Show Original Post
notes (@notes@bofh.social)
This is a problem, and another shining example at just how inefficient #Mastodon is.
We got DDoS'd because ... "a user with 8k followers and 33k posts, so if they mass-delete posts it will generate a whole lot of activity... "
Da fuq?
6000 requests an hour is not an acceptable rate. 47,000 MORE requests that we received after we blocked the source IP in our WAF is not an acceptable amount of traffic either. #Mastodon needs to seriously cut this back. We're not even a SMALL instance, hosted on low-end infrastructure.
Un-fucking-believable.
Going to ask my co-admin to submit a feature request to @Sharkey@shonk.social to request to rate limit incoming requests from servers.
Fucking bullshit.
#MastoAdmin #FediAdmin #DDOS #Administrivia
Show Original Post
vantablack (@vantablack@cyberpunk.lol)
ANARCHY PRIDE EMOJI PACK RELEASE!!!
our wonderful cyberpunk.lol moderator @rachaelspooky has put together the coolest fuckin emoji pack ever!!! chock full of anarchist symbols mixed with pride flags!!!!!! there's NINETY-SIX emojis in it
free to download here https://rachael.cafe/anarchy_pride.zip
consider donating to faer venmo! link and more info at https://cyberpunk.lol/@rachaelspooky/113500146952543645
#anarchy #anarchism #emoji #pride #FediAdmin
Show Original Post
vantablack (@vantablack@cyberpunk.lol)
ANARCHY PRIDE EMOJI PACK RELEASE!!!
our wonderful cyberpunk.lol moderator @rachaelspooky has put together the coolest fuckin emoji pack ever!!! chock full of anarchist symbols mixed with pride flags!!!!!! there's NINETY-SIX emojis in it
free to download here https://rachael.cafe/anarchy_pride.zip
consider donating to faer venmo! link and more info at https://cyberpunk.lol/@rachaelspooky/113500146952543645
#anarchy #anarchism #emoji #FediAdmin
Show Original Post
nil (@nil@fosstodon.org)
Related, if anyone has tips being a #FediAdmin let me know :) https://pet.tax/p/admin/763854796318816662
Show Original Post
notes (@notes@bofh.social)
Reason #791 for deploying a #Sharkey instance over a #Mastodon instance to the #fediverse:
User quotas.
I can set per-user quotas and limits for how much #S3 storage my users consume thereby allowing me to grow and manage my cloud storage costs organically.
#MastoAdmin #FediAdmin #ServerAdmin #InstanceAdmin
Show Original Post
notes (@notes@nsfw.lgbt)
Attention #NSFW users on .lgbt and .social...
As .lgbt and .social grows we have more and more users with varying tolerance levels for certain fetishes. Please remember to properly hash-tag your posts if they contain fetishes. We have always had Rule #9 since the launch of nsfw.lgbt and nsfw.social and while we haven't really been enforcing Rule 9 in the past, we are going to start enforcing Rule 9 going forward.
To recap, Rule 9 states:
Extreme content, including some fetishes, must be hash-tagged (#tag) so users have the agency to filter out and mute/ block the content from their timeline.If we come across a post that hasn't been tagged we will probably delete it. If extreme content is continued to be posted that isn't tagged we may limit the account.
Thanks for your understanding as we try to make nsfw.lgbt and nsfw.social a fun experience for everyone.
#FediAdmin #MastoAdmin #NSFWadmin
Show Original Post
notes (@notes@nsfw.social)
Attention #NSFW users on .social and .lgbt...
As .social and .lgbt grows we have more and more users with varying tolerance levels for certain fetishes. Please remember to properly hash-tag your posts if they contain fetishes. We have always had Rule #9 since the launch of nsfw.social and nsfw.lgbt and while we haven't really been enforcing Rule 9 in the past, we are going to start enforcing Rule 9 going forward.
To recap, Rule 9 states:
Extreme content, including some fetishes, must be hash-tagged (#tag) so users have the agency to filter out and mute/ block the content from their timeline.If we come across a post that hasn't been tagged we will probably delete it. If extreme content is continued to be posted that isn't tagged we may limit the account.
Thanks for your understanding as we try to make nsfw.social and nsfw.lgbt a fun experience for everyone.
#FediAdmin #MastoAdmin #NSFWadmin
Show Original Post