pcouy

publication croisée depuis : https://lemmy.pierre-couy.fr/post/653426

This is a guide I wrote for Immich's documentation. It features some Immich specific parts, but should be quite easy to adapt to other use cases.

It is also possible (and not technically hard) to self-host a protomaps release, but this would require 100GB+ of disk space (which I can't spare right now). The main advantages of this guide over hosting a full tile server are :

• it's a single nginx config file to deploy
• it saves you some storage space since you're only hosting tiles you've previously viewed. You can also tweak the maximum cache size to your needs
• it is easy to configure a trade-off between map freshness and privacy by tweaking the cache expiration delay

If you try to follow it, please send me some feedback on the content and the wording, so I can improve it

This is a guide I wrote for Immich's documentation. It features some Immich specific parts, but should be quite easy to adapt to other use cases.

It is also possible (and not technically hard) to self-host a protomaps release, but this would require 100GB+ of disk space (which I can't spare right now). The main advantages of this guide over hosting a full tile server are :

• it's a single nginx config file to deploy
• it saves you some storage space since you're only hosting tiles you've previously viewed. You can also tweak the maximum cache size to your needs
• it is easy to configure a trade-off between map freshness and privacy by tweaking the cache expiration delay

If you try to follow it, please send me some feedback on the content and the wording, so I can improve it

publication croisée depuis : https://lemmy.pierre-couy.fr/post/584644

While monitoring my Pi-Hole logs today, I noticed a bunch of queries for XXXXXX.bodis.com, where XXXXXX are numbers. I saw a few variations for the numbers, each one being queried several times.

Digging further, I found out these queries were caused by CNAME records on domains that look like they used to point to Lemmy/Kbin instances.

From what I understand, domain owners can register a CNAME record to XXXXXX.bodis.com and earn some money from the traffic it receives. I guess that each number variation is a domain owner ID in Bodis' database. I saw between 5 to 10 different number variations, each one being pointed to by a bunch of old Lemmy domains.

This probably means that among actors who snatch expired domains, several of them have taken a specific interest with expired domains of old Lemmy instances. Another hypothesis is that there were a lot of domains registered for hosting Lemmy during the Reddit API debacle (about 1 year ago), which started expiring recently.

Are there any other instance admins who noticed the same thing ? Is any of my two hypothesis more plausible than the other ? Should we worry about this trend ?

Anyway, I hope this at least serves as a reminder to not let our domains expire ;)

While monitoring my Pi-Hole logs today, I noticed a bunch of queries for XXXXXX.bodis.com, where XXXXXX are numbers. I saw a few variations for the numbers, each one being queried several times.

Digging further, I found out these queries were caused by CNAME records on domains that look like they used to point to Lemmy/Kbin instances.

From what I understand, domain owners can register a CNAME record to XXXXXX.bodis.com and earn some money from the traffic it receives. I guess that each number variation is a domain owner ID in Bodis' database. I saw between 5 to 10 different number variations, each one being pointed to by a bunch of old Lemmy domains.

This probably means that among actors who snatch expired domains, several of them have taken a specific interest with expired domains of old Lemmy instances. Another hypothesis is that there were a lot of domains registered for hosting Lemmy during the Reddit API debacle (about 1 year ago), which started expiring recently.

Are there any other instance admins who noticed the same thing ? Is any of my two hypothesis more plausible than the other ? Should we worry about this trend ?

Anyway, I hope this at least serves as a reminder to not let our domains expire ;)

Cross-posted from : https://lemmy.pierre-couy.fr/post/581642

Context : Immich default map tile provider (which gets sent a bunch of PII every time you use the map feature) is a company that I see no reason to trust. This is a follow-up to this post, with the ~~permanent~~ temporary fix I came up with. I will also summarize the general opinion from the comments, as well as some interesting piece of knowledge that commenters shared.

Hacky fix

This will use Nginx proxy module to build a caching proxy in front of Open Street Map's tileserver and to serve a custom style.json for the maps.

This works well for me, since I already proxy all my services behind a single Nginx instance. It is probably possible to achieve similar results with other reverse proxies, but this would obviously need to be adapted.

Caching proxy

Inside Nginx's http config block (usually in /etc/nginx/nginx.conf), create a cache zone (a directory that will hold cached responses from OSM) :

http {
     # You should not need to edit existing lines in the http block, only add the line below
    proxy_cache_path /var/cache/nginx/osm levels=1:2 keys_zone=osm:100m max_size=5g inactive=180d;
}

You may need to manually create the /var/cache/nginx/osm directory and set its owner to Nginx's user (typically www-data on Debian based distros).

Customize the max_size parameter to change the maximum amount of cached data you want to store on your server. The inactive parameter will cause Nginx to discard cached data that's not been accessed in this duration (180d ~ 6months).

Then, inside the server block that serves your Immich instance, create a new location block :

server {
    listen 443 ssl;
    server_name immich.your-domain.tld;

    # You should not need to change your existing config, only add the location block below

    location /map_proxy/ {
        proxy_pass https://tile.openstreetmap.org/;
        proxy_cache osm;
        proxy_cache_valid 180d;
        proxy_ignore_headers Cache-Control Expires;
        proxy_ssl_server_name on;
        proxy_ssl_name tile.openstreetmap.org;
        proxy_set_header Host tile.openstreetmap.org;
        proxy_set_header User-Agent "Nginx Caching Tile Proxy for self-hosters";
        proxy_set_header Cookie "";
        proxy_set_header Referer "";
    }
}

Reload Nginx (sudo systemctl reload nginx). Confirm this works by visiting https://immich.your-domain.tld/map_proxy/0/0/0.png, which should now return a world map PNG (the one from https://tile.openstreetmap.org/0/0/0.png )

This config ignores cache control headers from OSM and sets its own cache validity duration (proxy_cache_valid parameter). After the specified duration, the proxy will re-fetch the tiles. 6 months seem reasonable to me for the use case, and it can probably be set to a few years without it causing issues.

Besides being lighter on OSM's servers, the caching proxy will improve privacy by only requesting tiles from upstream when loaded for the first time. This config also strips cookies and referrer before forwarding the queries to OSM, as well as set a user agent for the proxy following OSM foundation's guidelines (according to these guidelines, you should add a contact information to this user agent)

This can probably be made to work on a different domain than the one serving your Immich instance, but this probably requires to add the appropriate headers for CORS.

Custom style.json

I came up with the following mapstyle :

{
  "version": 8,
  "name": "Immich Map",
  "sources": {
    "immich-map": {
      "type": "raster",
      "tileSize": 256,
      "tiles": [
        "https://immich.your-domain.tld/map_proxy/{z}/{x}/{y}.png"
      ]
    }
  },
  "sprite": "https://maputnik.github.io/osm-liberty/sprites/osm-liberty",
  "glyphs": "https://fonts.openmaptiles.org/{fontstack}/{range}.pbf",
  "layers": [
    {
      "id": "raster-tiles",
      "type": "raster",
      "source": "immich-map",
      "minzoom": 0,
      "maxzoom": 22
    }
  ],
  "id": "immich-map-dark"
}

Replace immich.your-domain.tld with your actual Immich domain, and remember the absolute path you save this at.

One last update to nginx's config

Since Immich currently does not provide a way to manually edit style.json, we need to serve it from http(s). Add one more location block below the previous one :

location /map_style.json {
    alias /srv/immich/mapstyle.json;
}

Replace the alias parameter with the location where you saved the json mapstyle. After reloading nginx, your json style will be available at https://immich.your-domain.tld/map_style.json

Configure Immich to use this

For this last part, follow steps 8, 9, 10 from this guide (use the link to map_style.json for both light and dark themes). After clearing the browser or app's cache, the map should now be loaded from your caching proxy. You can confirm this by tailing Nginx's logs while you zoom and move around the map in Immich

Summary of comments from previous post

Self-hosting a tile server is not realistic in most cases

People who have previously worked with maps seem to confirm that there are no tile server solution lightweight enough to be self hosted by hobbyists. There is maybe some hope with generating tiles on demand, but someone with deep knowledge of the file formats involved in the process should confirm this.

Some interesting links were shared, which seem to confirm this is not realistically self-hostable with the available software :

• OSM Foundation's policy on using their tile server
• Switch2OSM
• https://wiki.openstreetmap.org/wiki/Sequentially-generate-planet-mbtiles

General sentiment about this issue

In all this part, I want to emphasize that while there seems to be a consensus, this is only based on the few comments from the previous post and may be biased by the fact that we're discussing it on a non-mainstream platform. If you disagree with anything below, please comment this post and explain your point of view.

• Nobody declared that they had noticed the requests to a third-party server before
• A non-negligible fraction of Immich users are interested in the privacy benefits over other solutions such as Google photos. These users do not like their self-hosted services to send requests to third-party servers without warning them first
• The fix should consist of the following :
  • Clearly document the implications of enabling the map, and any feature that sends requests to third parties
  • Disable by default features that send requests to third parties (especially if it contains any form of geolocated data)
  • Provide a way to easily change the tile provider. A select menu with a few pre-configured style.json would be nice, along with a way to manually edit style.json (or at least some of its fields) directly from the Immich config page

Context : Immich default map tile provider (which gets sent a bunch of PII every time you use the map feature) is a company that I see no reason to trust. This is a follow-up to this post, with the ~~permanent~~ temporary fix I came up with. I will also summarize the general opinion from the comments, as well as some interesting piece of knowledge that commenters shared.

Hacky fix

This will use Nginx proxy module to build a caching proxy in front of Open Street Map's tileserver and to serve a custom style.json for the maps.

This works well for me, since I already proxy all my services behind a single Nginx instance. It is probably possible to achieve similar results with other reverse proxies, but this would obviously need to be adapted.

Caching proxy

Inside Nginx's http config block (usually in /etc/nginx/nginx.conf), create a cache zone (a directory that will hold cached responses from OSM) :

http {
     # You should not need to edit existing lines in the http block, only add the line below
    proxy_cache_path /var/cache/nginx/osm levels=1:2 keys_zone=osm:100m max_size=5g inactive=180d;
}

You may need to manually create the /var/cache/nginx/osm directory and set its owner to Nginx's user (typically www-data on Debian based distros).

Customize the max_size parameter to change the maximum amount of cached data you want to store on your server. The inactive parameter will cause Nginx to discard cached data that's not been accessed in this duration (180d ~ 6months).

Then, inside the server block that serves your Immich instance, create a new location block :

server {
    listen 443 ssl;
    server_name immich.your-domain.tld;

    # You should not need to change your existing config, only add the location block below

    location /map_proxy/ {
        proxy_pass https://tile.openstreetmap.org/;
        proxy_cache osm;
        proxy_cache_valid 180d;
        proxy_ignore_headers Cache-Control Expires;
        proxy_ssl_server_name on;
        proxy_ssl_name tile.openstreetmap.org;
        proxy_set_header Host tile.openstreetmap.org;
        proxy_set_header User-Agent "Nginx Caching Tile Proxy for self-hosters";
        proxy_set_header Cookie "";
        proxy_set_header Referer "";
    }
}

Reload Nginx (sudo systemctl reload nginx). Confirm this works by visiting https://immich.your-domain.tld/map_proxy/0/0/0.png, which should now return a world map PNG (the one from https://tile.openstreetmap.org/0/0/0.png )

This config ignores cache control headers from OSM and sets its own cache validity duration (proxy_cache_valid parameter). After the specified duration, the proxy will re-fetch the tiles. 6 months seem reasonable to me for the use case, and it can probably be set to a few years without it causing issues.

Besides being lighter on OSM's servers, the caching proxy will improve privacy by only requesting tiles from upstream when loaded for the first time. This config also strips cookies and referrer before forwarding the queries to OSM, as well as set a user agent for the proxy following OSM foundation's guidelines (according to these guidelines, you should add a contact information to this user agent)

This can probably be made to work on a different domain than the one serving your Immich instance, but this probably requires to add the appropriate headers for CORS.

Custom style.json

I came up with the following mapstyle :

{
  "version": 8,
  "name": "Immich Map",
  "sources": {
    "immich-map": {
      "type": "raster",
      "tileSize": 256,
      "tiles": [
        "https://immich.your-domain.tld/map_proxy/{z}/{x}/{y}.png"
      ]
    }
  },
  "sprite": "https://maputnik.github.io/osm-liberty/sprites/osm-liberty",
  "glyphs": "https://fonts.openmaptiles.org/{fontstack}/{range}.pbf",
  "layers": [
    {
      "id": "raster-tiles",
      "type": "raster",
      "source": "immich-map",
      "minzoom": 0,
      "maxzoom": 22
    }
  ],
  "id": "immich-map-dark"
}

Replace immich.your-domain.tld with your actual Immich domain, and remember the absolute path you save this at.

One last update to nginx's config

Since Immich currently does not provide a way to manually edit style.json, we need to serve it from http(s). Add one more location block below the previous one :

location /map_style.json {
    alias /srv/immich/mapstyle.json;
}

Replace the alias parameter with the location where you saved the json mapstyle. After reloading nginx, your json style will be available at https://immich.your-domain.tld/map_style.json

Configure Immich to use this

For this last part, follow steps 8, 9, 10 from this guide (use the link to map_style.json for both light and dark themes). After clearing the browser or app's cache, the map should now be loaded from your caching proxy. You can confirm this by tailing Nginx's logs while you zoom and move around the map in Immich

Summary of comments from previous post

Self-hosting a tile server is not realistic in most cases

People who have previously worked with maps seem to confirm that there are no tile server solution lightweight enough to be self hosted by hobbyists. There is maybe some hope with generating tiles on demand, but someone with deep knowledge of the file formats involved in the process should confirm this.

Some interesting links were shared, which seem to confirm this is not realistically self-hostable with the available software :

• OSM Foundation's policy on using their tile server
• Switch2OSM
• https://wiki.openstreetmap.org/wiki/Sequentially-generate-planet-mbtiles

General sentiment about this issue

In all this part, I want to emphasize that while there seems to be a consensus, this is only based on the few comments from the previous post and may be biased by the fact that we're discussing it on a non-mainstream platform. If you disagree with anything below, please comment this post and explain your point of view.

• Nobody declared that they had noticed the requests to a third-party server before
• A non-negligible fraction of Immich users are interested in the privacy benefits over other solutions such as Google photos. These users do not like their self-hosted services to send requests to third-party servers without warning them first
• The fix should consist of the following :
  • Clearly document the implications of enabling the map, and any feature that sends requests to third parties
  • Disable by default features that send requests to third parties (especially if it contains any form of geolocated data)
  • Provide a way to easily change the tile provider. A select menu with a few pre-configured style.json would be nice, along with a way to manually edit style.json (or at least some of its fields) directly from the Immich config page

Update : I made a follow-up post containing a Nginx-based solution to cache map tiles from OSM and limit the amount of PII you send

While monitoring the logs in Rethink DNS (awesome app BTW) today, I noticed the Immich app making requests to api-l.cofractal.com.

After reaching out on Immich's discord, the devs explained to me that it is used as a tile provider for the map feature. I can confirm it is not realistic to self-host a tile provider without heavily tuning down the level of details on the map (which would still require a lot of disk space and CPU time). I understand the need for a third-party service to provide the map tiles, but I'm concerned by this one.

Visiting cofractal.com only tells us that they're selling APIs. I did not find any details about the company, not even the country they're registered in. The website is also missing informations about what they are logging or not. Everything else seems gated behind a login page, but they "are not currently accepting new customers". The whois for the domain says they're in California. Digging a bit more, I find AS26073 which apparently is the same company.

This bothers me, because Cofractal gets sent every location you viewed (and the zoom level) on Immich's map, along with your client's IP address and a "Referrer" header pointing to your Immich instance. This sounds like a lot of PII to me. It's also behind cloudflare which gets to see the same stuff.

When asked about it, one dev (thanks to them for almost instantly replying to every concern/question I threw at them) explained that they personally know the people behind Cofractal. According to this Immich dev, Cofractal provides free access to its paid service to Immich's user base as a way to support the project, with the side benefit of load testing their platform.

This explanations seems plausible and reasonable to me. However, I do not personally know the people behind Cofractal, and by default, I do not trust for-profit companies to act in an altruistic way. Here's a summary of everything that makes me uneasy about this company :

• it does not say anything about the kind of data they are logging or not
• it requires digging through whois records to find the most basic info about the company
• it freely provides access to its normally paid service (for the whole Immich user base), but it does not communicate about it (or it is really hard to find)
• it does not communicate about anything : searching for its name only returns its home page and websites with informations on Autonomous Systems
• it is "not currently accepting new [paying] customers" while providing the service for free to a quite large user base (Immich v1.109.2 got 170k downloads in 5 days, v1.108.0 got 438k downloads in 13 days )
• It is not mentioned anywhere in the whole immich.app website (searching for site:immich.app "cofractal" gave me no result). Not even a "Thank You" or "Sponsor" note on the homepage for the free API
• (it is behind cloudflare)

The dev I talked to encouraged me to create a feature request, and seemed favorable to adding a switch for disabling maps client side. It is already possible to disable it server-wide, and the "URL to a style.json map theme" option seems to provide a way to customize the tile provider. Which leads to this post : I'm trying to collect feedback on this before creating the feature request.

• It should be made prominently clear to server admins that leaving maps enabled causes clients to send requests to a third party-server and give details about what is sent (viewed locations, zoom level, IP address, Immich instance URL). The Post Install Steps in the docs and a paragraph above the switch on the config page seem like good places to me. Are there other/more appropriate place for such a warning ?
• The "URL to a style.json map theme" option should probably be renamed to make it clearer that it allows changing tile providers. Or better yet, it could be reworked to make it easier to choose which third-party you decide to trust
• What do you think about the idea of providing instance admins with a list of choices for tile providers ? Maybe with a short pros/cons list in the docs for each provider. I'd be fine with using a more reputable provider with the extra step of configuring my own API key (which would probably require proxying requests to the tile provider to not share the API key with all clients)
• Should the Immich server proxy requests to the tile provider in any case ? Since the tile provider has access to the Referrer and Origin headers (which is probably required for CORS), they are currently able to link user IP addresses with Immich instances. Proxying requests with the Immich server should prevent that.
• I would go as far as making maps disabled by default for new installs. I understand that "disabling by default would be a significant downgrade for a majority of users", but I feel like there's a strong overlap between the self-hosting and privacy communities. So we should at least have some debate about it

I've also been told that I'm the first one to raise concerns about this, which leads to one more question : Did nobody complain because nobody noticed ? Or are my concerns unjustified ?

Pour référence : https://etudiant.lefigaro.fr/article/bac-philo-2023-qui-de-raphael-enthoven-ou-chatgpt-redige-la-meilleure-copie_a694c010-0a09-11ee-bd34-f2c2eadd1748/

(désolé pour le sponsor de la vidéo qui apparaît dans l'aperçu généré par lemmy)

cross-posted from : https://lemmy.pierre-couy.fr/post/350920

I am trying to come-up with a reusable template to quickly start new projects using my prefered tools and frameworks, and I'm happy with what I got. However, using Docker is quite new for me and I've probably done some weird or unconventional stuff in my docker-compose.yml or my Dockerfiles. I'd love to learn from people with more experience with Docker, so feel free to tell me everything that is wrong with my setup.

I'm more confident about the stuff I did with Python/Django and Nuxt, but all criticism is welcome. This also applies to the readme : I'd like to provide detailed instructions about working with this project template, so please report anything that is unclear or missing.

Thank you to anyone who takes the time to check it out and help me make this useful to as many people as possible.

publication croisée depuis : https://lemmy.pierre-couy.fr/post/350920

I am trying to come-up with a reusable template to quickly start new projects using my prefered tools and frameworks, and I'm happy with what I got. However, using Docker is quite new for me and I've probably done some weird or unconventional stuff in my docker-compose.yml or my Dockerfiles. I'd love to learn from people with more experience with Docker, so feel free to tell me everything that is wrong with my setup.

I'm more confident about the stuff I did with Python/Django and Nuxt, but all criticism is welcome. This also applies to the readme : I'd like to provide detailed instructions about working with this project template, so please report anything that is unclear or missing.

Thank you to anyone who takes the time to check it out and help me make this useful to as many people as possible.

I am trying to come-up with a reusable template to quickly start new projects using my prefered tools and frameworks, and I'm happy with what I got. However, using Docker is quite new for me and I've probably done some weird or unconventional stuff in my docker-compose.yml or my Dockerfiles. I'd love to learn from people with more experience with Docker, so feel free to tell me everything that is wrong with my setup.

I'm more confident about the stuff I did with Python/Django and Nuxt, but all criticism is welcome. This also applies to the readme : I'd like to provide detailed instructions about working with this project template, so please report anything that is unclear or missing.

Thank you to anyone who takes the time to check it out and help me make this useful to as many people as possible.