Social 📸 Photography

ijMy 3-node <a href="https://nerdculture.de/tags/Ceph" class="mention hashtag" rel="nofollow noopener" target="_blank">#Ceph</a> cluster is showing nice performance when running "bin/tootctl search deploy --only=accounts" during the Mastodon upgrade...

RachelPhysical test fit is good! (The 3.5in drives are not populated yet) <a href="https://transitory.social/tags/Homelab" rel="nofollow noopener" target="_blank">#Homelab</a> <a href="https://transitory.social/tags/Ceph" rel="nofollow noopener" target="_blank">#Ceph</a> <a href="https://transitory.social/tags/Kubernetes" rel="nofollow noopener" target="_blank">#Kubernetes</a> <a href="https://transitory.social/tags/Minilab" rel="nofollow noopener" target="_blank">#Minilab</a>

MichaelWell of course. My Ceph cluster would produce a scrub inconsistency error while I'm 400 km away from the Homelab. Let's see what this is about.<a href="https://social.mei-home.net/tags/HomeLab" class="mention hashtag" rel="nofollow noopener" target="_blank">#HomeLab</a> <a href="https://social.mei-home.net/tags/Ceph" class="mention hashtag" rel="nofollow noopener" target="_blank">#Ceph</a>

Marianne SpillerUpgrade des Prod-Clusters von <a href="https://konfigurationsmanufaktur.de/tags/Ceph" class="mention hashtag" rel="nofollow noopener" target="_blank">#Ceph</a> Reef zu Squid ✅ Und das sogar noch **vor** dem 1. August! 🎉 Lief unproblematisch.<a href="https://konfigurationsmanufaktur.de/tags/Proxmox" class="mention hashtag" rel="nofollow noopener" target="_blank">#Proxmox</a> <a href="https://konfigurationsmanufaktur.de/tags/Ceph" class="mention hashtag" rel="nofollow noopener" target="_blank">#Ceph</a>

OpenStackAre you running <a href="https://social.openinfra.dev/tags/OpenStack" class="mention hashtag" rel="nofollow noopener" target="_blank">#OpenStack</a> and <a href="https://social.openinfra.dev/tags/Ceph" class="mention hashtag" rel="nofollow noopener" target="_blank">#Ceph</a>? Share your story at <a href="https://social.openinfra.dev/tags/Cephalocon" class="mention hashtag" rel="nofollow noopener" target="_blank">#Cephalocon</a> In Vancouver on October 28! The CFP is currently open and closes this Sunday, July 13 at 11:59pm PT.<a href="https://ceph.io/en/community/events/2025/cephalocon-2025/" rel="nofollow noopener" translate="no" target="_blank">https://ceph.io/en/community/events/2025/cephalocon-2025/</a>

Stuart Longland (VK4MSL)Seems some of my disks have seen a few writes! `smartctl -a ${DEVICE}` stats for the <a href="https://mastodon.longlandclan.id.au/tags/Ceph" class="mention hashtag" rel="nofollow noopener" target="_blank">#Ceph</a> and virtual machine host cluster here.The 2TB SSDs are OSDs in Ceph, the others are OS disks or local VM cache disks. Ceph is moaning about the disks in "oxygen" and "fluorine".Numbers here are Total Drive Writes in GB.``` $ for h in *; do for d in ${h}/tmp/*.txt; do lbas=$( grep -F Total_LBAs_Written ${d} | cut -c 88- ); if [ -n "${lbas}" ]; then dev=$( basename "${d}" .txt ); dev=/dev/${dev##*-}; sect=$( grep '^Sector Size' ${d} | cut -c 18-21 ); printf "%-20s %8s %6d %s\n" ${h} ${dev} $(( ( ${lbas} * ${sect} ) / 1024 / 1024 / 1024 )) "$( grep '^Device Model' ${d} )"; fi; done; done beryllium.chost.lan /dev/sda 0 Device Model: WD Green 2.5 240GB boron.chost.lan /dev/sdb 76370 Device Model: Samsung SSD 870 EVO 2TB carbon.chost.lan /dev/sda 113892 Device Model: Samsung SSD 870 EVO 2TB carbon.chost.lan /dev/sdb 157993 Device Model: Samsung SSD 860 EVO 2TB fluorine.chost.lan /dev/sda 111939 Device Model: Samsung SSD 870 QVO 2TB helium.chost.lan /dev/sda 100476 Device Model: Samsung SSD 870 QVO 2TB hydrogen.chost.lan /dev/sda 184564 Device Model: Samsung SSD 860 EVO 2TB hydrogen.chost.lan /dev/sdb 58602 Device Model: Samsung SSD 870 EVO 2TB lithium.chost.lan /dev/sda 0 Device Model: WD Green 2.5 240GB magnesium.chost.lan /dev/sda 0 Device Model: WD Green 2.5 240GB neon.chost.lan /dev/sdb 146926 Device Model: Samsung SSD 860 EVO 2TB nitrogen.chost.lan /dev/sda 99473 Device Model: Samsung SSD 870 EVO 2TB oxygen.chost.lan /dev/sdb 108748 Device Model: Samsung SSD 870 QVO 2TB sodium.chost.lan /dev/sda 0 Device Model: WD Green 2.5 240GB ```<a href="https://mastodon.longlandclan.id.au/tags/HomeServer" class="mention hashtag" rel="nofollow noopener" target="_blank">#HomeServer</a>

Stuart Longland (VK4MSL)I'm a glutton for punishment this week… I'm poking the octopus again, moving to Ceph 18.So far, we're doing the monitor daemons. A fun fact, they switched from LevelDB to RocksDB some time back and in Ceph 17, they dropped support for LevelDB.I found this out when updating the first monitor node, it crashed. Refused to start. All I had was a cryptic:ceph_abort_msg("MonitorDBStore: error initializing keyvaluedb back storage")Not very helpful. The docs for Ceph 17 (which I didn't read as I was going from 16→18 direct, which the Ceph 18 docs says you _can_ do), merely state:> LevelDB support has been removed. WITH_LEVELDB is no longer a supported build option. Users should migrate their monitors and OSDs to RocksDB before upgrading to Quincy. (<a href="https://docs.ceph.com/en/latest/releases/quincy/" rel="nofollow noopener" translate="no" target="_blank">https://docs.ceph.com/en/latest/releases/quincy/</a>)How? 0 suggestions as to the procedure.What worked here, was to manually remove the monitor node, rename the /var/lib/ceph/mon/* directory, then re-add the monitor using the manual instructions.<a href="https://docs.ceph.com/en/latest/rados/operations/add-or-rm-mons/" rel="nofollow noopener" translate="no" target="_blank">https://docs.ceph.com/en/latest/rados/operations/add-or-rm-mons/</a><a href="https://mastodon.longlandclan.id.au/tags/Ceph" class="mention hashtag" rel="nofollow noopener" target="_blank">#Ceph</a>

Stuart Longland (VK4MSL)We're mostly done now… just a few placement groups to re-map and backfill.Looking at the steps to go next. I want to get to Debian 12 and Ceph 19.There are no builds of Ceph 19 for Debian 11, which is what I'm on now. Looks like once again, I'll have to use the Debian repository Ceph binary, which is for Ceph 16.2.15. So I need to first comment out the Ceph repositories, and update to Debian 12, *then* I can jump to Ceph 19.Would it kill them to build a couple for more than one Debian release?<a href="https://mastodon.longlandclan.id.au/tags/Ceph" class="mention hashtag" rel="nofollow noopener" target="_blank">#Ceph</a> <a href="https://mastodon.longlandclan.id.au/tags/Debian" class="mention hashtag" rel="nofollow noopener" target="_blank">#Debian</a>

Stuart Longland (VK4MSL)Finally got to the post office to pick up the 6TB HDD for backing up the cluster, and last night took a back-up of all RBD volumes.I'm part way migrated to Ceph v16 now. Monitors are updated, managers are updated, all the OSDs have been re-started. I've flipped the switch for OMAP usage stats migration… and now I re-start each OSD one by one and wait for it to convert and return to online status.<a href="https://mastodon.longlandclan.id.au/tags/Ceph" class="mention hashtag" rel="nofollow noopener" target="_blank">#Ceph</a> <a href="https://mastodon.longlandclan.id.au/tags/HomeServer" class="mention hashtag" rel="nofollow noopener" target="_blank">#HomeServer</a>

Marianne SpillerFriendly reminder, dass <a href="https://konfigurationsmanufaktur.de/tags/Ceph" class="mention hashtag" rel="nofollow noopener" target="_blank">#Ceph</a> Reef (18.x) ab August EOL ist. Das Sommerloch wäre doch ein guter Moment, das Upgrade auf Squid (19.x) in Angriff zu nehmen... 😎

Benoit🚨 <a href="https://mastodon.benoit.jp.net/tags/Hiring" class="mention hashtag" rel="nofollow noopener" target="_blank">#Hiring</a> — Platform Engineer @ Rakuten Japan 🇯🇵 Join my team! I’m leading this Platform Engineering group focused on <a href="https://mastodon.benoit.jp.net/tags/Kubernetes" class="mention hashtag" rel="nofollow noopener" target="_blank">#Kubernetes</a> & Linux systems.💼 Stack: <a href="https://mastodon.benoit.jp.net/tags/Chef" class="mention hashtag" rel="nofollow noopener" target="_blank">#Chef</a> <a href="https://mastodon.benoit.jp.net/tags/Terraform" class="mention hashtag" rel="nofollow noopener" target="_blank">#Terraform</a> <a href="https://mastodon.benoit.jp.net/tags/GCP" class="mention hashtag" rel="nofollow noopener" target="_blank">#GCP</a> <a href="https://mastodon.benoit.jp.net/tags/OnPrem" class="mention hashtag" rel="nofollow noopener" target="_blank">#OnPrem</a> <a href="https://mastodon.benoit.jp.net/tags/RockyLinux" class="mention hashtag" rel="nofollow noopener" target="_blank">#RockyLinux</a> <a href="https://mastodon.benoit.jp.net/tags/Rook" class="mention hashtag" rel="nofollow noopener" target="_blank">#Rook</a> <a href="https://mastodon.benoit.jp.net/tags/Ceph" class="mention hashtag" rel="nofollow noopener" target="_blank">#Ceph</a>📍 Tokyo | 4 days/week in-office (only one day of WFH!)If we know each other (Fediverse or IRL), I can refer you! <a href="https://mastodon.benoit.jp.net/tags/GetFediHired" class="mention hashtag" rel="nofollow noopener" target="_blank">#GetFediHired</a> <a href="https://mastodon.benoit.jp.net/tags/FediTechJobs" class="mention hashtag" rel="nofollow noopener" target="_blank">#FediTechJobs</a>🔗 <a href="https://rakuten.wd1.myworkdayjobs.com/RakutenInc/job/Tokyo-Japan/Platform-Engineer---Merchandising-and-Advertisement-Department--MAD-_1026705" rel="nofollow noopener" translate="no" target="_blank">https://rakuten.wd1.myworkdayjobs.com/RakutenInc/job/Tokyo-Japan/Platform-Engineer---Merchandising-and-Advertisement-Department--MAD-_1026705</a>

Marianne SpillerMal wieder was ins <a href="https://konfigurationsmanufaktur.de/tags/Blog" class="mention hashtag" rel="nofollow noopener" target="_blank">#Blog</a> gepackt: zu <a href="https://konfigurationsmanufaktur.de/tags/Ceph" class="mention hashtag" rel="nofollow noopener" target="_blank">#Ceph</a> und den BLUESTORE_SLOW_OP_ERRORs, die da seit einem der letzten Updates herumschwirren. Spoiler: es ist weniger dramatisch, als ursprünglich angenommen, und man kann damit umgehen 🙂<a href="https://www.unixe.de/ceph-bluestore-slow-op-error/" rel="nofollow noopener" translate="no" target="_blank">https://www.unixe.de/ceph-bluestore-slow-op-error/</a>

JdeBP<a href="https://sfba.social/@4d3fect" class="u-url mention" rel="nofollow noopener" target="_blank">@4d3fect</a> <a href="https://mastodonapp.uk/@bytebro" class="u-url mention" rel="nofollow noopener" target="_blank">@bytebro</a> Ceph, and a lot of DASD.And as I noted elsethread, their scale factors all appear to be powers of 10, so the fucktonnes are apparently metric.<a href="https://epcc.ed.ac.uk/somerville-high-performance-data-intensive-storage-service-astronomy" rel="nofollow noopener" translate="no" target="_blank">https://epcc.ed.ac.uk/somerville-high-performance-data-intensive-storage-service-astronomy</a><a href="https://mastodonapp.uk/tags/VeraRubinObservatorium" class="mention hashtag" rel="nofollow noopener" target="_blank">#VeraRubinObservatorium</a> <a href="https://mastodonapp.uk/tags/LSST" class="mention hashtag" rel="nofollow noopener" target="_blank">#LSST</a> <a href="https://mastodonapp.uk/tags/astronomy" class="mention hashtag" rel="nofollow noopener" target="_blank">#astronomy</a> <a href="https://mastodonapp.uk/tags/Chile" class="mention hashtag" rel="nofollow noopener" target="_blank">#Chile</a> <a href="https://mastodonapp.uk/tags/VeraRubin" class="mention hashtag" rel="nofollow noopener" target="_blank">#VeraRubin</a> <a href="https://mastodonapp.uk/tags/Ceph" class="mention hashtag" rel="nofollow noopener" target="_blank">#Ceph</a> <a href="https://mastodonapp.uk/tags/metric" class="mention hashtag" rel="nofollow noopener" target="_blank">#metric</a>

Heinlein SupportJetzt zum Nachlesen: SLAC-Vortragsfolien sind online!Du warst auf der SLAC 2025 und möchtest nochmal in einen Vortrag eintauchen? Oder du konntest dieses Jahr leider nicht auf unserer Linux-Konferenz dabei sein? Die Folien zu den Vorträgen & Workshops sind ab sofort online und für Dich in den jeweiligen Programmslots zum Download hinterlegt. Schau vorbei und wirf einen Blick in Vorträge zu <a href="https://social.heinlein-support.de/tags/Proxmox" class="mention hashtag" rel="nofollow noopener" target="_blank">#Proxmox</a>, <a href="https://social.heinlein-support.de/tags/Ceph" class="mention hashtag" rel="nofollow noopener" target="_blank">#Ceph</a>, <a href="https://social.heinlein-support.de/tags/OpenVox" class="mention hashtag" rel="nofollow noopener" target="_blank">#OpenVox</a> & Co. 👉 <a href="https://www.heinlein-support.de/slac/programm-2025" rel="nofollow noopener" translate="no" target="_blank">https://www.heinlein-support.de/slac/programm-2025</a> <a href="https://social.heinlein-support.de/tags/slac2025" class="mention hashtag" rel="nofollow noopener" target="_blank">#slac2025</a> <a href="https://social.heinlein-support.de/tags/LinuxVortr%C3%A4ge" class="mention hashtag" rel="nofollow noopener" target="_blank">#LinuxVorträge</a> <a href="https://social.heinlein-support.de/tags/LinuxWorkshops" class="mention hashtag" rel="nofollow noopener" target="_blank">#LinuxWorkshops</a>

Stuart Longland (VK4MSL)I was planning on doing a back-up of the <a href="https://mastodon.longlandclan.id.au/tags/Ceph" class="mention hashtag" rel="nofollow noopener" target="_blank">#Ceph</a> cluster today in preparation of migrating it to a later release… but of course, I've misplaced the PSU for my external 4TB back-up drive.I noticed it was getting pretty tight on space too, and given it's now 5 years old, figured it was time for a new one. I've got a 6TB one coming.In the meantime, I had another crack at the Ericsson F5521GW 3G module in the Toughpad.If I boot with init=/bin/bash on the kernel command line, then modprobe cdc-acm, I see 3 ACM devices appear, and I can use `screen` with them.I managed to remove the old PIN code on the SIM card I have in there but couldn't get the GPS going, most I got out of it was a "Operation not allowed" error.Doing this in early init seems to be enough to convince the module to stay enumerated. Something though, tells it to go bugger off early in the boot sequence unless I talk to it first.A bit of a nuisance to be honest… but I'm somewhat closer.<a href="https://mastodon.longlandclan.id.au/tags/ToughPad" class="mention hashtag" rel="nofollow noopener" target="_blank">#ToughPad</a> <a href="https://mastodon.longlandclan.id.au/tags/EricssonF5521gw" class="mention hashtag" rel="nofollow noopener" target="_blank">#EricssonF5521gw</a>

Stuart Longland (VK4MSL)I have gotten through all but one… just doing one at a time. Last night I got two done. This morning I got one done before I headded to work.During the work day, I had one scripted to wait for `ceph health` to report `HEALTH_OK`, wait a hour, then do its update unattended. I checked on it later from work and it had updated just fine.So I scheduled the same to happen with one of my bigger nodes, "hydrogen" (they're all a periodic table naming scheme as I started with Intel Atoms). I've just finished it now, leaving one left: "carbon". It, like "hydrogen" has two disks.I'll do the same there… wait for `HEALTH_OK`, wait a hour longer to be sure, then mark its disks out and begin the update.I expect it'll do it overnight, and by morning I should have a Debian 11 install there.<a href="https://mastodon.longlandclan.id.au/tags/HomeServer" class="mention hashtag" rel="nofollow noopener" target="_blank">#HomeServer</a> <a href="https://mastodon.longlandclan.id.au/tags/Ceph" class="mention hashtag" rel="nofollow noopener" target="_blank">#Ceph</a>

Stuart Longland (VK4MSL)20:56:10 up 514 days, 6:15, 1 user, load average: 0.94, 0.85, 0.64Just updating a couple of my <a href="https://mastodon.longlandclan.id.au/tags/Ceph" class="mention hashtag" rel="nofollow noopener" target="_blank">#Ceph</a> nodes… it's been a long time on Debian 10, but I need to move on. Ansible no longer works with it.Procedure so far is remove the old Ceph repositories (as they don't support anything beyond Debian 10) and lean on the fact that Debian 11 ships Ceph 14.It's a slightly older release of Ceph 14, but it'll be enough… I can get the OS updated on all the nodes by doing them one-by-one… then when I get them all up to Debian 11… I should be able to jump to the next stable release of Ceph… then do another round of OS updates to move to Debian 12.Ohhh boy.<a href="https://mastodon.longlandclan.id.au/tags/HomeServer" class="mention hashtag" rel="nofollow noopener" target="_blank">#HomeServer</a>

RachelOk probably another week to go until I get these drives running I should probably use that to figure out backups for this current pool..... if the bulk storage poll in ceph works well enough I'll ship the current file server off to the parent's but for now I'll be able to use the LAN to do initial syncs S3-compatible backup via Velero is one option, with MinIO, or <a href="https://garagehq.deuxfleurs.fr/" rel="nofollow noopener" target="_blank">Garage</a> running in a container backed by ZFS (I am not building a remote ceph cluster 😅) Anyone have thoughts/suggestions on backup strategy here? Probably backing up 2-3TB of data total (lots of photos) I'll end up with local snapshots, and remote backups, huge bonus if they can be recovered without needing a ceph cluster to restore to in the case of something catastrophic. <a href="https://transitory.social/tags/HomeLab" rel="nofollow noopener" target="_blank">#HomeLab</a> <a href="https://transitory.social/tags/Ceph" rel="nofollow noopener" target="_blank">#Ceph</a>

Thoralf Will 🇺🇦🇮🇱🇹🇼Test von <a href="https://soc.umrath.net/tags/ceph" class="mention hashtag" rel="nofollow noopener" target="_blank">#ceph</a> auf meinem <a href="https://soc.umrath.net/tags/Proxmox" class="mention hashtag" rel="nofollow noopener" target="_blank">#Proxmox</a>-Cluster abgeschlossenen.Es hat funktioniert, aber der Performance-Overhead ist gewaltig und steht in keinem sinnvollen Verhältnis zum Nutzen, der definitiv vorhanden wäre.Also alles wieder auf zfs zurückgebaut. Evtl. behalte ich einen kleinen Cluster für Backups, etc.

Thoralf Will 🇺🇦🇮🇱🇹🇼Eine Idee für das "Wie?" in meinem Kopf wäre:1. Auf jedem Node eine Disk aus dem <a href="https://soc.umrath.net/tags/zfs" class="mention hashtag" rel="nofollow noopener" target="_blank">#zfs</a>-Mirror heraustrennen. 2. <a href="https://soc.umrath.net/tags/ceph" class="mention hashtag" rel="nofollow noopener" target="_blank">#ceph</a> mit den 3 Disks aufbauen. 3. Volumes von zfs auf ceph migrieren. 4. Alles prüfen. 5. zfs zerlegen 6. Die weiteren Platten ceph zur Verfügung stellen.Ginge das?Alternativ hätte ich auch noch 3 Platten rumliegen (1, 2 und 4 TB), mit denen ich eine initiale ceph-Umgebung aufbauen könnte, einen Node komplett umziehen könnte und mich dann Stück für Stück durcharbeite ...

Recent searches

Search options

Administered by:

Server stats:

#ceph