"I love billymg"

2022-05-25 13:56:01

billymg PagesAbout (Public Key)Bitdash Crawler VTreeLogotron VTreeMP-WP VTreeMP-WP-Tests VTreeRecent PostsA Beginner's Guide to Installing Gentoo: Part One Bitdash Crawler: Geolocation, Time Series Data Collection, and an IRC Bot Straightening a Bent 1U Server Chassis Running MP-WP on a Modern Musl-Based Gentoo Building TRB on a 2022 Vintage Musl Gentoo Recent Commentsbillymg on A Beginner's Guide to Installing Gentoo: Part Onecrtdaydreams on A Beginner's Guide to Installing Gentoo: Part OneJacob Welsh on Building TRB on a 2022 Vintage Musl Gentoobillymg on Building TRB on a 2022 Vintage Musl GentooJacob Welsh on Building TRB on a 2022 Vintage Musl GentooArchivesMarch 2022February 2022January 2022September 2021August 2021May 2021June 2020May 2020April 2020March 2020January 2020December 2019November 2019September 2019August 2019March 2019January 2019December 2018October 2018Categoriesbitcoin (1)bitdash.io (6)costa rica (2)crawler (3)gentoo (1)guides (3)logotron (4)mp-wp (16)patches (8)photos (3)pizarro (1)planning (1)postgres (1)real estate (1)testing (3)travel (2)trb (1)user research (1)v (7)wireframes (1)MetaLog inEntries RSSComments RSSWordPress.org A Beginner's Guide to Installing Gentoo: Part One March 12th, 2022 This guide will walk you through setting up a base Gentoo system. It is intended for beginners who want to move to an operating system that gives them more control over what is installed. It will produce a reasonably sane Gentoo "daily driver" that will allow you to run TRB and other republican goodies. It will allow you to install xorg and a tiling manager without too much trouble. It will NOT be as strict as a Dulap when it comes to banning all systemdisms and other malware, so use accordingly.1First of all, a huge thank you to asciilifeform for producing an ultra-hygienic Gentoo and making it available to the world. Dulap remains my go-to for server applications and any box where the need for security is greater than the desire for comfort. The first part of this guide, up to the chroot, is more or less lifted directly from the Dulap construction kit.PrerequisitesGather these items before beginning with the guide.Any random laptop or assembled assortment of PC parts22 USB thumb drives, at least 2GBA Gentoo Stage 3 tarball: obtained hereA Gentoo Minimal Installation CD image: obtained hereCreate the Bootable USB DriveNote the path to your copy of the Gentoo Minimal Installation CD ISO file and check the drive device that corresponds to your USB drive with fdisk -l on Linux or diskutil list on macOS. Create the bootable USB with the following command, adjusted to your ISO file and USB drive:dd if=install-amd64-minimal-20220308T160629Z.iso of=/dev/sdb bs=4096Obtain a Stage 3 TarballSelect a Stage 3 tarball from the Gentoo Downloads page.3 Download it and verify the checksum. Copy it to your other USB drive. If you're working on macOS or Window you'll need to format the drive as FAT so that your Linux build can access it.Booting From the USB Drive and Formatting the Target DriveBoot from the bootable install disk you created in the previous step. You'll likely need to enter your BIOS and rearrange the boot order so that USB is tried first before any internal drives.4 Once this option is set, exit the BIOS saving your changes, the machine should boot from the USB drive.You are now in a mini Gentoo prepared specially for installing Gentoo. You'll first need to format your target drive so that you can proceed with the install. Run fdisk -l to view your drives and verify that your main drive (where you intend to install Gentoo) is in the first position. This will be /dev/sda if it's a SATA drive and /dev/nvme0n1 if it's an NVMe drive. If it is not one of these, reboot and adjust your BIOS boot order again until it is.Using the correct device, run the following commands:# Set the partition schemeparted -s /dev/sda mklabel gpt# Create the partitionsparted -s -a optimal /dev/sda unit s mkpart boot ext3 64 262143parted -s -a optimal /dev/sda unit s mkpart primary ext4 262144 100%# Enable the boot flag on the first partitionparted -s -a optimal /dev/sda set 1 boot onsync# Format the partitionsmkfs.ext3 /dev/sda1mkfs.ext4 /dev/sda2sync# First create the mount point if it does not existmkdir /mnt/gentoo# Mount the main partition to a mount point of your choice, here it's '/mnt/gentoo'mount /dev/sda2 /mnt/gentooNow your target drive should be ready. Try running fdisk -l to see a summary of the new partitions.Next, connect and mount your other USB drive, the one containing the Stage 3 tarball. You can mount this one to /mnt/usb or anywhere else you choose. If you're not sure which device is the USB drive you just plugged in run fdisk -l again and find it among the list.# Mount the USB drive at /dev/sdb2 to /mnt/usb. Change to your own values if necessary.mount /dev/sdb2 /mnt/usbNow run lsmod and dmesg and save the output to your USB drive with the Stage 3 tarball. You will need this information later when configuring the kernel.lsmod > /mnt/usb/lsmodout.txtdmesg > /mnt/usb/dmesgout.txtExtracting the Tarball and Chroot'ing Into the New SystemNavigate to the root of your new drive and extract the Stage 3 tarball.cd /mnt/gentootar xpvf /mnt/usb/stage3-*.tar.xz --xattrs-include='*.*' --numeric-ownerAfter this completes you can run ls -l and you'll see your new system at home on its new drive.Use the included net-setup tool to get network access. Then after you have a connection, copy the DNS info to the new environment.# Run 'net-setup'net-setup# Verify that you have a connectionping thebitcoin.foundation# Copy the DNS info to the new environmentcp --dereference /etc/resolv.conf /mnt/gentoo/etc/You are now ready to chroot into the new system.# Mount the boot partitionmount /dev/sda1 /mnt/gentoo/bootmount -o bind /dev /mnt/gentoo/devmount -o bind /proc /mnt/gentoo/procmount -o bind /sys /mnt/gentoo/syschroot /mnt/gentoo /bin/bashYou are now "inside" the new system, meaning what was previously /mnt/gentoo is now simply /. For example, ls / will display the contents of the partition at /dev/sda2 that you created earlier, not the contents of the bootable USB drive. Everything you do from here on out will affect the new system.5Configuring Portage and Obtaining the Kernel SourceAfter chrooting, run emerge-webrsync to get a snapshot of the latest upstream repo. If you downloaded a Musl Stage 3 tarball then you will want to add the Musl overlay at this point. If not, then skip this step.Add the Musl Overlay (skip if building a glibc-based system)Overlays6 are a feature of Portage, Gentoo's package management system, that allow you to add additional packages not tracked in the main upstream repository. They can be thought of as sheets of vellum paper overlaying the default repository. They leave the original package list intact while adding one or more new packages (or versions of packages). This will be covered more in Part Two but for now—if you opted for a musl-based system—you will need to enable at least one overlay. If you opted for a glibc-based system, proceed to the next step.# Emerge 'eselect-repository' and 'git'emerge -av app-eselect/eselect-repositoryemerge -av dev-vcs/git# Create the config directorymkdir -p /etc/portage/repos.conf # Enable and sync the 'musl' overlayeselect repository enable muslemaint sync -r muslThis will give you an up-to-date musl overlay, which is necessary for emerging7 the correct musl versions of packages for your musl-based build.Apply the Ban ListCreate a file called 'crapolade'8 /etc/portage/package.mask/crapolade and include in it asciilifeform's ban list:# Create this file at /etc/portage/package.mask/crapolade# systemdismssys-apps/systemdsys-apps/systemd-tmpfilessys-fs/udevvirtual/tmpfilessys-auth/consolekitsys-auth/polkitsys-apps/dbusmedia-video/libavgnome-base/gconf>=app-crypt/gnupg-2.0.22app-accessibility/at-spi2-atkapp-accessibility/at-spi2-coregnome-extra/polkit-gnomedev-util/gdbus-codegengnome-base/dconf>=x11-libs/gtk+-3.0.0#>=sys-devel/gcc-5.0The way this works is that Portage will check all entries in /etc/portage/package.mask when emerging packages and refuse to install packages that you have "masked". This also applies to packages brought in as dependencies by other packages, so it's a pretty effective way of keeping unwanted cruft out of your machine. You may have noticed that one item, >=sys-devel/gcc-5.0, is commented out. This is because any Stage 3 tarball that you download today will already include a GCC newer than 5.0, so the only result of leaving that line in would be some error spew about "Masked package already installed" every time emerge is run.Later in this guide you will have to comment out other lines. As I mentioned in the beginning, this guide is not designed to produce a military grade Gentoo, but rather a comfortable daily driver that will run TRB and other republican software.9In addition to the output of lsmod and dmesg that we saved earlier, there's another utility, lspci, available that I find useful for identifying hardware components. Emerge with:emerge --ask --verbose sys-apps/pciutilsYou will later use the output of lspci and the saved output from lsmod and dmesg to get an idea of what you'll need to enable in the kernel.If you're on a more recent model laptop you may also need to install the Linux firmware package to enable graphics cards or wifi chips. If your laptop has Intel wifi, for example, you'll probably need this.emerge -av sys-kernel/linux-firmwareFinally, emerge the Linux kernel source.emerge -av sys-kernel/gentoo-sources# Create a symlink to simplify things, change the version/filename to match what you have in /usr/srcln -s /usr/src/linux-5.15.11-gentoo /usr/src/linuxConfiguring and Building the KernelIf this is your first time configuring a Linux kernel, which would not be unreasonable considering you're reading "A Beginner's Guide to Installing Gentoo", then it may feel a little daunting. Given the nearly endless combinations of possible hardware configurations there's also no way to create a step-by-step guide—you're on your own for this.The strategy I would recommend going into this for the first time would be: don't overthink it and don't overdo it. There's a non-zero chance that you won't need to make any modifications at all and your kernel will boot with the defaults. Other knobs will be obvious. If you have an NVMe drive, be sure to enable support for NVMe drives. If you have an Nvidia graphics card it's probably ok to disable graphics support for other vendors, etc.10 cd /usr/src/linux# Launch the config UI, you will make all your changes heremake menuconfig#Compile the kernelmake && make modules_install# Copies the kernel and config to /bootmake installBuilding the InitramfsGet genkernel to build the initramfs.emerge --ask sys-kernel/genkernelBuild the initramfs. Be absolutely sure that the path you pass to --kernel-config points to the same config used to compile the kernel. This config should be present at /boot/config-5.15.11-gentoo (or whatever version kernel you used). If not, adjust accordingly.genkernel --install --kernel-config=/boot/config-5.15.11-gentoo initramfsConfigure the File Systems Tab (fstab)Edit /etc/fstab to set boot and primary disks.# SAMPLE fstab/dev/sda1 /boot ext3 defaults 0 2/dev/sda2 / ext4 noatime,discard 0 1Configure the BootloaderYou will have to choose a bootloader. The two main options are LILO and GRUB.11 You should try to get LILO to work first, it's got a much smaller footprint than GRUB and less moving parts means less that can potentially go wrong. If you can't get LILO to work12 or your just don't like LILO for some reason then you can use GRUB.Option One: LILOEmerge LILO.emerge -av sys-boot/liloEdit /etc/lilo.conf and adjust the values to match your system. Then run lilo to install the bootloader.liloThat should be it.Option Two: GRUBEmerge GRUB 2.emerge -av sys-boot/grubExit the chroot and enable EFI if you’re using that. If not, skip to installing GRUB.# Exit the chrootexit# Enable EFImodprobe efivarsmodprobe efivarfs# Re-enter the chrootchroot /mnt/gentooInstall grub.# For EFIgrub-install --target=x86_64-efi --efi-directory=/boot --debug# For Non-EFI (adjust device to match your system)grub-install --target=i386-pc /dev/sda --debugGenerate the GRUB config.grub-mkconfig -o /boot/grub/grub.cfgBooting Your New OSBefore rebooting set a root user password. If you do not set this you will be locked out and will have to boot from your USB drive and chroot in all over again.passwdIf you’re on a machine that only has a wireless connection you’ll also need to grab the wpa_supplicant and a DHCP client before you boot into your new system otherwise you will not have internet connectivity.emerge -av net-wireless/wpa_supplicantemerge -av net-misc/dhcpcdAttempt to reboot to your newly installed Gentoo.#Exit chroot and rebootexitrebootIf everything works you can log in as root with the password you set.A Few Remaining BasicsYour system now works but before you can start having fun with it there are a few small things to take care of.Make Your HiDPI Display UsableIf you are on a HiDPI display your console font is likely really tiny, to the point where it is practically illegible. Increase it by choosing a different font.setfont latarcyrheb-sun32Make this permanent by editing /etc/conf.d/consolefont and adding to boot services.rc-update -v show | grep consolefontrc-update add consolefont bootSee list of available fonts in /usr/share/consolefonts if you want to experiment with different options.Enable Wireless and Connect to a WiFi NetworkThe wpa_supplicant replaces that little WiFi icon in the top-right corner of your old OS as the primary means of finding, selecting, and connecting to WiFi networks.# Add 'wpa_supplicant' to the default and boot runlevelsrc-update add wpa_supplicant defaultrc-update add wpa_supplicant boot# Start the servicerc-service wpa_supplicant startIf this doesn’t work it’s likely because your kernel driver isn’t loaded.# Load the kernel modulemodprobe iwlwifi# Try starting the service againrc-service wpa_supplicant startTo make the kernel module load at boot, create the following file:# /etc/modules-load.d/networking.confiwlwifiYou can now use wpa_cli to control your wireless interface. First try scanning for available networks.# Scan for networkswpa_cli scan# Wait a few seconds then view resultswpa_cli scan_resultsNote the SSID of the network you would like to connect to. To generate the WPA PSK for the network run the following command with the SSID and password of the network:wpa_passphrase my-network-5g mynetworkpassThe output will look something like this:network={ ssid="my-network-5g" #psk="mynetworkpass" psk=9f1315f6253481f27965068df528064ae85e6cbb06540090c79e3817cb8f12a7}Copy the entire block and add it to the config at /etc/wpa_supplicant/wpa_supplicant.conf. Also add ctrl_interface and update_config knobs to the top of the file. Your file should look something like this:# WPA Supplicant config at '/etc/wpa_supplicant/wpa_supplicant.conf'ctrl_interface=DIR=/run/wpa_supplicant GROUP=wheelupdate_config=1network={ ssid="my-network-5g" #psk="mynetworkpass" psk=9f1315f6253481f27965068df528064ae85e6cbb06540090c79e3817cb8f12a7}Save the config, then use wpa_cli again to load the config and list saved networks.# Reload the configwpa_cli reconfigure# List saved networkswpa_cli list_networks# Select the network, where the number is the ID of the network returned by the previous commandwpa_cli select_network 0You should now have internet access again.Create a Non-Root User and Install 'sudo'Now that you have an internet connection you should create a user account and install/configure sudo so that you’re not doing everything as root. First create your new user.# Create a new user, '-m' creates a home folder for this useruseradd -m mynewuser# Set the password for this userpasswd mynewuseremerge -av app-admin/sudoConfigure with visudovisudoYou can read more elsewhere about this file but an easy way to control who has sudo access is with the "sudo" group. Uncomment the line for this configuration so that it looks like:%sudo ALL=(ALL) ALLYou can also set a longer timeout if you like (the default is only 5 minutes).# Include this line to set the default sudo timeout to 30 minutesDefaults env_reset, timestamp_timeout=30Create the sudo group and add your user to the groupgroupadd sudosudo usermod -a -G sudo mynewuserNow you can log out of the root account with exit and log back in to your new user. Congratulations, you now have a basic working Gentoo linux machine with wireless internet access. You can now sudo emerge any package you like. In part two we’ll set up a minimal graphical environment plus some command line niceties. If you tried using this guide and noticed errors or parts that were unclear please leave feedback so that I can make improvements.If you intend to build a web server or have no need for a browser or any graphical environment then I suggest following this guide to build a Dulap instead. [↩]I won't bother with listing "minimum specs" because they aren't high enough to be worth mentioning and because it mostly depends on what you want to run. Gentoo isn't macOS or Windows, you don't need 4GB of RAM just for the OS. Any cheapo laptop from the last 15 years should do. If your goal is to run TRB then you need at least a 1TB SSD. [↩]At this point in time, if you use a current glibc-based system you will need to create a separate chroot environment later on in order to build TRB with the rotor due to a version incompatibility. If you select a musl-based system then you will be able to build TRB just fine, with a few tweaks, but you may have a harder time running other popular programs that don't yet play nice with musl. [↩]This varies by machine, depending on your motherboard. Typically F1, F11, or DEL gets you into the BIOS— the post splash screen should say which key, something like "Press F1 to enter setup". From there just keyboard navigate until you find what you're looking for. [↩]Unless you exit the chroot. [↩]Now called "ebuild repositories". [↩]Installing [↩]You can call it whatever you want but this name fits. [↩]So why include the ban list in the first place? The idea is to make you think twice about what you're installing, and only unmask the bare minimum that you absolutely need. Often if you are blocked because of a masked package you can get around it by installing an older version the package you want, or manually installing an older version of the specific dependency that's asking to bring in the masked package. Or you can find another program that has similar functionality to what you want but that doesn't depend on a masked package. If you conclude that you really must have a package that is blocked on a certain masked dependency, then you can unmask only that dependency. [↩]Although don't worry too much about optimization at this point, you can always slim the kernel later on. [↩]There may be others, I don't know. [↩]I was unable to get LILO to successfully install the bootloader on my ThinkPad but it's what I use on my FX-8350 desktop. YMMV. [↩] Posted in gentoo, guides | 2 Comments » Bitdash Crawler: Geolocation, Time Series Data Collection, and an IRC Bot March 7th, 2022 The Bitdash Crawler has been running steadily since August of last year. The first release was minimal—only the basic crawling functionality, and only collecting data returned directly by the probed nodes. This patch adds additional data collection, as well as a new IRC interface for basic status reports.Geolocation DataWhen I first looked into adding this I was reluctant because I thought the only way was to connect to a third-party service via API keys. I didn't want the crawler to have to rely on any third-party service, as those come with the risk that they will change out from under you at any moment and without notice. Thankfully a user in #asciilifeform pointed me to Maxmind's excellent GeoLite2 geolocation database. I was almost surprised at how simple it was and that there was no catch. Yes, you have to provide an email and create an account with them, but after that they just give you their geolocation database1, in CSV format that you can take wherever you like. They even provide clear documentation for creating the schema and importing the CSV data. Overall a very nice offering and I'd recommend it to anyone who needs to add this to their project.Time Series DataI'm working on a redesign of the www interface for the crawler2 and it includes charts. Previously the crawler was only storing current snapshots of the data it collected, now it's also generating some aggregate statistics3, taken at set intervals, and storing them indefinitely. This job is handled separately from the main crawler script, via a set of SQL queries run from a cron job.IRC Bot InterfaceWith the addition of geolocation and time series data collection the crawler can finally provide some reports its underlying library alone cannot. Soon this data will be available via the www interface but in the meantime I wanted to make it available in the chans via an IRC bot. The bot is based on asciilifeform's logotron bot, with which I've become familiar since using it to power the logger at logs.bitdash.io. It works roughly like this: billymg | !c helpcrawlerbot | billymg: my valid commands are: src, uptime, help, net-summary, version, trb-status billymg | !c net-summarycrawlerbot | Bitcoin Network (IPv4 Nodes Active Within the Last 48 hours) Global: 8166; TRB-Compatible: 61; TRB: 13crawlerbot | TRB-Compatible by Country: United States: 26; Canada: 4; Singapore: 4; Romania: 4; Russia: 3; France: 2; United Kingdom: 2; Italy: 1; Lithuania: 1; Norway: 1; Australia: 1; Germany: 1; Chile: 1; Belgium: 1; Spain: 1; Ukraine: 1; Netherlands: 1; Finland: 1; Sweden: 1; Switzerland: 1; Bulgaria: 1; Mexico: 1; South Africa: 1;crawlerbot | TRB by Country: United States: 7; Canada: 1; Romania: 1; Singapore: 1; Lithuania: 1; France: 1; Norway: 1;Patches and SignaturesI unfortunately had to regrind the genesis patch and the small bug fix patch that followed because there was a typo in the root directory's name. I'll leave the original two patches up for archival purposes but this patch is built on a new tree, with the genesis regrind also including the small fix from the original second patch.bitdash_crawler_genesis_regrind.vpatch (sig)bitdash_crawler_add_bot_geo_and_time_series.vpatch (sig)Presumably a subset of the data included in their full commercial offering, but plenty good enough for my needs. [↩]Which at the moment is nothing more than a few Sketch mockups. [↩]Network breakdown by major user agent version, network breakdown by country, TRB breakdown by country, and TRB-compatible breakdown by country. All stats are collected both for recently active nodes (48hrs) and for recently active nodes returning at least one non-self peer (referred to as "participating"). [↩] Posted in bitdash.io, crawler, patches | No Comments » Straightening a Bent 1U Server Chassis February 17th, 2022 I recently acquired a new (used) server which was damaged in shipping.1 Luckily the damage wasn't anything serious, just a slightly bent frame.In order to have any hope of fixing this I'd first have to remove the motherboard. As you can see, none of the internals appear damaged. It seems like the PCI riser's bracket may have provided some extra sturdiness to the location just at the edge of the motherboard, preventing the case from bending closer to the middle. In any case, buena suerte. Let's get started by first removing the plastic airflow cover and the PCI riser.Now on to all of the fan and power cables. The fans were no problem but it was a bit tight around the mains, which required some extra finesse. The first thing that got in the way was the zip tie closest to the connectors. It made it difficult to get my fingers underneath the cables and from the other side it looked like it also might have been wrapped around some cables I did not want to disturb.With a sharp knife I carefully sawed at it until it broke. I also removed one of the heat sinks, since that was getting in the way.2 To help with some of the clips I used a flat head screwdriver to squeeze them open.With all of the cables unplugged, all that was left was to unscrew and remove the motherboard. There were no surprises here and inspecting the motherboard upon removal further reassured me that it hadn't suffered any damage during transit.Now with the case (mostly) empty I was ready to move it to the bodega3 for phase two. I scanned the walls and shelves until I found two vices that seemed perfect for the operation.4I also noticed an old sheet which was being used as a rag, so I made some space on a workbench and laid it flat just to avoid adding any extra scratches to the chassis while working on it. I then attached the vices to either side and started gently tightening the one on the right side to straighten the frame.As I was doing this I noticed the left side starting to arc up so I moved the left vice inward to prevent adding a new deformity while fixing the original.I tightened the right side all the way but, as you may have noticed in the photo above, the flat surface isn't going to cut it. In order to fully straighten out the bend I'd need to add some leverage. I once again searched the bodega for something that would do the job. A piece of wood, some cardboard, or a folded piece of paper might do. Instead I found something even better, a small section of 4 AWG wire. I placed it right under the bend and proceeded to slowly tighten the right side again. I again noticed some unwanted bowing so I moved the right vice inward, closer to the wire, and continued tightening.That was it. The case looked pretty straight to me now so I brought it back inside to take a closer look and reassemble the internals. As you can see from the before and after photos below, the operation seems to have been a success. Perhaps not perfect but hopefully good enough to slide into a rack somewhere.BeforeAfterHere it is below with the motherboard back in place and the old thermal paste cleaned off of the CPUs.And finally all put back together and loaded with 256GB of DDR3 ECC RAM.I also loaded the SSDs (not visible in the photo) but unfortunately wasn't able to test anything past powering it on since I have no way to connect it to a monitor.5 The next step, after installing Dulap on it (or perhaps Pentacle), is to find a home for it here in Costa Rica. Oh, and I guess I'll also need a hardware RAID controller for it. If anyone has any recommendations I'd much appreciate it as this will be my first time buying one.Getting things shipped to Costa Rica can be a pain. This item had to be smuggled in a suitcase, and although the impromptu importer went to great lengths to make sure it was packed properly, apparently the baggage handlers went to even greater lengths to ensure that that didn't matter. [↩]It would've had to have been removed later anyway, as it was attached to the chassis. [↩]Meaning in this case garage/workshop/storage area, not a convenience store in NYC. [↩]The bodega in its current state is more like my personal junkyard than anything else. It needs to be thoroughly cleaned and organized, and it will be, just probably not this year. [↩]I mistakenly ordered an HDMI to VGA cable, rather than the VGA to HDMI cable that I need. [↩] Posted in photos | No Comments » « Older Entries Powered by mp-wp. Theme BMG_Minimal by billymg. Entries (RSS) and Comments (RSS).