private-schrijfsels-en-noti.../2024/So you wanna do FreeBSD 14....

403 lines
20 KiB
Markdown
Raw Permalink Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

---
tags:
- freebsd
- homelab
- selfhosting
---
Environment:
* FreeBSD 14.1-RELEASE-p3 AMD64
* Python 3.11.9
* Home Assistant 2024.3.3
2024.4 or higher not possible due too needed python 3.12 see https://www.home-assistant.io/blog/2024/04/03/release-20244/#backward-incompatible-changes
I have encountered several project now which are Linux container/docker "native" installation. Like PhotoPrism and HomeAssistant. Yeah software dependencies are hard for developers and maintainers. But ending your software on just one install method is a "quick-fix". Apple macOS tries to "fix this" by putting everything of an application in a folder suffixed as `.app`. No this are no containers but all application dependencies reside in this folder. The price you pay is more disk space when multiple applications use same "Apple .framework" or libraries and other executables.
It seems nowadays the quick-fix is to enforce people using Docker for production and when that is not available it will be wrapped again in a native Linux virtual machine. So you are almost out of luck running it otherwise. Gitlab is another beast, it runs on ruby and it has "omnibus" host installer. Which just basicly Gitlab-fies your whole OS.
Why on earth do people think we can get rid of ALL software written in C/C++ converted to the promised better language Rust. As Python is executing `cc` ?
```
last pid: 5386; load averages: 1.48, 1.51, 1.61 up 35+20:47:26 17:47:34
14 processes: 2 running, 12 sleeping
CPU: 24.9% user, 20.4% nice, 6.5% system, 0.0% interrupt, 48.2% idle
Mem: 810M Active, 773M Inact, 506M Laundry, 1627M Wired, 385M Buf, 111M Free
ARC: 282M Total, 98M MFU, 50M MRU, 599K Anon, 6378K Header, 123M Other
75M Compressed, 415M Uncompressed, 5.57:1 Ratio
Swap: 2689M Total, 500M Used, 2190M Free, 18% Inuse
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
4564 homeassist 1 135 0 239M 159M CPU1 1 0:41 99.88% cc
78755 homeassist 1 20 0 17M 5072K select 1 0:01 0.20% tmux
2726 homeassist 1 20 0 14M 2980K CPU2 2 0:00 0.17% top
96466 homeassist 10 20 0 183M 115M kqread 0 0:06 0.03% python3.11
674 homeassist 1 68 0 208M 180M wait 0 1:43 0.00% python3.11
97272 homeassist 1 20 0 99M 77M piperd 1 0:05 0.00% python3.11
99599 homeassist 1 68 0 57M 40M wait 0 0:01 0.00% python3.11
25 homeassist 1 21 0 16M 3616K pause 2 0:00 0.00% csh
89135 homeassist 1 20 0 13M 2612K wait 2 0:00 0.00% sh
81771 homeassist 1 20 0 13M 2664K ttyin 1 0:00 0.00% sh
79030 homeassist 1 63 0 13M 2672K wait 0 0:00 0.00% sh
1361 homeassist 1 24 0 13M 2752K wait 3 0:00 0.00% sh
98234 homeassist 1 20 0 13M 2656K ttyin 1 0:00 0.00% sh
81501 homeassist 1 20 0 14M 3644K select 1 0:00 0.00% tmux
```
The "modern" software landscape is getting messy!
It is such a mess to run HomeAssistant native on FreeBSD you need to install some binary packages with `pkg` from FreeBSD binary ports. Then Python `pip` install some extra packages in a virtual environment and finally the application Python runs the `cc` (c compiler) itself! Even using `meson` build system as intermediate. We need Python `cryptography` dependency to use the `Rust` compiler as the devs have decided to not proceed with (accelerated) crypto implemented in C. I think if you further want to get into the dependency hell we need Library X written in C++ and wrap it under Python. Then a C++ runtime is also needed...
https://dan.langille.org/2022/08/27/getting-home-assistant-running-in-a-freebsd-13-1-jail/
Run the stuffs in venv:
`(homeassistant) homeassistant@mango:/data/homeassistant % ./bin/hass --ignore-os-check -v -v -v -v`
https://devhumor.com/media/monkeyuser-npm-package-delivery
https://developer.apple.com/library/archive/documentation/General/Conceptual/DevPedia-CocoaCore/Framework.html
ImportError: /data/homeassistant/lib/python3.11/site-packages/numpy/core/_multiarray_tests.cpython-311.so: Undefined symbol "ccosl"
https://github.com/numpy/numpy/issues/24873
```
note: This error originates from a subprocess, and is likely not a problem with pip.
2024-11-08 19:00:58.632 INFO (SyncWorker_5) [homeassistant.util.package] Attempting install of webrtc-noise-gain==1.2.3
2024-11-08 19:01:13.568 ERROR (SyncWorker_5) [homeassistant.util.package] Unable to install package webrtc-noise-gain==1.2.3: error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [20 lines of output]
Traceback (most recent call last):
File "/data/homeassistant/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
main()
File "/data/homeassistant/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/homeassistant/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
return hook(config_settings)
^^^^^^^^^^^^^^^^^^^^^
File "/tmp/pip-build-env-ih3jdwtb/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 333, in get_requires_for_build_wheel
return self._get_build_requires(config_settings, requirements=[])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/pip-build-env-ih3jdwtb/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 303, in _get_build_requires
self.run_setup()
File "/tmp/pip-build-env-ih3jdwtb/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 521, in run_setup
super().run_setup(setup_script=setup_script)
File "/tmp/pip-build-env-ih3jdwtb/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 319, in run_setup
exec(code, locals())
File "<string>", line 384, in <module>
ValueError: Unsupported system: freebsd
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
```
```
2024-11-08 19:02:25.055 ERROR (SyncWorker_0) [homeassistant.util.package] Unable to install package ha-av==10.1.1: error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [11 lines of output]
Package libavformat was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavformat.pc'
to the PKG_CONFIG_PATH environment variable
Package 'libavformat' not found
Package 'libavcodec' not found
Package 'libavdevice' not found
Package 'libavutil' not found
Package 'libavfilter' not found
Package 'libswscale' not found
Package 'libswresample' not found
pkg-config could not find libraries ['avformat', 'avcodec', 'avdevice', 'avutil', 'avfilter', 'swscale', 'swresample']
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
```
```
pkg install py311-ffmpeg-python-0.2.0_1
```
```
2024-11-08 19:06:55.875 ERROR (SyncWorker_5) [homeassistant.util.package] Unable to install package webrtc-noise-gain==1.2.3: error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [20 lines of output]
Traceback (most recent call last):
File "/data/homeassistant/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
main()
File "/data/homeassistant/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/homeassistant/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
return hook(config_settings)
^^^^^^^^^^^^^^^^^^^^^
File "/tmp/pip-build-env-i8fc26m0/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 333, in get_requires_for_build_wheel
return self._get_build_requires(config_settings, requirements=[])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/pip-build-env-i8fc26m0/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 303, in _get_build_requires
self.run_setup()
File "/tmp/pip-build-env-i8fc26m0/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 521, in run_setup
super().run_setup(setup_script=setup_script)
File "/tmp/pip-build-env-i8fc26m0/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 319, in run_setup
exec(code, locals())
File "<string>", line 384, in <module>
ValueError: Unsupported system: freebsd
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
```
```
ImportError: Error importing numpy: you should not try to import numpy from
its source directory; please exit the numpy source tree, and relaunch
your python interpreter from there.
```
```
Great Thanks [@LukaszMoskala](https://github.com/LukaszMoskala),
For Android Termux :
# outside of homeassistant venv
git clone [https://github.com/numpy/numpy.git](https://github.com/numpy/numpy.git) numpy-git
cd numpy-git
git checkout v1.26.0
git cherry-pick [040ed2d](https://github.com/numpy/numpy/commit/040ed2dc9847265c581a342301dd87d2b518a3c2)
git submodule update --init
cd ..
# now enter homeassistant venv
LDFLAGS=" -lm -lpython3" pip install numpy-git/
```
https://github.com/numpy/numpy/issues/24873#issuecomment-1807458661
`setenv LDFLAGS "-lm -lpython3" ; pip install --config-settings=builddir=build dl/numpy-git-build-freebsd-14.1 dl/numpy-git/`
```
(homeassistant) homeassistant@mango:/data/homeassistant % setenv LDFLAGS "-lm -lpython3" ; pip install dl/numpy-git/
Processing ./dl/numpy-git
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... error
error: subprocess-exited-with-error
× Preparing metadata (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [12 lines of output]
+ /data/homeassistant/bin/python /data/homeassistant/dl/numpy-git/vendored-meson/meson/meson.py setup /data/homeassistant/dl/numpy-git /data/homeassistant/dl/numpy-git/.mesonpy-8palx956/build -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/data/homeassistant/dl/numpy-git/.mesonpy-8palx956/build/meson-python-native-file.ini
The Meson build system
Version: 1.2.99
Source dir: /data/homeassistant/dl/numpy-git
Build dir: /data/homeassistant/dl/numpy-git/.mesonpy-8palx956/build
Build type: native build
Project name: NumPy
Project version: 1.26.0
../../meson.build:1:0: ERROR: Compiler cc cannot compile programs.
A full log can be found at /data/homeassistant/dl/numpy-git/.mesonpy-8palx956/build/meson-logs/meson-log.txt
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
```
Okay so it means `meson` build system is also written in Python....
https://github.com/tprelog/iocage-homeassistant/issues/64#issuecomment-1936432832
Thanks meson and pip for throwing away the NEEDED log:
```
(homeassistant) homeassistant@mango:/data/homeassistant % cat /data/homeassistant/dl/numpy-git/.mesonpy-4rpzl4ag/build/meson-logs/meson-log.txt
cat: /data/homeassistant/dl/numpy-git/.mesonpy-4rpzl4ag/build/meson-logs/meson-log.txt: No such file or directory
```
Follow-up: the way to not clean up the build dir (and hence the log file) is:
```
pip install . --config-settings=builddir=build
```
https://github.com/mesonbuild/meson-python/issues/246#issuecomment-1354426272
So probably:
```
setenv LDFLAGS "-lm -lpython3" ; pip install --config-settings=builddir=dl/numpy-git-build-freebsd-14.1 dl/numpy-git/
```
Log is preserved...
```
A full log can be found at /data/homeassistant/dl/numpy-git/dl/numpy-git-build-freebsd-14.1/meson-logs/meson-log.txt
[end of output]
```
```
-----
Sanity check compile stderr:
ld: error: unable to find library -lpython3
cc: error: linker command failed with exit code 1 (use -v to see invocation)
-----
../../meson.build:1:0: ERROR: Compiler cc cannot compile programs.
```
Will get then
```
setenv LDFLAGS "-lm -lpython3.11" ; pip install --config-settings=builddir=dl/numpy-git-build-freebsd-14.1 dl/numpy-git/
```
Due too shared object (.so) naming:
```
(homeassistant) homeassistant@mango:/data/homeassistant % pkg list python311 | grep libpython
/usr/local/lib/libpython3.11.so
/usr/local/lib/libpython3.11.so.1
/usr/local/lib/libpython3.11.so.1.0
/usr/local/lib/libpython3.11.so.1.0-gdb.py
/usr/local/lib/python3.11/config-3.11/libpython3.11.a
```
Still could not find it try adding `/usr/local/lib` search path:
```
(homeassistant) homeassistant@mango:/data/homeassistant % setenv LDFLAGS "-lm -lpython3.11 -L/usr/local/lib" ; pip install --config-settings=builddir=dl/numpy-git-build-freebsd-14.1 dl/numpy-git/
```
Before installation with this cherry-picked numpy fix we need to uninstall numpy:
```
(homeassistant) homeassistant@mango:/data/homeassistant % pip uninstall numpy
Found existing installation: numpy 1.26.0
Uninstalling numpy-1.26.0:
Would remove:
/data/homeassistant/bin/f2py
/data/homeassistant/lib/python3.11/site-packages/numpy-1.26.0.dist-info/*
/data/homeassistant/lib/python3.11/site-packages/numpy/*
Proceed (Y/n)? y
Successfully uninstalled numpy-1.26.0
```
Finally homebrew patched numpy
```
Processing ./dl/numpy-git
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: numpy
Building wheel for numpy (pyproject.toml) ... done
Created wheel for numpy: filename=numpy-1.26.0-cp311-cp311-freebsd_14_1_RELEASE_p3_amd64.whl size=24156181 sha256=165ac15d3ddf5d2336520934b1321b4eb26074ddb11205398b450375693d56ba
Stored in directory: /home/homeassistant/.cache/pip/wheels/11/b7/76/343a9fc71c70dd4931749d568222522502d6af19e277d2e1d6
Successfully built numpy
Installing collected packages: numpy
Successfully installed numpy-1.26.0
```
Need to fix `webrtc-noise-gain==1.2.3` dependency probably
```
2024-11-08 19:42:54.330 INFO (MainThread) [homeassistant.requirements] Multiple attempts to install webrtc-noise-gain==1.2.3 failed, install will be retried after next configuration check or restart
2024-11-08 19:42:54.330 ERROR (MainThread) [homeassistant.setup] Setup failed for 'mobile_app': Requirements for assist_pipeline not found: ['webrtc-noise-gain==1.2.3'].
2024-11-08 19:42:54.339 INFO (MainThread) [homeassistant.setup] Setup of domain tts took 0.1 seconds
2024-11-08 19:42:54.351 INFO (MainThread) [homeassistant.requirements] Multiple attempts to install webrtc-noise-gain==1.2.3 failed, install will be retried after next configuration check or restart
2024-11-08 19:42:54.351 ERROR (MainThread) [homeassistant.setup] Setup failed for 'assist_pipeline': Requirements for assist_pipeline not found: ['webrtc-noise-gain==1.2.3'].
```
Interesting is
```
2024-11-08 19:42:54.330 ERROR (MainThread) [homeassistant.setup] Setup failed for 'mobile_app': Requirements for assist_pipeline not found: ['webrtc-noise-gain==1.2.3'].
```
https://www.reddit.com/r/termux/comments/178kt7e/need_help_installing_webrtcnoisegain_for_home/
Setuptools is also calling the `cc` ?? Really
Needed for iOS app....
In `~/.homeassistant/configuration.yaml` we need to add to root `# Mobile application mobile_app:` to enable iOS login.
Thanks again for more dependency hell when almost everything is "working":
```
2024-11-08 19:48:52.382 WARNING (MainThread) [homeassistant.bootstrap] Support for the running Python version 3.11.9 is deprecated and will be removed in Home Assistant 2024.4; Please upgrade Python to 3.12
```
Thanks...
https://github.com/home-assistant/core/issues/102645#issuecomment-1792234494
```
(homeassistant) homeassistant@mango:/data/homeassistant/build % pip install --config-settings=builddir=webrtc webrtc-noise-gain==1.2.3
Collecting webrtc-noise-gain==1.2.3
```
```
(homeassistant) homeassistant@mango:/data/homeassistant/build % pip install --config-settings=builddir=webrtc webrtc-noise-gain==1.2.5
Collecting webrtc-noise-gain==1.2.5
Downloading webrtc_noise_gain-1.2.5.tar.gz (3.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.1/3.1 MB 7.9 MB/s eta 0:00:00
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: webrtc-noise-gain
Building wheel for webrtc-noise-gain (pyproject.toml) ... \
```
Hard pinned to version:
```
root@mango:/data/homeassistant # ag webrtc-noise-gain
lib/python3.11/site-packages/homeassistant/package_constraints.txt
62:webrtc-noise-gain==1.2.3
lib/python3.11/site-packages/homeassistant/components/assist_pipeline/manifest.json
9: "requirements": ["webrtc-noise-gain==1.2.3"]
```
```
(homeassistant) homeassistant@mango:/data/homeassistant % cp lib/python3.11/site-packages/homeassistant/package_constraints.txt lib/python3.11/site-packages/homeassistant/package_constraints.txt.orig
```
https://www.reddit.com/r/termux/comments/178kt7e/need_help_installing_webrtcnoisegain_for_home/
Seems not to work
```
2024-11-08 20:18:12.020 INFO (SyncWorker_4) [homeassistant.util.package] Attempting install of webrtc-noise-gain==1.2.3
2024-11-08 20:18:14.406 ERROR (SyncWorker_4) [homeassistant.util.package] Unable to install package webrtc-noise-gain==1.2.3: ERROR: Cannot install webrtc-noise-gain==1.2.3 becaus
e these package versions have conflicting dependencies.
```
Inside the HA assist_pipeline the dependency is also pinned to exact version
```
(homeassistant) homeassistant@mango:/data/homeassistant % cp lib/python3.11/site-packages/homeassistant/components/assist_pipeline/manifest.json lib/python3.11/site-packages/homeassistant/components/assist_pipeline/manifest.json.orig
(homeassistant) homeassistant@mango:/data/homeassistant % vim lib/python3.11/site-packages/homeassistant/components/assist_pipeline/manifest.json
vim: Command not found.
(homeassistant) homeassistant@mango:/data/homeassistant % nvim lib/python3.11/site-packages/homeassistant/components/assist_pipeline/manifest.json
```
Hacking it to use which one compiled: `webrtc-noise-gain==1.2.5`
```
$ cd /data/homeassistant
$ ./bin/hass --ignore-os-check -v -v -v -v
```
TODO:
* https://github.com/davidjb/home-assistant-rc.d/blob/master/homeassistant
* https://github.com/tprelog/iocage-homeassistant/issues/64
* https://gist.github.com/robvanoostenrijk/0978f3f919a540cb0b280a6978cbe8aa
* https://community.home-assistant.io/t/2024-12-crashes-system-for-many-users/806155