398 lines
19 KiB
Markdown
398 lines
19 KiB
Markdown
---
|
||
tags:
|
||
- freebsd
|
||
- homelab
|
||
- selfhosting
|
||
---
|
||
Environment:
|
||
* FreeBSD 14.1-RELEASE-p3 AMD64
|
||
* Python 3.11.9
|
||
* Home Assistant 2024.3.3
|
||
|
||
I have encountered several project now which are Linux container/docker "native" installation. Like PhotoPrism and HomeAssistant. Yeah software dependencies are hard for developers and maintainers. But ending your software on just one install method is a "quick-fix". Apple macOS tries to "fix this" by putting everything of an application in a folder suffixed as `.app`. No this are no containers but all application dependencies reside in this folder. The price you pay is more disk space when multiple applications use same "Apple .framework" or libraries and other executables.
|
||
|
||
It seems nowadays the quick-fix is to enforce people using Docker for production and when that is not available it will be wrapped again in a native Linux virtual machine. So you are almost out of luck running it otherwise. Gitlab is another beast, it runs on ruby and it has "omnibus" host installer. Which just basicly Gitlab-fies your whole OS.
|
||
|
||
Why on earth do people think we can get rid of ALL software written in C/C++ converted to the promised better language Rust. As Python is executing `cc` ?
|
||
|
||
```
|
||
last pid: 5386; load averages: 1.48, 1.51, 1.61 up 35+20:47:26 17:47:34
|
||
14 processes: 2 running, 12 sleeping
|
||
CPU: 24.9% user, 20.4% nice, 6.5% system, 0.0% interrupt, 48.2% idle
|
||
Mem: 810M Active, 773M Inact, 506M Laundry, 1627M Wired, 385M Buf, 111M Free
|
||
ARC: 282M Total, 98M MFU, 50M MRU, 599K Anon, 6378K Header, 123M Other
|
||
75M Compressed, 415M Uncompressed, 5.57:1 Ratio
|
||
Swap: 2689M Total, 500M Used, 2190M Free, 18% Inuse
|
||
|
||
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
|
||
4564 homeassist 1 135 0 239M 159M CPU1 1 0:41 99.88% cc
|
||
78755 homeassist 1 20 0 17M 5072K select 1 0:01 0.20% tmux
|
||
2726 homeassist 1 20 0 14M 2980K CPU2 2 0:00 0.17% top
|
||
96466 homeassist 10 20 0 183M 115M kqread 0 0:06 0.03% python3.11
|
||
674 homeassist 1 68 0 208M 180M wait 0 1:43 0.00% python3.11
|
||
97272 homeassist 1 20 0 99M 77M piperd 1 0:05 0.00% python3.11
|
||
99599 homeassist 1 68 0 57M 40M wait 0 0:01 0.00% python3.11
|
||
25 homeassist 1 21 0 16M 3616K pause 2 0:00 0.00% csh
|
||
89135 homeassist 1 20 0 13M 2612K wait 2 0:00 0.00% sh
|
||
81771 homeassist 1 20 0 13M 2664K ttyin 1 0:00 0.00% sh
|
||
79030 homeassist 1 63 0 13M 2672K wait 0 0:00 0.00% sh
|
||
1361 homeassist 1 24 0 13M 2752K wait 3 0:00 0.00% sh
|
||
98234 homeassist 1 20 0 13M 2656K ttyin 1 0:00 0.00% sh
|
||
81501 homeassist 1 20 0 14M 3644K select 1 0:00 0.00% tmux
|
||
```
|
||
|
||
The "modern" software landscape is getting messy!
|
||
|
||
It is such a mess to run HomeAssistant native on FreeBSD you need to install some binary packages with `pkg` from FreeBSD binary ports. Then Python `pip` install some extra packages in a virtual environment and finally the application Python runs the `cc` (c compiler) itself! Even using `meson` build system as intermediate. We need Python `cryptography` dependency to use the `Rust` compiler as the devs have decided to not proceed with (accelerated) crypto implemented in C. I think if you further want to get into the dependency hell we need Library X written in C++ and wrap it under Python. Then a C++ runtime is also needed...
|
||
|
||
https://dan.langille.org/2022/08/27/getting-home-assistant-running-in-a-freebsd-13-1-jail/
|
||
|
||
Run the stuffs in venv:
|
||
`(homeassistant) homeassistant@mango:/data/homeassistant % ./bin/hass --ignore-os-check -v -v -v -v`
|
||
|
||
https://devhumor.com/media/monkeyuser-npm-package-delivery
|
||
|
||
https://developer.apple.com/library/archive/documentation/General/Conceptual/DevPedia-CocoaCore/Framework.html
|
||
|
||
ImportError: /data/homeassistant/lib/python3.11/site-packages/numpy/core/_multiarray_tests.cpython-311.so: Undefined symbol "ccosl"
|
||
|
||
https://github.com/numpy/numpy/issues/24873
|
||
|
||
```
|
||
note: This error originates from a subprocess, and is likely not a problem with pip.
|
||
2024-11-08 19:00:58.632 INFO (SyncWorker_5) [homeassistant.util.package] Attempting install of webrtc-noise-gain==1.2.3
|
||
2024-11-08 19:01:13.568 ERROR (SyncWorker_5) [homeassistant.util.package] Unable to install package webrtc-noise-gain==1.2.3: error: subprocess-exited-with-error
|
||
|
||
× Getting requirements to build wheel did not run successfully.
|
||
│ exit code: 1
|
||
╰─> [20 lines of output]
|
||
Traceback (most recent call last):
|
||
File "/data/homeassistant/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
|
||
main()
|
||
File "/data/homeassistant/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
|
||
json_out['return_val'] = hook(**hook_input['kwargs'])
|
||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||
File "/data/homeassistant/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
|
||
return hook(config_settings)
|
||
^^^^^^^^^^^^^^^^^^^^^
|
||
File "/tmp/pip-build-env-ih3jdwtb/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 333, in get_requires_for_build_wheel
|
||
return self._get_build_requires(config_settings, requirements=[])
|
||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||
File "/tmp/pip-build-env-ih3jdwtb/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 303, in _get_build_requires
|
||
self.run_setup()
|
||
File "/tmp/pip-build-env-ih3jdwtb/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 521, in run_setup
|
||
super().run_setup(setup_script=setup_script)
|
||
File "/tmp/pip-build-env-ih3jdwtb/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 319, in run_setup
|
||
exec(code, locals())
|
||
File "<string>", line 384, in <module>
|
||
ValueError: Unsupported system: freebsd
|
||
[end of output]
|
||
|
||
note: This error originates from a subprocess, and is likely not a problem with pip.
|
||
error: subprocess-exited-with-error
|
||
```
|
||
|
||
```
|
||
2024-11-08 19:02:25.055 ERROR (SyncWorker_0) [homeassistant.util.package] Unable to install package ha-av==10.1.1: error: subprocess-exited-with-error
|
||
|
||
× Getting requirements to build wheel did not run successfully.
|
||
│ exit code: 1
|
||
╰─> [11 lines of output]
|
||
Package libavformat was not found in the pkg-config search path.
|
||
Perhaps you should add the directory containing `libavformat.pc'
|
||
to the PKG_CONFIG_PATH environment variable
|
||
Package 'libavformat' not found
|
||
Package 'libavcodec' not found
|
||
Package 'libavdevice' not found
|
||
Package 'libavutil' not found
|
||
Package 'libavfilter' not found
|
||
Package 'libswscale' not found
|
||
Package 'libswresample' not found
|
||
pkg-config could not find libraries ['avformat', 'avcodec', 'avdevice', 'avutil', 'avfilter', 'swscale', 'swresample']
|
||
[end of output]
|
||
|
||
note: This error originates from a subprocess, and is likely not a problem with pip.
|
||
error: subprocess-exited-with-error
|
||
```
|
||
|
||
```
|
||
pkg install py311-ffmpeg-python-0.2.0_1
|
||
```
|
||
|
||
```
|
||
2024-11-08 19:06:55.875 ERROR (SyncWorker_5) [homeassistant.util.package] Unable to install package webrtc-noise-gain==1.2.3: error: subprocess-exited-with-error
|
||
|
||
× Getting requirements to build wheel did not run successfully.
|
||
│ exit code: 1
|
||
╰─> [20 lines of output]
|
||
Traceback (most recent call last):
|
||
File "/data/homeassistant/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
|
||
main()
|
||
File "/data/homeassistant/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
|
||
json_out['return_val'] = hook(**hook_input['kwargs'])
|
||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||
File "/data/homeassistant/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
|
||
return hook(config_settings)
|
||
^^^^^^^^^^^^^^^^^^^^^
|
||
File "/tmp/pip-build-env-i8fc26m0/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 333, in get_requires_for_build_wheel
|
||
return self._get_build_requires(config_settings, requirements=[])
|
||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||
File "/tmp/pip-build-env-i8fc26m0/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 303, in _get_build_requires
|
||
self.run_setup()
|
||
File "/tmp/pip-build-env-i8fc26m0/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 521, in run_setup
|
||
super().run_setup(setup_script=setup_script)
|
||
File "/tmp/pip-build-env-i8fc26m0/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 319, in run_setup
|
||
exec(code, locals())
|
||
File "<string>", line 384, in <module>
|
||
ValueError: Unsupported system: freebsd
|
||
[end of output]
|
||
|
||
note: This error originates from a subprocess, and is likely not a problem with pip.
|
||
error: subprocess-exited-with-error
|
||
```
|
||
|
||
```
|
||
ImportError: Error importing numpy: you should not try to import numpy from
|
||
its source directory; please exit the numpy source tree, and relaunch
|
||
your python interpreter from there.
|
||
```
|
||
|
||
```
|
||
Great Thanks [@LukaszMoskala](https://github.com/LukaszMoskala),
|
||
For Android Termux :
|
||
|
||
# outside of homeassistant venv
|
||
|
||
git clone [https://github.com/numpy/numpy.git](https://github.com/numpy/numpy.git) numpy-git
|
||
cd numpy-git
|
||
git checkout v1.26.0
|
||
git cherry-pick [040ed2d](https://github.com/numpy/numpy/commit/040ed2dc9847265c581a342301dd87d2b518a3c2)
|
||
git submodule update --init
|
||
cd ..
|
||
|
||
# now enter homeassistant venv
|
||
|
||
LDFLAGS=" -lm -lpython3" pip install numpy-git/
|
||
```
|
||
https://github.com/numpy/numpy/issues/24873#issuecomment-1807458661
|
||
|
||
`setenv LDFLAGS "-lm -lpython3" ; pip install --config-settings=builddir=build dl/numpy-git-build-freebsd-14.1 dl/numpy-git/`
|
||
|
||
```
|
||
(homeassistant) homeassistant@mango:/data/homeassistant % setenv LDFLAGS "-lm -lpython3" ; pip install dl/numpy-git/
|
||
Processing ./dl/numpy-git
|
||
Installing build dependencies ... done
|
||
Getting requirements to build wheel ... done
|
||
Preparing metadata (pyproject.toml) ... error
|
||
error: subprocess-exited-with-error
|
||
|
||
× Preparing metadata (pyproject.toml) did not run successfully.
|
||
│ exit code: 1
|
||
╰─> [12 lines of output]
|
||
+ /data/homeassistant/bin/python /data/homeassistant/dl/numpy-git/vendored-meson/meson/meson.py setup /data/homeassistant/dl/numpy-git /data/homeassistant/dl/numpy-git/.mesonpy-8palx956/build -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/data/homeassistant/dl/numpy-git/.mesonpy-8palx956/build/meson-python-native-file.ini
|
||
The Meson build system
|
||
Version: 1.2.99
|
||
Source dir: /data/homeassistant/dl/numpy-git
|
||
Build dir: /data/homeassistant/dl/numpy-git/.mesonpy-8palx956/build
|
||
Build type: native build
|
||
Project name: NumPy
|
||
Project version: 1.26.0
|
||
|
||
../../meson.build:1:0: ERROR: Compiler cc cannot compile programs.
|
||
|
||
A full log can be found at /data/homeassistant/dl/numpy-git/.mesonpy-8palx956/build/meson-logs/meson-log.txt
|
||
[end of output]
|
||
|
||
note: This error originates from a subprocess, and is likely not a problem with pip.
|
||
error: metadata-generation-failed
|
||
|
||
× Encountered error while generating package metadata.
|
||
╰─> See above for output.
|
||
|
||
note: This is an issue with the package mentioned above, not pip.
|
||
hint: See above for details.
|
||
```
|
||
|
||
Okay so it means `meson` build system is also written in Python....
|
||
|
||
https://github.com/tprelog/iocage-homeassistant/issues/64#issuecomment-1936432832
|
||
|
||
Thanks meson and pip for throwing away the NEEDED log:
|
||
```
|
||
(homeassistant) homeassistant@mango:/data/homeassistant % cat /data/homeassistant/dl/numpy-git/.mesonpy-4rpzl4ag/build/meson-logs/meson-log.txt
|
||
cat: /data/homeassistant/dl/numpy-git/.mesonpy-4rpzl4ag/build/meson-logs/meson-log.txt: No such file or directory
|
||
```
|
||
|
||
|
||
Follow-up: the way to not clean up the build dir (and hence the log file) is:
|
||
|
||
```
|
||
pip install . --config-settings=builddir=build
|
||
```
|
||
|
||
https://github.com/mesonbuild/meson-python/issues/246#issuecomment-1354426272
|
||
|
||
So probably:
|
||
|
||
```
|
||
setenv LDFLAGS "-lm -lpython3" ; pip install --config-settings=builddir=dl/numpy-git-build-freebsd-14.1 dl/numpy-git/
|
||
```
|
||
|
||
Log is preserved...
|
||
```
|
||
A full log can be found at /data/homeassistant/dl/numpy-git/dl/numpy-git-build-freebsd-14.1/meson-logs/meson-log.txt
|
||
[end of output]
|
||
```
|
||
|
||
```
|
||
-----
|
||
Sanity check compile stderr:
|
||
ld: error: unable to find library -lpython3
|
||
cc: error: linker command failed with exit code 1 (use -v to see invocation)
|
||
|
||
-----
|
||
|
||
../../meson.build:1:0: ERROR: Compiler cc cannot compile programs.
|
||
```
|
||
|
||
Will get then
|
||
```
|
||
setenv LDFLAGS "-lm -lpython3.11" ; pip install --config-settings=builddir=dl/numpy-git-build-freebsd-14.1 dl/numpy-git/
|
||
```
|
||
|
||
Due too shared object (.so) naming:
|
||
```
|
||
(homeassistant) homeassistant@mango:/data/homeassistant % pkg list python311 | grep libpython
|
||
/usr/local/lib/libpython3.11.so
|
||
/usr/local/lib/libpython3.11.so.1
|
||
/usr/local/lib/libpython3.11.so.1.0
|
||
/usr/local/lib/libpython3.11.so.1.0-gdb.py
|
||
/usr/local/lib/python3.11/config-3.11/libpython3.11.a
|
||
```
|
||
|
||
Still could not find it try adding `/usr/local/lib` search path:
|
||
```
|
||
(homeassistant) homeassistant@mango:/data/homeassistant % setenv LDFLAGS "-lm -lpython3.11 -L/usr/local/lib" ; pip install --config-settings=builddir=dl/numpy-git-build-freebsd-14.1 dl/numpy-git/
|
||
```
|
||
|
||
Before installation with this cherry-picked numpy fix we need to uninstall numpy:
|
||
```
|
||
(homeassistant) homeassistant@mango:/data/homeassistant % pip uninstall numpy
|
||
Found existing installation: numpy 1.26.0
|
||
Uninstalling numpy-1.26.0:
|
||
Would remove:
|
||
/data/homeassistant/bin/f2py
|
||
/data/homeassistant/lib/python3.11/site-packages/numpy-1.26.0.dist-info/*
|
||
/data/homeassistant/lib/python3.11/site-packages/numpy/*
|
||
Proceed (Y/n)? y
|
||
Successfully uninstalled numpy-1.26.0
|
||
```
|
||
|
||
Finally homebrew patched numpy
|
||
|
||
```
|
||
Processing ./dl/numpy-git
|
||
Installing build dependencies ... done
|
||
Getting requirements to build wheel ... done
|
||
Preparing metadata (pyproject.toml) ... done
|
||
Building wheels for collected packages: numpy
|
||
Building wheel for numpy (pyproject.toml) ... done
|
||
Created wheel for numpy: filename=numpy-1.26.0-cp311-cp311-freebsd_14_1_RELEASE_p3_amd64.whl size=24156181 sha256=165ac15d3ddf5d2336520934b1321b4eb26074ddb11205398b450375693d56ba
|
||
Stored in directory: /home/homeassistant/.cache/pip/wheels/11/b7/76/343a9fc71c70dd4931749d568222522502d6af19e277d2e1d6
|
||
Successfully built numpy
|
||
Installing collected packages: numpy
|
||
Successfully installed numpy-1.26.0
|
||
```
|
||
|
||
Need to fix `webrtc-noise-gain==1.2.3` dependency probably
|
||
|
||
```
|
||
2024-11-08 19:42:54.330 INFO (MainThread) [homeassistant.requirements] Multiple attempts to install webrtc-noise-gain==1.2.3 failed, install will be retried after next configuration check or restart
|
||
2024-11-08 19:42:54.330 ERROR (MainThread) [homeassistant.setup] Setup failed for 'mobile_app': Requirements for assist_pipeline not found: ['webrtc-noise-gain==1.2.3'].
|
||
2024-11-08 19:42:54.339 INFO (MainThread) [homeassistant.setup] Setup of domain tts took 0.1 seconds
|
||
2024-11-08 19:42:54.351 INFO (MainThread) [homeassistant.requirements] Multiple attempts to install webrtc-noise-gain==1.2.3 failed, install will be retried after next configuration check or restart
|
||
2024-11-08 19:42:54.351 ERROR (MainThread) [homeassistant.setup] Setup failed for 'assist_pipeline': Requirements for assist_pipeline not found: ['webrtc-noise-gain==1.2.3'].
|
||
```
|
||
|
||
Interesting is
|
||
```
|
||
2024-11-08 19:42:54.330 ERROR (MainThread) [homeassistant.setup] Setup failed for 'mobile_app': Requirements for assist_pipeline not found: ['webrtc-noise-gain==1.2.3'].
|
||
```
|
||
|
||
https://www.reddit.com/r/termux/comments/178kt7e/need_help_installing_webrtcnoisegain_for_home/
|
||
|
||
Setuptools is also calling the `cc` ?? Really
|
||
|
||
Needed for iOS app....
|
||
|
||
In `~/.homeassistant/configuration.yaml` we need to add to root `# Mobile application mobile_app:` to enable iOS login.
|
||
|
||
Thanks again for more dependency hell when almost everything is "working":
|
||
|
||
```
|
||
2024-11-08 19:48:52.382 WARNING (MainThread) [homeassistant.bootstrap] Support for the running Python version 3.11.9 is deprecated and will be removed in Home Assistant 2024.4; Please upgrade Python to 3.12
|
||
```
|
||
|
||
Thanks...
|
||
https://github.com/home-assistant/core/issues/102645#issuecomment-1792234494
|
||
|
||
```
|
||
(homeassistant) homeassistant@mango:/data/homeassistant/build % pip install --config-settings=builddir=webrtc webrtc-noise-gain==1.2.3
|
||
Collecting webrtc-noise-gain==1.2.3
|
||
```
|
||
|
||
```
|
||
(homeassistant) homeassistant@mango:/data/homeassistant/build % pip install --config-settings=builddir=webrtc webrtc-noise-gain==1.2.5
|
||
Collecting webrtc-noise-gain==1.2.5
|
||
Downloading webrtc_noise_gain-1.2.5.tar.gz (3.1 MB)
|
||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.1/3.1 MB 7.9 MB/s eta 0:00:00
|
||
Installing build dependencies ... done
|
||
Getting requirements to build wheel ... done
|
||
Preparing metadata (pyproject.toml) ... done
|
||
Building wheels for collected packages: webrtc-noise-gain
|
||
Building wheel for webrtc-noise-gain (pyproject.toml) ... \
|
||
```
|
||
|
||
Hard pinned to version:
|
||
|
||
```
|
||
root@mango:/data/homeassistant # ag webrtc-noise-gain
|
||
lib/python3.11/site-packages/homeassistant/package_constraints.txt
|
||
62:webrtc-noise-gain==1.2.3
|
||
|
||
lib/python3.11/site-packages/homeassistant/components/assist_pipeline/manifest.json
|
||
9: "requirements": ["webrtc-noise-gain==1.2.3"]
|
||
```
|
||
|
||
```
|
||
(homeassistant) homeassistant@mango:/data/homeassistant % cp lib/python3.11/site-packages/homeassistant/package_constraints.txt lib/python3.11/site-packages/homeassistant/package_constraints.txt.orig
|
||
```
|
||
|
||
https://www.reddit.com/r/termux/comments/178kt7e/need_help_installing_webrtcnoisegain_for_home/
|
||
|
||
Seems not to work
|
||
|
||
```
|
||
2024-11-08 20:18:12.020 INFO (SyncWorker_4) [homeassistant.util.package] Attempting install of webrtc-noise-gain==1.2.3
|
||
2024-11-08 20:18:14.406 ERROR (SyncWorker_4) [homeassistant.util.package] Unable to install package webrtc-noise-gain==1.2.3: ERROR: Cannot install webrtc-noise-gain==1.2.3 becaus
|
||
e these package versions have conflicting dependencies.
|
||
```
|
||
|
||
Inside the HA assist_pipeline the dependency is also pinned to exact version
|
||
```
|
||
(homeassistant) homeassistant@mango:/data/homeassistant % cp lib/python3.11/site-packages/homeassistant/components/assist_pipeline/manifest.json lib/python3.11/site-packages/homeassistant/components/assist_pipeline/manifest.json.orig
|
||
(homeassistant) homeassistant@mango:/data/homeassistant % vim lib/python3.11/site-packages/homeassistant/components/assist_pipeline/manifest.json
|
||
vim: Command not found.
|
||
(homeassistant) homeassistant@mango:/data/homeassistant % nvim lib/python3.11/site-packages/homeassistant/components/assist_pipeline/manifest.json
|
||
```
|
||
|
||
Hacking it to use which one compiled: `webrtc-noise-gain==1.2.5`
|
||
|
||
|
||
```
|
||
$ cd /data/homeassistant
|
||
$ ./bin/hass --ignore-os-check -v -v -v -v
|
||
```
|
||
|
||
TODO:
|
||
https://github.com/davidjb/home-assistant-rc.d/blob/master/homeassistant |