• 0 Posts
  • 386 Comments
Joined 3 years ago
cake
Cake day: June 16th, 2023

help-circle








  • Hardware I’m running:

    • 8/16 AMD CPU
    • 32GB system RAM
    • 12GB GPU VRAM (AMD)

    I’m mainly using these MoE models:

    Qwen-3.6-35B-A3B

    • Q4_K_M quant quality
    • 128k context (conversation length before it compacts)
    • Gives me about 260 prefill and 19 token gen speeds

    Gemma4-26B-A4B

    • Q8 quant quality
    • 128k context length
    • Gives me about 190 prefill and 14 token gen speeds

  • 9tr6gyp3@lemmy.worldtoSelfhosted@lemmy.world•how do you manage your server?
    link
    fedilink
    English
    arrow-up
    14
    arrow-down
    3
    ·
    edit-2
    20 days ago

    I created git repos on my main workstation for each homelab server/service I maintain that keeps:

    • documentation
    • notes
    • lessons learned
    • scripts, configs
    • runbooks
    • backup details
    • security audit details
    • log items that need attention
    • infrastructure

    I just point a local LLM (offline model that runs on my workststion) into those repos and ask it to perform certain things on those servers. It can do things like update packages, install packages, make config changes, set/check permissions, read logs (and fix errors in real time), and check the health of the overall system.

    I have it run pre backups before making changes, then post backups once its done.

    Once changes are in place and everything is running okay, I ask it to update documentation in the repo and tag the release.

    I use opencode that connects to a llama.cpp service. opencode lets me gate the AI so that any elevated commands that it needs to run (e.g. sudo or ssh), I have to approve it. It cant just go around making changes without permission.






  • 9tr6gyp3@lemmy.worldtoLinux@lemmy.world•Proposed KDE LLM guidelines
    link
    fedilink
    English
    arrow-up
    2
    arrow-down
    1
    ·
    25 days ago

    Those are good things to ask prior to using it, but thats just the beginning.

    • If the training data matches what you would look up on the internet anyway, is it a deal breaker?
    • If it cannt be used for production, what about for organizational purposes? Documentation? Resource tracking? Upstream changes? Merging considerations? Security auditing?
    • It sounds like you would rather do the coding part, correct me if I’m wrong. This way you have full control of the license. For someone who isn’t a coder, they would probably have to consider that the code it generates could closely resemble an existing licensed work since the models are trained on open-source data.
    • I think the getting dumber argument is going to depend on the user, and its not an LLM issue specifically. You could definitely bypass the thinking process, but you could also make it explain things step by step, which could also accelerate learning stuff.

    But you’re right, to each their own.