I use Manjaro Linux with the Cinnamon desktop and sometimes run into system-level issues, but I have no idea how to properly debug them. It doesn’t feel as straightforward as debugging a normal program. What’s the best way or resource to learn system debugging on Linux?

  • DigitalDilemma@lemmy.mlEnglish
    3·
    21 days ago

    Sysadmin here, this is my usual flow for various distros

    1. as /u/FigMcLargeHuge mentions, recent logfiles in /var/log. Notably /var/log/messages (EL) and syslog (Debian) but anything that’s recent.

    2. journalctl - More and more things are moving to binary logging. If you know the process, then journalctl -u processname restricts to just that. also add a -f for tailing it for ongoing logs.

    3. dmesg -T - especially at system level, this captures any hardware/low level logs. (-T reports actual times, not just seconds since boot)

    4. Once you have some logs that you think are related, but don’t know WTF they actually mean, you have two options. The first is to google likely strings. This is… ineffective much of the time - accidental misinformation and outdated advice is increasingly common. The answer might be there, but it takes time and can be frustrating to weed out the cruft.

    The better way, (IMO, and people downvote me for saying this) is to use AI. Get a few lines of logs with the errors, check them for confidential information, and simply paste the suspect lines into chatgpt, gemini, claude, co-pilot, whatever. No need for context, it’ll figure that out. The LLM will, 4 times out of 5, identify the problem very quickly.

    Now, once it’s identified that, it will offer to fix it for you. This is where you’ve got to be on your toes as LLMs are really really quick to give bad advice at this level. But that first triage is nearly always worth doing and helps shape your own mind as to what’s going on. AI is still useful for fixing it, but do understand what it’s telling you to do.

    • Holytimes@sh.itjust.works
      2·
      19 days ago

      If there’s one thing AI is good at is trawling though a hell scape of random data from across time and space. And outputting… Something reasonably close.

      Least close enough to actually put you in the right path 9 times out of 10. Even if the ai is wrong, it’s still vastly more time effective then going to stack overflow for this was solved with no solution present.

      Hell I had a issue trying to find documentation for a old game I was modding from 10 years ago that maybe sold 1000 copies the wiki was dead the devs site was dead and the way back machine didn’t have the pages I needed.

      Some fucking how chat gpt knew the game, was able to identify it correctly from a script I gave it then spat out the exact correct context and documentation I needed.

      It was flabbergasting. Cause at the same time it told me the game came out in 1995 when it came out in 2015… It got basic ass info wrong but the technical details right.

      I was so fucking confused how it managed that.

    • BCsven@lemmy.ca
      1·
      21 days ago

      I have resorted to the AI step also, if Stract.com doesn’t give me a good link, because if I paste a minidlna crash log Google responds with:

      • Mini Cooper on sale
      • Buy your DAC device here
      • want to sign up to streaming music
      • network and NAS comparisons

      Useless.

      At least AI said: based on your error it appears a file in your database has metdata tags it cannot parse properly. Sure enough the tagger I used had applied a tag to a wmv file and Minidlna couldn’t deal with tag1 area vs tag 2 areas used in other file formats.