this post was submitted on 15 Jan 2025
82 points (97.7% liked)

Programming

17832 readers
67 users here now

Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!

Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.

Hope you enjoy the instance!

Rules

Rules

  • Follow the programming.dev instance rules
  • Keep content related to programming in some way
  • If you're posting long videos try to add in some form of tldr for those who don't want to watch videos

Wormhole

Follow the wormhole through a path of communities !webdev@programming.dev

founded 2 years ago
MODERATORS
snowe@programming.dev
Ategon@programming.dev
MaungaHikoi@lemmy.nz
82
Am I crazy in thinking that bash is good enough for production? (lemm.ee)
submitted 3 days ago by Badland9085@lemm.ee to c/programming@programming.dev
129 comments fedilink hide all child comments
 

This may make some people pull their hair out, but I’d love to hear some arguments. I’ve had the impression that people really don’t like bash, not from here, but just from people I’ve worked with.

There was a task at work where we wanted something that’ll run on a regular basis, and doesn’t do anything complex aside from reading from the database and sending the output to some web API. Pretty common these days.

I can’t think of a simpler scripting language to use than bash. Here are my reasons:

  • Reading from the environment is easy, and so is falling back to some value; just do ${VAR:-fallback}; no need to write another if-statement to check for nullity. Wanna check if a variable’s set to something expected? if [[ <test goes here> ]]; then <handle>; fi
  • Reading from arguments is also straightforward; instead of a import os; os.args[1] in Python, you just do $1.
  • Sending a file via HTTP as part of an application/x-www-form-urlencoded request is super easy with curl. In most programming languages, you’d have to manually open the file, read them into bytes, before putting it into your request for the http library that you need to import. curl already does all that.
  • Need to read from a curl response and it’s JSON? Reach for jq.
  • Instead of having to set up a connection object/instance to your database, give sqlite, psql, duckdb or whichever cli db client a connection string with your query and be on your way.
  • Shipping is… fairly easy? Especially if docker is common in your infrastructure. Pull Ubuntu or debian or alpine, install your dependencies through the package manager, and you’re good to go. If you stay within Linux and don’t have to deal with differences in bash and core utilities between different OSes (looking at you macOS), and assuming you tried to not to do anything too crazy and bring in necessary dependencies in the form of calling them, it should be fairly portable.

Sure, there can be security vulnerability concerns, but you’d still have to deal with the same problems with your Pythons your Rubies etc.

For most bash gotchas, shellcheck does a great job at warning you about them, and telling how to address those gotchas.

There are probably a bunch of other considerations but I can’t think of them off the top of my head, but I’ve addressed a bunch before.

So what’s the dealeo? What am I missing that may not actually be addressable?

top 50 comments
sorted by: hot top controversial new old
[–] locuester@lemmy.zip 1 points 17 hours ago* (last edited 17 hours ago) (1 children)

What gave you the impression that this was just for development? Bash is widely used in production environments for scripting all over enterprises. The people you work with just don’t have much experience at lots of shops I would think.

It’s just not wise to write an entire system in bash. Just simple little tasks to do quick things. Yes, in production. The devops world runs on bash scripts.

[–] Badland9085@lemm.ee 1 points 10 hours ago (1 children)

I’ve never had that impression, and I know that even large enterprises have Bash scripts essentially supporting a lot of the work of a lot of their employees. But there are also many very loud voices that seems to like screaming that you shouldn’t use Bash almost at all.

You can take a look at the other comments to see how some are entirely turned off by even the idea of using bash, and there aren’t just a few of them.

[–] locuester@lemmy.zip 1 points 8 hours ago (1 children)

This Lemmy thread isn’t representative of the real world. I’ve been a dev for 40 years. You use what works. Bash is a fantastic scripting tool.

[–] Badland9085@lemm.ee 1 points 6 hours ago (1 children)

I understand that. I have coworkers with about 15-20 years in the industry, and they frown whenever I put a bash script out for, say, a purpose that I put in my example: self-contained, clearly defined boundaries, simple, and not mission critical despite handling production data, typically done in less than 100 lines of bash with generous spacing and comments. So I got curious, since I don’t feel like I’ve ever gotten a satisfactory answer.

Thank you for sharing your opinion!

[–] locuester@lemmy.zip 2 points 5 hours ago

My #1 rule for the teams I lead is “consistency”. So it may fall back to that. The standard where you work is to use a certain way of doing things so everyone becomes skilled at the same thing.

I have the same rule, but I always let a little bash slide here and there.

[–] melezhik@programming.dev 13 points 2 days ago* (last edited 2 days ago) (1 children)

We are not taking about use of Bash in dev vs use Bash in production. This is imho incorrect question that skirts around the real problem in software development. We talk about use of Bash for simple enough tasks where code is rarely changed ( if not written once and thrown away ) and where every primitive language or DSL is ok, where when it comes to building of medium or complex size software systems where decomposition, complex data structures support, unit tests, error handling, concurrency, etc is a big of a deal - Bash really sucks because it does not allow one to deal with scaling challenges, by scaling I mean where you need rapidly change huge code base according changes of requirements and still maintain good quality of entire code. Bash is just not designed for that.

[–] Badland9085@lemm.ee 6 points 1 day ago (2 children)

But not everything needs to scale, at least, if you don’t buy into the doctrine that everything has to be designed and written to live forever. If robust, scalable solutions is the nature of your work and there’s nothing else that can exist, then yeah, Bash likely have no place in that world. If you need any kind of handling more complicated than just getting an error and doing something else, then Bash is not it.

Just because Bash isn’t designed for something you want to do, doesn’t mean it sucks. It’s just not the right tool. Just because you don’t practice law, doesn’t mean you suck; you just don’t do law. You can say that you suck at law though.

[–] melezhik@programming.dev 5 points 1 day ago* (last edited 1 day ago)

Yep. Like said - "We talk about use of Bash for simple enough tasks ... where every primitive language or DSL is ok", so Bash does not suck in general and I myself use it a lot in proper domains, but I just do not use it for tasks / domains with complexity ( in all senses, including, but not limited to team work ) growing over time ...

[–] tleb@lemmy.ca 9 points 1 day ago (1 children)

If your company ever has >2 people, it will become a problem.

[–] Badland9085@lemm.ee 1 points 1 day ago

You’re speaking prophetically there and I simply do not agree with that prophecy.

If you and your team think you need to extend that bash script to do more, stop and consider writing it in some other languages. You’ve move the goalpost, so don’t expect that you can just build on your previous strategy and that it’ll work.

If your “problem” stems from “well your colleagues will not likely be able to read or write bash well enough”, well then just don’t write it in bash.

[–] vext01@lemmy.sdf.org 25 points 2 days ago

Honestly, if a script grows to more than a few tens of lines I'm off to a different scripting language because I've written enough shell script to know that it's hard to get right.

Shellcheck is great, but what's greater is a language that doesn't have as many gotchas from the get go.

[–] zygo_histo_morpheus@programming.dev 18 points 2 days ago (2 children)

One thing that I don't think anyone else has mentioned is data structures. Bash does have arrays and hashmaps at least but I've found that working with them is significantly more awkward than in e.g. python. This is one of several reasons for why bash doesn't scale up well, but sure for small enough scripts it can be fine (if you don't care about windows)

[–] syklemil@discuss.tchncs.de 6 points 2 days ago (1 children)

I think I mentioned it, but inverse: The only data type I'm comfortable with in bash are simple string scalars; plus some simple integer handling I suppose. Once I have to think about stuff like "${foo[@]}" and the like I feel like I should've switched languages already.

Plus I rarely actually want arrays, it's way more likely I want something in the shape of

@dataclass(frozen=True)
class Foo:
    # …

foos: set[Foo] = …
[–] lurklurk@lemmy.world 1 points 1 day ago (1 children)

I use the same heuristic... if I need a hashmap or more complex math, I need a different language

Also if the script grows beyond 100 lines, I stop and think about what I'm doing. Sometimes it's OK, but it's a warning flag

[–] syklemil@discuss.tchncs.de 2 points 1 day ago

Yeah agreed on the 100 lines, or some other heuristic in the direction of "this script will likely continue to grow in complexity and I should switch to a language that's better suited to handle that complexity".

load more comments (1 replies)
[–] furrowsofar@beehaw.org 9 points 2 days ago (5 children)

Just make certain the robustness issues of bash do not have security implications. Variable, shell, and path evalutions can have security issues depending on the situation.

load more comments (5 replies)
[–] ShawiniganHandshake@sh.itjust.works 22 points 2 days ago (1 children)

I've worked in bash. I've written tools in bash that ended up having a significant lifetime.

Personally, you lost me at

reading from the database

Database drivers exist for a reason. Shelling out to a database cli interface is full of potential pitfalls that don't exist in any language with a programmatic interface to the database. Dealing with query parameterization in bash sounds un-fun and that's table stakes, security-wise.

Same with making web API calls. Error handling in particular is going to require a lot of boilerplate code that you would get mostly for free in languages like Python or Ruby or Go, especially if there's an existing library that wraps the API you want to use in native language constructs.

load more comments (1 replies)
[–] FizzyOrange@programming.dev 24 points 3 days ago (6 children)

I'm afraid your colleagues are completely right and you are wrong, but it sounds like you genuinely are curious so I'll try to answer.

I think the fundamental thing you're forgetting is robustness. Yes Bash is convenient for making something that works once, in the same way that duct tape is convenient for fixes that work for a bit. But for production use you want something reliable and robust that is going to work all the time.

I suspect you just haven't used Bash enough to hit some of the many many footguns. Or maybe when you did hit them you thought "oops I made a mistake", rather than "this is dumb; I wouldn't have had this issue in a proper programming language".

The main footguns are:

  1. Quoting. Trust me you've got this wrong even with shellcheck. I have too. That's not a criticism. It's basically impossible to get quoting completely right in any vaguely complex Bash script.
  2. Error handling. Sure you can set -e, but then that breaks pipelines and conditionals, and you end up with really monstrous pipelines full of pipefail noise. It's also extremely easy to forget set -e.
  3. General robustness. Bash silently does the wrong thing a lot.

instead of a import os; os.args[1] in Python, you just do $1

No. If it's missing $1 will silently become an empty string. os.args[1] will throw an error. Much more robust.

Sure, there can be security vulnerability concerns, but you’d still have to deal with the same problems with your Pythons your Rubies etc.

Absolutely not. Python is strongly typed, and even statically typed if you want. Light years ahead of Bash's mess. Quoting is pretty easy to get right in Python.

I actually started keeping a list of bugs at work that were caused directly by people using Bash. I'll dig it out tomorrow and give you some real world examples.

[–] lurklurk@lemmy.world 1 points 1 day ago (1 children)

I don't disagree with your point, but how does set -e break conditionals? I use it all the time without issues

Pipefail I don't use as much so perhaps that's the issue?

[–] FizzyOrange@programming.dev 1 points 1 day ago (1 children)

It means that all commands that return a non-zero exit code will fail the script. The problem is that exit codes are a bit overloaded and sometimes non-zero values don't indicate failure, they indicate some kind of status. For example in git diff --exit-code or grep.

I think I was actually thinking of pipefail though. If you don't set it then errors in pipelines are ignored, which is obviously bad. If you do then you can't use grep in pipelines.

[–] lurklurk@lemmy.world 1 points 1 day ago

My sweet spot is set -ue because I like to be able to use things like if grep -q ...; then and I like things to stop if I misspelled a variable.

It does hide failures in the middle of a pipeline, but it's a tradeoff. I guess one could turn it on and off when needed

[–] JamonBear@sh.itjust.works 6 points 2 days ago (3 children)

Agreed.

Also gtfobins is a great resource in addition to shellcheck to try to make secure scripts.

For instance I felt upon a script like this recently:

#!/bin/bash
# ... some stuff ...
tar -caf archive.tar.bz2 "$@"

Quotes are OK, shellcheck is happy, but, according to gtfobins, you can abuse tar, so running the script like this: ./test.sh /dev/null --checkpoint=1 --checkpoint-action=exec=/bin/sh ends up spawning an interactive shell...

So you can add up binaries insanity on top of bash's mess.

[–] lurklurk@lemmy.world 2 points 1 day ago

I imagine adding -- so it becomes tar -caf archive.tar.bz2 -- "$@" would fix that specific case

But yeah, putting bash in a position where it has more rights than the user providing the input is a really bad idea

[–] syklemil@discuss.tchncs.de 1 points 1 day ago

Quotes are OK, shellcheck is happy, but, according to gtfobins, you can abuse tar, so running the script like this: ./test.sh /dev/null --checkpoint=1 --checkpoint-action=exec=/bin/sh ends up spawning an interactive shell…

This runs into a part of the unix philosophy about doing one thing and doing it well: Extending programs to have more (absolutely useful) functionality winds up becoming a security risk. The shell is generally geared towards being a collection of shortcuts rather than a normal, predictable but tedious API.

For a script like that you'd generally want to validate that the input is actually what you expect if it needs to handle hostile users, though. It'll likely help the sleepy users too.

load more comments (1 replies)
load more comments (4 replies)
[–] Die4Ever@programming.dev 42 points 3 days ago (8 children)

I just don't think bash is good for maintaining the code, debugging, growing the code over time, adding automated tests, or exception handling

load more comments (8 replies)
[–] jollyroberts@jolly-piefed.jomandoa.net 39 points 3 days ago

"Use the best tool for the job, that the person doing the job is best at." That's my approach.

I will use bash or python dart or whatever the project uses.

[–] toynbee@lemmy.world 9 points 2 days ago

Over the last ten - fifteen years, I've written lots of scripts for production in bash. They've all served their purposes (after thorough testing) and not failed. Pretty sure one of my oldest (and biggest) is called temporary_fixes.sh and is still in use today. Another one (admittedly not in production) was partially responsible for getting me my current job, I guess because the interviewers wanted to see what kind of person would solve a coding challenge in bash.

However, I would generally agree that - while bash is good for many things and perhaps even "good enough" - any moderately complex problem is probably better solved using a different language.

[–] syklemil@discuss.tchncs.de 13 points 3 days ago (5 children)

At the level you're describing it's fine. Preferably use shellcheck and set -euo pipefail to make it more normal.

But once I have any of:

  • nested control structures, or
  • multiple functions, or
  • have to think about handling anything else than simple strings that other programs manipulate (including thinking about bash arrays or IFS), or
  • bash scoping,
  • producing my own formatted logs at different log levels,

I'm on to Python or something else. It's better to get off bash before you have to juggle complexity in it.

[–] sabin@lemmy.world 2 points 1 day ago

Set don't forget set -E as well to exit on failed subshells.

[–] 0x0@lemmy.dbzer0.com 3 points 1 day ago

If you're writing a lot of shell scripts and checking them with Shellcheck, and you're still convinced that it's totally safe... I tip my hat to you.

load more comments (3 replies)
[–] synae@lemmy.sdf.org 5 points 2 days ago* (last edited 2 days ago) (1 children)

As I've matured in my career, I write more and more bash. It is absolutely appropriate for production in the right scenarios. Just make sure the people who might have to maintain it in the future won't come knocking down your door with torches and pitchforks...

load more comments (1 replies)
load more comments
view more: next ›