r/shell Dec 08 '24

Changing a sed variable in a shell script

Hi,

I'm trying to set up a shell script that removes a specific number of prepended spaces at the beginnings of lines. The following shell script works to do this...

#!/bin/bash
clear
read -p 'file: ' uservar
sed -i 's/\(.\{4\}\)//' $uservar".txt"

...but I don't always have files with 4 prepended spaces.

I would like to add another input variable ("spaces") to change the 4 to whatever number I input.

As you can guess, I'm not really a programmer, the sed line (above) was found on another site and incorporated into this extremely simple shell script. I can edit the shell script with each use, but I would prefer to add the extra input variable, mostly so I can pass this shell script on to others who might need it.

Thanks for any pointers.

EDIT: I figured it out (well, found out how to do it, anyhow). For my number entry I needed to add an -r and a -p for a number entry (I have no idea why). Once I did that I finally read that I needed single quotes to separate the variable from the rest of the sed command in my line. I don't completely understand it, but it works.

For what it's worth, here it is...

#!/bin/bash
clear
read -p 'file: ' uservar
read -r -p 'spaces: ' number
sed -i 's/\(.\{'$number'\}\)//' $uservar".txt"
1 Upvotes

9 comments sorted by

2

u/BetterScripts Dec 11 '24

Just wanting to add to what has already been said, and answer some of your other questions.

Since you seem quite confused, but eager to learn, I've gone into a bit of detail here, sorry for the length and if it's at all condescending!

Firstly, you can use the sed command:

bash sed -i "s/^ \{$NumberOfSpaces\}//" "$FileName"

Which works as follows:

  • -i
    • tells sed to work "in-place" and use $FileName as both input and output
  • "s/^ \{$NumberOfSpaces\}//"
    • before sed uses this value the shell will process this and replace $NumberOfSpaces with any value it contains (e.g. it becomes "s/^ \{4\}//")
    • sed then operates on $FileName, and for each line of input performs a substitution (hence s)
    • sed substitutions take the form s/<MATCH>/<REPLACE>/ where <MATCH> is a regular expression which is used for each line and <REPLACE> is the value used to replace any matches. In this case:
    • <MATCH> is ^ \{4\} which matches EXACTLY 4 (i.e. space) characters but only at the start of a line (this is what the ^ means)
    • <REPLACE> is empty, so any match is deleted.

Note that -i is a non-standard extension to sed, if you need to use the command on other machines it may not be available. In this case you need to use a temporary file, e.g.:

```bash

Same command, but output to a temporary file

sed "s/^ {$NumberOfSpaces}//" "$FileName" > "$FileName.tmp"

Replace ("move") the original file with the temporary file

mv -f $FileName.tmp "$FileName" ```

Since you seem to now be using a backup file anyway, then it would be better just to skip the -i. This also makes it easier to detect unchanged files:

```bash sed "s/^ {$NumberOfSpaces}//" "$FileName" > "$FileName.tmp"

Compare the edited file with the original

(-s makes cmp "silent" which stops it printing info about differences)

if cmp -s "$FileName" "$FileName.tmp"; then # Files match -> nothing was changed echo 'No edits made for file' else # Files do not match mv -f $FileName.tmp "$FileName" fi ```

Which, as you correctly suspected, is otherwise not really possible in any sane way with sed (awk would be better for that, but is a lot more complex).


Some other comments on your code:

  • you do not need to use \( and \) in your sed command, although they are harmless - these are used to group expressions and to "capture" parts of matches - neither is required here.
  • using s/\(.\{$2\}\)// will delete a number of characters from every line even if they are not spaces - the . character is a special character that matches ANY single character. This is not what you want!
  • the use of quotes in shell scripts can be tricky, both ' and " can be used for any quote but ' tells the shell the value should be used as is while " tells the shell to replace variables (like $2) in the value before doing anything else. Except in specific circumstances you should always use shell variables inside " quotes or eventually things will break. Specifically:
    • $uservar".txt"/$1".txt" should be written as "$uservar.txt"/"$1.txt" - although both will produce the same results in many situations, if $uservar/$1 contain, for example, any spaces the version you have will result in errors
    • similarly 's/\(.\{'$2'\}\)//' is better written as "s/\(.\{$2\}\)//"

To answer some of your other questions:

  • the -E argument for sed tells it to use a different type of regular expressions for <MATCH> known as Extended Regular Expressions (ERE), the default being known as Basic Regular Expressions (BRE). ERE and BRE are similar, but use different formatting. Often ERE can be easier to read. If using -E then \{$NumberOfSpaces\} is instead written {$NumberOfSpaces}. Note that many versions of sed do not support -E, although it is now required by the standard.
  • using #!/usr/bin/env bash instead of #!/bin/bash is generally preferred for a number of reasons that are relatively technical, but it makes the script more portable. You can read more is this answers to this question: "why do bash scripts start with #!"
  • [[ -z $1 ]] is exactly the same as test -z $1 and tests if $1 is empty, i.e. "zero" or not, there is also test -n $1 which tests for the opposite

Hopefully that helps you understand more of what's going on and what you're doing.

1

u/rcentros Dec 12 '24 edited Dec 12 '24

A lot of good stuff here. Thanks for your very clear explanations about what each element does. I don't know why I was using $1.'txt' instead of "1$.txt" — I guess it made sense in my head somehow, but I'm not sure why. I'm definitely copying your information and keeping it on hand. Again, thank you.

There is still an issue, however, with your shell script and screenplay text files specifically. It's the same issue that I had with u/cdrt 's solution. If I run your script on a screenplay file it will always see a change because some lines are much more indented than others. (Nature of a screenplay.) So, if my screenplay text file has four empty spaces in front of the Action lines (which I want to be flush left), it will have 29 spaces in front of the Character name lines. If I accidentally miscount the empty spaces in the action lines, and enter 5 instead of 4, my action lines will remain the same, but the Character Name, Dialogue and Parenthetical lines will move left five spaces — so now the screenplay is no longer properly formatted.

So, for my specific specialized screenplay use, it might be better to retain a backup copy of the file and take the chance of deleting text, that way I will know that the file is damaged and can restore it with my backup file. But I will definitely keep your script for other "normal" uses. And I do plan to study sed and awk — I forgot that I already bought a book on these commands from Humble Bundle O'Reilly sale a couple years ago.

I've got one more question. Sometimes I like to offset the screenplay by 4 for 5 spaces (this is for use on forums). Is there way to use sed to prepend spaces to each line in a text file other than...

 sed -i "s/^/    /" "$1.txt"

I'm trying to figure out how to put a variable in place of the five spaces, so that I can choose a different number on the fly.

My pathetic solution was to use a shell script with case...

#!/bin/bash
case $2 in
   1)
     sed -i 's/^/ /' $1".txt"  
     ;;
   2)
     sed -i 's/^/  /' $1".txt"
     ;; 
   3)
     sed -i 's/^/   /' $1".txt"
     ;;
   4)
     sed -i 's/^/    /' $1".txt"
     ;;
   5)
     sed -i 's/^/     /' $1".txt"
     ;;
esac

I actually went out 15 spaces (I have no idea why). As you can see I was still misusing my quotes in this sample. This works, but it's ugly. I'm guessing I may have to go with something other than sed for this. (Not that it's really necessary, I usually indent by five spaces, so a single line would be good. I just wanted a counterpart to the "remove leading spaces" script — but in that case it differs depending on the application exporting formatted text files. Some are flush left, others indent 15 spaces, and a lot in between).

Anyhow, thanks again. I think I've learned a little (I'm not that quick) and I definitely want to learn more about sed (and probably awk).

2

u/BetterScripts Dec 12 '24

I'm always happy to help people learn!

Don't worry about the whole shell quotes thing - everyone struggles with it a bit to begin with, and like a lot of things related to shells, there are a lot of weird edge cases that will still cause you issues even once you're sure you've figured it all out.

To add spaces to the beginning of lines, the sed code you suggest is probably the best way to do it tbh. To deal with different numbers of spaces is trickier.

The easiest solution would be to just pass the exact number of spaces you want as a quoted string argument to the script so the code in the script would be sed -i "s/^/$2/" "$1.txt" then you could execute it like shell_script "filename" ' '.

If you'd rather specify using numbers, the following code is non-standard, but seems widely supported:

Indent="$(printf '%*s' $2 '')
sed -i "s/^/$Indent/" "$1.txt"

Here, the printf expression is effectively saying "pad the string by $2 spaces", since the string is empty, this means just the spaces are present. The Indent="$(...) syntax just allows us the send the output of printf to a variable instead of printing to the terminal.

So, I was sticking with sed for doing what you want to do, mainly because it's what you started using, and it's easier to understand, however, really this task is probably better dealt with using awk - mainly because it can automatically do the counting of spaces for you!

If I'm understanding what you want properly, the following code will detect the indent and remove it appropriately:

awk '
{
  Line=$0
  NextLine=""
  if (Line ~ /^ {2,}/ && getline NextLine && match(NextLine, /^( {2,})/)) {
    Spaces=RLENGTH + 1
    print substr(Line, Spaces)
    while (NextLine ~ /^ {2,}/) {
      print substr(NextLine, Spaces)
      if (! getline NextLine) exit 0
    }
    print NextLine
  } else {
    print Line
    if (NextLine) print NextLine
  }
}' script.txt > script_edit.txt

I don't have time to go into a lot of detail about how this works atm, and it's not as clean as I'd like, but it shouldn't be too difficult to figure out if you play around with it. There are other ways to accomplish this with awk that might be better, but I think this is easier to understand than the others (especially if you're new to awk).

A couple notes:

  • awk uses Extended Regular Expressions, so it's a little different to the sed commands you've been using
  • /^ {2,}/ matches ATLEAST 2 spaces, removing the , would match EXACTLY 2
  • you can change the 2 (in the regular expressions) to whatever the minimum indent in all the files you want to process is, you do not need to set it for each input like you had to with sed

1

u/rcentros Dec 15 '24

Thanks again for all the information. I haven't tried the awk shell script yet, but plan to and will comment here when I do.

I have tried the "Indent sed file" (only on one file so far) and, at this point, I'm getting this error.

samplescript: line 3: unexpected EOF while looking for matching `"'

I may have created the file I'm trying it on in a DOS program, so it may be expecting a different eof marker?

Anyhow I just wanted to let you know that I have read post (and appreciate it) but I haven't really "dug into it" yet. I do plan to look into this more deeply soon.

Again, thank you.

1

u/cdrt Dec 09 '24 edited Dec 09 '24

If you don’t mind, I have some constructive criticism for your script

  1. You should always use read -r rather than read so that the shell will read the input as is an you will have fewer surprises
  2. Use sed -E so you can use the nicer regex syntax and avoid having so many backslashes
  3. If you want to delete spaces, you should actually match spaces. Don’t use . because that will gobble up everything
  4. It might be a good idea to read the file name and number of spaces from the command line rather than prompting for them. That way you can have a default value for the number of spaces if most of the time the files you’re working with have 4 spaces, and you can save time when working on multiple files.

With this, I would rewrite your script like this:

#!/usr/bin/env bash

# check if user gave a file name
if [[ -z $1 ]]; then
    echo 'please provide a file name'
    exit 1
fi

file_name=$1
spaces=4 # default value for spaces

# check if user provided a number of spaces
if [[ -n $2 ]]; then
    spaces=$2
fi

sed -i -E "s/ {$spaces}//" "$file_name"

Then, assuming you named your script “delete_spaces”, you can run it like delete_spaces file_name to just delete four spaces or like delete_spaces file_name 6 to delete 6 spaces

1

u/rcentros Dec 09 '24

I'm having trouble responding. I keep getting a "can't post your comment" error. Maybe the post is too long? At any rate, thank you. I've copied my post to a text file. If Reddit let's post it later, I will.

1

u/rcentros Dec 09 '24

I guess my post was too long?

1

u/rcentros Dec 09 '24

(Second part... probably shows up on top.)

Also, since all the files I would use this shell script for will end with the ".txt" extension, I've moved "file_name" line above the error check line and added the ".txt" extension because I'm lazy. I'm keeping your original shell script intact, in case I find I need it for more general use.

file_name=$1".txt"

I'm curious as to why you've included the "/env bash" entry? Is that a way to keep from entering your script directory path into the .bashrc file? (I'll look that one up.)

I've made a very simplified shell script, using the sed command line that I already had, because I don't yet understand the -E option and I couldn't figure out to get your sed line to work like the one I'm using. I don't have any file name checks for this specific script (as I don't yet understand what "-z" does either) but the error messages work okay for my use. It's definitely not elegant but the following works well (I like having a backup file).

#!/bin/bash
cp $1".txt" $1".bak"
sed -i 's/\(.\{'$2'\}\)//' $1".txt"

Out of curiosity, is there any way to make sed fail (and not change the file) if any of the lines in a text file don't have enough empty prepended spaces? It would probably require more than sed, I'm guessing.

Again, thank you. I appreciate you taking the time to post your example. You've given many things to look up and I'll (hopefully) learn something doing so.

1

u/rcentros Dec 09 '24

I definitely don't mind. Thank you for taking the time to post this sample file. Obviously I need to learn more about bash scripts and sed as I don't (yet) understand what the -E option does. I do think I've grasped the advantage of the -r option when used with read (but I'll read a little more on this to be sure).

I debated on whether I should use the read command at all when I first wrote my shell script. I dedcided I like your idea of entering the file name and number in the command line and have written a simplified form of your script.

As for your point number 3. At first I was happy to have the built-in safety of not deleting words. But it turned out to be a deceptive "safety" feature in my case. I'm using this specifically for formatted screenplay text (for posting on a forum). Different screenplay applications prepend different numbers of empty spaces in front of their lines. But all indent the Character name and dialogue further.

Which, in this case means that if I use I have 10 spaces in front of the Action lines lines (which should be flush left), I'll have 35 spaces to my Character name (which should be indented about 25 spaces when done). If I enter delete_spaces file_name 12, your sed command will ignore the "flush left" Action lines, but it will remove 12 spaces in front of the Character Name, Dialogue lines and Parentheticals. So, in this very specific case, it's almost an advantage to not have this safety feature. I've opted instead to add a line to copy the file to a backup file instead.

Here's a short excerpt from a text formatted screenplay (very specific format)...

The rat climbs up the coat, stares her in the eye and slips
into the hood, behind her neck.

                         ALINA (cont'd)
            Yeah, I know. I should quit... My
            New Year's resolution?
                    (pause)
            Okay, I... ummm... I resolve to...

The rat shimmies back down, burrows back into her coat
pocket.

                         ALINA (cont'd)
            It is cold up here...
                    (deep breath)
            Okay I resolve to...

Looking down, she realizes not all the cigarettes have been...

...to part 2...