Unix shell script to truncate a large file

I am trying to write a Unix script which will truncate/empty a file which is continuously being written/open by an application when it reaches say 3GB of space. I know that the below command would do it :

cp /dev/null [filename]

But I am going to run this in an production environment automatically as a cron job - just posting here to see if you guys faced any issues while doing something similar to this.

 Answers

Just to add another answer,

: > filename

: is a no-op in bash, so this essentially just opens the file for writing (which of course truncates the file) and then immediately closes it.

EDIT: as shellter commented, you don't actually need a command to go along with the redirection:

$ echo foo > foo.txt
$ cat foo.txt
foo
$ > foo.txt
$ cat foo.txt
$

A simple redirection all by itself will clear the file.

---

That seems reasonable to me. Unix, of course, would let you do this about 50 different ways. For example,

echo -n "" >filename
cat /dev/null >filename

Replace a string in file using shell script

Suppose my file a.conf is as following

Include /1
Include /2
Include /3

I want to replace "Include /2" with a new line, I write the code in .sh file : 

line="Include \\/2"
rep=""

sed "s/${line}/${rep}/g" /root/new_scripts/a.conf > /tmp/a.conf-new

mv /tmp/a.conf-new /root/new_scripts/a.conf 

shell script echo new line to file

var1="Hello"

var2="World!"
logwrite="$var1\n$var2"
echo -e "$logwrite"  >> /Users/username/Desktop/user.txt

Explanation:
The \n escape sequence indicates a line feed. Passing the -e argument to echo enables interpretation of escape sequences.
It may even be simplified further:

var1="Hello"
var2="World!"
echo -e "$var1\n$var2"  >> /Users/username/Desktop/user.txt

or even:

echo -e "Hello\nWorld! "  >> /Users/username/Desktop/user.txt

How to Format Date for Display or Use In a Shell Script

How do I format date to display on screen on for my shell scripts as per my requirements on Linux or Unix like operating systemsYou need to use the standard date command to format date or time. You can use the same command with the shell script.
 The syntax is

1. date +FORMAT 
2. date +"%FORMAT"
3. date +"%FORMAT%FORMAT" 
4. date +"%FORMAT-%FORMAT"

Task: Display date in mm-dd-yy format

Open a terminal and type the following date command:
$ date +"%m-%d-%y"Sample output:

02-27-07

To turn on 4 digit year display:
$ date +"%m-%d-%Y"Just display date as mm/dd/yy format:
$ date +"%D"

Task: Display time only

Type the following command:
$ date +"%T"Outputs:

19:55:04

To display locale's 12-hour clock time, enter:
$ date +"%r"Outputs:

07:56:05 PM

To display time in HH:MM format, type:
$ date +"%H-%M"Sample outputs:

00-50

How do I save time/date format to the shell variable?

$ NOW=$(date +"%m-%d-%Y"To display a variable use echo / printf command:
$ echo $NOW

A sample shell script

#!/bin/bash
NOW=$(date +"%m-%d-%Y")
FILE="backup.$NOW.tar.gz"
echo "Backing up data to /nas42/backup.$NOW.tar.gz file, please wait..."
# rest of script
# tar xcvf /nas42/backup.$NOW.tar.gz /home/ /etc/ /var
 

A complete list of FORMAT control characters supported by the date command

FORMAT controls the output. It can be the combination of any one of the following:

%FORMAT String	Description
%%	a literal %
%a	locale's abbreviated weekday name (e.g., Sun)
%A	locale's full weekday name (e.g., Sunday)
%b	locale's abbreviated month name (e.g., Jan)
%B	locale's full month name (e.g., January)
%c	locale's date and time (e.g., Thu Mar 3 23:05:25 2005)
%C	century; like %Y, except omit last two digits (e.g., 21)
%d	day of month (e.g, 01)
%D	date; same as %m/%d/%y
%e	day of month, space padded; same as %_d
%F	full date; same as %Y-%m-%d
%g	last two digits of year of ISO week number (see %G)
%G	year of ISO week number (see %V); normally useful only with %V
%h	same as %b
%H	hour (00..23)
%I	hour (01..12)
%j	day of year (001..366)
%k	hour ( 0..23)
%l	hour ( 1..12)
%m	month (01..12)
%M	minute (00..59)
%n	a newline
%N	nanoseconds (000000000..999999999)
%p	locale's equivalent of either AM or PM; blank if not known
%P	like %p, but lower case
%r	locale's 12-hour clock time (e.g., 11:11:04 PM)
%R	24-hour hour and minute; same as %H:%M
%s	seconds since 1970-01-01 00:00:00 UTC
%S	second (00..60)
%t	a tab
%T	time; same as %H:%M:%S
%u	day of week (1..7); 1 is Monday
%U	week number of year, with Sunday as first day of week (00..53)
%V	ISO week number, with Monday as first day of week (01..53)
%w	day of week (0..6); 0 is Sunday
%W	week number of year, with Monday as first day of week (00..53)
%x	locale's date representation (e.g., 12/31/99)
%X	locale's time representation (e.g., 23:13:48)
%y	last two digits of year (00..99)
%Y	year
%z	+hhmm numeric timezone (e.g., -0400)
%:z	+hh:mm numeric timezone (e.g., -04:00)
%::z	+hh:mm:ss numeric time zone (e.g., -04:00:00)
%:::z	numeric time zone with : to necessary precision (e.g., -04, +05:30)
%Z	alphabetic time zone abbreviation (e.g., EDT)

Running PHP scripts as shell scripts

As well as running PHP scripts via a web server such as Apache, you can also run PHP scripts from the command line (it's also possible to write GUI apps with PHP using PHP-GTK, but I'll maybe look at that another time). This post looks at the two ways you can run a PHP script from a *nix based computer (Linux, UNIX, BSD, OSX) and the single way you can do it from Windows.

Using a shebang

UNIX shell scripts are set as executable and the first line in the file contains a "shebang" line which shows the path to the executable which will parse the script.

If your distro's PHP binary is at e.g. /usr/bin/php you would add this to the first line of your script:

#!/usr/bin/php

and then have your PHP script under it.

To run a "hello world script" you'd do this:

#!/usr/bin/php
<?php
  echo "hello world\n";

The script then needs to be marked as executable. To set it so it can be run as anyone do this:

chmod 0755 /path/to/script.php

To make it so only you can run it (and so no one else can even read it) do this:

chmod 0700 /path/to/script.php

Read the chmod manpage for more details about file permissions if you don't know about them. 

Calling the PHP binary, passing the script filename

To avoid permission issues, or to run the PHP script from the command line on Windows (the above method does not work on Windows unless run via cygwin), you need to execute the PHP binary passing the script filename as the parameter.

If you are in the same directory as the script, the script is named e.g. myscript.php, and the PHP binary is at /usr/bin/php you would do this to run it:

/usr/bin/php myscript.php

If the PHP binary is in your executable path (it usually will be on most Linux distros) then you should just need to run "php" and then the script:

php myscript.php

If you're running it from cron it's always a good idea to put the full path to both:

/usr/bin/php /path/to/myscript.php

When using this method you don't need to have the shebang line at the top of the script.

An example of doing this on Windows:

c:\php\php.exe c:\path\to\myscript.php

Conclusion

It's easy to run a PHP script from the command line, which is useful for batch processing or database cleanup for your website. Combined with automatic processing via the cron system and you have a powerful tool at your disposal if you are a PHP programmer and/or your website is programmed in PHP.

Related posts:

• PHP CLI counter for long running processes
• Command line arguments for the PHP CLI
• Command line arguments with a PHP CLI script
• Determine whether PHP is being run via HTTP or CLI

Run PHP Files From the Command Line

I've been brushing up on my shell scripting lately. I just got a MacBook and never felt compelled to spend too much time with Cygwin. I'm learning quite a bit now but there are still some tasks that I'd need to accomplish sooner rather than later. I know how to accomplish the task using PHP so I've got that to my advantage. What's awesome is that I can quickly and easily run my PHP files from the command line.


The Shell Script
php doTask.php
I'm not sure whether I consider using PHP instead of a straight shell script as being resourceful or using PHP as a crutch. What are your thoughts?

Shell Scripting in Linux

Shell Scripting – It is file or program containing the sequence of the commands executing in it.

Shell – It is an command line interpreter which reads the program line by line and converts it into the machine language. So if we talk about linux, the shells like

/bin/sh
/bin/bash
/sbin/nologin
/bin/dash
/bin/ksh
/bin/csh
/usr/bin/perl

which are used to read your commands and execute them on the system. So we actually use them as per the requirements. For example  “/usr/bash” is for “Linux” and “/bin/sh” is used for “Unix” etc.

Scripting – Scripting is a program containing the multiple number of commands  in it. Which is used to complete a task with a single command. You can do repeated tasks in short period of time and avoid running several commands every time.

We actually put all the linux commands in a file and execute it on shell.

Following are the steps to execute any shell script on linux:-

1. Shebang Value –  This is the very first line to be written in every shell script. It makes the linux kernel to understand that which shell is to be used before running this script. eg.     #!/bin/bash   or   #!/bin/sh .  Sometime people write their scripts without shebang value in it but this is not the correct practice. It will work but not effectively like once you encrypt your shell script code and put it in the /usr/bin directory to make it a executable command for linux, then the same script will not work as you wanted. Try it 
2. Make file executable – Once we saved the shell script file, just need to make it executable as mentioned below:-
   chmod +x filename
   OR
   chmod 755 filename
3. Run as a command – Copy the same file to /usr/bin directory so that you can run it directly as “filename” on linux like as all other commands.
4. Other methods – You can also tun the shell script from the same place where it is being created like as  “./filename   OR  sh filename”.
5. Debug Mode –  Sometime we need to check if the script is showing some errors while running then we can have the below option to run the same script in debug mode.
   sh  -X  filename

Examples of basic scripts:-

Objective: Create a script to see date,  calendar and to check the status of the commands.

[chetan.jangir@ip-172-21-0-221 ~]$ vi  testscript.sh
#!/bin/bash
date
cal
echo $?

.
.
.
.

:wq!

[chetan.jangir@ip-172-21-0-221 ~]$ chmod +x testscript.sh

[chetan.jangir@ip-172-21-0-221 ~]$ sh testscript.sh
Mon Jun 20 05:46:47 UTC 2016
June 2016
Su Mo Tu We Th Fr Sa
1 2 3 4
5 6 7 8 9 10 11
12 13 14 15 16 17 18
19 20 21 22 23 24 25
26 27 28 29 30

0

#####################

At the end  “0”  shows that the above commands did run successfully.

Import Data From CSV File to MySQL With Bash Script

Here is what you need to this simple demonstration on how to import data from CSV file to MySQL database. Fist we need a comma separated value file like:

1 2 3 4 5 6 7 8 9 10 11

$ cat input.csv 54,68,74 54,67,77 54,67,78 54,67,76 54,67,79 54,67,74 54,67,75 54,67,68 54,67,63 54,65,63

Next, you need MySQL database with at least one table. Our database is called “mydata” and contains a single table named “cost”:

1 2 3 4 5 6 7 8 9 10

mysql> describe cost; +---------+---------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +---------+---------+------+-----+---------+----------------+ | id | int(11) | NO | PRI | NULL | auto_increment | | column1 | int(11) | NO | | NULL | | | column2 | int(11) | NO | | NULL | | | column3 | int(11) | NO | | NULL | | +---------+---------+------+-----+---------+----------------+ 4 rows in set (0.00 sec)

Lastly, we need a bash script to import our comma separated CSV file into our table. The file can look like the one below:

1 2 3 4 5 6 7 8 9

$ cat import-SQL.sh #!/bin/bash IFS=, while read column1 column2 column3 do echo "INSERT INTO cost (column1,column2,column3) VALUES ('$column1', '$column2', '$column3');" done < input.csv | mysql -u myusername -p mypassword mydata;

Most of the stuff above is self explanatory except IFS variable. IFS or Internal field separator is a bash buildin bash variable which is by default to set to contain space like “ ”. It is used to read input and since now our input is comma separated we set IFS to “,”. Ok, enough talk, let’s run the script:

1	$ ./import-SQL.sh

No ouptut, good output. Let’s check our database:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

mysql> select * from cost; +----+---------+---------+---------+ | id | column1 | column2 | column3 | +----+---------+---------+---------+ | 1 | 54 | 68 | 74 | | 2 | 54 | 67 | 77 | | 3 | 54 | 67 | 78 | | 4 | 54 | 67 | 76 | | 5 | 54 | 67 | 79 | | 6 | 54 | 67 | 74 | | 7 | 54 | 67 | 75 | | 8 | 54 | 67 | 68 | | 9 | 54 | 67 | 63 | | 10 | 54 | 65 | 63 | +----+---------+---------+---------+ 10 rows in set (0.00 sec)