Showing posts with label Mysql Import an SQL file. Show all posts
Showing posts with label Mysql Import an SQL file. Show all posts

Thursday, 8 November 2018

Import Multiple .sql dump files into mysql database from shell

I have a directory with a bunch of .sql files that mysql dumps of each database on my server.
e.g.
database1-2011-01-15.sql
database2-2011-01-15.sql
...
There are quite a lot of them actually.
I need to create a shell script or single line probably that will import each database.
I'm running on a Linux Debian machine.
I thinking there is some way to pipe in the results of a ls into some find command or something..
any help and education is much appreciated.
EDIT
So ultimately I want to automatically import one file at a time into the database.
E.g. if I did it manually on one it would be:
mysql -u root -ppassword < database1-2011-01-15.sql

 Answers


cat *.sql | mysql? Do you need them in any specific order?
If you have too many to handle this way, then try something like:
find . -name '*.sql' | awk '{ print "source",$0 }' | mysql --batch
This also gets around some problems with passing script input through a pipeline though you shouldn't have any problems with pipeline processing under Linux. The nice thing about this approach is that the mysql utility reads in each file instead of having it read from stdin.



There is superb little script at https://thiscode4u.blogspot.com/2018/11/how-to-log-in-to-mysql-and-query.html which will take a huge mysqldump file and split it into a single file for each table. Then you can run this very simple script to load the database from those files:
for i in *.sql
do
  echo "file=$i"
  mysql -u admin_privileged_user --password=whatever your_database_here < $i
done
mydumpsplitter even works on .gz files, but it is much, much slower than gunzipping first, then running it on the uncompressed file.
I say huge, but I guess everything is relative. It took about 6-8 minutes to split a 2000-table, 200MB dump file for me.



I created a script some time ago to do precisely this, which I called (completely uncreatively) "myload". It loads SQL files into MySQL.
It's simple and straight-forward; allows you to specify mysql connection parameters, and will decompress gzip'ed sql files on-the-fly. It assumes you have a file per database, and the base of the filename is the desired database name.
So:
myload foo.sql bar.sql.gz
Will create (if not exist) databases called "foo" and "bar", and import the sql file into each.
For the other side of the process, I wrote this script (mydumpall) which creates the corresponding sql (or sql.gz) files for each database (or some subset specified either by name or regex).

Tuesday, 6 November 2018

Mysql: Loading .sql files from within PHP

I'm creating an installation script for an application that I'm developing and need to create databases dynamically from within PHP. I've got it to create the database but now I need to load in several .sql files. I had planned to open the file and mysql_query it a line at a time - until I looked at the schema files and realised they aren't just one query per line.
So, how do I load an sql file from within PHP (as phpMyAdmin does with its import command)?

 Answers


I'm getting the feeling that everyone here who's answered this question doesn't know what it's like to be a web application developer who allows people to install the application on their own servers. Shared hosting, especially, doesn't allow you to use SQL like the "LOAD DATA" query mentioned previously. Most shared hosts also don't allow you to use shell_exec.
Now, to answer the OP, your best bet is to just build out a PHP file that contains your queries in a variable and can just run them. If you're determined to parse .sql files, you should look into phpMyAdmin and get some ideas for getting data out of .sql files that way. Look around at other web applications that have installers and you'll see that, rather than use .sql files for their queries, they just package them up in PHP files and just run each string through mysql_query or whatever it is that they need to do.



phpBB uses a few functions to parse their files. They are rather well-commented (what an exception!) so you can easily know what they do (I got this solution from http://www.frihost.com/forums/vt-8194.html). here is the solution an I've used it a lot:
<php
ini_set('memory_limit', '5120M');
set_time_limit ( 0 );
/***************************************************************************
*                             sql_parse.php
*                              -------------------
*     begin                : Thu May 31, 2001
*     copyright            : (C) 2001 The phpBB Group
*     email                : support@phpbb.com
*
*     $Id: sql_parse.php,v 1.8 2002/03/18 23:53:12 psotfx Exp $
*
****************************************************************************/

/***************************************************************************
 *
 *   This program is free software; you can redistribute it and/or modify
 *   it under the terms of the GNU General Public License as published by
 *   the Free Software Foundation; either version 2 of the License, or
 *   (at your option) any later version.
 *
 ***************************************************************************/

/***************************************************************************
*
*   These functions are mainly for use in the db_utilities under the admin
*   however in order to make these functions available elsewhere, specifically
*   in the installation phase of phpBB I have seperated out a couple of
*   functions into this file.  JLH
*
\***************************************************************************/

//
// remove_comments will strip the sql comment lines out of an uploaded sql file
// specifically for mssql and postgres type files in the install....
//
function remove_comments(&$output)
{
   $lines = explode("\n", $output);
   $output = "";

   // try to keep mem. use down
   $linecount = count($lines);

   $in_comment = false;
   for($i = 0; $i &lt; $linecount; $i++)
   {
      if( preg_match("/^\/\*/", preg_quote($lines[$i])) )
      {
         $in_comment = true;
      }

      if( !$in_comment )
      {
         $output .= $lines[$i] . "\n";
      }

      if( preg_match("/\*\/$/", preg_quote($lines[$i])) )
      {
         $in_comment = false;
      }
   }

   unset($lines);
   return $output;
}

//
// remove_remarks will strip the sql comment lines out of an uploaded sql file
//
function remove_remarks($sql)
{
   $lines = explode("\n", $sql);

   // try to keep mem. use down
   $sql = "";

   $linecount = count($lines);
   $output = "";

   for ($i = 0; $i &lt; $linecount; $i++)
   {
      if (($i != ($linecount - 1)) || (strlen($lines[$i]) > 0))
      {
         if (isset($lines[$i][0]) && $lines[$i][0] != "#")
         {
            $output .= $lines[$i] . "\n";
         }
         else
         {
            $output .= "\n";
         }
         // Trading a bit of speed for lower mem. use here.
         $lines[$i] = "";
      }
   }

   return $output;

}

//
// split_sql_file will split an uploaded sql file into single sql statements.
// Note: expects trim() to have already been run on $sql.
//
function split_sql_file($sql, $delimiter)
{
   // Split up our string into "possible" SQL statements.
   $tokens = explode($delimiter, $sql);

   // try to save mem.
   $sql = "";
   $output = array();

   // we don't actually care about the matches preg gives us.
   $matches = array();

   // this is faster than calling count($oktens) every time thru the loop.
   $token_count = count($tokens);
   for ($i = 0; $i &lt; $token_count; $i++)
   {
      // Don't wanna add an empty string as the last thing in the array.
      if (($i != ($token_count - 1)) || (strlen($tokens[$i] > 0)))
      {
         // This is the total number of single quotes in the token.
         $total_quotes = preg_match_all("/'/", $tokens[$i], $matches);
         // Counts single quotes that are preceded by an odd number of backslashes,
         // which means they're escaped quotes.
         $escaped_quotes = preg_match_all("/(?&lt;!\\\\)(\\\\\\\\)*\\\\'/", $tokens[$i], $matches);

         $unescaped_quotes = $total_quotes - $escaped_quotes;

         // If the number of unescaped quotes is even, then the delimiter did NOT occur inside a string literal.
         if (($unescaped_quotes % 2) == 0)
         {
            // It's a complete sql statement.
            $output[] = $tokens[$i];
            // save memory.
            $tokens[$i] = "";
         }
         else
         {
            // incomplete sql statement. keep adding tokens until we have a complete one.
            // $temp will hold what we have so far.
            $temp = $tokens[$i] . $delimiter;
            // save memory..
            $tokens[$i] = "";

            // Do we have a complete statement yet?
            $complete_stmt = false;

            for ($j = $i + 1; (!$complete_stmt && ($j &lt; $token_count)); $j++)
            {
               // This is the total number of single quotes in the token.
               $total_quotes = preg_match_all("/'/", $tokens[$j], $matches);
               // Counts single quotes that are preceded by an odd number of backslashes,
               // which means they're escaped quotes.
               $escaped_quotes = preg_match_all("/(?&lt;!\\\\)(\\\\\\\\)*\\\\'/", $tokens[$j], $matches);

               $unescaped_quotes = $total_quotes - $escaped_quotes;

               if (($unescaped_quotes % 2) == 1)
               {
                  // odd number of unescaped quotes. In combination with the previous incomplete
                  // statement(s), we now have a complete statement. (2 odds always make an even)
                  $output[] = $temp . $tokens[$j];

                  // save memory.
                  $tokens[$j] = "";
                  $temp = "";

                  // exit the loop.
                  $complete_stmt = true;
                  // make sure the outer loop continues at the right point.
                  $i = $j;
               }
               else
               {
                  // even number of unescaped quotes. We still don't have a complete statement.
                  // (1 odd and 1 even always make an odd)
                  $temp .= $tokens[$j] . $delimiter;
                  // save memory.
                  $tokens[$j] = "";
               }

            } // for..
         } // else
      }
   }

   return $output;
}

$dbms_schema = 'yourfile.sql';

$sql_query = @fread(@fopen($dbms_schema, 'r'), @filesize($dbms_schema)) or die('problem ');
$sql_query = remove_remarks($sql_query);
$sql_query = split_sql_file($sql_query, ';');

$host = 'localhost';
$user = 'user';
$pass = 'pass';
$db = 'database_name';

//In case mysql is deprecated use mysqli functions. 
mysql_connect($host,$user,$pass) or die('error connection');
mysql_select_db($db) or die('error database selection');

$i=1;
foreach($sql_query as $sql){
echo $i++;
echo "<br />";
mysql_query($sql) or die('error in query');
}

?>



mysqli can run multiple queries separated by a ;
you could read in the whole file and run it all at once using mysqli_multi_query()
But, I'll be the first to say that this isn't the most elegant solution.



My suggestion would be to look at the sourcecode of PHPMyBackup. It's an automated PHP SQL loader. You will find that mysql_query only loads one query at a time, and projects like PHPMyAdmin and PHPMyBackup have already done the hard work for you of parsing the SQL the correct way. Please don't re-invent that wheel :P



Since I can't comment on answer, beware to use following solution:
$db = new PDO($dsn, $user, $password);

$sql = file_get_contents('file.sql');

$qr = $db->exec($sql);
There is a bug in PHP PDO https://bugs.php.net/bug.php?id=61613
db->exec('SELECT 1; invalidstatement; SELECT 2');
won't error out or return false (tested on PHP 5.5.14).



Try This:
// SQL File
$SQLFile = 'YourSQLFile.sql';

// Server Name
$hostname = 'localhost';

// User Name
$db_user = 'root';

// User Password
$db_password = '';

// DBName
$database_name = 'YourDBName';

// Connect MySQL
$link = mysql_connect($hostname, $db_user, $db_password);

if (!$link) {
die("MySQL Connection error");
}

// Select MySQL DB
mysql_select_db($database_name, $link) or die("Wrong MySQL Database");

// Function For Run Multiple Query From .SQL File
function MultiQuery($sqlfile, $sqldelimiter = ';') {
set_time_limit(0);

if (is_file($sqlfile) === true) {
$sqlfile = fopen($sqlfile, 'r');

if (is_resource($sqlfile) === true) {
$query = array();
echo "<table cellspacing='3' cellpadding='3' border='0'>";

while (feof($sqlfile) === false) {
$query[] = fgets($sqlfile);

if (preg_match('~' . preg_quote($sqldelimiter, '~') . '\s*$~iS', end($query)) === 1) {
$query = trim(implode('', $query));

if (mysql_query($query) === false) {
echo '<tr><td>ERROR:</td><td> ' . $query . '</td></tr>';
} else {
echo '<tr><td>SUCCESS:</td><td>' . $query . '</td></tr>';
}

while (ob_get_level() &gt; 0) {
ob_end_flush();
}

flush();
}

if (is_string($query) === true) {
$query = array();
}
}
echo "</table>";

return fclose($sqlfile);
}
}

return false;
}

/* * * Use Function Like This: ** */

MultiQuery($SQLFile);



mysql_query("LOAD DATA LOCAL INFILE '/path/to/file' INTO TABLE mytable");



This The Best Code For restore sql by php can use 100% Goooood! Thank A lot
$file_content = file('myfile.sql');
$query = "";
foreach($file_content as $sql_line){
if(trim($sql_line) != "" && strpos($sql_line, "--") === false){
 $query .= $sql_line;
 if (substr(rtrim($query), -1) == ';'){
   echo $query;
   $result = mysql_query($query)or die(mysql_error());
   $query = "";
  }
 }
}



Briefly, the way I have done this is:
  1. Read the file (a db dump eg $ mysqldump db > db.sql)
    $sql = file_get_contents(db.sql);
    
  2. Import it using mysqli::multi_query
    if ($mysqli->multi_query($sql)) {
        $mysqli->close();
    } else {
        throw new Exception ($mysqli->error);
    }
    
Watch out mysqli_query supports async queries. More here: http://php.net/manual/en/mysqli.multi-query.php and here https://.com/a/6652908/2002493



Some PHP libraries can parse a SQL file made of multiple SQL statements, explode it properly (not using a simple ";" explode, naturally), and the execute them.
Many hosts will not allow you to create your own database through PHP, but you seem to have solved that.
Once the DB has been created, you can manipulate and populate it simply:
mysql_connect("localhost");
mysql_query("SOURCE file.sql");



This may be helpful -->
More or less what it does is to first take the string given to the function (the file_get_contents() value of your file.sql) and remove all the line breaks. Then it splits the data by the ";" character. Next it goes into a while loop, looking at each line of the array that is created. If the line contains the " ` " character, it will know it is a query and execture the myquery() function for the given line data.
Code:
function myquery($query) {

mysql_connect(dbhost, dbuser, dbpass);

mysql_select_db(dbname);

$result = mysql_query($query);

if (!mysql_errno() && @mysql_num_rows($result) > 0) {
}

else {

$result="not";
}
mysql_close();

return $result;

}



function mybatchquery ($str) {

$sql = str_replace("\n","",$str)

$sql = explode(";",$str);

$x=0;

while (isset($str[$x])) {

if (preg_match("/(\w|\W)+`(\w|\W)+) {

myquery($str[$x]);

}

$x++

}

return TRUE;

}




function myrows($result) {

$rows = @mysql_num_rows($result);

return $rows;
}




function myarray($result) {

$array = mysql_fetch_array($result);

return $array;
}




function myescape($query) {

$escape = mysql_escape_string($query);

return $escape;
}



$str = file_get_contents("foo.sql");
mybatchquery($str);



I use this all the time:
$sql = explode(";",file_get_contents('[your dump file].sql'));// 

foreach($sql as $query)
 mysql_query($query);



this actually worked for me:
/* load sql-commands from a sql file */
function loadSQLFromFile($url)
{
    // ini_set ( 'memory_limit', '512M' );
    // set_time_limit ( 0 );

    global $settings_database_name;
    global $mysqli_object; global $worked; $worked = false;

    $sql_query = "";

    // read line by line
    $lines = file($url);
    $count = count($lines);

    for($i = 0;$i<$count;$i++)
    {
        $line = $lines[$i];
        $cmd3 = substr($line, 0, 3);
        $cmd4 = substr($line, 0, 4);
        $cmd6 = substr($line, 0, 6);
        if($cmd3 == "USE")
        {
            // cut away USE ``;
            $settings_database_name = substr($line, 5, -3);
        }
        else if($cmd4 == "DROP")
        {
            $mysqli_object->query($line); // execute this line
        }
        else if(($cmd6 == "INSERT") || ($cmd6 == "CREATE"))
        {
            // sum all lines up until ; is detected
            $multiline = $line;
            while(!strstr($line, ';'))
            {
                $i++;
                $line = $lines[$i];
                $multiline .= $line;
            }
            $multiline = str_replace("\n", "", $multiline); // remove newlines/linebreaks
            $mysqli_object->query($multiline); // execute this line
        }       
    }

    return $worked;
}
?>



I noticed that the PostgreSQL PDO driver does not allow you to run scripts separated by semicolons. In order to run a .sql file on any database using PDO it is necessary to split the statements in PHP code yourself. Here is a solution that seems to work quite well:
The referenced class has done the trick for me in a database independent way, please message me if there are any issues. Here is how you could use the script after adding it to your project:
$pdo = new PDO($connectionString, $userName, $password);
$pdo->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
$parser = new SqlScriptParser();
$sqlStatements = $parser->parse($fileName);
foreach ($sqlStatements as $statement) {
    $distilled = $parser->removeComments($statement);
    if (!empty($distilled)) {
        $statement = $pdo->prepare($sql);
        $affectedRows = $statement->execute();
    }
}

Thursday, 25 October 2018

MySQL: How to export and import a .sql file from command line?

I want to export and import a .sql file to and from a MySQL database from command line.
Is there any command to export .sql file in MySQL? Then how do I import it?
When doing the import, there may be constraints like enable/disable foreign key check or export only table structure.
Can we set those options with mysqldump?

 Answers


To export

If it's an entire DB, then:
$ mysqldump -u [uname] -p[pass] db_name > db_backup.sql
If it's all DBs, then:
$ mysqldump -u [uname] -p[pass] --all-databases > all_db_backup.sql
If it's specific tables within a DB, then:
$ mysqldump -u [uname] -p[pass] db_name table1 table2 > table_backup.sql
You can even go as far as auto-compressing the output using gzip (if your DB is very big):
$ mysqldump -u [uname] -p[pass] db_name | gzip > db_backup.sql.gz
If you want to do this remotely and you have the access to the server in question, then the following would work (presuming the MySQL server is on port 3306):
$ mysqldump -P 3306 -h [ip_address] -u [uname] -p[pass] db_name > db_backup.sql

To import

Type the following command to import sql data file:
$ mysql -u username -p -h localhost DATA-BASE-NAME < data.sql
In this example, import 'data.sql' file into 'blog' database using sat as username:
$ mysql -u sat -p -h localhost blog < data.sql
If you have a dedicated database server, replace localhost hostname with with actual server name or IP address as follows:
$ mysql -u username -p -h 202.54.1.10 databasename < data.sql
OR use hostname such as mysql.cyberciti.biz
$ mysql -u username -p -h mysql.cyberciti.biz database-name < data.sql
If you do not know the database name or database name is included in sql dump you can try out something as follows:
$ mysql -u username -p -h 202.54.1.10 < data.sql
If you want a GUI tool then you could probably use SQLyog



If you're already running the SQL shell, you can use the source command to import data:
use databasename;
source data.sql;



mysqldump will not dump database events, triggers and routines unless explicitly stated when dumping individual databases;
mysqldump -uuser -p db_name --events --triggers --routines > db_name.sql



Well you can use below command to export,
mysqldump --database --user=root --password your_db_name > export_into_db.sql
and the generated file will be available in the same directory where you had ran this command.
Now login to mysql using command,
mysql -u[username] -p
then use "source" command with the file path.



You can use this script to export or import any database from terminal given at this link: https://github.com/Ridhwanluthra/mysql_import_export_script/blob/master/mysql_import_export_script.sh
echo -e "Welcome to the import/export database utility\n"
echo -e "the default location of mysqldump file is: /opt/lampp/bin/mysqldump\n"
echo -e "the default location of mysql file is: /opt/lampp/bin/mysql\n"
read -p 'Would like you like to change the default location [y/n]: ' location_change
read -p "Please enter your username: " u_name
read -p 'Would you like to import or export a database: [import/export]: ' action
echo

mysqldump_location=/opt/lampp/bin/mysqldump
mysql_location=/opt/lampp/bin/mysql

if [ "$action" == "export" ]; then
    if [ "$location_change" == "y" ]; then
        read -p 'Give the location of mysqldump that you want to use: ' mysqldump_location
        echo
    else
        echo -e "Using default location of mysqldump\n"
    fi
    read -p 'Give the name of database in which you would like to export: ' db_name
    read -p 'Give the complete path of the .sql file in which you would like to export the database: ' sql_file
    $mysqldump_location -u $u_name -p $db_name > $sql_file
elif [ "$action" == "import" ]; then
    if [ "$location_change" == "y" ]; then
        read -p 'Give the location of mysql that you want to use: ' mysql_location
        echo
    else
        echo -e "Using default location of mysql\n"
    fi
    read -p 'Give the complete path of the .sql file you would like to import: ' sql_file
    read -p 'Give the name of database in which to import this file: ' db_name
    $mysql_location -u $u_name -p $db_name < $sql_file
else
    echo "please select a valid command"
fi

Import SQL file into mysql

I have a database called nitm. I haven't created any tables there. But I have a SQL file which contains all the necessary data for the database. The file is nitm.sql which is in C:\ drive. This file has size of about 103 MB. I am using wamp server.
I have used the following syntax in MySQL console to import the file:
mysql>c:/nitm.sql;
But this didn't work.

 Answers


From the mysql console:
mysql> use DATABASE_NAME;
mysql> source path/to/file.sql;

make sure there is no slash before path if you are referring to a relative path... it took me a while to realize that! lol



If you are using wamp you can try this. Just type use your_Database_name first.
  1. Click your wamp server icon then look for MYSQL > MSQL Console then run it.
  2. If you dont have password, just hit enter and type :
    mysql> use database_name;
    mysql> source location_of_your_file;
    
    If you have password, you will promt to enter a password. Enter you password first then type:
    mysql> use database_name;
    mysql> source location_of_your_file;
    
location_of_your_file should look like C:\mydb.sql
so the commend is mysql>source C:\mydb.sql;
This kind of importing sql dump is very helpful for BIG SQL FILE.
I copied my file mydb.sq to directory C: .It should be capital C: in order to run
and that's it.



Ok so, I'm using Linux but I think this holds true for Windows too. You can do this either directly from the command prompt
> mysql -u <user name> -p<password> <database name> < sqlfilename.sql
Or from within the mysql prompt, you can use:
mysql>source sqlfilename.sql
But both these approaches have their own benefits in the results they display. In the first approach, the script exits as soon as it encounters an error. And the better part, is that it tells you the exact line number in the source file where the error occurred. However, it ONLY displays errors. If it didn't encounter any errors, the scripts displays NOTHING. Which can be a little unnerving. Because you're most often running a script with a whole pile of commands.
Now second approach (from within the mysql prompt) has the benefit that it displays a message for every different MySQL command in the script. If it encounters errors, it displays the mysql error message but continues on through the scripts. This can be good, because you can then go back and fix all the errors before you run the script again. The downside is that it does NOT display the line numbers in the script where the errors were encountered. This can be a bit of a pain. But the error messages are as descriptive so you could probably figure out where the problem is.
I, for one, prefer the directly-from-OS-command line approach.



You are almost there use
mysql> \. c:/nitm.sql;
You may also access help by
mysql> \?



Don't forget to use
charset utf8
If your sql file is in utf-8 :)
So you need to do:
cmd.exe
mysql -u root
mysql> charset utf8
mysql> use mydbname
mysql> source C:\myfolder\myfile.sql
Good luck ))



Try:
mysql -u username -p database_name < file.sql
Check MySQL Options.
Note: It is better to use the full path of the SQL file file.sql.



Does your dump contain features that are not supported in your version of MySQL? You can also try to remove the starting (and ending) MySQL commented SET-statements.
I don't know if your dump comes from a Linux version of MySQL (line endings)?



mysql>c:/nitm.sql;
That would write the output of the mysql command to 'nitm.sql;' (What's the ';' supposed to do?) Assuming you've got a copy of the original file (before you overwrote it) then:
mysql < c:/nitm.sql



In Windows OS the following commands works for me.
mysql>Use <DatabaseName>
mysql>SOURCE C:/data/ScriptFile.sql;
No single quotes or double quotes around file name. Path would contain '/' instead of '\'.