Great source for deleting millions of files in a ubuntu directory when you get an error:
/bin/rm: Argument list too long.
Well, simply navigate to the directory with the problem and execute this:
ls -1 | wc -l && time find . -type f -delete
Apparently, ‘find’ can handle large number of files better than ‘rm’ and it even has a -delete feature!
I just deleted 3 million files using this method.
Special thanks to Deleting tons of files in Linux Argument list too long | SteveKamerman.com.
I was explaining on Hacker News why Windows fell behind Linux in terms of operating system kernel performance and innovation. And out of nowhere an anonymous Microsoft developer who contributes to the Windows NT kernel wrote a fantastic and honest response acknowledging this problem and explaining its cause. His post has been deleted! Why the censorship? I am reposting it here. This is too insightful to be ignored. [Edit: The anonymous poster himself deleted his post as he thought it was too cruel and did not help make his point, which is about the social dynamics of spontaneous contribution. However he let me know he does not mind the repost at the condition I redact the SHA1 hash info, which I did.]
Full Story Below…
Not quite happy with the available disk usage analyzers and looking for a fun project to get used to C programming, I started working on ncdu: A disk usage analyzer with an ncurses interface, aimed to be run on a remote server where you don’t have an entire gaphical setup, but have to do with a simple SSH connection. ncdu aims to be fast, simple and easy to use, and should be able to run in any minimal POSIX-like environment with ncurses installed.
install:
#apt-get install ncdu
run:
#ncdu
Get is here: NCurses Disk Usage.
I have to transfer a bazillion files from one server to another via FTP and I don’t know how to do this. I’ve been using the archaic FTP since the early 90s and have never figure this out, or for that matter needed to until today when I had to upload about a terabyte of files to a new server.
Thanks you NcFTP! You can do a simple command like this to upload all directories recursively!
mput -R *
first, get a copy of ncftp for ubuntu
apt-get install ncftp
locally, go to the directory you want to upload from.
cd /var/www/vhosts/somedomain.com/
user ncftp to connect to the remote server
ncftp -u username -p userpassword domain_or_ip
NcFTP 3.2.5 (Feb 02, 2011) by Mike Gleason (http://www.NcFTP.com/contact/).
Connecting to xxx.xxx.xx.x…
——— Welcome to Pure-FTPd [privsep] [TLS] ———-
You are user number 5 of 50 allowed.
Local time is now 08:51. Server port: 21.
IPv6 connections are also welcome on this server.
You will be disconnected after 15 minutes of inactivity.
Logging in…
OK. Current restricted directory is /
navigate to the directory you want to upload
cd public_html
OK. Current directory is /public_html
finally mput(multi put) all of your files from your local directory. add the -R flag for recursive.
ncftp /public_html > mput -R *
voila!!! you’re on your way!
add_company.php: 1.00 kB 81.08 kB/s
add_user.php: 2.42 kB 205.00 kB/s
crossdomain.xml: 405.00 B 17.47 kB/s
css/jquery.imageZoom.css: 851.00 B 51.27 kB/s
css/prettyPhoto.css: 12.01 kB 933.22 kB/s
databae.sql: 1.62 kB 133.73 kB/s
delete_company.php: 1.14 kB 68.09 kB/s
delete_file.php: 298.00 B 26.93 kB/s
delete_user.php: 817.00 B 66.54 kB/s
download_file.php: 1.98 kB 159.13 kB/s
edit_company.php: 1.44 kB 102.02 kB/s
edit_user.php: 3.27 kB 136.30 kB/s
files/ft2.php: 130.49 kB 2.06 MB/s
files/AdvancePierre_Foods/Carl’s Jr.ai: 1.99 MB 4.90 MB/s
…/AdvancePierre_Foods/advancepierre/APFnsmFin.pptx: 14.46 MB 5.83 MB/s
…_Foods/powerpoint videos/SEQ_AmandaSmellerV03.wmv: 64.51 MB 6.19 MB/s
files/AdvancePierre_Foods/powerpoint videos/MP4s.zip: 77.74 MB 6.33 MB/s
…rre_Foods/powerpoint videos/SEQ_StevenKiddV02.wmv: 16.09 MB 6.15 MB/s
…&J/partners club 2012/partnersclubtrip2012v05.wmv: 49.59 MB 6.29 MB/s
thanks NcFTP!
NcFTP Software.
SparkleShare – Self hosted, instant, secure file sync.
SparkleShare creates a special folder on your computer in which projects are kept. All projects are automatically synced to their respective hosts (you can have multiple projects connected to different hosts) and to your team’s SparkleShare folders when someone adds, removes or edits a file.
The idea of SparkleShare sprouted at the GNOME Usability Hackfest in London, where a couple of designers came to the conclusion that they didn’t have a good (Open Source) collaboration tool to share their work (for more background, read “The one where the designers ask for a pony”). They didn’t like how the good collaboration tools were proprietary, and that using them meant having to give up privacy, control and other rights. What they needed was something that they could run and control themselves, without having to depend on other companies.
Getting some error like:
magicspam-plesk depends on libmilter1.0.1;
Trying to install MagicSpam on Unbuntu 12.04 and Plesk 11?
apt-get install libmilter-dev
root@blue:/tmp# sudo dpkg -i magicspam-plesk-1.0.5-3-precise64-postfix10.amd64.deb
(Reading database … 188972 files and directories currently installed.)
Preparing to replace magicspam-plesk 1.0.5-3 (using magicspam-plesk-1.0.5-3-precise64-postfix10.amd64.deb) …
Unpacking replacement magicspam-plesk …
dpkg: dependency problems prevent configuration of magicspam-plesk:
magicspam-plesk depends on libmilter1.0.1; however:
Package libmilter1.0.1 is not installed.
dpkg: error processing magicspam-plesk (–install):
dependency problems – leaving unconfigured
Processing triggers for ureadahead …
Errors were encountered while processing:
magicspam-plesk
So you’ve upgrade to Plesk and your users can’t upload large files any more. Upon looking at your log files, you see an fcgi error
[warn] [client x.x.x.x] mod_fcgid: HTTP request length 16777256 (so far) exceeds MaxRequestLen (16777216)
Where is maxrequestlen (16777216) set in Parallels Plesk 11 and Ubuntu
It’s in a 2 locations… Edit the Apache2, virtualhost config and the virtual host setup files.
edit these 2 files:
/etc/apache2/mods-available/fcgid.conf
/var/www/vhosts/your_domain_name_here/conf/last_httpd_ip_default.include
and add these lines:
<IfModule mod_fcgid.c> MaxRequestLen 2147483648 FcgidMaxRequestLen 2147483648 </IfModule>
If you have other config info in your module, leave it. I’ve added 2147483648 bytes which is 2gb.
Apparently fcgid settings ARE ALSO in your virtual hosts file here:
/usr/local/psa/admin/conf/templates/default/domain/domainVirtualHost.php
<?php if ($VAR->domain->physicalHosting->php || $VAR->domain->physicalHosting->phpHandlerType == 'fastcgi'): ?>
<IfModule mod_fcgid.c>
FcgidInitialEnv PP_CUSTOM_PHP_INI <?php echo $VAR->domain->physicalHosting->vhostDir ?>/etc/php.ini
FcgidMaxRequestLen 2147483648
</IfModule>
<?php endif; ?>
edit the FcgidMaxRequestLen value. Mine is 2gb now.
Let’s reconfigure your domain with Plesk
/usr/local/psa/admin/bin/httpdmng --reconfigure-domain yourdomain.com
Finally, I restarted PSA and Apache2
% service apache2 restart and % service psa restart
Go back to your web page and you should be able to upload large files now.
mod_fcgid: HTTP request length (so far) exceeds MaxRequestLen
mod_fcgid MaxRequestLen ubuntu 12 plesk 11
After updating to Ubuntu 12.04 my DNS networking is whacked.
Postfix is getting these errors: Host or domain name not found. Name service error
Apparently you can’t edit the /etc/resolv.conf file directly because it will be overwritten by the ‘system’
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8) # DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
Step 1 – edit this file: /etc/resolvconf/resolv.conf.d/head and add your name server to the file. I’m using Google’s DNS server:
nameserver 8.8.8.8
Step 2 – Restart resolvconf
sudo resolvconf -u
Now, when you look at the /etc/resolv.conf file your entry DNS entry will be in it:
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8) # DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN nameserver 8.8.8.8
I restarted my networking, “just in case.” Not sure if this step is required
service networking restart
I read somewhere that you should add your name servers directly into /etc/network/interfaces. I did this and restarted resolvconf but it didn’t write the changes to /etc/resolv.conf
iface eth0 inet static
address 192.168.3.3
netmask 255.255.255.0
gateway 192.168.3.1
dns-search example.com
dns-nameservers 8.8.8.8
Command To See Postfix Queue
% mailq
or
% postqueue -p
FLUSH Messages In Queue
% postfix flush
or
% postqueue -f
Hope this helps.
Woke up this morning and the Postfix SMTP server on Ubuntu 10.04 and Plesk stopped running.
I could start and stop via service
# service postfix start
and it appeared everything was working fine. However, it wasn’t until I looked at the syslog /var/log/syslog that I saw an error occurring on start up.
Nov 26 12:51:27 blue postfix/tlsmgr[28017]: fatal: open database /var/lib/postfix/smtpd_scache.db: Invalid argument
Nov 26 12:51:29 blue postfix/master[28012]: warning: process /usr/lib/postfix/tlsmgr pid 28017 exit status 1
Nov 26 12:51:29 blue postfix/master[28012]: warning: /usr/lib/postfix/tlsmgr: bad command startup — throttling
SOLUTION:
1. stopped postfix: # service postfix stop
2. deleted the cache files friom /var/lib/postfix
# rm /var/lib/postfix/smtpd_scache.db
and
# rm /var/lib/postfix/smtp_scache.db
3. restarted postfix and voila, everything works: # service postfix start
hope this helps!
Years ago, I was proud to own an 80MHz, 16MB RAM PC with a huge 512MB harddisk. But even on this incredibly huge harddisk, my diskspace seemed to disappear over time. So I wrote a little tool to find out where all my diskspace had gone: I called it gettree, as it showed all files in a directory and also calculated the size of subtrees. Over the years, the OS I used changed, but the problem stayed. Using Linux, there are some tools to deal with it, but I couldn’t find anything that was clearly laid out, intuitive to browse, and doesn’t need X-Windows.
http://kb.parallels.com/en/111283
Another nice page to help me remember where all those log files and config files are for Parallels Plesk Panel for Linux services logs and configuration files
What an AWESOME project. I’m so thankful to have found this. Scan your server for malware. We had a server getting slammed with a malware file. Linux Malware Detect found it. Thanks, THANKS!
Linux Malware Detect | R-fx Networks.
Linux Malware Detect (LMD) is a malware scanner for Linux released under the GNU GPLv2 license, that is designed around the threats faced in shared hosted environments. It uses threat data from network edge intrusion detection systems to extract malware that is actively being used in attacks and generates signatures for detection. In addition, threat data is also derived from user submissions with the LMD checkout feature and from malware community resources. The signatures that LMD uses are MD5 file hashes and HEX pattern matches, they are also easily exported to any number of detection tools such as ClamAV.
php5 cronjob fuser designed to clean out php sessions is eating the cpu on ubuntu boxes.
This is the content of /etc/cron.d/php5 on 11.10:
09,39 * * * * root [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +$(/usr/lib/php5/maxlifetime) ! -execdir fuser -s {} 2>/dev/null \; -delete
And this is the content on 11.04 where it worked fine:
09,39 * * * * root [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +$(/usr/lib/php5/maxlifetime) -delete
There’s no fuser in the old 11.04 cron. replace your 11.10 with the old 11.04 cron.
solution from here:
http://www.flynsarmy.com/2011/11/fuser-using-100-cpu-in-ubuntu-11-10/
also noted on ubuntu’s bug page
https://bugs.launchpad.net/ubuntu/+source/php5/+bug/876387
kill -9 `ps xawo state=,pid=|sed -n ‘s/Z //p’`
mutt – email client
http://www.mutt.org/
“All mail clients suck. This one just sucks less.” -me, circa 1995
htop – interactive process viewer
http://htop.sourceforge.net/
mysqldump -v -u username -p database_name > dumpfile.sql
To Import a MySQL Database run:
mysql -v -u username -p database_name < dumpfile.sql
lshw -C MEMORY
or,
dmidecode –type 5,6,16,17
