Usefull Links
> When to submit to the ArXiv
> How to reflect your bib citations in arxiv submission
> Career advice by Terence Tao
> Miscellanies about computers (Prof. Koudai Sugimoto)
.bashrc and .bash_profile
difference between .bashrc and .bash_profile: .bashrc is sourced every sinlge time you log in, whereas .bash_profile is sourced only for the first log in.
Terminal
Text Encoding with Asian characters: Uncheck the ``set local variables automatically'' in the Environment Preferences, and add ``export LANG="ja_JP.UTF-8" in the .bash_profile
ref1,
ref2
Turn on the GPU renderer to speed iTerm2 up:
ref1
Search all files under the current directories including their sub-directories:
find ./ | xargs grep "cfl"
ssh
ssh using password instead of key authentication: ssh username@serverid -o PreferredAuthentications=keyboard-interactive,password -o PubkeyAuthentication=no
ssh agent forwarding: ssh -A username@serverid
Keep running a background job in remote sshed machine:
nohup python myscript.py &
ps aux |grep python
kill <relevant PID>
(cf,
reference1)
scp
scp using password instead of key authentication:scp -o PreferredAuthentications="password" username@serverid
type passwords only once while copying multiple files:
scp xxx@yyy:/home/zzz/
\{file1,file2,file3...}
echo
ecp options: echo -n: force no line break at the end of the output; echo -e: explicitly state \\n where you want to insert a line break.
Makefile
Write makefiles
Fast Fourier Transform (FFT)
FFTW website
General Introduction of Fourier Transform:
ref1,
ref2
Hermite Conjugate:
ref1,
ref2
Installation FFTW onto Linux:
ref1 ,
ref2 ,
ref3 ,
ref4
Error of linking to FFTW library:
ref1 ,
ref2
Most basic knowledge required to perform FFTW from Fortran:
ref1 (program f90 file),
ref2 (compile),
ref3 (real to complex transform: refer to bottom 2 examples)
Format:
ref1
Call FFTW from Fortran:
ref1 (Standard 1D transformation but the out array should be size of N/2+1),
ref2,
ref3 (Compile option guidance)
Compile options when you call FFTW from Fortran:
ref1,
ref2
What is the correct output array size?:
ref1,
ref2,
ref3
How the Fourier complex are stored in the output array?:
ref1
Analyze the output from FFTW:
ref1,
ref2
Complex array in Fortran:
ref1,
ref2
Random Seed Generation
Quality of randomness;
ref1,
ref2
LCG (Park and Miller):
ref1
Character Encoding
nkf: convert character encoding
nkf -g, nkf -w --overwrite
File Handling
ecp options: do a same action on all the files in your directory
sed examples:
ref 1,
ref 2,
ref 3,
ref 4,
ref 5
example1:
find . -maxdepth 1 -name "*dat" | sed 'p; s/SMBH/GW/' | xargs -n 2 mv
example2 (replace texts within files):
sed -i '' 's/aaa/bbb/' TargetFile
ref)
sed on Mac
example3:
for file in xyz_*.dat; do
sed -e '$d' $file > ${file%.txt}_copy
rm $file
mv ${file%.txt}_copy $file
done
Other references:
find+sed+xargs,
find,
mkdir consecutive-numbered directories
To list directories only: find ./* -type d -name "*"
Remove all files except some specific files
ls | grep -v -E 'file1.txt|file2.txt|file3.dat|file4.py' | xargs rm -rf
Rename files: find aabbcc*dat -maxdepth 1 | sed 'p;s/bb/dd/' | xargs -n 2 mv
if test
if test options:
if [ -e $file ]
then mkdir xxx
else echo "$file not found" fi
-f for checking whether or not it is file, -d for checking whether or not it is directory
(cf,
reference (in Japanese): very helpful with some corrections needed; -d option has to be -e, mkdir_if_not_exist(){} stetement is not needed.)
if/else comparison with a wild card:
if ls ./000*dat > /dev/null 2 >& 1
then
echo "OK"
tar -czvf ./000data.tar.gz ./000*dat
else
echo "NG"
fi
another ref
directories
ls only directories: du -altrh | grep ^d
du and df: du is for directory usage, df is for disk availability
shell script
How to pass variables:$1 for the first system argument, $2 for second, and so on.
Rewrite shell scripts while keeping their running jobs unaffected: cp them to tmp files and mv back to original names:
ref1,
ref2 (how to use bash brace expansion)
cp and mv to edit a file that other programs are using but do not violate those operations
for i in file{000..102}.sh; cp -i $i{,.tmp}; done
for i in file{000..102}.sh; do mv $i{.tmp,}; done
Generate files and directories with serial numbers
using seq (e.g., for i in `seq -f %03g 1 10`; do mkdir testdir${i}; done)
using brackets {} (e.g., touch sample0{0..9})
read:
input from command line
combine variables and texts:
ref1
Command line
erase already typed charaters: CTRL+u
Combining data from multiple files into one file:
paste
Image Magick
how to compress EPS data size (in Japanese): convert xxx.jpeg eps2:yyy.eps
convert in GrayScale (in Japanese): convert -type GrayScale xxx.jpeg yyy.jpeg
Vim
Ultimate Vim configuration;
Enable cursor in amix Vim
Increment numbers by CTRL+A (or CTRL+V CTRL+A for multiple lines), or by defining macros:
press q
Count word numbers: g + Ctrl-g, :%s/\i\+/\&/gn;
ref1,
ref2
colorscheme:
ref1,
ref2
Check your current colorscheme, help group-names:
ref1,
ref2
Color table:
ref1
Ubuntu
Installation:
ref1,
ref2,
ref3
SSH server:
ref1,
ref2
BIOS:
ref1
GUI-driver (NVIDIA):
ref1,
ref2
ubuntu-drivers devices
sudo ubuntu-drivers autoinstall (if installing the recommended driver shown by the previous command)
sudo reboot
Network Card (NIC):
ref1
ref2
Rebooting Ubuntu: Alt+PrintScreen+RSEIUO:
ref1,
ref2
Miscellanies
IP address:
ref1,
ref2
Fortran
Cray ftn debug options:
-Rb,
-G
Subroutine does not necessarily require return while function does:
ref1,
ref2
Multiple IF conditions:
valid: ( (N .EQ. 7) .OR. (N .EQ. 2) )
invalid: ( N.EQ.7.OR.2 )
int, ceiling, floor
Check if the file exists:
access:
ref1,
ref2
inquire:
ref1,
ref2
ex)
INTEGER*4 :: access
INTEGER*4 :: mystatus
mystatus = access('./test_access.txt', ' ')
if(mystatus.eq.0) then
print *, mystatus,"exists"
open(12, file="test_access.txt", status="old", position="append", action="write")
write(12,"(a)") "Time to add a new line"
close(12)
else if(mystatus.ne.0) then
print *, mystatus,"do not exist"
open(12, file="test_access.txt", status="new", action="write")
write(12,"(a)") "Time to start a new file"
close(12)
end if
Defining a function with some optial inputs:
ref
Format: binary I/O
(
ref1,
ref2)
Format:
Fortran cannot print three digits exponent unless explicitly specified
Move_alloc (reallcation of arrays):
ref1,
ref2,
ref3,
ref4,
ref5,
ref6
Reshape arrays:
ref1,
ref2
Exit stated in do-loop
Stop function
C array starts its index from 0, whereas 1 for Fortran
How to create and use modules
Check file existance by inquire function:
reference1,
reference2
How to obtain system arguments from the commandline:
reference1,
reference2,
reference3,
reference4,
reference5
[ifort debug options]
ref1,
ref2
Traceback:
ref1,
ref2
ifort xxx.f90 -traceback
I/O related error messages:
list
Debugging
how to gdb
how to check memory (valgrind)
Microsoft PowerPoint
Play multiple movies simultaneously
Gnuplot
Basic how-to:
ref1,
ref2,
ref3
color map
Generate PDF directly from gnuplot:
ref1,
ref2
set terminal postscript enhanced color
set output '| ps2pdf - plot.pdf'
Histogram:
ref1,
ref2,
ref3,
ref4,
ref5
Histogram Normalization:
ref1,
ref2,
ref3,
ref4,
ref5,
ref6
Fitting (e.g., Gaussian):
ref1,
ref2
set lmargin 10
set tics font ",20"
C++
Reference instead of pointer:
ref1,
ref2,
ref3,
ref4
Binary output: fwrite:
ref1
fp = fopen(filename.c_str(), "wb");
for (int k=ks; k<=ke; k++) {
for (int j=js; j<=je; j++) {
for (int i=is; i<=ie; i++) {
out_tmp = out_array(k,j,i);
fwrite(&out_tmp, sizeof(out_tmp), 1, fp);
fclose(fp);
}}}
fopen option:
ref1
Binary output: fread:
ref1,
ref2
Python
Installation through pip:
ref1,
ref2,
installation options,
python3,
Memory allocation:
ref1
Installation onto Ubuntu:
ref1,
ref2,
ref3
sudo apt update
sudo apt install build-essential libbz2-dev libdb-dev libreadline-dev libffi-dev libgdbm-dev liblzma-dev libncursesw5-dev libsqlite3-dev libssl-dev zlib1g-dev uuid-dev tk-dev
cd /tmp
wget https://www.python.org/ftp/python/3.x.y/Python-3.x.y.tar.xz
tar -xf Python-3.x.y.tgz
cd Python-3.x.y
./configure (--prefix=/home/user/xxx if you want to install into a specific directory)
make
sudo make altinstall
add the directory to the $PATH
Installing packages:
ref1,
ref2
Do not run pip3 or pip3.x but simply ``pip''.
Run without the ``-targert'' option leads you to simply install numpy under the directory that python3.x is installed.
e.g.,) python3.x -m pip install numpy
in case if you have python3.x under $HOME/.local/, then numpy will be installed under $HOME/.local/lib/python3.x/site-packages/
Basisc of import and from:
ref1,
ref2
File open and read:
ref1
for line in datafile:
line_d = line.split()
print(line_d[0])
Binary file: read and write:
ref1,
ref2
import struct
path = '/tmp/a.bin'
f1 = open(path,'wb')
val = 3.1400000000
f1.write(struct.pack('f', val))
f1.close()
f2 = open(path,'rb')
val2 = struct.unpack('f', f2.read())
print(val2)
f2.close()
Basics of pyplot:
ref1,
ref2,
ref3,
ref4,
ref5 (Hierarchical structure of ``Figure'', ``Axes'', and ``Axis'')
Format: print '{0:d} {1:05.3e} {2:s} {3:f}'.format(ii, ngh, 'CHARACTER', vel)
ref1,
ref2
Log scale ploet including negative value:
plt.xscale("symlog")
rasterization in matplotlib:
m=ax.plot(xx,yy); m.set_rasterized(True) (cf,
reference)
1D histogram: option: density (pyhotn3) or normed (python2)
2D histogram (hist2d):
ref1,
ref2,
ref3
qdel:
release the memory space. The variable is no longer accessible so that you have to define again if needed.
(cf,
reference1,
reference2,
reference3)
Log-normal fitting:
ref1,
ref2,
ref3,
ref4,
ref5
Extract non-zero elements from an array: numpy.nonzero()
ref1,
ref2,
ref3
Note that this does not keep the original multi-dimensional geometry.
Sum, min, max, mean, std, etc. without numpy.NaN elements
ref1,
ref2,
ref3
numpy.min() in a fixed direction, by designating "axis": numpy.min(testarr,axis=0)
ref1
Makedir only when the directory does not exist: Combination of os.path.exists and os.makedirs.
import os
def write(filename, text):
file_path = os.path.dirname(filename)
if not os.path.exists(file_path):
os.makedirs(file_path)
(cf,
Difference between os.makedir and os.makedirs:makedir is for a directory, whereas makedirs is for creating a hierarchy.)
read() and readline(): read() reads word by word whereas readline reads line by line.
Read characters in a file and put them into list:
with open('data.txt') as f:
for line in f:
int_list = [int(i) for i in line.split()]
print int_list
Use numpy sum instead of sum for summing multi-dimensional array
numpy array: ndim, shape, size, and len
Append files by using numpy savetxt
Break nested for-loops by using StopInteration()
Numpy arange/linspace/logspace/zeros/ones/mgrid/ogrid
Pyplot scatter edgecolor:edgecolors='none'
Roll/Shift array elements numpy roll
Repeat elements: tile, repeat [
ref1,
ref2]
Replace characters in string list:
import string
src = 'Boston'
dst = src.translate(string.maketrans('stn', 'STN')) # 'BoSToN'
Random index generator:
import random
random.sample(range(1, 100), 3)
[77, 52, 45]
Aspect ratio for axes object: ax.set_aspect(aspect, adjustable=None, anchor=None)
aspect = 'equal' sets the ratio as same scaling as data
[Labels, Spines and Ticks]
Sepcify tick locations on a figure in logarithmic scale
ax.set_xticks([20, 200, 500])
ax.get_xaxis().set_major_formatter(matplotlib.ticker.ScalarFormatter())
Manually substitute tick labels: plt.xticks([1, 2, 3], ["A", "B", "C"])
Multiple colors in a single label:
ref1,
ref2
from matplotlib.offsetbox import AnchoredOffsetbox, TextArea, VPacker
ybox1 = TextArea("lab1", textprops=dict(color="red", size=10,rotation='vertical'))
ybox2 = TextArea("lab2", textprops=dict(color="blue",size=10,rotation='vertical'))
ybox = VPacker(children=[ybox1,ybox2],align="center", pad=0, sep=5)
anchored_ybox = AnchoredOffsetbox(loc=8, child=ybox, pad=0., frameon=False,bbox_to_anchor=(-0.14, 0.42), bbox_transform=ax.transAxes, borderpad=0.)
ax.add_artist(anchored_ybox)
savefig('./multicolor.pdf',bbox_extra_artists=(lgd,text), bbox_inches='tight')
Show minor tick labels
Change the label of minor ticks
Remove ticks and tick labels:
ref1,
ref2
Customize dashed lines as dash-dot-dot lines
[legends]
layout of the legend
handletextpad: spacing between markers and labels
labelspacing, scatterpoints
handlelength: set the length of legend lines
get_lineshandlelength: set the line width of legend lines
legendHandles: set the line transparency of legend lines
legend shown horizontally: mode=``expand'' + borderaxespad + ncol
Origin of imshow:
ref1
2D color map: [
pcolormesh,
dimension,
alternative:
histogram2d]
Colorbar adjustment:
[additional some detail ref in Japanese,
ref1 (very comprehensive),
ref2]
If you want a color bar adjusted to this modified figure, they calculate xyratio = (np.max(yy)-np.min(yy))/(np.max(xx)-np.min(xx)) and plt.colorbar(sc,shrink=xyratio)
Color scales easily read by those with colorblindness:
ref1(virids),
ref2
Get and set color map: plt.get_cmap('xxx'):
ref1,
ref2
Make your own cmap:
ref1 (continuous colormap),
ref2
Change the color cycle in line plots: plt.sytle.use('xxx'):
ref1,
ref2,
ref3,
ref4,
ref5
Convert the form elements in a numpy array:xxxarray.astype('int')
colormath:
a Python module from which you can use a wide range of color spaces (e.g., specifying CMYK colors instead of ordinary RGB.)
(cf,
reference1)
(cf,
reference2)
<Inquiry>
I would very appreciate if any of you know and teach me how to convert CMYK tuples to RGB tuples by Python colormath. The followings are the command I run:
mycolor_CMYK = color_objects.CMYKColor(0.75,0.0,0.0,0.25);
mycolor_CMY = color_conversions.convert_color(mycolor_CMYK,CMYColor);
mycolor_RGB = color_conversions.convert_color(mycolor_CMY,BaseRGBColor);
ax.plot(xxx,yyy,color=mycolor_RGB);
I was stacked at the following error messages and could not proceed:
ValueError: to_rgba: Invalid rgba arg "BaseRGBColor (rgb_r:0.1875 rgb_g:0.7500 rgb_b:0.7500)"
to_rgb: Invalid rgb arg "BaseRGBColor (rgb_r:0.1875 rgb_g:0.7500 rgb_b:0.7500)"
cannot convert argument to rgb sequence
Finally I ended up with writing a conversion function as follows:
def mycolorconv_cmyk_to_rgb(c,m,y,k):
r = (1.0-c)*(1.0-k)
g = (1.0-m)*(1.0-k)
b = (1.0-y)*(1.0-k)
return r,g,b
This works with the following commands:
my_red,my_green,my_blue = mycolorconv_cmyk_to_rgb(75.,0.,0.,25.);
ax.plot(xxx,yyy,color=(my_red,my_green,my_blue));
c.f., Conversion from CMYK to RGB:
ref1,
ref2,
ref3,
ref4
Search the path to imported package directories: sys.path
Latex
How to approach errors?
Journal Abbreviation
(cf,
reference)
subfigure cannot recognize \textwidth
Font size of section titles
Space between paragraphs
introduce a linebreak in the enumerate environment
add border around both figure and caption with adjustbox:
ref1,
ref2
Movie
iMovie
Export an mp4 file from a series of png files:
ref1,
ref2
ffmpeg
Export an mp4 file from a series of png files:
ref1,
ref2,
ref3,
ref4.
example: ffmpeg -framerate 5 -start_number 1 -i test_%03d.png -vcodec libx264 -pix_fmt yuv420p -vf "scale=trunc(iw/2)*2:trunc(ih/2)*2" -r 30 out.mp4
-framerate: output frame per second
-start_number: starting number of the input files
-i: designate input files
-vcodec: designate the encoder
-pix_fmt: designate the format
-vf: re-scale the pixel size
-r: the number of input files (total frame numbers)
ref1,
ref2,
Computer Job Organization
CPUs and cores:
ref1
Memory usage:
ref1
kill all the jobs of a specific user: pgrep xxxxx -u 'username' | xargs kill -9:
ref1,
ref2
Parallel Computing
[OPENMPI]
sudo apt install openmpi-bin libopenmpi-dev:
ref1
[HDF5]
Source Code:
The HDF Group; other refs:
ref1,
ref2,
ref3
Download and make: https://portal.hdfgroup.org/display/support/HDF5%201.12.0
wget https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.N/hdf5-1.N.N/src/distribution.tar.gz
gzip -cd distribution.tar.gz | tar xvf -
CC=/usr/bin/mpicc ./configure --prefix=/usr/local/phdf5 --enable-parallel
make
make check
sudo make install
sudo make check-install
Add the followings in the .bashrc
export PATH=/usr/local/phdf5/bin:$PATH
export PATH=/usr/local/phdf5/include:$PATH
export LD_LIBRARY_PATH=/user/local/phdf5/lib:$LD_LIBRARY_PATH
(Please refer to the INSTALL file in the ./hdf5-1.N.N/release_docs/ directory.)
MacBook
Computer Name: how to change name:
ref1,
ref2
Enebale key repeating while holding a key:
ref
Preview app to crop a PDF file by removing the regions outside your selection:
ref:
first crop, second save as PostScript, third open the PS file in preview and save as a PDF.
Paste texts without anyformatting: Kyeboard setting (for most of your apps) and Command + Control + V (for MS offices):
ref1,
ref2,
ref3
[Mail App]
Definition of IMAP and POP:
ref1
What does "synchronize" mean?:
ref1,
ref2
Make the local copy of IMAP mails:
ref1,
ref2
MacPorts
Migration after OS X upgrade:
Official weisite
reference 1
reference 2
Trello
Synchronize your Trello calendear to Google calendar:
ref1,
ref2
awk
Count letters: cat xxx.txt | awk '{count += (split($0, a, "starwars") - 1)} END{print count}'
Add the line number (divided by 10): cat yyy.txt | awk '{i=1; print NR/10.0 " " $1 $2 $3; i++}' > zzz.txt